Last week I attended the Data Management Association (DAMA) – Rocky Mountain Chapter (RMC) 2024 Q3 Event and had the privilege of hearing John O’Brien, Principal Advisor and CEO at Radiant Advisors, speak on his recent report, The 2024 Modern Data Management & Data Fabric Market Study. According to his report, over 40% of respondents have allocated budgets towards enhancing their data management capabilities, a financial dedication that calls attention to the importance of data management in driving business success [1]. Let’s dive into the survey findings and explore the current focus and future direction of modern data management as voted on by our industry peers.
The 2024 Modern Data Management & Data Fabric Market Study was conducted in March 2024 and provides a comprehensive overview of trends in data management. The survey was produced by Unisphere Research and Radiant Advisors and sponsored by Informatica, Denodo, and Quest. Respondents of the survey are North American (95.1%), and the majority are involved in the IT, tech, and software industry. Visit Denodo’s website for a copy of the full report.
Current Trends
Primary Focus: Operational Efficiency
According to the survey, 61.3% of respondents identified enhancing operational efficiency as their primary focus [1]. This emphasis highlights the strategic importance of automating processes to advance data-driven decisions. This is no surprise as streamlining processes and automating routine tasks are related to cost reduction, enhanced productivity, and improved operations. Ultimately adding efficiencies enables faster response times to market changes and customer needs, allowing businesses to stay competitive.
Continued Adoption: Cloud Solutions
Cloud solutions started to gain popularity in the early 2000s, when Amazon introduced Elastic Cloud Compute [2]. According to the market survey, 76.4% of respondents are currently using or planning to use the cloud in their data management environment. This result supports the trend of an ever-increasing adoption of cloud-based solutions as organizations stride towards more flexible, scalable data frameworks [1].
The adaptability offered by the cloud has been pivotal to managing varying workloads and optimizing costs. Businesses only pay for what they use as they can scale resources up or down based on demand quickly. Cloud-based data architectures facilitate integration with diverse data sources, enhancing the ability to consolidate and analyze data from multiple platforms. This approach supports innovation, allowing companies to rapidly deploy new applications and services without the constraints of traditional on-premises infrastructure. Additionally, cloud solutions enhance collaboration and accessibility, providing remote and even global teams with real-time access to data and analytics tools.
Cloud-based solutions offer flexibility, scalability, and access that directly support improvements to business efficiencies and operations. Gartner predicts that by 2028, cloud computing will become a necessary component for maintaining business competitiveness [3].
Key Priorities: Data Quality, Governance, Master Data Management, Data Catalogs, & Lineage
Top priorities for 2024 enterprise data management include data quality, governance, integration, and management platforms to ensure data reliability and usability [1]. Trusted data is crucial for driving data literacy within organizations, as employees are more likely to engage with and utilize data when they believe in its integrity. Master Data Management (MDM), data catalogs, and data lineage are recognized for providing comprehensive views across domains.
MDM provides an authoritative source of truth for critical business data by integrating and consolidating data from various sources, eliminating inconsistencies and redundancies. Implementing an MDM strategy to the data infrastructure allows for accessibility needed for better governance and data quality checks.
Data catalogs act as comprehensive inventories of data assets, providing metadata and context. They facilitate simplified data discovery and access, allowing users to understand the transformations, and usage of data within the organization. Adding in lineage is a common next step as it offers traceability, providing the origin story of the data.
MDM, data catalogs, and lineage together become a triple threat, ensuring that users can trust the data they are using, knowing it has been responsibly managed and vetted.
Figure 1. Data Management Technologies in Use or Planned in 2024, from The 2024 Modern Data Management & Data Fabric Market Study [1]
Future Direction
Data Fabric
Data fabric is gaining acceptance, with respondents noting its potential to innovate data management. According to the survey, 52% of participants are neutral on data fabric adoption, while 26% acknowledge its benefits for IT data management and 6% for business analytics and discovery [1].
Data fabric is a clear next step with a top focus of organizations to enhance operational efficiency and priority topics of data quality and data governance at the forefront. Data Fabric promotes a centralized approach to data management. It integrates various data sources, systems, and technologies into a strategically architected framework (commonly a zone-based architecture), creating a unified data platform that simplifies access and control. To manage data across the organization, data fabric leverages metadata, data catalogs, and automated data management processes to create a comprehensive and consistent view of the data landscape.
Active Metadata
Active metadata is noted for its ability to improve data governance and quality monitoring. Unlike passive metadata, which is static and often used only for documentation purposes, active metadata continuously interacts with data systems and applications to provide contextual, operational, and governance insights as data is used and modified. Since active metadata is dynamically updated and utilized in real-time, it’s the next level to enhancing data management and analytics processes.
It is likely that organizations who are designing active metadata within their framework already have a handle on static metadata living within data catalogs and have a data governance program in place.
Analytics Catalogs
An analytics catalog is a centralized repository that organizes and provides access to an organization’s analytical assets, such as reports, dashboards, data visualizations, and analytical models. It helps users discover, understand, and utilize their organizations’ analytical resources. The analytics catalog typically includes metadata about the analytical assets, such as their purpose, data sources, authors, usage statistics, and relevant documentation.
Although analytics catalogs are further out in the timeline to adoption, they may be beneficial as the push for data literacy across organizations continues.
Challenges
Integration complexity and stakeholder engagement are significant challenges, identified by 40.9% of survey respondents as barriers to unifying data views and obtaining enterprise-wide support for data initiatives [1]. Despite organizations claiming a proactive stance on data management, a considerable disparity remains between their strategic goals and actual implementation.
Steps Towards Implementation
To overcome these challenges, an iterative approach is necessary. At Datalere, we suggest these steps toward modernizing your data management framework:
- Prioritize 2 to 3 use cases that can be built in 8 to 12 weeks (about 3 months).
- Have the use cases span most of the intended architecture, so include, ingestion, transformations, materialized data zones, quality, governance, reports and dashboards (and possibly ML).
- Pivot quickly.
- Use the cloud so you can build quickly and collaboratively without committing to anything that can’t be replaced.
If your organization is looking for support in implementing a modern data management strategy, reach out to our team at Datalere. We offer expert consulting services in designing and developing custom data solutions tailored to your needs.
[2] https://level365.com/history-of-the-cloud-based-solutions/
