As examined in the post The Intersection Between MDM, PIM and ESG, environmental data management is becoming an important aspect of the offerings provided in solutions for Master Data Management (MDM) and Product Information Management (PIM). Consequentially, this will also apply to the Data Quality Management (DQM) capabilities that are either offered as part of these solutions or as standalone solutions for data quality.
This site has an interactive Select Your Solution service for potential buyers of MDM / PIM / DQM solutions. The service has a questionnaire and an undelaying model for creating a longlist, shortlist or direct PoC suggestion for the best candidate(s) according to the context, scope, and requirements of an intended solution.
In line with the rise of the Environmental, Social and Governance (ESG) theme, the questionnaire and the underlying selection model must in the first place be enhanced with environmental data management aspects, as the environmental part of ESG is the one with the currently most frequently and comprehensive experienced data management challenges.
There is already established a good basis for this.
However, if you as either one from a solution end user organization with environmental data management challenges or a solution provider have input to environmental data management aspects to be covered, you are more than welcome to make a comment here on the blog or use the below contact form:
Today’s guest blog post is from Marcin Chudeusz of DEXT.AI. a company specializing in creating Artificial Intelligence-powered Software for Data Platforms.
Have you ever experienced the frustration of missing crucial pieces in your data puzzle? The feeling of the weight of responsibility on your shoulders when data issues suddenly arise and the entire organization looks to you to save the day? It can be overwhelming, especially when the damage has already been done. In the constantly evolving world of data management, where data warehouses, data lakes, and data lakehouses form the backbone of organizational decision-making, maintaining high-quality data is crucial. Although the challenges of managing data quality in these environments are many, the solutions, while not always straightforward, are within reach.
Data warehouses, data lakes, and lakehouses each encounter their own unique data quality challenges. These challenges range from integrating data from various sources, ensuring consistency, and managing outdated or irrelevant data, to handling the massive volume and variety of unstructured data in data lakes, which makes standardizing, cleaning, and organizing data a daunting task.
Today, I would like to introduce you to Digna, your AI-powered guardian for data quality that’s about to revolutionize the game! Get ready for a journey into the world of modern data management, where every twist and turn holds the promise of seamless insights and transformative efficiency.
Digna: A New Dawn in Data Quality Management
Picture this: you’re at the helm of a data-driven organization, where every byte of data can pivot your business strategy, fuel your growth, and steer you away from potential pitfalls. Now, imagine a tool that understands your data and respects its complexity and nuances. That’s Digna for you – your AI-powered guardian for data quality.
Goodbye to Manually Defining Technical Data Quality Rules
Gone are the days when defining technical data quality rules was a laborious, manual process. You can forget the hassle of manually setting thresholds for data quality metrics. Digna’s AI algorithm does it for you, defining acceptable ranges and adapting as your data evolves. Digna’s AI learns your data, understands it, and sets the rules for you. It’s like having a data scientist in your pocket, always working, always analyzing.
Figure 1: Learn how Digna’s AI algorithm defines acceptable ranges for data quality metrics like missing values. Here, the ideal count of missing values should be between 242 and 483, and how do you manually define technical rules for that?
Seamless Integration and Real-time Monitoring
Imagine logging into your data quality tool and being greeted with a comprehensive overview of your week’s data quality. Instant insights, anomalies flagged, and trends highlighted – all at your fingertips. Digna doesn’t just flag issues; it helps you understand them. Drill down into specific days, examine anomalies, and understand the impact on your datasets.
Whether you’re dealing with data warehouses, data lakes, or lakehouses, Digna slips in like a missing puzzle piece. It connects effortlessly to your preferred database, offering a suite of features that make data quality management a breeze. Digna’s integration with your current data infrastructure is seamless. Choose your data tables, set up data retrieval, and you’re good to go.
Figure 2: Connect seamlessly to your preferred database. Select specific tables from your database for detailed analysis by Digna.
Navigate Through Time and Visualize Data Discrepancies
With Digna, the journey through your data’s past is as simple as a click. Understand how your data has evolved, identify patterns, and make informed decisions with ease. Digna’s charts are not just visually appealing; they’re insightful. They show you exactly where your data deviated from expectations, helping you pinpoint issues accurately.
With Digna, every column in your data table gets attention. Switch between columns, unravel anomalies, and gain a holistic view of your data’s health. It doesn’t just monitor data values; it keeps an eye on the number of records, offering comprehensive analysis and deep insights with minimal configuration. Digna’s user-friendly interface ensures that you’re not bogged down by complex setups.
Figure 3: Observe how Digna tracks not just data values but also the number of records for comprehensive analysis. Transition seamlessly to Dataset Checks and witness Digna’s learning capabilities in recognizing patterns.
Real-time Personalized Alert Preferences
Digna’s alerts are intuitive and immediate, ensuring you’re always in the loop. These alerts are easy to understand and come in different colors to indicate the quality of the data. You can customize your alert preferences to match your needs, ensuring that you never miss important updates. With this simple yet effective system, you can quickly assess the health of your data and stay ahead of any potential issues. This way, you can avoid real-life impacts of data challenges.
Whether you prefer inspecting your data directly from the dashboard or integrating it into your workflow, I invite you to commence your data quality journey. It’s more than an inspection; it’s an exploration—an adventure into the heart of your data with a suite of features that considers your data privacy, security, scalability, and flexibility.
Automated Machine Learning
Digna leverages advanced machine learning algorithms to automatically identify and correct anomalies, trends, and patterns in data. This level of automation means that Digna can efficiently process large volumes of data without human intervention, erasing errors and increasing the speed of data analysis.
The system’s ability to detect subtle and complex patterns goes beyond traditional data analysis methods. It can uncover insights that would typically be missed, thus providing a more comprehensive understanding of the data.
This feature is particularly useful for organizations dealing with dynamic and evolving data sets, where new trends and patterns can emerge rapidly.
Domain Agnostic
Digna’s domain-agnostic approach means it is versatile and adaptable across various industries, such as finance, healthcare, and telcos. This versatility is essential for organizations that operate in multiple domains or those that deal with diverse data types.
The platform is designed to understand and integrate the unique characteristics and nuances of different industry data, ensuring that the analysis is relevant and accurate for each specific domain.
This adaptability is crucial for maintaining accuracy and relevance in data analysis, especially in industries with unique data structures or regulatory requirements.
Data Privacy
In today’s world, where data privacy is paramount, Digna places a strong emphasis on ensuring that data quality initiatives are compliant with the latest data protection regulations.
The platform uses state-of-the-art security measures to safeguard sensitive information, ensuring that data is handled responsibly and ethically.
Digna’s commitment to data privacy means that organizations can trust the platform to manage their data without compromising on compliance or risking data breaches.
Built to Scale
Digna is designed to be scalable, accommodating the evolving needs of businesses ranging from startups to large enterprises. This scalability ensures that as a company grows and its data infrastructure becomes more complex, Digna can continue to provide effective data quality management.
The platform’s ability to scale helps organizations maintain sustainable and reliable data practices throughout their growth, avoiding the need for frequent system changes or upgrades.
Scalability is crucial for long-term data management strategies, especially for organizations that anticipate rapid growth or significant changes in their data needs.
Real-time Radar
With Digna’s real-time monitoring capabilities, data issues are identified and addressed immediately. This prompt response prevents minor issues from escalating into major problems, thus maintaining the integrity of the decision-making process.
Real-time monitoring is particularly beneficial in fast-paced environments where data-driven decisions need to be made quickly and accurately.
This feature ensures that organizations always have the most current and accurate data at their disposal, enabling them to make informed decisions swiftly.
Choose Your Installation
Digna offers flexible deployment options, allowing organizations to choose between cloud-based or on-premises installations. This flexibility is key for organizations with specific needs or constraints related to data security and IT infrastructure.
Cloud deployment can offer benefits like reduced IT overhead, scalability, and accessibility, while on-premises installation can provide enhanced control and security for sensitive data.
This choice enables organizations to align their data quality initiatives with their broader IT and security strategies, ensuring a seamless integration into their existing systems.
Conclusion
Addressing data quality challenges in data warehouses, lakes, and lakehouses requires a multifaceted approach. It involves the integration of cutting-edge technology like AI-powered tools, robust data governance, regular audits, and a culture that values data quality.
Digna is not just a solution; it’s a revolution in data quality management. It’s an intelligent, intuitive, and indispensable tool that turns data challenges into opportunities.
I’m not just proud of what we’ve created at DEXT.AI; I’m most excited about the potential it holds for businesses worldwide. Join us on this journey, schedule a call with us, and let Digna transform your data into a reliable asset that drives growth and efficiency.
Cheers to modern data quality at scale with Digna!
This article was written by Marcin Chudeusz, CEO and Co-Founder of DEXT.AI. a company specializing in creating Artificial Intelligence-powered Software for Data Platforms. Our first product, Digna offers cutting-edge solutions through the power of AI to modern data quality issues.
Contact me to discover how Digna can revolutionize your approach to data quality and kickstart your journey to data excellence.
Today’s guest blog post is from Benjamin Cutler of Winpure. In here Benjamin goes through a few things that you in a nutshell should know about master data management.
People
People have multiple phone numbers and multiple email addresses and in 2022 there must be several decades of historic contact information available for any one person. Most of us move at least once, every few years. Sometimes we go by different nicknames in different situations, some people even change their names. We hold different titles throughout the course of our careers and we change companies every few years. Only a few people in our lives know exactly how to get a hold of us, at any given time. Many of us change vehicles just as often as we change our hair color. Many of us are employees, most of us are also customers, many of us are spouses and sometimes we are grandparents, parents, aunts, uncles, and children at the same time. Sometimes we’re out enjoying ourselves and sometimes we just want to be left alone. We each have unique interests and desires, but we also have many things in common with other groups of people.
Products
Products have many different descriptions, they come in many different variations, different sizes, different colors, and different packaging materials. Similar products are often manufactured by different manufacturers, and they can be purchased from many different commercial outlets, at different price points. Any one product on the market at any one time will likely be available in several variations, but that product will also likely change over time as the manufacturer makes improvements. Products can be purchased therefore they can also be sold. They can also be returned or resold to other buyers, so there are different conditions and ways to determine product value. There are SKU and UPC numbers and other official product identification and categorization systems including UNSPSC and others, but none of them speak the same language.
Companies
Companies are made up of many different people who come and go over time. The company may change names or change ownership. It may have multiple locations which means multiple addresses and phone numbers, and they probably offer many different ways to contact them. Depending on where you look, there are probably more than a dozen different ways to find their contact information, but only some of those company listings will be correct. Companies have tax IDs and Employer IDs and DUNS IDs in the US, and there are many different systems worldwide.
Addresses
Addresses are the systems we use to identify locations. Each country and territory has its own system so each system is different. In the US we use premise numbers, street names with and without street prefixes and suffixes, we use unit numbers, states, counties, cities, towns and 5 and 9 digital numerical postal codes. Addresses and address systems can change over time, and they are inherently one of the most inconsistent forms of identification. Addresses are usually riddled with errors, misspellings, different structures and formatting, and they can be very difficult to work with. What makes this even more difficult is that the same address represented in multiple internal business systems will often be represented differently, and will rarely match the way the same address is represented externally.
Data
Data is a digital description of all of these things. Data usually comes in columns and rows and all shapes and sizes. Data about these things is captured, stored in business systems and it’s used to get work done. Need to call a contact? Check your contact data. Need to know a company’s billing address? Check your company data. Need to know something about a product? Check your product information. Need to know something about where your customers live and work or where to deliver the product? Check your address information. But here’s the thing: the information rarely matches from system to system and it’s very hard to keep up to date. This is especially difficult for a few reasons. Internally your company probably has many different business systems and many different ways of storing and representing these things, so it rarely matches internally, plus, the way that your company stores and represents this information will almost never match external information. How can you know the best way to contact your customer who has multiple phone numbers and multiple email addresses? If you’re searching some external system for updated information about some product or contact and the information doesn’t match, how do you find the new information? How can you know if your own information is correct and up to date? How can you scale your efforts to communicate with hundreds or thousands of customers at a time, communicating information that is specifically relevant for each of them? If the information doesn’t match or is not correct, how can you know who is who?
Relationships
The relationships across people, other groups of people, products, other groups of products, companies, other groups of companies, addresses, and other addresses, is where the rubber hits the road. Business value comes from connecting companies and products or services with other people and companies, and other products and services, at scale. Customers purchasing products might be interested in purchasing related products. Customers often buy things based on location. Companies selling to customers might be able to sell more, if they target similar customers in similar locations. Products and services also sell well based on location, and companies can optimize sales territories and delivery routes based on the relative proximity to other locations.
People and Technology
The people and technology between all of this, finds it difficult to keep up. People do things one by one and we’re good with ambiguity. We program computers and business systems to do things faster. Computers do things programmatically and very quickly but they’re not good with ambiguity. People can see similarity between things that are similar, but computers and business systems cannot. People might be good with troubleshooting and critical thinking, but computers and business systems are not. A computer program might be able to find the same customer in multiple systems and might be able to update that customer’s information all at once, but how can you know if the new information is the best information? Knowing that your customer probably has multiple phone numbers and multiple addresses and multiple nicknames, how can you know which information is correct? Doing this at scale can be very, very difficult.
In Conclusion
Master Data Management is very difficult but it’s fundamental in scaling your business. People can sell products door-to-door, but data and technology allow us to market, sell, deliver, and service our products and services, to tens and hundreds of thousands of people in milliseconds, regardless of the distance. Most organizations still view data as a cost of doing business but with the right investments in people, process, technology, and in data management, we can scale as worldwide organizations.
Many analyst market reports in the Master Data Management (MDM), Product Information Management (PIM) and Data Quality Management (DQM) space have a generic ranking of the vendors.
The trouble with generic ranking is that one size does not fit all.
On this list there is no generic ranking. Instead there is a service where you can provide your organization’s context, scope and requirements and within 2 to 48 hours get Your Solution List.
The selection model includes these elements:
Your context in terms of geographical reach and industry sector.
Your scope in terms of data domains to be covered and organizational scale stretching from specific departments over enterprise-wide to business ecosystem wide (interenterprise).
Your specific requirements covering the main requirements that differentiate the vendors on market.
Vendor capabilities.
A model that combines those facts into a rectangle where you can choose to:
Go ahead with a Proof of Concept with the best fit vendor
Make an RFP with the best fit vendors in a shortlist
Examine a longlist of best fit vendors and other alternatives like combining more than one solution.
The vendors included are both the major players on the market as well as emerging solutions with innovative offerings.
Any implementation of a Master Data Management (MDM), Product Information Management (PIM) and/or Data Quality Management (DQM) solution will need a business case to tell if the intended solution has a positive business outcome.
Prior to the solution selection you will typically have:
Identified the vision and mission for the intended solution
Nailed the pain points the solution is going to solve
Framed the scope in terms of the organizational coverage and the data domain coverage
Gathered the high-level requirements for a possible solution
Estimated the financial results achieved if the solution removes the pain points within the scope and adhering to the requirements
The solution selection (utilizing the Select Your Solution service on this site) will then inform you about the Total Cost of Ownership (TCO) of the best fit solution(s).
From here you can, put very simple, calculate the Return of Investment (ROI) by withdrawing the TCO from the estimated financial results.
You can check out more inspiration about ROI and other business case considerations on The Resource List.
![Business Case ROI TCO MDM PIM DQM [2]](https://mdmlist.files.wordpress.com/2020/05/business-case-roi-tco-mdm-pim-dqm-2.png?w=300)
The solution selection (utilizing the 
The Disruptive MDM / PIM / DQM List 