MIS: Data Management in Operational Systems

Contact Data in Operational Systems

• Standardizing a customer's name in operational systems is crucial.

Data Cleaning

• Data cleaning involves weeding out and fixing or discarding inconsistent, incorrect, or incomplete data.
• Specialized software tools are used for analyzing, standardizing, correcting, matching, and consolidating data.

The Challenge of Perfect Data

• Achieving perfect data is almost impossible due to the trade-offs in data quality.
• Companies may prioritize accuracy over completeness, or vice versa.
• Examples: a birth date of 2/31/25 is complete but inaccurate, while an address with "Denver, Colorado" without a zip code is accurate but incomplete.

Data Quality Audits

• Companies perform data quality audits to determine the accuracy and completeness of data.
• Most organizations set acceptable thresholds to balance quality and cost.
• Example: achieving 85% accuracy and 65% completeness for making good decisions at a reasonable cost.

Impact on Decision Making

• Low-quality data can significantly affect decision-making processes.
• Businesses must formulate strategies to maintain clean and high-quality data.

Maintaining Data Quality

• Regular audits and cleansing processes are essential.
• Specialized software tools help analyze, standardize, correct, match, and consolidate data.
• Ensuring data quality across multiple databases and systems, both internal and external, is crucial.

Dirty Data Problems

• Over 25% of critical data in Fortune 1000 companies will continue to be flawed (Gartner Inc.).
• Data may be inaccurate, incomplete, or duplicated.

The Problem of Dirty Data

• Dirty data is essential for maintaining quality data in data warehouses or data marts.
• It increases the effectiveness of decision-making.

Data Cleansing Process

• Data cleansing occurs first during the ETL (Extract, Transform, Load) process.
• It occurs again once the data is in the data warehouse.
• Ideally, scrubbed data is accurate and consistent.

Functionality of Data Lakes

• Data lakes can be queried for all relevant data when a business question arises.
• They provide a smaller dataset for analysis to help answer the question.
• Data lakes are often associated with Hadoop storage.

Hadoop Data Lakes

• Hadoop data lakes comprise one or more Hadoop clusters.
• They process and store non-relational data (e.g., log files, clickstream records, sensor data, images, social media posts).
• Hadoop data lakes support analytics applications, not transaction processing.

Data Lakes vs. Data Warehouses

• Data lakes are different from data warehouses in terms of their functionality and storage.

Importance of Data Quality

• Low-quality data costs U.S. businesses $600 billion annually (The Data Warehousing Institute).
• It affects decision-making, especially in advertising strategies.
• Dirty data is erroneous or flawed data that cannot be completely removed.