Data Governance and Profiling

Data Governance

• Data profiling involves identifying distributions, key candidates, foreign-key candidates, functional dependencies, embedded value dependencies, and performing inter-table analysis.

Data Profiling

• Structure discovery: validating data consistency, format, and structure, and examining simple statistics (minimum, maximum, means, medians, and standard deviations).
• Examples: identifying date patterns (YYYY-MM-DD or YYYY/DD/MM) and phone number formats (correct number of digits).

Content Discovery

• Examining individual attributes and data values to identify data quality issues.
• Helps find null values, empty fields, duplicates, incomplete values, outliers, and anomalies.
• Example: 'State' field containing two-letter abbreviations or fully spelled-out city names, and validating databases with predefined rules.

Relationship Discovery

• Discovering relationships between data parts, critical for designing database schemas, data warehouses, or ETL flows.
• Examples: key relationships between database tables, references between cells or lookup cells in spreadsheets, and joining tables based on key relationships.

Data Profiling Techniques

Data Completeness

• Involves identifying missing values and calculating the percentage of missing values for each variable.
• Helps determine the extent of missing data and its potential impact on analysis and decision-making.

Data Uniqueness

• Identifying duplicate records to maintain data integrity.
• Highlights data quality issues, such as data entry errors or system glitches.

Data Patterns

• Analyzing data patterns to assess format, structure, and regularities in data values.
• Useful for understanding naming conventions, data formats, and potential data quality issues.

Data Anomalies

• Detecting anomalies to identify unexpected or erroneous data values.
• Outliers, unexpected distributions, and extreme values can indicate data quality issues or data entry errors.

Data Dependencies

• Analyzing relationships and dependencies between variables to understand how different variables or attributes are related.
• Reveals correlations, associations, or dependencies between variables, useful for data exploration and modeling.

Data Quality Management

• Data quality: the fitness of data for its intended use, encompassing accuracy, completeness, consistency, timeliness, and relevance.
• Data quality management: ensuring data meets desired quality standards.
• Poor data quality can lead to incorrect insights, flawed decision-making, and operational inefficiencies.

Data Quality Management Process

• Profiling: reviewing source data, understanding data structure, content, and interrelationships.
• Cleansing and Remediation: correcting data quality issues.
• Monitoring and Validation: ensuring data quality standards are met.
• Maintenance and Verification: ongoing data quality management.