Module 2:Data Preprocessing1: Improving Data Quality

• The key learning outcomes for the week are to recognize major tasks in data preprocessing and to perform data cleaning and integration.

Data Quality and Preprocessing

• Preprocessing improves data quality, especially for accuracy, which refers to the correctness of data. Incorrect attribute values can arise from faulty instruments or human error.
• Improves completeness of data, meaning that full information is avialable
• Improves consistency; data is same across all sources
• Improves Timeliness: time difference and delay
• Improves Believability: reflects how much the data is trusted by users
• Improves Interpretability: reflects how easy the data are understood.

Major Tasks in Data Preprocessing

• Data cleaning fills in missing values, smooths noisy data, identifies/removes outliers, and resolves inconsistencies.
• Data integration integrates data from multiple sources like databases, data cubes, or files.
• Data reduction obtains a smaller representation of the dataset that still produces the same results.
• Data transformation converts data into appropriate forms to improve mining results.

Data Cleaning

• Real-world data is often dirty, with potentially incorrect information due to instrument faults, or human/computer errors.
• Incomplete data lacks attribute values and may contain only aggregate data. For example: Occupation=“” represents missing data.
• Noisy data contains noise, errors, or outliers, like Salary=“-10."
• Inconsistent data contains discrepancies in codes or names. An example can be Age="42" wheras Birthday=“03/07/2010."

Incomplete Data

• Data is not always available.
• Missing data can occur because:
  • Due to equipment malfunction
  • Inconsistencies
  • Data entry errors
  • Data irrelevance
  • Change in history

Handling Missing Values

• Ignore the sample where the percentage of ignored samples is too high, which is not effective
• Fill in manually, which is tedious and infeasible
• Fill in automatically, which requires:
  • Global constant, like "unknown"
  • Attribute mean for all samples or for a specific class
  • The most probable value using statistical models

Noisy Data

• Noise is random error in data.
• Incorrect attribute values arise from faulty instruments, data entry/transmission problems, technology limitations, and naming convention inconsistencies.
• Other data problems wich require cleaning includes incomplete data, duplicate records and inconsistent data

Handling Noisy Data

• Binning is a technique that involved first sorting the data and partitioning into equal-frequency binds.
• Regression smooths the data by fitting it into regression functions.
• Clustering detects and removes outliers.
• Combined computer and human inspection deals with possible outliers by detecting suspicious values and checking manually.

Data Cleaning as a Process

• Data discrepancy detection involves:
  • Utilizing metadata (domain, dependency, distribution)
  • Checking field overloading, uniqueness/consecutive/null rules.
  • Data scrubbing using domain knowledge
  • Using data auditing for outlier detection via rules and relationship analysis.
• Data migration and integration allows transformation tools and ETL tools for specifying transformations graphically.
• Integration of the processes are best when iterative and interactive leveraging Potter's Wheels.

Data Integration

• Combines data from multiple sources into a coherent store via data integration.
• Schema integration: e.g., A.cust-id = B.cust-# requires integration metadata from various sources
• Entity identification involves identifying real-world entities accross sources.
• Detecting and resolving data value conflicts can arise since attribute values may differ, possibly due to different scales like metric vs British units.

Handling Redundancy in Data Integration

• Occurs when integrated from multiple data sources
• Object and Derivable data can contribute to this
• Integration should reduce redundancies to improve mining speed and quality.

Redundancy Detection

• Can use correlation analysis for redundancy detection
• Use a chi-square test (X²) for nominal (categorical) attributes to assess their correlation, focusing on the differences between actual and expected counts to identify dependencies.
• Apply correlation coefficients and covariance to numeric attributes to determine how values vary across attributes, understanding the strength and direction of their relationships.

X² Correlation Test

• The test utilizes a contingency table that lists data tuples described by distinct values of attributes A and B.
• X2= the sum ( (observed - expected)^2 / expected )
• A statistic tests the hypothesis of independence between A and B. The test’s degrees of freedom are defined by [(r-1) x (c-1)], dependent on a significance level.

Correlation Example

• In a group surveyed (n=1500), data reveals that 250 men prefer fiction, 50 prefer non-fiction, and 200 women prefer fiction, versus 1000 for non-fiction.
• For 1 degree of freedom, the X2 value needed to reject the hypothesis is 10.828
• The computed value, X² = 507.93 indicates correlation between gender and preferred reading type.

Covariance

• The covariance and correlation coefficient measure the extent to which two numeric attributes change together.
• covariance : The sum of (ai - A) (bi – B)/number of tuples
• A positive covariance means that if A is larger than its expected value, B tends to be larger than its expected value - and vice versa. Negative = smaller.
• Independence, COVA,B = 0 but the converse is not true

Co-Variance Example

• Two stocks (A, B) have the following values in one week: (2, 5), (3, 8), (5, 10), (4, 11), (6, 14).
• The mean for A is: 4. For B: 9.6
• Covariance(A,B) results in 4
• These variables rise together. Cov shows that their relationship is > 0

Weka Tool

• WEKA provides many algorithms, datasets, tools for transforming datasets, ability to train, preprocess and analyze without writting program code.
• Complete Date Mining with WEKA can be viewed ad: https://www.youtube.com/user/WekaMOOC)