Data Cleaning in Machine Learning

Feature Engineering Module 2: Data Preprocessing

• Real-time data is often noisy, missing values, and inconsistent due to large size (often gigabytes or more) and heterogeneous sources.
• Irrelevant features significantly decrease model accuracy, requiring preprocessing.

Major Tasks in Data Preprocessing

• Data cleaning (removing noise and inconsistent data)
• Data integration (combining data from multiple sources into a coherent store)
• Data reduction (reducing data size by aggregation, eliminating redundant features, or clustering)
• Data transformations (e.g., normalization, scaling data to a specific range, like 0.0 to 1.0)

Data Preprocessing Techniques

• Data cleaning: Removing noise, fixing inconsistencies (missing values, outliers, etc.) in data.
• Data integration: Combining data from various sources into a consistent data store.
• Data reduction: Decreasing data size, e.g. through aggregation, feature elimination or clustering.
• Data transformations: Rescaling, normalizing, or applying other transformations to data.

Reasons for Inaccurate Data

• Faulty data collection instruments
• Human or computer errors during data entry
• Deliberate submission of incorrect data
• Errors in data transmission (e.g., limited buffer size)
• Inconsistent data formats or naming conventions
• Duplicate data

Reasons for Incomplete Data

• Missing data due to the attribute not being available.
• Missing data due to misunderstanding or equipment malfunctions
• Inconsistent data leading to deletion
• Data history or modifications being overlooked
• Missing data needing inference

Data Quality Factors

• Timeliness: Expected accessibility and availability of the data. Measured as the time between when it's expected and when it's readily available for use.
• Believability: Reflects how much users trust the data.
• Interpretability: How easy the data is understood.

Handling Missing Values

• Deletion: Removing rows or columns with missing values. Can be done by removing the entire row or column that contains missing values.
• Imputation: Replacing missing values with estimated ones (mean, median, average, polynomial interpolation). Methods: manual imputation or methods such as SimpleImputer, KNNImputer or IterativeImputer

Handling Noisy Data

• Noise: Random error or variance in a measured variable.
• Outlier detection: Identifying unusual, extreme values.
• Data smoothing: Techniques that mitigate the noise. Methods such as binning.

Binning

• Binning Methods: Distributing sorted data into buckets/bins.
• Smoothing by bin means: Replacing each value in a bin with the bin's mean.
• Smoothing by bin median: Replacing each value with the bin's median.
• Smoothing by bin boundary: Replacing each value with the nearest bin boundary (minimum/maximum).

Identifying or Removing Outliers

• Outliers: Data points significantly deviating from other values.
• Detection methodologies: Clustering techniques, box plots, Z-score.
• Outliers can be the result of an error or can be a relevant piece of data.

Resolving Inconsistencies

• Discrepancies: Inconsistent data values, formats, representations.
• Causes: Poorly designed data entry, human error, deliberate mistakes, data decay, inconsistent use of codes, instrumentation errors, and system errors.
• Resolution: Data cleaning using tools like data scrubbing and data auditing.

Data examination

• Examining data regarding unique, consecutive, and null rules.
• Unique rule: each value being different from all others in the attribute
• Consecutive rule: no missing values between lowest and highest values, and all values are unique
• Null rule: specifying how blanks, question marks, or special characters (indicating missing values) are handled.

Data Cleaning Tools

• Commercial tools: Data scrubbing, data auditing, data migration and ETL (extraction, transformation, loading) tools.
• Examples: Potter's Wheel, publicly available data cleaning tools that integrate discrepancy detection and transformation.

Methods Used

• Interactive manipulation: allowing dynamic correction of the data in real time
• Visual feedback: visual cues to identify problems such as anomalies or missing data
• Customizable rules: enabling rules of correction to be tailor made to the specifics of the data set/organization.
• Transformations: to enable correction of problems by changing data values, combining columns, or applying calculations