Nettoyage des Données - Concepts Clés

Introduction to Data Mining - Chapter 2: Data Preprocessing

• Data preprocessing is crucial for effective data mining
• Real-world data often contains errors, inconsistencies, missing values, and outliers
• Data cleaning tasks include handling missing values, smoothing noisy data, identifying and removing outliers, and resolving inconsistencies

Why Data Preprocessing?

• Data in the real world is often dirty, needing cleansing
• Incomplete data lacks certain attributes or contains only aggregate data (e.g., occupation="")
• Noisy data contains errors or outliers (e.g., Salary="-10")
• Inconsistent data has discrepancies in codes or names (e.g., age, birthday, ratings, discrepancies in duplicate records)

What is Data?

• Data is a collection of data objects and their attributes
• An attribute is a property or characteristic of an object, like eye color, temperature etc
• Attributes/Features are variables, or fields, or characteristics.
• Objects are also known as records, points, cases, samples, entities, or instances

Types of Attributes

• Nominal attributes represent categories (e.g., ID numbers, eye color, zip codes)
• Ordinal attributes involve ordered categories (e.g., rankings, grades, height in tall, medium, short)
• Interval attributes use a scale with a constant difference between units (e.g., calendar dates, temperatures in Celsius)
• Ratio attributes have a true zero point, allowing for ratios between values (e.g. temperature, length, time, counts)

Discrete and Continuous Attributes

• Discrete attributes have finite or countable infinite values (e.g., zip codes, counts, vocabulary)
• Continuous attributes have real numbers as values (e.g., temperature, height, weight)

Data Quality

• Data quality problems include noise, outliers, missing values, and duplicates
• Noise refers to modifications in original values (e.g., distortion of voice, screen "snow")
• Outliers are data objects with significantly different characteristics from others
• Missing values occur when information is not collected or attributes are not applicable to all cases
• Duplicate data sets include identical or similar data objects

Major Tasks in Data Preprocessing

• Data cleaning: fill in missing values, remove outliers, smooth noisy data, resolve inconsistencies
• Data integration: merges data from multiple sources
• Data transformation: Normalization, Aggregation and Generalization, Feature construction. These normalize data, summarize data or generalize data to a higher level (like from cities/states to regions), or create new features.
• Data reduction: reduce data volume, like through aggregation, sampling, dimensionality reduction, or creation of new features.

Forms of Data Preprocessing

• Illustrations showing the process of data integration and data transformation
• Diagrams demonstrate a visual representation of processes

###Data Cleaning

• Importance: Data cleaning is one of the biggest problems in data warehousing.

• Tasks: Fill missing values, smooth noisy data, identify and remove outliers(using methods like binning, regression, or clustering), correct inconsistent data, resolve redundancy.

How to Handle Missing Data

• Ignoring tuples: Mostly in classification tasks when class labels are missing, but not effective if missing values per attribute vary.
• Manually filling values
• Automatically filling with global constants ("unknown"), attribute means, class specific means, most probable values( using statistical formulas like Bayesian or regression)

How to Handle Noisy Data

• Binning (partition data into bins)

• Regression (fitting to a function)

• Clustering (group similar data points)

• Combined computer and human inspection (checks by human)

Binning Methods

• Data sorting and partitioning (equal frequency/equi-depth bins: data with equal frequency are grouped together)

• Smoothing by bin means, median, or boundaries (calculation of average values, median values, and boundaries for data bin)

• Illustrations showing the process.

Data Integration

• Combine data from multiple sources
• Schema integration: integrate metadata from different sources
• Entity identification problem: identify real world entities from multiple data sources ( e.g., Bill Clinton = William Clinton)
• Detecting and resolving data value conflicts (identifying if value conflicts exist): differences in values for same entities from different sources. Explains that conflicts can be caused by different representations of the data, or scales used.

Handling Redundancy

• Redundant data arises when integrating multiple databases

• Data objects/attribute names might be different across sources

• Derivable data (one attribute may be derived from another).

• Correlation analysis

• Carefully integrating data can reduce and avoid inconsistencies, improving data mining speed and quality.

Correlation Analysis

• Correlation coefficient (Pearson's product moment) helps find relationships.

• Examines cross products to identify relationship between variables. Numerical data (use a correlation coefficient)

• Categorical data (use X2 or chi-square test)

• X2 test measures association between categorical variables

• Values closer to 1 or -1 shows stronger correlation (larger values of X2 imply stronger correlation)

• Correlation does not imply causality (e.g. hospitals, car thefts)

Data Transformation

• Smoothing: reduce data noise

• Aggregation: summarize data (e.g., building a data cube)

• Generalization: move to higher level concept hierarchy (e.g., cities to regions)

• Normalization: scale a range to a specific, small range (min-max normalization, z-score normalization, normalization by decimal scaling)

• Attribute/feature construction: build new attributes based on given ones.

Normalization

• Min-max normalization: scale values between certain minimums and maximums

• Z-score normalization: converts data values to their z-scores. Use mean and deviation

• Decimal scaling: Adjust by the power of 10.

• Illustrative examples of each normalization method

Data Reduction

• Purpose: reduce data volume but preserve important information (e.g., use reduced dataset instead of complete dataset, reducing processing time)

• Strategies: Aggregation, Sampling, Dimensionality reduction, Feature subset selection, Feature creation, Discretization, Attribute Transformation. These each differ in the process by which they reduce the data.

Data Reduction : Aggregation

• Combining attributes or objects
• Can reduce attribute counts, or change unit of data
• Example: Aggregating cities into regions
• Aggregated data tends to be less variable.

Data Reduction: Sampling

• A main strategy to reduce data

• Sample often for preliminary investigation or final data analysis

• Sampling Types: simple random, without replacement, with replacement

Data Reduction: Dimensionality Reduction

• Techniques: PCA, Singular Value Decomposition, other supervised and nonlinear approaches

• Goal: capture most variation in data through projection into new space.

• Explanations of the techniques and their uses

Data Reduction: Feature Subset Selection

• Choosing subset of features that are sufficient/relevant

• Redundant features (duplicate information): use example of purchase price vs. sales tax

• Irrelevant features (contain no relevant information): use example of student ID to predict GPA

• Techniques: Brute-force approach, filter approaches, wrapper approaches

Data Reduction: Feature Creation

• Create new attributes

• Three methodologies: Feature extraction (domain-specific), mapping data to a new space, feature construction (combining features)

Data Reduction: Mapping Data to a New Space

• Mapping to new space for analysis (fourier transform, wavelet)

Data Reduction: Discretization

• Transforming continuous data to discrete (categorical)

  • Entropy-based approach using class labels
  • Discretization without using class labels (e.g., equal interval width, equal frequency, K-means)

Data Reduction: Attribute Transformation

• Function mapping values (e.g., x², log(x), eˣ, |x|)
• Standardizing and normalizing data is also a key transformation

Testez vos connaissances sur le nettoyage des données, y compris les tâches essentielles, les méthodes pour traiter les données manquantes et l'importance du nettoyage dans l'entreposage des données. Ce quiz vous permettra d'évaluer votre compréhension des techniques liées à l'analyse de données et à l'intégration des données.