ICT 462-3 Week 2: Data Preparation Techniques

The process of transforming raw data into a format suitable for analysis and modeling.

It involves cleaning, transforming, and integrating data to improve its quality and make it more suitable for analysis.

A crucial step in data preprocessing, as it involves identifying and correcting errors, handling missing values, and resolving inconsistencies in the data.

It involves replacing missing values with sensible estimates based on available data or using algorithms.

These values can be due to errors, incomplete data collection, or other factors. They can significantly impact the accuracy of models.

Improving data quality by identifying and resolving inconsistencies or errors in the data.

Data transformation methods involve changing the format or representation of the data to make it more suitable for analysis.

The process of combining data from multiple sources into a single, unified dataset.

Replacing missing values with the average (numerical) or most frequent value (categorical) in the dataset.

Predicting missing values based on the values of other features using machine learning algorithms like K-Nearest Neighbors (KNN) or Random Forest.

Data points that significantly deviate from the general trend or pattern in the dataset.

Random errors or variance in a measured variable. It can be introduced by measurement tools, data processing, or experts during data collection.

A technique used to reduce noise in the data by smoothing out fluctuations while preserving essential patterns.

A smoothing technique that partitions data into intervals and replaces each value with the average of the values within its interval.

A smoothing technique that uses regression models to fit a line or curve to the data and then replaces original values with predicted values from the model.

Ignoring tuples with missing class labels, often used in classification tasks. However, it's not very effective unless there are numerous missing attribute values in the tuple. This method leads to losing potentially valuable information.

Replacing missing values with a reasonable substitute, aiming to retain most of the dataset's information. Used to prevent data loss and avoid bias.

Replacing missing values with the average of the existing values in the column. Works best for numerical data with a normal distribution.

Replacing missing values with the median of the existing values in the column. Suitable for numerical data that is skewed.

Replacing missing values with the most frequent value in the column. Useful for categorical data.

Replacing missing values with a chosen fixed value. Applicable across all data types. Useful for nominal features where 'not answered' could be used as a fixed value.

Using the value before or after the missing value for time series data. Takes advantage of the order of data.

Replacing missing values with the minimum or maximum value within a defined range. Useful when data must fit within specific boundaries.

Values falling outside this range are considered outliers. The formula is: Lower Bound = Q1 - 1.5 * IQR, Upper Bound = Q3 + 1.5 * IQR

It isolates data points that require fewer splits to be separated from the rest of the data in a decision tree.

Grouping similar data points (using distance or density) and treating outliers as separate clusters. Outliers don't fit well within a cluster.

A process that reduces the volume or dimensionality of data while preserving as much information as possible.

It makes analysis more efficient, reduces storage costs, and allows algorithms to run faster with large datasets.

Techniques that reduce the number of features (dimensions) in a dataset while preserving key information. This simplifies models, decreases overfitting, and speeds up processing.

A statistical technique that transforms original features into uncorrelated components. It works by finding the directions of maximum variance in the data.

A non-linear method for visualizing high-dimensional data in 2D or 3D. It preserves local structure, keeping similar data points close together after reduction.

The process of selecting the most relevant features, discarding irrelevant or redundant ones, to improve model performance.

Techniques that utilize statistics to determine the importance of features. Examples include correlation analysis, variance thresholding, and hypothesis testing.

A step in PCA where the data is transformed into a new coordinate system defined by the principal components. This creates a new space that captures the most variance in the data.

The measure of how much variables change together. A high covariance indicates a strong relationship between features.

Values that determine the direction of maximum variance in the data. They represent the principal components in PCA.

Techniques like forward selection, backward elimination, and recursive feature elimination (RFE) are used to test subsets of features to determine the best combination.

These methods, such as regularization (e.g., Lasso), automatically perform feature selection during the model training process.

A technique that reduces large datasets by selecting a smaller random sample, representing the larger dataset.

Rescales data to fit within a specific range, usually between 0 and 1. It's used when features have different scales or ranges, ensuring no feature dominates due to its scale.

Transforms data to have a mean of 0 and a standard deviation of 1, assuming the data follows a normal distribution.