Data Mining Review and CRISP-DM Lifecycle

NumPy arrays store data contiguously in memory, making access and manipulation faster than lists due to locality of reference.

Analyzing data with one variable at a time.

A model's ability to balance error from simplifying assumptions (bias) with error from random fluctuations in the training data (variance).

The average squared difference between predicted and actual values in a regression model.

A range of values that likely contains the true value of a population parameter.

Data mining involves tasks like estimation, prediction (using regression, classification, or association), and clustering.

A standard process for data mining projects, including understanding the business problem, exploring the data, preparing the data, modeling the data, evaluating the results, and deploying the solution.

This stage of data mining involves cleaning, transforming, and preparing raw data for analysis to improve quality and consistency.

Techniques used to improve the quality of data, including removing problematic entries, columns, or clusters.

The process of preparing data for use in data mining, handling inconsistencies, missing values, outliers, and other issues to ensure accuracy.

Preprocessing data reduces issues like garbage input (GIGO) that can lead to inaccurate results by dealing with incomplete, noisy, and poorly formatted data.

Factors impacting the growing use of data mining include product commercialization, ongoing technology improvements, and external influences.

Data mining objectives are translating business requirements and goals into specific datamining problems that can be solved through the data.

The average of a dataset. Calculated by summing all the values and dividing by the number of values.

The middle value of a sorted dataset. If there are an even number of values, it's the average of the two middle values.

The most frequent value in a dataset.

A measure of the asymmetry of a distribution. Positive skewness means the tail is longer on the right, negative skewness means the tail is longer on the left.

Scaling data to a specific range, typically between 0 and 1. This ensures that all features have the same scale.

NumPy arrays store data in a contiguous block of memory, allowing for faster access and manipulation compared to lists. This efficiency is due to the principle of locality of reference.

Analyzing a single variable at a time to understand its characteristics and distribution. This helps identify patterns, trends, and outliers within that specific variable.

A range of values calculated from sample data that is likely to contain the true value of a population parameter with a certain level of confidence.

A graphical representation of data that summarizes its distribution through the minimum, maximum, median, and quartiles (25th and 75th percentiles).

The complexity of a model refers to its ability to fit the training data precisely. A model with more parameters and flexibility can capture intricate relationships, leading to high complexity.

Data mining focuses on extracting useful knowledge from data. Common tasks involve estimation, prediction (through regression, classification, association, or clustering), and finding patterns.

This crucial stage of data mining involves cleaning, transforming, and preparing raw data to improve its quality and consistency for analysis. This ensures the data is useful for modeling.

Raw data often needs preprocessing to address issues like incomplete or inconsistent data. This helps ensure accuracy in analysis and minimizes 'garbage in, garbage out' (GIGO).

Data cleaning removes problematic entries, columns, or clusters from the dataset. This improves the quality and reliability of the data.

Data mining has grown in popularity due to factors like commercialization of data-driven products, rapid technological advancements, and external pressures to leverage data for informed decision-making.

The initial stage of the CRISP-DM process involves understanding the business problem and acquiring knowledge about the available data.

A dataset where the classes are not equally represented. For example, in a dataset of customer reviews, there might be many more positive reviews than negative reviews.

A misleading metric for imbalanced datasets because it's heavily influenced by the performance on the majority class. For example, in a dataset with 90% positive cases, a model that predicts everything as positive would have 90% accuracy, but it's not a good model.

Out of all the positive predictions made by a model, how many were actually positive. It measures how precise the model is at identifying positive cases.

Out of all the actual positive cases, how many were correctly identified by the model. It measures how well the model can identify true positives.

A balanced metric that considers both precision and recall. It is the harmonic mean of the two. It's useful for balancing the trade-off between precision and recall, particularly in imbalanced datasets.

Resampling is a technique used to adjust the distribution of training data in order to minimize the impact of class imbalance.

Over-sampling involves adding more examples to the minority class in an imbalanced dataset. This is done to increase the representation of the less frequent class and make the model more sensitive to it.

SMOTE (Synthetic Minority Over-sampling Technique) is a method for generating synthetic examples of the rare class by combining existing examples. It uses a nearest-neighbor approach to create new data points.

Under-sampling involves removing examples from the majority class to reduce its dominance. This helps to reduce the bias towards the majority class and improve the model's performance on the minority class.

Tomek Links are pairs of examples from opposite classes that are very close together. Under-sampling techniques like Tomek Links identify and remove majority class examples from these pairs, helping to clarify the decision boundary and improve model performance.

Handling missing data is crucial because it can significantly affect the accuracy of analyses. Removing records with missing values can lead to biased results, as the pattern of missing data might be systematic, losing valuable information.

This method replaces missing values with a predetermined constant, like 0.0 for numeric values or "Missing" for categorical fields.

Missing values are replaced with the mean (for numeric data) or the mode (for categorical data).

While replacing missing values with mean or mode seems plausible, it can lead to overconfident results. Statistical measures can be artificially lowered, and the true variability of the data might be obscured.

Data imputation methods provide more sophisticated techniques to handle missing data, often incorporating relationships between variables to estimate missing values.

A single value that summarizes the central tendency of a dataset, indicating the typical value within the data distribution.

A method of standardizing data by converting raw values to a standardized score, where the mean is 0 and standard deviation is 1.

Metrics describing how spread out the data values are, providing insight into the variability within a dataset.

(X - mean(X)) / SD(X)

Yes, the mean is strongly influenced by extreme values (outliers) in a dataset, because it accounts for all values equally.

The simplest measure of spread, calculated by subtracting the minimum value from the maximum value in a dataset.

A normalization technique that scales data values by dividing by a power of 10, where the power is determined by the number of digits in the largest absolute value.

A measure of spread indicating how much data values typically deviate from the mean, taking into account the magnitude of deviations.

X* = X / 10^d

Normalization helps to ensure that all features have the same scale, preventing features with larger values from dominating the analysis.

A data scaling technique that transforms data values to a range between 0 and 1, by adjusting them based on the minimum and maximum values in the dataset. This ensures all features have the same scale.

'X' represents the original data value that you want to normalize. It is the individual value you are transforming to fit within the 0 to 1 range.

Normalization or standardization is used to prepare data for analysis or machine learning algorithms. It helps to prevent features with larger scales from dominating, ensuring all features have equal influence on the model.

The standard deviation measures how spread out the data points are from the mean. A higher standard deviation indicates greater variability in the data.

Raw data often contains inconsistencies, missing values, and outliers. Preprocessing cleans and prepares data for analysis, minimizing errors and improving accuracy.

Data cleaning involves removing problematic entries, columns, or clusters from a dataset to improve its quality and reliability.

A type of machine learning where the algorithm is given labeled data (input and desired output) to learn a mapping between features and target variables.

A type of machine learning where the algorithm is given unlabeled data and must find patterns or structures without explicit guidance.

A unsupervised learning technique that groups data points based on their similarity, creating clusters of related data.

The variable that the machine learning model aims to predict or understand in supervised learning.

The independent variables used to make predictions about the target variable in supervised learning.

Cross-validation is a technique used to estimate the performance of a machine learning model on unseen data. It helps prevent overfitting by evaluating the model on a separate test set.

A spurious artifact is a pattern in the training data that is not representative of the real world and could lead the model to make inaccurate predictions on new data.

A data analyst ensures that the training and test sets are independent, meaning they contain different samples of data, to reduce the likelihood of spurious patterns.

Model evaluation is crucial to determine how well the model will generalize to new, unseen data. It helps identify areas where the model needs improvement and ensures its reliability.

The goal is to minimize the error of the model on the test set, ensuring it makes accurate predictions on new data.

Ensuring the training and test sets have similar distributions of important features to avoid bias and improve model generalization. This prevents the model from performing well on the training data but poorly on unseen data.

It helps mitigate bias by training and testing on different folds of data, giving a more robust model evaluation. Each data point is used in the test set exactly once, making it efficient.

To uncover hidden patterns, insights, and valuable knowledge from large datasets to support decision-making, predict future trends, and improve business processes.

Missing values can bias results and lead to inaccurate models. Removing records with missing values can lose valuable information, and replacing them with simple constants might be misleading.

Scaling data to a similar range to ensure that all features have equal influence on the analysis and prevent features with larger values from dominating the learning process.

When a model learns the training data too well and fails to generalize to new data.

When a model is too simple and doesn't capture the underlying patterns in the data.

The model complexity that minimizes error on the test set, balancing model accuracy with generalizability.

The goal is to find the sweet spot where the model is complex enough to capture the patterns in the data, but not too complex that it overfits and loses its ability to generalize.