CRISP-DM Framework and Industry 4.0 Components

• Exam is on Monday, March 11th, 2024.
• Focus on easy questions, manage time wisely, and review thoroughly.
• The quiz has multi-response questions with at least one correct and one incorrect option.
• Approach each question critically and consider all possibilities before answering.

CRISP-DM Framework

• Direct data interaction is present during the "Data Understanding" and "Data Preparation" phases.
• Data Understanding involves getting acquainted with the data's content, quality, and structure.
• Data Preparation includes constructing the final dataset from raw data through tasks like table selection, cleaning, and transformation.
• Business Understanding focuses on project objectives and converting knowledge into a data mining plan.
• Modeling involves selecting and applying modeling techniques and calibrating their parameters.
• Evaluation assesses how well the model meets business objectives.

Industry 4.0 Key Components

• Key components include: Internet of Things (IoT), Big Data Analytics, Cyber-Physical Systems, and Cloud Computing.
• Traditional Manufacturing Processes are not a driving factor.
• IoT connects devices and machines in factories for real-time data exchange.
• Big Data Analytics analyzes large data volumes to uncover patterns for better decision-making.
• Cyber-Physical Systems integrate computational processes with physical processes.
• Cloud Computing provides the infrastructure for data storage and processing.
• Industry 4.0 revolutionizes traditional manufacturing through digital transformation.

Big Data '4Vs'

• Essential characteristics are Velocity and Volume.
• Velocity refers to the speed at which data is generated and processed.
• Volume denotes the amount of data, which is beyond traditional databases' capacity.
• Variety represents the different types of data, not Versatility.
• Veracity refers to the quality and credibility of data, not Validity.
• Volatility relates to how data's relevance changes over time.

Addressing Missing Values

• Important for accuracy of statistical models, completeness for analysis, and preventing biases.
• Missing values distort statistical properties.
• Incomplete data affects reliability of conclusions.
• Unaddressed missing values can introduce biases.
• Addressing missing values does not inherently increase the number of features nor reduce computational load.

Identifying and Addressing Outlier Values

• Enhances the robustness of statistical models.
• Prevents skewed interpretations of data trends.
• Ensures accurate predictions and analyses.
• Improves the quality of data visualization by optimizing the scale.
• Removing or handling outliers makes models more representative.
• Addressing outliers does not inherently increase the data set's size.

Combining and Analyzing DataFrames (df1 and df2)

• Merge df1 and df2 on EmployeeID to get a complete view of employees and their projects.
• Use a left join to merge df1 with df2 also on EmployeeID to ensure all employees are listed.
• Aggregating df1 by Department does not directly relate to combining information from both data frames in a meaningful way.
• Summarizing df2 to calculate the average number of projects per employee does not consider the context of df1.
• Concatenating df1 and df2 vertically would not align employee information with their respective project.

Data Cleaning and Preprocessing

• Impute missing numerical values using the median of the column to avoid the influence of outliers.
• Remove rows with outliers in numerical columns after defining a threshold.
• Impute missing categorical values using the mode of the column or a placeholder value like 'Unknown'.
• Use robust scaling techniques on numerical data to mitigate the effect of outliers on scaling.
• The median is less affected by outliers and is suitable for inputting.
• Outliers can skew data results and need identifying.
• Missingness needs to be addressed.
• Approaches ensure integrity.

Principal Component Analysis (PCA)

• It reduces dimensionality while preserving variability.
• Transforms original variables into linear combinations.
• PCA is not a classification method.
• It seeks to reduce the number of variables not increase
• PCA aims to preserve variability not minimize.
• Compute eigenvectors and eigenvalues from data's covariance matrix
• Eigenvectors represent direction of maximum variance
• Eigenvalues show captured variance be each principal component
• Covariance Matrix is used to know correlations between variables
• PCA assumes components are orthogonal and uncorrelated

Multidimensional Scaling (MDS)

• Represents high-dimensional data in a lower-dimensional space, preserving distances.
• Visualizes similarity/dissimilarity in an easy-to-interpret way.
• Uncovers the underlying structure by analyzing the distance matrix.
• It does not cluster data into groups.
• It does not predict outcomes using linear regression.

Distance Measures and MDS

• The choice of distance measure impacts the MDS output.
• Euclidean distance is effective for geometric distances.
• Cosine distance is suitable for comparing directions/orientations.
• Different distance measures will give different results.

Dataset Splitting

• Evaluate the model's performance on unseen data and avoid overfitting.
• Fine-tune model parameters and select the best model architecture using the validation set.
• Splitting the data doesn't create new data it partitioned it.
• Primary point isn't to minimize data use. It's for evaluation
• Test set is a proxy for new test data. Crucial for accuracy
• Validation set is used to test for configurations

K-Fold Cross-Validation

• Increases reliability by averaging results, decreasing variance.
• Shuffles the dataset before splitting to assess performance better.
• Models are trained and validated on multiple datasets.
• It does not test what the code is for training
• It test exactly one for all dataset usages

Simple Linear Regression

• Assumes a linear relationship between variables.
• Assumes normally distributed residuals.
• Assumes homoscedasticity.
• For modeling linear predictor
• One predictor only

Supervised Learning

• Requires input features and target labels.
• Used for classification and regression.
• Aims to make predictions on new data.
• Models are evaluated on test data accuracy.
• Not used for clustering

Classification vs. Regression

• Classification is for categorical outcomes, while regression is for continuous numerical outcomes, predicting classes rather than values. They use categorical too

Regression Analysis

• Use to predict a store's value regarding economic trends
• Not for categorising trends/spam identifying
• Best for outcomes

Regression Perforamnce

• Mean Squared Error measures difference between actual
• Not accuracy

Multiple Regression Model

• There is also the error term
• The betas all showcase impact
• In the intercept the intercept, not data

XGBoost

• Handles missing data automatically.
• Offers efficient gradient boosting implementations.
• Supports linear model solvers and tree learning.
• Does not help for JavaScript directly
• Cannot get without extra preprocessing

Deep Learning

• Approximates nonlinear functions.
• Neural networks can be adapted for tasks
• Convolutional means can be effective and continuous
• Functions require appropriate design
• Can be catered as numerical

Clustering Methods

• Groups objects by similarity; identifies patterns.
• A unsupervised learning technique.
• Uses various similarity measures.
• Can be hierarchical or partitional.
• Clusters are helpful data that give you an idea for data

K-Means Clustering

• Minimizes within-cluster variances but doesn't guarantee a global optimum.
• Requires the number of clusters to be specified in advance.
• The dataset is required to be specified on clusters, which is wrong -Not highly effective for managing noise
• The system fails if too much noise is involved

Partitioning vs. Density-Based Clustering

• K-means requires specified cluster, density can determine data density
• Density are meant for managing data