ML & Data Science: Bias-Variance and Imbalanced Data

• This document contains cheat sheets for Machine Learning and Data Science topics asked during interviews.

Bias-Variance Tradeoff

• Bias measures the error between average prediction and ground truth, indicating the model's capacity to predict values.
• Variance represents the variability in model predictions for a given dataset and how the model adjusts to dataset changes.
• High bias indicates an overly simplified model (under-fitting), while high variance signifies an overly complex model (over-fitting).
• Increasing bias can reduce variance and vice-versa.
• Error is calculated by: Error = bias² + variance + irreducible error.
• The best model reduces error through a compromise between bias and variance.

Imbalanced Data in Classification

• Accuracy, while a common metric, does not always provide correct insights for trained models.
• Precision measures the exactness of the model and Recall its completeness.
• The F1 Score combines Precision and Recall.
• In class 1, Precision = TP / (TP + FP) or True Positives / (True Positives + False Positives)
• In class 1, Recall, Sensitivity is (TP)/(TP+FN) or True Positives / (True Positives + False Negatives)
• Specificity = TN / (TN + FP) or True Negatives / (True Negatives + False Positives)
• False Positive Rate = FP / (TN+FP) or False Positives / (True Negatives + False Positives)
• Accuracy is (TP+TN) / (TP+TN+FP+FN) or (True Positives + True Negatives) / (True Positives + True Negatives + False Positives + False Negatives)
• Data replication, synthetic data generation, modified loss functions, and algorithm adjustments can address imbalanced data.
• Synthetic Data involves image rotation, dilation, cropping, and noise addition to create new data.
• Modify loss to reflect greater error when misclassifying smaller sample sets to address modified loss.

PCA Dimensionality Reduction

• PCA finds a new set of orthogonal feature vectors in a dataset to maximize data spread in the feature vector direction.
• Feature vectors are ranked in decreasing order of data spread (variance).
• Datapoints show maximum variance in the first feature vector and minimum variance in the last.
• Variance of datapoints in the feature vector direction indicates information measure.
• Steps: Standardize datapoints, find the covariance matrix, perform eigenvalue decomposition, and sort.
• Dimensionality Reduction Steps: Apply the steps above, keep the first m feature vectors from the sorted eigenvector matrix, transform the data for new basis, and note the importance of a feature vector is proportional to the magnitude of the eigenvalue.

Bayes Theorem and Classifier

• Bayes' Theorem describes event probability based on prior knowledge of related conditions.
• Bayes' Theorem is: P(A|B) = (P(B|A) * P(A)) / P(B).
• MAP Estimation: The MAP estimate of a random variable y, given observed data, involves accommodating prior knowledge during estimation.
• MLE (Maximum Likelihood Estimation) is a special case of MAP where the prior is uniform.
• Naive Bayes' assumes the features are independent.

Regression Analysis

• Regression analysis involves fitting a function f(.) to data points y=f(xi) under an error function.
• Linear Regression fits a line minimizing the sum of mean-squared error for each data point.
• Polynomial Regression fits a polynomial of order k minimizing the sum of mean-squared error.
• Bayesian Regression fits a Gaussian distribution by minimizing the mean-squared error for each data point.
• Ridge Regression fits a line or polynomial minimizing the sum of mean-squared error and the weighted L2 norm of the function parameters.
• LASSO Regression fits a line or polynomial minimizing the mean-squared error and the weighted L1 norm.
• Logistic Regression fits a line or polynomial with sigmoid activation, minimizing binary cross-entropy loss.

Regularization in ML

• Regularization addresses over-fitting in ML, reducing model variance.
• L2 Regularization prevents weights from becoming too large. Larger weights increase model complexity and chances of overfitting.
• L1 Regularization prevents weights from becoming too large (defined by L1 norm) and introduces sparsity.
• Entropy regularization is used for probability output models, pushing distribution towards uniformity.
• Data augmentation can be applied to modify data sampling
• K-fold Cross-validation divides the data into k groups, training on (k-1) and testing on 1.
• Injecting noise, adding random noise to the weights during learning.
• Dropout involves randomly dropping connections between consecutive layers in neural networks.

Convolutional Neural Network

• Data enters CNN via the input layer, passing through hidden layers to the output, and backpropagation updates the weights.
• CNN layers commonly include convolutional, pooling, normalization, activation, and loss calculation.
• Convolutional layers apply filters across feature maps generating dot products.
• Transposed Convolutional Layers increase output feature map size (Up-sampling).
• Pooling layers are non-trainable and change feature map size.
• Max/Average Pooling decreases spatial size, selecting max/average values defined by the kernel.
• UnPooling increases spatial size by placing input pixels in defined receptive fields.
• Various normalization approaches limit unbounded activation.
• Regression and Classification are two main types of Loss Functions

Famous CNNs

• AlexNet (2012): Consists of 5 CONV layers and 3 FC layers using ReLU activation and Local Response Localization.
• VGGNet (2014): Reduces parameters in CONV layers and improves training time using small conv kernels (3x3) and maxpool kernels (2x2).
• ResNet (2015): Connects layers to solve vanishing gradient problems, with multiple versions (ResNet50, ResNet101).
• Inception (2014): Uses lager kernels for more global features while smaller detect area-specific, needing kernels of different sizes.

Ensemble Learning in ML

• Ensemble Learning combines multiple weak models to improve bias, variance, and/or accuracy.
• Bagging trains N weak models with subsets in parallel. It reduces the variance in the prediction.
• Boosting trains N weak models sequentially, weighting misclassified points and decreasing bias.
• Stacking trains N models of different types with one subset, using a meta-learner for final prediction and improves accuarcy.

Autoencoder & Variational Autoencoder

• Autoencoders learns to find efficient embeddings of unlabeled data, with an encoder and decoder.
• Autoencoders compress data from higher to lower dimensions and back and is trained on loss function for reconstruction.
• VAE (Variational autoencoder) addresses non-regularized latent space, outputting parameters of a pre-defined distribution and imposes a constraint on this function, forcing it to be a normal distribution.
• Latent variable is smooth and continuous and defined by training loss which is the sum of the reconstruction loss and KL divergence between the unit gaussian and decoder output distribution.

Data Structures

• List: Ordered collection of elements accessed by index, in any order.
• Linked List: Elements with values and pointers, traversed sequentially.
• Stack: A sequential data structure accessed in LIFO order.
• Queue: A sequential data structure accessed in FIFO order.
• HashTable: Paired assignments accessed in constant time.
• Tree: Hierarchical relation between root, parent, child, and leaf nodes.
• Graph: A pair of sets (V, E) of vertices and edges, which can be cyclic.

Coding Interview Preparation

• Timeline: Start preparing 2-3 months, easier after some experience
• Review: Lists/Arrays, Linked List, Hash Table/Dictionary, Tree, Graph, Heap, Queue.
• Practice: on LeetCode.com, InterviewBit.com, HackerRank.com
• Listen, Talk, Discuss, Start Coding, Discuss, Optimize and Repeat

Behavioral Interview Preparation

• The STAR method can be used to organize the stories one tells during and interview

• Situation: provide the necessary context

• Task: Explain what you were responsible for

• Action: Provide steps that you took to address issue

• Result: State outcome of your actions