Deep Learning: Data Handling and Augmentation

• Computer Vision presentation by Naeemullah Khan, held on February 17, 2025, at KAUST Academy
• The presentation covers Data Handling, Augmentation, Transfer Learning, Ensembling, Dropout, Batch Normalization and Full Training Workflow.
• The goal is to implement efficient techniques, apply augmentation strategies, utilize transfer learning, understand ensemble methods, apply regularization techniques, and design a deep learning training workflow
• Practical implementation of Deep Learning algorithms involves both art and science, focusing on building upon previous knowledge
• The presentation will look at the important tools used in the practical implementation of Deep Learning algorithms

DataLoaders

• Deep Learning has been made possible by large amounts of data and computational resources
• Data handling is an important aspect: It involves handling large data, reading components from different parts of storage, and feeding data to SGD algorithms in a streamlined manner
• PyTorch provides Dataset and DataLoaders for efficient data handling
• There will be an extension of the Dataset and DataLoaders class to construct custom Dataloaders
• Datasets in PyTorch: torch.utils.data.Dataset class is an abstract base and used to define and manage datasets for training and evaluation
• Custom Dataset Creation: A custom dataset is created by subclassing Dataset, implementing __len__() to return dataset size, and __getitem__() to fetch a single data sample
• torch.utils.data.DataLoader is used to load data in mini-batches, shuffle data, and utilize multiprocessing for efficiency
• Built-in Datasets: PyTorch provides datasets in torchvision.datasets and torchtext.datasets, which facilitates loading common datasets like MNIST, CIFAR-10, and IMDB

Data Augmentation

• Data is fundamental for any machine learning algorithm
• Data Augmentation techniques improve model performance when access to unlimited data is unavailable
• Focus on data rather than architecture search in deep learning
• Data augmentation creates virtual training samples, including horizontal flips, random crops, color casting, geometric distortion, translation, and rotation
• Multiple augmentations exist, so choose the right ones for the task
• Error Analysis: Identify model weaknesses and apply augmentations that address those issues
• Steps of Error Analysis:
• Train a baseline model.
• Make predictions on validation data.
• Inspect the worst predictions to identify model weaknesses.
• Apply relevant augmentations to address these issues.
• Types of Errors:
• Failure with small objects means to Use Scale Augmentation.
• Failure with different colors/environments means to Use Color Augmentations.
• Failure with rotated images means to Use Rotation Augmentations.
• Failure with blurry images means to Use Noise Augmentations.
• Data augmentation is only applied to Training data for proper evaluation and performance metrics
• Test-Time Augmentation (TTA) is where multiple augmented versions of the same image are passed through the model separately, and the predictions are averaged to improve accuracy
• Applying TTA requires custom scripts to generate augmented versions of test images, pass them through the model, and aggregate predictions

Transfer Learning

• Fine-Tuning is a strategy in transfer learning where a pre-trained model is adapted to a new task.
• Using fine-tuning, is starting from a model already trained on a related task instead of training from scratch
• Fine-Tuning saves time, computational resources and Improves performance, especially with limited data
• When to Finetune your model
• New dataset is small and similar distribution to original dataset: Freeze (or partially freeze) feature extraction layers and fine-tune the classifier
• New dataset is small and different distribution to original dataset: Use the pretrained network as a generic feature extractor and train a light classifier on top (e.g., SVM) or freeze earlier layers and selectively fine-tune later layers
• New dataset is large, regardless of the original data distribution: Fine-tune the entire network (both features extractor and classifier).

Ensembling

• No single model is perfect; different models make different types of errors
• Combining the predictions of diverse models can reduce errors and improve accuracy
• Bagging (Bootstrap Aggregating): Train multiple models on different subsets of data (e.g., Random Forest model)
• Boosting: Train models sequentially, where each model focuses on the errors of the previous one (e.g., AdaBoost, Gradient Boosting)
• Hard Voting combines classifier predictions where each model votes for a class, and the class with the majority of votes is selected
• Soft Voting combines classifier predictions where the predicted probabilities from each model are averaged, and the class with the highest average probability is selected
• Team-work is the best policy, and thus training multiple networks for the same task then ensemble to get better results
• Simple Analysis with N elements and an ensemble of M models: probability of error of the label by p(e) is Independent and Identically Distributed (i.i.d) : For example, with M = 3 and e = 0.01

Dropout

• Dropout is a regularization technique used to prevent overfitting in neural networks
• During training, it randomly drops (sets to 0) a fraction of neurons in each layer based on a specified probability for every forward pass.
• Dropping a subset of neurons during each forward pass, helps the network not rely on specific connections, and dropout acts as an efficient ensemble of multiple smaller networks, leading to less overfitting
• Neurons are dropped during training, but during inference, all neurons are used
• Problem: Inconsistent values during inference compared to training can cause the network to produce worse results Solution: Scale the output during inference by p to keep the same expected activation as in training
• In PyTorch, dropout behavior is controlled by the model's mode, that means training and evaluation modes
  • Training Mode: Enables dropout
  • Evaluation Mode: Disables dropout, scales activations

Batch Normalization

• Networks train better when inputs are normalized therefore normalizing intermediate layers too
• Problem: forcing every layer to output zero-mean, unit-variance values might be too restrictive (not always optimal)
• Solution: Let the network learn if it wants normalization
• Give the network control using control γ (scale) and β (shift) and can amplify or reduce the normalized values, and move the values away from zero
• These are learned during training like normal weights!
• After normalization outputs are always zero-mean, unit-variance
• Two extremes the network can learn: ""Keep normalization"": γ ≈ 1, β ≈ 0 and ""Undo normalization"": γ and β restore original scale and shift
• μj: the mean across batch dimension
• σ²j: variance across the batch dimension
• Batch Normalization is usually inserted after Fully Connected or Convolutional layers, and before nonlinearity
• During Training: Use statistics from the current batch
• During Inference: Use stored running averages (μrunning, σ²running) and and these values are fixed from the training, so no batch statistics are needed
• Batch normalization behavior is controlled by model's mode in PyTorch, same as Dropout (training and evaluation)
• Advantages includes: Makes deep networks much easier to train, Improves gradient flow, Allows higher learning rates/faster convergence, Networks become more robust to initialization, Acts as regularization during training, Zero overhead at test-time or can be fused with conv!
• Disadvantages: Behaves differently during training and testing which is a very common source of bugs!

Full Training Workflow

• Step 1: Initial Setup
  • Start with a pretrained model and just finetune when possible; it saves time and improves results
  • Define an initial architecture without regularization (e.g., dropout) or augmentations
  • Setup validation strategy and choose an appropriate evaluation loss and metric
  • Train the model to get a baseline score
• Step 2: Improvement Process
  • Overfitting is common at the start; use regularization techniques like Dropout and batch normalization
  • Perform error analysis to identify weaknesses and choose appropriate augmentations or preprocessing
  • Tune hyperparameters: layers, epochs, learning rate, batch size, etc
  • Optionally, use ensembling to boost scores (requires more resources).
  • Key Tip: Track scores and improvements at every step to measure progress effectively.
• Step 3: Finalization
  • Save the optimized model for deployment
  • Use the model for inference in real-world applications