Machine Learning for Computer Vision

Deep Learning for Computer Vision

• Artificial Intelligence (AI) and Computer Vision (CV) have various application domains, including:
  • Automotive: self-driving cars, driver assistance
  • Manufacturing: industrial inspection, quality assurance
  • Security and Defense: surveillance, access control, facial recognition
  • Agriculture: crop monitoring, precision agriculture, pest control
  • Retail: customer tracking, theft detection, automatic checkout
  • Healthcare: medical image analysis, computer-aided diagnosis
  • Entertainment: cinema, digital games

Artificial Intelligence

• Artificial Intelligence (AI) consists of:
  • Natural Language Processing (NLP)
  • Machine Learning (ML)
  • Deep Learning (DL)
  • Computer Vision (CV)
  • Expert Systems
  • Fuzzy Logic

Computer Vision

• Image Acquisition:
  • Cameras have a human-eye model
  • Pinhole camera model: f (focal length) and c (center of the camera)
  • Camera sensor: converts light into electrical signals
  • Bayer filter: used in color cameras to capture color images
  • Three-sensor cameras: used for high-quality color images

Computer Vision Tasks

• Image Classification: classifying images into categories
• Object Detection: detecting objects within images
• Semantic Segmentation: segmenting images into semantic regions
• Instance Segmentation: segmenting individual objects within images
• Tracking: tracking objects across frames

Machine Learning

• Machine Learning is a subset of Artificial Intelligence (AI)
• Supervised Learning: training a model on labeled data
• Evaluation Metrics: used to evaluate the performance of a model
  • Confusion Matrix: a table used to evaluate the performance of a model
  • Precision: the ratio of true positives to true positives plus false positives
  • Recall: the ratio of true positives to true positives plus false negatives
  • Accuracy: the ratio of true positives plus true negatives to total instances
  • F1-score: the harmonic mean of precision and recall

Neural Networks

• Neural Networks are used for classification in Computer Vision

• Evaluation and Metrics: used to evaluate the performance of a neural network

• Training a Neural Network: training a model on a dataset

• Implementation Challenges: challenges faced when implementing a neural network

• Neural Networks for other Computer Vision tasks: used for other tasks such as object detection and segmentation### Neural Networks

• Neural Networks are a type of Deep Learning model

• Types of Neural Networks include:

  • Recurrent Neural Networks (RNN)
  • Long Short-Term Memory (LSTM)
  • Gated Recurrent Unit (GRU)
  • Convolutional Neural Networks (CNN)
  • Transformers
  • Generative Adversarial Networks (GAN)
  • Stable Diffusion

Neurons

• A neuron is a linear function with an optional non-linear activation
• The output of a neuron is calculated using the formula: yi = Σ xj*wij + bi

Neural Network

• A neural network is a linear function in the form yi = Σ xj*wij + bi
• Neural networks can be used for classification in Computer Vision

Deep Neural Network

• A deep neural network is a neural network with multiple layers
• The deeper the neural network, the more complex the learning

Activations

• Activations are used for intermediate neurons
• Examples of activations include sigmoid, tanh, and ReLU

Training Neural Networks

• Training involves optimizing the network's parameters to produce outputs close to the ground truth, using examples with corresponding ground truth labels.
• The output neurons are supposed to estimate the ground truth labels.

Class Encoding

• In 2-class problems, the label for each sample is either 0 or 1, and there is typically only 1 output neuron.
• The output neuron provides the probability of class 1 (p) and conversely, the probability of class 0 is 1-p.

Multiclass Problems

• Labels may be specified as integers or as "one-hot" vectors.
• In one-hot encoding, each class is represented by a binary vector with a single 1 and all other elements being 0.
• Neural networks for classification usually generate one-hot vectors on the output.

Image Classification

• Standard networks for image classification include AlexNet, VGG, GoogLeNet, ResNet, SqueezeNet, DenseNet, MobileNet, NASNet, and EfficientNet.
• Standard datasets for image classification include ImageNet, MNIST, Fashion MNIST, Pascal VOC, CIFAR10, CIFAR100, and KITTI.

Convolutions

• Convolutions can be applied to grayscale or RGB images.
• There are different types of convolutions, including normal convolution, normal convolution with no padding and stride of 2, atrous convolution, and transpose convolution.
• A typical CNN structure consists of building blocks, including convolution, and can be used for image classification tasks.

Evaluation Metrics

• Evaluation strategy: dataset split, stratified split, and cross-validation
  • Dataset split: training set (~60%), validation set (~20%), test set (~20%)
  • Stratified split: considering the classes
  • Cross-validation: successively train and evaluate on different sets of data

Classification Metrics

• True Positives (TP): correctly identified positives (class 1) instances
• True Negatives (TN): correctly identified negatives (class 0) instances
• False Positives (FP): incorrectly classified as positives (class 1) that are really negatives (class 0)
• False Negatives (FN): incorrectly classified as negatives (class 0) that are really positives (class 1)
• Confusion Matrix: a table that summarizes the predictions against the actual true labels
• Confusion Matrix - normalized: normalized by the total number of instances
• Precision: TP / (TP + FP)
• Recall: TP / (TP + FN)
• Accuracy: (TP + TN) / (TP + TN + FP + FN)
• F1-score: 2 * (Precision * Recall) / (Precision + Recall)

Regression Metrics

• MSE (Mean Squared Error): 1/n * σ (y - y')^2
• MAE (Mean Absolute Error): 1/n * σ |y - y'|

Object Detection Metrics

• Intersection-over-Union (IoU) - Jaccard index: (A ∩ B) / (A ∪ B)
• Dice index: 2(A ∩ B) / (|A| + |B|)

Object Detection Metrics (1 class)

• Intersection-over-Union
• [email protected] / [email protected]
• [email protected] / [email protected]
• Average Precision (AP) / Average Recall (AR) – IoU=0.50:0.05:0.95
• Keras implementation
• MSCOCO Python Toolbox implementation

Object Detection Metrics (multiclass)

• mean Average Precision (mAP)/mean Average Recall (mAR)
• mean of AP/AR for all classes

Semantic Segmentation Metrics

• Pixel-wise classification metrics:
  • Precision, Recall, F-Score, Accuracy
• Segmentation Area Metrics:
  • Mean Intersection-over-Union
  • IoU for each class
  • Average over classes
• Keras implementation

Tracking Metrics

• MOTP (Multiple object tracker precision): error in estimated position for matches over all frames, averaged by total number of matches
• MOTA (Multiple object tracker accuracy): 1 - (FN + FP + MM) / GT

Training Neural Networks

• Training means optimizing the parameters so that the network's output is equal (or close) to the ground truth
• Steps: initialize weights randomly, define a loss function, apply gradient descent on the weight values to minimize the sum of errors
• Gradient descent: an optimization algorithm used to minimize the loss function by adjusting the model's parameters
• Learning rate: a hyperparameter that controls how quickly the model learns from the training data
• Backpropagation: an algorithm used to compute the gradients of the loss function with respect to the model's parameters

HOTA (Higher Order Tracking Accuracy)

• A metric for evaluating the performance of multi-object tracking algorithms

Batch Normalization

• Normalizes activations of the previous layer across a batch
• Applies a transformation to maintain mean output close to 0 and output standard deviation close to 1

Normalization Norms

• L1
• L2

Dropout

• Main scientific advance of the Deep Learning era
• Introduced in AlexNet, NIPS 2012
• Randomly cancels features during training
• Forces the network to learn in a more generic way when information is incomplete
• A regularization strategy that helps the network avoid overfitting

Types of Dropout

• SpatialDropout1D/2D/3D: drops entire feature maps in 1D, 2D, 3D
• GaussianDropout: multiplies with 1-centered Gaussian noise
• GaussianNoise: adds 0-centered Gaussian noise

Loss Functions

• Probabilistic losses
• Regression losses
• Hinge losses for "maximum-margin" classification

Probabilistic Losses

• Binary Cross-entropy (log-loss, binary problems)
  • Formula: −(1/N) ∑ (ygt.log(ypred)+(1−ygt).log(1−ypred))
• Categorical Cross-entropy (log-loss, multiple classes, one-hot representation)
  • Formula: −(1/N) ∑ ygt.log(ypred)
  • Shape of ypred and ygt is [batch_size, num_classes]
• Sparse Categorical Cross-entropy (log-loss, multiple classes, labels provided as integers)
  • Shape of ygt is [batch_size], shape of ypred is [batch_size, num_classes]

Layer Types

• Core (Input, Dense, Activation…)
• Convolution (Conv1D, Conv2D, Conv3D…)
• Pooling (MaxPooling1D/2D/3D, AveragePooling1D/2D/3D, GlobalMaxPooling1D/2D/3D)
• Reshaping (Reshape, Flatten, Cropping1D/2D/3D, UpSampling1D/2D/3D, ZeroPadding1D/2D/3D…)
• Merging (Concatenate, Average, Maximum, Minimum…)
• Normalization (BatchNormalization, LayerNormalization)
• Regularization (Dropout, SpatialDropout1D/2D/3D, GaussianDropout, GaussianNoise, …)

Data Normalization

• Changes the range of input values
• Stabilizes the model's behavior in training and speeds up training
• Normalization process:
  • Normalize inputs and outputs
  • Train model with normalized inputs and outputs
• Inference process:
  • Normalize inputs
  • Run inputs through the model to get normalized outputs
  • Denormalize outputs

Normalization Layers

• LayerNormalization: normalizes the activations of the previous layer for each given example
• Applies a transformation to maintain the mean activation within each example close to 0 and the activation standard deviation close to 1

Batch Normalization

• Normalizes activations of the previous layer across a batch
• Applies a transformation to maintain mean output close to 0 and output standard deviation close to 1

Normalization Norms

• L1
• L2

Dropout

• Main scientific advance of the Deep Learning era
• Introduced in AlexNet, NIPS 2012
• Randomly cancels features during training
• Forces the network to learn in a more generic way when information is incomplete
• A regularization strategy that helps the network avoid overfitting

Types of Dropout

• SpatialDropout1D/2D/3D: drops entire feature maps in 1D, 2D, 3D
• GaussianDropout: multiplies with 1-centered Gaussian noise
• GaussianNoise: adds 0-centered Gaussian noise

Loss Functions

• Probabilistic losses
• Regression losses
• Hinge losses for "maximum-margin" classification

Probabilistic Losses

• Binary Cross-entropy (log-loss, binary problems)
  • Formula: −(1/N) ∑ (ygt.log(ypred)+(1−ygt).log(1−ypred))
• Categorical Cross-entropy (log-loss, multiple classes, one-hot representation)
  • Formula: −(1/N) ∑ ygt.log(ypred)
  • Shape of ypred and ygt is [batch_size, num_classes]
• Sparse Categorical Cross-entropy (log-loss, multiple classes, labels provided as integers)
  • Shape of ygt is [batch_size], shape of ypred is [batch_size, num_classes]

Layer Types

• Core (Input, Dense, Activation…)
• Convolution (Conv1D, Conv2D, Conv3D…)
• Pooling (MaxPooling1D/2D/3D, AveragePooling1D/2D/3D, GlobalMaxPooling1D/2D/3D)
• Reshaping (Reshape, Flatten, Cropping1D/2D/3D, UpSampling1D/2D/3D, ZeroPadding1D/2D/3D…)
• Merging (Concatenate, Average, Maximum, Minimum…)
• Normalization (BatchNormalization, LayerNormalization)
• Regularization (Dropout, SpatialDropout1D/2D/3D, GaussianDropout, GaussianNoise, …)

Data Normalization

• Changes the range of input values
• Stabilizes the model's behavior in training and speeds up training
• Normalization process:
  • Normalize inputs and outputs
  • Train model with normalized inputs and outputs
• Inference process:
  • Normalize inputs
  • Run inputs through the model to get normalized outputs
  • Denormalize outputs

Normalization Layers

• LayerNormalization: normalizes the activations of the previous layer for each given example
• Applies a transformation to maintain the mean activation within each example close to 0 and the activation standard deviation close to 1

Tensorboard

• Tensorboard can automatically generate a graph for the metrics.
• Tensorboard can be activated as a callback.

Command Line

• The command line to use Tensorboard is tensorboard --logdir logs/fit.

Custom Callback

• A custom callback can be created by defining a class that inherits from tf.keras.callbacks.Callback.
• The class can have methods such as on_train_begin and on_batch_end to track batch losses and accuracies.

Training

• When training with callbacks, the validation loss and accuracy are initially better than the training loss and accuracy.
• This is because the validation metrics are only evaluated at the end of the epoch, after all the updates.
• The reported training loss and accuracy are the average over the whole epoch, and are negatively affected by the initial (untrained) parameters.

Agenda

• Artificial Intelligence and Computer Vision can be achieved with Intel i7/i9 and GeForce RTX.
• Synthetic data can be generated using games.
• For organizations, buying a physical server with multiple GPUs or renting a cloud server (AWS, Azure, etc.) is an option.
• Distributed training can be used.
• Pretrained backbones can be used and fine-tuned on new data.

Data Augmentation

• Data augmentation involves reusing real examples with small random changes/effects.
• This produces realistic additional examples at a very low cost.
• Common augmentation strategies include:
  • Random translation (horizontal/vertical)
  • Random rotation
  • Random flip (horizontal/vertical)
  • Random zoom
  • Random skew/tilt/stretch
  • Random noise addition
  • Random Distortion
• Augmentation is a form of regularization.

Training with Own Data

• When training with own data, it's likely that you will have your own data that you want to feed the network during training.
• You may also want to automatically apply augmentation to your data.

Training Approach

• Deep Learning is unreasonably effective, and throwing good data at a suitable network can make it learn from it.
• To get good data, you need to compromise between quantity and quality.
• Abundant and accessible data is often low-quality, while high-quality data may need to be hand-labeled.
• You can get a pretrained network and retrain it on your data.

Training Challenges

• Dataset building involves large datasets and data quality.
• Training hardware involves compute capability and memory size.
• Dataset building tricks include data harvesting and data augmentation.
• Training tricks include using decent hardware and laptop for mortals.

Mobile/Embedded AI

• Implementing AI in devices with limited resources involves pruning and quantization
• TensorFlow Lite, PyTorch Mobile, and PyTorch Edge are popular frameworks for mobile/embedded AI
• Getting started with AI on Jetson Nano is a course offered by NVIDIA

TinyML

• On-device TinyML applications typically rethink network architecture
• SqueezeNet is an example of a network architecture that achieves AlexNet-level accuracy with 50x fewer parameters

Inference Challenges

• Model size vs memory size is a challenge in inference
• Compute capability vs ops per image is another challenge
• Model simplification and model compression are approaches to address these challenges

Model Simplification/Model Compression

• Pruning involves removing redundant weights or kernels
• Quantizing involves using less bits to store weights and features
• Knowledge Distillation involves training a weaker smaller network to provide outputs similar to a good large network

Model Pruning

• Reduces computation time at the cost of reduced accuracy
• Removing a neuron implies removing its weights, bias, and memory storage
• Removing a kernel implies removing the kernel, resulting feature map, and input channel of all kernels of the following layer
• Several possible strategies for pruning include:
  • Removing kernels with lower values (L1/L2)
  • Structured pruning
  • Smallest effect on activations of next layer
  • Minimize feature map reconstruction error of next layer
• Network pruning as architecture search

Model Pruning Resources

• TensorFlow Model Optimization is a toolkit for model pruning
• Yann LeCun's paper "Optimal Brain Damage" (1989) is a seminal work on model pruning
• Other papers on model pruning include "Rethinking the Value of Network Pruning" (ICLR 2019) and "Permute, Quantize, and Fine-Tune: Efficient Compression of Neural Networks" (CVPR 2021)

Quantization

• Weights are normally stored and used as 32-bit floating point numbers
• Simplifying weights to use integers with less bits (reduced precision) reduces model size and increases operation speed
• Different possibilities for quantization balance include:
  • 8 bits for weights and features
  • 4 bits for weights and features
  • 2 bits for weights, 6 bits for features
  • 1 bit weights, 8 bit features
  • 1 bit weights, 32 bit features

DL4CV Study Notes

Artificial Intelligence and Computer Vision

• Application domains: Automotive, Manufacturing, Security and Defense, Agriculture, Retail, Healthcare, Media
• Tasks: AI, ML, Deep Learning, Computer Vision tasks, Traditional Approach vs Deep Learning Approach

Machine Learning and Deep Learning

• Supervised Learning
• Evaluation and Metrics overview
• Features and Classifiers

Neural Networks

• Neurons and Neural Networks
• Deep Neural Networks
• Activations and Label Encoding
• Convolutional Neural Networks

Neural Networks for Classification in Computer Vision

• LetNet
• AlexNet
• GoogLeNet
• VGG
• ResNet

Evaluation and Metrics

• Classification
• Object detection/Segmentation
• Tracking

Training Neural Networks

• Gradient descent and parameter updates
• Forward pass and backward pass
• Normalization
• Loss functions
• Optimizers
• Learning rate
• Generators
• Callbacks

Implementation Challenges

• Training challenges
• Transfer Learning
• Data Augmentation
• Synthetic Datasets
• Inference challenges
• Model Compression

Neural Networks for other Computer Vision tasks

• Classification
• Object detection
• Semantic segmentation
• Instance segmentation

Demos

• Audio Recognition
• Autoencoder
• Generative Adversarial Network
• Stable Diffusion
• Inference with YOLOv8
• Inference with DeepLabv3+
• Training YOLOv8
• Training Unet on ISBI

Homework

• Train Unet (Tensorflow)
• Data (GTA5 part 1)
• Evaluation (scikit-learn functions)

3D Deep Learning

• PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
• PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
• Neural Radiance Fields (NeRFs)
• Instant-NGP
• Nerfstudio

Audio

• Possible approaches: Take spectrograms of slices of input and treat them as a sequence, Take spectrogram of the input and treat it as an image
• Use a Deep Neural Network to process the input

SmartPhoneHeadScanner

• No additional information provided

Generative Adversarial Networks

• Goodfellow et al., 2014
• StyleGAN: A Style-Based Generator Architecture for Generative Adversarial Networks
• StyleGAN2: TensorFlow 1.14
• Analyzing and Improving the Image Quality of StyleGAN
• Image-to-Image Translation with Conditional Adversarial Networks
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks

DL4NLP

• Probabilistic modeling of word occurrences
• Models are typically trained to output the probability of the next word in the sentence
• Word embeddings – distributed representation
• Transformers: Self-Attention Layer, Multiple heads, Self-attention constructs a tensor

Stable Diffusion

• Denoising approach
• Text-to-image task
• Robin Rombach, et al., “High-Resolution Image Synthesis with Latent Diffusion Models”, CVPR 2022

Visual Content Generation

• DALL-E: text-to-image
• SORA: text-to-video
• Zero123: image-to-3D
• DreamFusion: text-to-3D using 2D Diffusion
• Magic3D: Text-to-3D

Deepfakes

• Morgan Freeman
• Deepfake: Video generated by AI, Voice by human imitator

Sound Generation

• AudioCraft
• MusicGen: text-to-music
• AudioGen: text-to-sound
• EnCodec: neural audio codec
• Multi Band Diffusion: decoder using diffusion
• MAGNeT: text-to-music and text-to-sound

Music Generation

• UDIO.com: Text prompt -> 30 second segments with lyrics
• Suno.com: Text prompt -> ~2 minute songs with lyrics

This quiz covers the basics of deep learning and its applications in computer vision. It includes topics such as artificial intelligence, machine learning, and computer vision tasks.