CNN Architectures: Datasets, LeNet, AlexNet, VGG

• CNN Architectures topic includes Datasets, LeNet-5, AlexNet, VGG, Residual Networks, Autoencoders / U-Net, Detectors, and Information Theory.

MNIST Dataset

• Modified National Institute of Standards and Technology database is used.
• It contains 60,000 training images and 10,000 testing images.
• Images are black and white, represented as 0 or 1.

Fashion MNIST

• Similar to MNIST but contains images of clothing items.
• It includes ten classes such as dress, coat, and shirt.
• MNIST is considered overused and "too easy".

CIFAR Dataset

• Maintained by the Canadian Institute for Advanced Research.
• It consists of overall 60,000 images like airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck.
• The 60,000 images are divided into 50,000 training and 10,000 testing images.
• Image size is 32 x 32 x 3.
• CIFAR-10 is split into total 10 classes.
• CIFAR-100 is split into total 100 classes.

ImageNet Dataset

• Contains 15 million labelled high-resolution images.
• Includes localization, detection data, and video.
• The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) is an yearly contest.
• CNNs have dominated the top positions since 2012.

COCO Dataset

• Tasks include Detection, Captions and Keypoints.
• It consists of 330,000 images with 80 object categories
• Has 1.5 million object instances.

LeNet-5

• Used for handwritten digit (0-9) classification (MNIST).
• It could be extended to classify handwritten characters (a-z).
• Can automatically digitize documents, letters, and books
• Developed in 1998.
• While not the first solution, with alternatives like K-NN, PCA + quadratic, and SVM, it was the first to reach a 0.7% error rate.
• The LeNet-5 architecture includes an input layer of size 32x32, followed by convolutional layers, subsampling layers, and fully connected layers.
• It comprised two convolutional layers, 2 fully connected layers and 1 fully connected layer, further MSE loss.
• Max pooling is used to downsample twice.
• Output has 10 dimensions, one for each digit.
• It has approximately is ~60,000 parameters.
• Uses scaled hyperbolic tangent as the active function: g(x) = Atanh(Sx).
• Not all feature maps are fully connected between layers because of lower computational cost and break of symmetry.
• Ignoring this is now more for ease of implementation.

Max Pooling

• Max Pooling is used to reduce resolution, leading to less memory usage and faster processing.
• It propagates the most "activated" features by removing noisy features.

Tanh Nonlinearity

• Sigmoid function maps inputs to a range of [0, 1]: s(x) = 1 / (1 + e^-x).
• The hyperbolic tangent function maps inputs to a range of [-1, 1]: tanh(x) = (e^x - e^-x) / (e^x + e^-x)
• Scaled and stretched sigmoid: tanh(x) = 2s(2x) - 1.
• Hyperbolic Tangent is slightly preferred over sigmoid due to symmetry around 0.

ReLU

• ReLU (Rectified Linear Unit) is defined as ReLU(x) = max(0, x).
• It mitigates vanishing gradients
• Leaky ReLU: defined as LReLU(x) = x if x ≥ 0, and ax if x < 0.
• ELU ( Exponential Linear Unit) function: ELU(x) = x if x ≥ 0 and a(e^x – 1), if x < 0

AlexNet

• It is designed as Much larger network than LeNet5.
• Has 60 million parameters.
• Uses two "parallel” networks for more efficient GPU parallelization.
• It contains of 5 convolutional layers, and 3 fully connected layers.
• It has the training time for 5 - 6 days with two GTX 580 GPUs.
• Used the ImageNet training set, comprising 1.2 million color images across 1000 classes.
• Inputs are sized at 224 x 224 x 3.
• Top-1 error rate stood at 37.5%, while the top-5 error rate was 17.0%
• AlexNet uses ReLU nonlinearity (activation function).
• The visualizations of the first layer filters show smooth, converged traditional filters with gradients, checkerboards, and blurs

Dropout

• Used in AlexNet to prevent overfitting and greatly improve test set performance.
• Randomly "drops" connections during training
• Uses all connections at inference time (val. and test).

ILSVRC Performance

• Describes the classification errors and average precision in object detection over the years on the ILSVRC competition.

VGG

• Developed by Visual Geometry Group, University of Oxford.
• VGG is a Deeper architecture than AlexNet.
• Has 16 - 19 layers.
• Was the Winner of 2014 ImageNet Challenge.
• It uses smaller convolutional filters (3x3).
• The learned representations generalize well.
• Has multiple configurations which range from 133 - 144 million parameters.
• The architecture incorporates ReLU activation and uses SGD with momentum.
• Dropout is used to avoid overfitting and glorot Initialization is used for weight initialization.
• Also have the weight Decay (L2 regularization).

Deep Neural Networks

• Are challenging to train because the input from prior layers change after weight updates.
• Techniques that help include Residual Networks and Batch normalisation.

Batch Normalisation

• Driven by significance of network depth.
• An obstacle to answering this question was the notorious problem of vanishing/exploding gradients, which hamper convergence.
• It allows SGD to converge and works fine for tens of layers.

Residual Networks

• Introduces "residual” connections lres = x + F(x).
• It was Winner of 2015 ILSVRC (ensemble).
• Easier to train.
• Achieves better propagation of gradients.
• Has deeper networks, with configurations like 152 layers for ImageNet or Up to 1000 layers for CIFAR 10.

Autoencoders

• Map input to a different space and back.
• Undercomplete autoencoders have fewer dimensions by dimensionality reduction, compression.
• Overcomplete autoencoders: have same size (or larger) dimensions by feature learning, denoising images / signals.
• The equation is represented as: f(x) and reconstruction as x = g(y) = g(f(x)).
• The ideal autoencoder model balances sensitivity to inputs for accurate reconstruction with insensitivity to avoid memorization/overfitting of the training data.
• Optimise with a reconstruction loss: L(x, x) with a regulariser to discourage memorisation.
• Models can learn key data attributes like the most important features.

U-Nets

• Autoencoders with skip connections
• Feature maps are concatenated.
• It leads Local detail is better preserved
• Initially employed for biomedical image segmentation.
• Extensively adaptable to various imaging problems.

R-CNN Family

• Tasks includes Object Detection which localizing items and localizing box segment and classifying.
• Used on Datasets such as PASCAL VOC, MS COCO, Cityscapes and KITTI.

R-CNN

• Finds Regions of Interest, an image is “scan” with potentially 2,073,600 evaluations using the Full HD image that could be done by CNN
• Then these areas are classified using SVM and CNN with the method R-CNN
• It is not based on algorithms. It uses objectness, selective search, and Category-independent object proposals
• It uses CNN to extract image feature.
• It finds possible objects, and then scales them to a consistent aspect ratio.
• It uses a CNN is used to classify objects.

IoU: Intersection over Union

• This helps to compare bounding boxes.
• IoU indicates how much overlap the ground truth bounding box has.
• A result of 1 means a perfect prediction, 0 vice versa.

NMS: Non-Maximum Suppression

• Used when the predicted regions often overlap.
• The R-CNN predicts classifications for 2000 Rols.
• Discrads the boxes with low classification confidence
• Greedily selects boxes depending on the score assigned.

Problems with R-CNN

• R-CNN needs lots of time to train due to thousands of iterations.
• The selective search algorithm is fixed, requiring excessive amount of time to train.
• This means it cannot be implemented real time
• Requires around 47 seconds for testing for each image.

Fast R-CNN

• It intakes an entire image with a lot of object proposals
• During the object proposal step to detect an area a region of inter, that is known as the of (RoI) pooling layer extracts a fixed-length feature vector from the feature map
• Training is end to end as the whole model is differentiable
• The training becomes faster 8.75hr vs 84 hr as well as the inference rate 0.32s vs 47s than the previous iterations that R-CNN used to complete.
• Improved precision.

Faster R-CNN

• Introduces Region Proposal Network
• Replaces Region of Interest (Rol) proposal algorithms with CNN
• It is a trainable proposal method and that uses a binary classifier to predict whether an object is present or not.

Evaluation

• Set two thresholds, Confidence which the model has correctly classified objects or sections and IoU intersection of unit whether this can be localized well.
• To compare two different types of recall in terms of precision and *IOU. * Precision = Good Predictions / All Predictions
• *Recall = Good Predictions / All Targets_

YOLO

• Stands for You Only Look Once
• Is lightweight and very Fast.
• has Lower accuracy.

Other CNN Detectors

• Includes Single Shot Detector (SSD), similar to YOLO

Explore CNN architectures, datasets like MNIST, Fashion MNIST, CIFAR & ImageNet. Discusses LeNet-5, AlexNet, VGG, Residual Networks, Autoencoders/U-Net, Detectors, and Information Theory. Understand the structure and applications of each dataset.