Neural Networks Overview

Neural Networks (NN)

• A neural network (NN) is a network of interconnected neurons.
• Neurons are simple processing units.
• The network structure is inspired by biological systems but is highly abstracted.
• The expressive power of the network comes from its architecture, not individual neurons.
• Networks are trained with data.

Units and Layers

• Artificial neurons (units) aggregate weighted inputs plus a bias.
• The equation for a unit is: y = σ(∑Wi xi + b).
• A linear unit is a simple weighted sum.
• A non-linear unit applies an activation function (ReLU, Sigmoid, or Tanh).

Activation Functions

• Sigmoid: σ(x) = 1 / (1 + e^-x)
• Tanh: tanh(x)
• ReLU: max(0, x)
• Leaky ReLU: max(0.1x, x)
• Maxout: max(w1x + b1, w2x + b2)
• ELU: x if x ≥ 0, a(e^x - 1 )otherwise

Layers

• Dense (fully connected): Each unit is connected to all units in the previous layer. Units have separate parameters.
• Convolutional: Uses shared weights. Important for images. Their units have the same arity and share parameters.
• Recurrent (RNN, LSTM): Maintains a memory/state across time steps.

Computational Capabilities

• Continuous and parallel processing.
• Universal Approximation Theorem: NNs with a single hidden layer can approximate any function.
• Deep networks improve expressiveness (feature hierarchy).

Training Neural Networks

• Supervised Learning: Learning from labeled data.
• Unsupervised Learning: Learning patterns without labels.
• Training: Optimizing model weights to minimize error.
• Loss Functions: Define the quantity to be minimized. Quantifies the difference between the predicted output and the ground truth.

Gradient Descent

• Gradient Descent (GD): Optimizes weights by computing gradients and iteratively updating them.
• Stochastic GD (SGD): Uses one sample at a time.
• Mini-batch GD: Uses small batches (common in DL).
• Momentum, Adam, RMSprop: Advanced optimizers for stability.
• Backpropagation: Uses the chain rule to propagate errors backward and adjust weights.

Models as Computation Graphs

• Forward Pass: Data flows from input to output.
• Backward Pass: Gradients propagate backward based on the chain rule.
• Implemented in DL frameworks (TensorFlow, PyTorch).

Other Relevant Concepts

• Hyperparameters vs. Parameters:
  • Parameters: Learned during training (e.g., weights, biases).
  • Hyperparameters: Set before training (e.g., learning rate, number of layers).
• Indeterminism: Random Initialization affects training & batch order shuffling changes optimization path.

Convolutional Neural Networks (CNNs)

• Specialized for image processing.
• Preserve spatial structure.
• Local receptive fields used for feature detection.

CNN Architecture - Key Concepts

• Convolutional Layer: The core layer for feature extraction.
• Applied to a kernel or filter over an image to detect edges or textures.
• Key Hyperparameters:
  • Kernel size (e.g. 3x3, 5x5).
  • Stride (step size of the filter).
  • Padding (handling image edges).

Activation Functions (CNNs)

• ReLU (Rectified Linear Unit).
• Commonly used to prevent the vanishing gradient problem.
• Better than sigmoid and Tanh for preventing saturation/slow learning.

Pooling Layer (CNNs)

• Max-pooling. Select the maximum value from a small region preserving dominant features.
• Average-pooling –reduces computation, helps prevent overfitting, model is translation invariant.

Fully Connected Layer (CNNs)

• Flattens feature maps.
• Connected to dense layers for classification.
• Traditional CNNs end with a softmax layer for multi-class classification.

CNN Properties

• Weight sharing: Filters shared across spatial positions (parameter optimization & global feature learning).
• Translation invariance: CNNs can recognise objects irrespective of their position in the image.
• Feature hierarchy: Earlier layers detect simple features (e.g. edges), deeper layers detect more complex structures (objects).

Training CNNs

• CNNs use backpropagation and gradient descent.
• Steps involve forward pass, calculating loss, and backpropagation.

Famous CNN Architectures

• LeNet-5 (1998): First successful CNN for digit recognition.
• AlexNet (2012): Deep network trained on ImageNet (1.2M images, 1000 classes).
• VGG-16/19 (2014): Used small 3x3 convolutions for efficiency.
• GoogLeNet/Inception (2014): Uses multiple filters (1x1, 3x3, 5x5) in parallel.
• ResNet (2015): Uses residual connections to solve vanishing gradient problems.
• DenseNet (2017): Connects each layer to every other layer to encourage feature reuse.

Fully Convolutional Networks (FCNs)

• Do not use dense layers.
• Useful for image segmentation & denoising

Image Segmentation

• Different types of segmentation using CNNs:
  • Semantic segmentation: Labels each pixel with a class.
  • Instance segmentation: Differentiates multiple objects of the same class.
  • Panoptic segmentation: Combines semantic and instance segmentation.

CNNs for Image Denoising

• CNNs can learn mappings from noisy images to clean images.
• Denoising Autoencoders: CNN-based models to remove noise while preserving image detail.

Generative Adversarial Networks (GANs)

• GANs generate new data samples that resemble real data.
• Two networks compete: Generator (creates fake data) and Discriminator (distinguishes).
• Goal: Train Generator to generate data indistinguishable from real data.
• Training involves an iterative adversarial process.

GAN Training Algorithm

• Iterate over the following steps:
  • Sample from real data or the prior distribution.
  • Generate fake samples.
  • Train D (discriminator) to maximize the probability of classifying real vs. fake correctly.
  • Train G (generator) to minimize the probability of the discriminator correctly classifying fake as real.

Common Problems in GAN Training

• Mode Collapse: Generator produces a limited number of variations.
• Vanishing/Exploding Gradients: Instability in learning.
• Divergence and Instability: Unstable adversarial process makes training difficult.

Variants of GANs

• Conditional GANs (cGANs): Condition generation on input data.
• CycleGANs: Learn mappings between domains without paired examples.
• Deep Convolutional GANs (DCGAN): Use CNNs instead of fully connected networks.
• Wasserstein GANs (WGANs): Fixes mode collapse and instability issues.

Applications of GANs

• Image generation
• Image-to-image translation
• Super-resolution
• Denoising
• Video & Music generation
• Text-to-image generation

Evaluating GANs

• Common Metrics:
  • Inception Score (IS)
  • Fréchet Inception Distance (FID)

Autoencoders (AEs)

• Used in unsupervised learning.
• Aim to learn a compressed (latent) representation of the input data.
• Two main components: Encoder and Decoder.

Why Use Autoencoders

• No labeled data required.
• Feature extraction, dimension reduction.
• Anomaly detection.
• Denoising.
• Data compression.

Autoencoder Architecture

• Input layer: Original data.
• Encoder: Reduces input dimension and extracts features.
• Latent space (Z): Compressed representation.
• Decoder: Reconstructs data from latent representation.
• Output layer: Reconstructed input.

Loss Function for Autoencoders

• Mean Squared Error (MSE): Penalizes large differences between input and reconstructed output.
• Cross-Entropy Loss: Used for binary or normalized data.

Types of Autoencoders

• Denoising AE: Removes noise from data.
• Sparse AE: Enforces sparsity in activations.
• Variational AE: A probabilistic extension.
• Contractive AE: Enforces robustness.
• Wasserstein AE: Different loss function.

Training Challenges in Autoencoders

• Overfitting: Memorizes input data.
• Poor Generalization: Bad performance on unseen data.
• Mode Collapse: All samples map to the same latent vector.

Applications of Autoencoders

• Image denoising.
• Anomaly detection.

Recurrent Neural Networks (RNNs)

• Designed for sequential data.
• Maintain a hidden state to carry information across time steps.
• Process sequences one step at a time.

Challenges in Training RNNs

• Vanishing gradients.
• Exploding gradients.
• Short-term memory.

Long Short-Term Memory (LSTM) Networks

• Uses gates to control information flow.
• Combats vanishing gradient problems.
• Has cell state and hidden state.

Gated Recurrent Units (GRUs)

• Simpler alternative to LSTMs.
• Merges forget and input gates into a single update gate.
• Fewer parameters than LSTMs.
• Well-suited for small datasets.

RNN Applications

• Sequence-to-sequence models (e.g., machine translation, text summarization).
• Speech recognition.
• Handwriting recognition.
• Music generation.

RNN Training Techniques

• Backpropagation Through Time (BPTT): Computes gradients across the entire sequence length.
• Gradient clipping: Limits the maximum values of gradients.
• Signal Regularization: Enforces sparsity in activations.