Deep Learning Architecture and Concepts Quiz

Lecture 2: Deep Feedforward Networks

• Deep learning uses modular components
• Deep learning uses nonlinearities to generate complex mappings
• Deep learning utilizes gradient-based learning methods
• Backpropagation utilizes the chain rule for efficient computation

Lecture Overview

• Deep learning involves modularity
• Deep learning uses nonlinearities
• Gradient-based learning is a core technique
• The chain rule is fundamental
• Backpropagation is a key technique

Last Time

• Neural networks transform inputs to outputs
• The input is weighted and summed
• This sum triggers an activation function
• The output is a result of an activation function

From Linear Functions to Nonlinear Functions

• Linear functions (f=ReLU(Ax)) are based on matrix multiplication and the ReLU function
• Non-linear architectures are required for complex mappings

How Deep Neural Networks Do It

• Deep networks utilize multiple layers
• ReLU (Rectified Linear Unit): a non-linear activation function
• ReLU(x) = max(0, x)
• For example ReLU(3) = 3
• For example ReLU(-3) = 0

Deep Feedforward Networks

• Feedforward neural networks are a type of neural network architecture
• They are also called multi-layer perceptrons (MLPs)
• The goal is to approximate a function, f
• The network defines a mapping y=f(x; θ)
• The network learns parameters to best approximate a function
• No feedback connections

Deep Feedforward Networks as a Composite Function

• A deep network is a series of composable functions
• y=f(x; θ) = h₁∘h₂∘…∘h_(l)∘x (where θₗ is the parameter for the l-th layer)

Neural Networks in Blocks

• Feedforward networks can be visualized as blocks
• Inputs flow through hidden layers to produce an output
• Each hidden layer transforms the input from the previous layer

What is a Module?

• A module is a building block for function transformation
• It takes input data or the output of other modules
• Modules use an activation function with or without trainable parameters (w).
• Examples: f = Ax, f = exp(x)

Requirements

• Activation functions need to be differentiable in most contexts
• Be careful with cycles

Feedforward Model

• The majority of models are feedforward
• Almost all CNNs/Transformers
• Very simple architecture

Non-Linear Feature Learning Perspective

• Linear models include logistic regression and linear regression
• They are convex with a closed-form solution
• They can be fitted reliably and efficiently
• They have limited capacity

Non-linear Feature Learning Perspective

• Deep learning aims to find good feature representations
• Deep learning involves finding the best parameters to achieve accurate feature representation from x
• No longer a convex training problem
• Design families for q(x; θ)
• Utilize human knowledge about the problem domain for better generalization

Directed Acyclic Graph Models

• Mix the network architectures to match the task domain
• This method makes sense for problems with multiple inputs or modalities (e.g., RGB and LIDAR)
• Interweaved & skip connections

Hierarchies of Modules

• Data-efficient modules and hierarchies are used for better model efficiency
• Trade-off between model complexity and efficiency, more training iterations with a 'weaker' model often better
• ReLUs are often the activation function of choice as they help train faster
• GPU memory is a practical constraint
• Modules need to be computed in the correct order

Loopy Connections

• Past outputs can affect future inputs
• Such cycles are common in recurrent networks
• Loops must be unfolded to train the model properly

How to get w? Gradient-Based Learning

• The non-linearity results in a non-convex loss function
• Optimizers are used with iterative gradient calculations for complex function mappings

Cost Function

• Maximum likelihood is common in training neural functions
• Taking the logarithm leads to minimizing negative log-likelihood, equivalent to cross-entropy

Cost Functions

• Euclidean loss suitable for regression problems
• Sensitive to outliers, magnifying errors quadratically
• Other functions cross-entropy, KL-divergence

Cost Functions

• Cost functions define what the model should learn
• The gradient must be sufficiently large and predictable
• Functions that saturate may become predictable with poor results
• The negative log-likelihood helps to avoid the problem of saturating functions due to output exponentiation (eg. softmax)

Deep Learning Modules

• Study of deep learning modules

Activation Functions

• Activation functions transform weighted input sums to outputs
• Outputs with a limited range are called "squashing" functions
• The activation function choice significantly impacts a network's performance and capabilities
• Often a single function is utilized for all layers in a given network
• Functions need to be differentiable at most points
• Linear and ReLU are commonly used

Linear Units

• Identity function with no activation function saturation
• Strong and stable gradients
• Reliable learning with modules

Rectified Linear Unit (ReLU)

• ReLU = max(0,x)
• Sparse activation
• Better gradient propagation
• Efficient computation (addition and multiplication)
• Scale invariant

Rectified Linear Unit (ReLU) Potential Problems

• Non-differentiable at zero,
• Not zero-centered
• Unbounded

Leaky ReLU

• Leaky ReLU is differentiable at zero.
• Allows a small, positive gradient when the unit isn't active
• Parametric ReLU includes a learnable parameter (a)

Exponential Linear Unit (ELU)

• A smooth approximation to the rectifier
• Non-monotonic (gradient change)
• Default activation function for many models such as BERT

Gaussian Error Linear Unit (GELU)

• Similar to ELU but not monotonic with a change in the gradient
• A default function for Vision Transformers and state of the art models

Sigmoid and Tanh

• Tanh (x) has output range [-1, +1]
• Data centered around zero
• Less positive bias
• Saturates at the extremes (resulting in zero gradients)
• Easy to become overconfident at the extreme values
• Gradients are less than optimal leading to problems with middle layers

Softmax

• Outputs probability distributions
• Normalizes output, avoids large or small values for better stability
• Useful at the output layer

How to Choose an Activation Function

• ReLU or GELU are typical choices for hidden layers
• Linear, sigmoid, and tanh are common output layers (regression, binary classification, multiclass classification)
• Choose appropriate functionality based on the type of classification task

New Modules

• Any function that is differentiable is a valid module
• Modules of modules can be effectively implemented
• They should be implemented as cascades of simple modules

Architecture Design

• The structure of a neural network and how units connect to each other
• Networks are composed of organized units, called layers
• The chain layer is how successive layers are utilized to map input into output layers
• Each layer's calculation utilizes a given function with specific network parameters

Width and Depth

• Universal approximation theorem
• Large MLPs can represent any function, providing adequate hidden units
• Deeper networks often generalize better and reduce the number of units required to accurately model a given function

Width and Depth

• The number of parameters in layers of convolutions without an increase of depth isn't the most effective strategy to model better results
• Deeper networks often generalize better and reduce the amount of generalization errors in learning

Deeper Networks: Hierarchical Pattern Recognition

• Deeper networks show a division of labor between layers
• Layers learn different features, resulting in a hierarchical pattern understanding of input data

A Neural Network Jungle

• Detailed list of neural network architectures (e.g., Perceptrons, MLPs, RNNs, LSTMs, GRUs, Autoencoders, Convolutional Nets, Transformers, Generative Adversarial Nets, Deep Residual Nets, Neural Turing Machines)

Intermezzo: Chain Rule

• The chain rule is a fundamental concept in calculus used to calculate derivatives of functions formed by composing other functions
• It's used in deep learning for finding gradients of the cost function by utilizing calculated derivative values
• Useful technique for complex non-linear functions and calculations

Computational Graph

• Shows operations to calculate outputs with nodes representing variables
• A simple calculation function or variable represents an operation or a node within the graph

Example

• Show example calculations and graphs utilizing the chain rule in derivative calculations

Example

• Shows example calculations with diagrams to highlight the importance of differentiability in neural networks