Deep Learning and Neural Networks Quiz

Lecture 2: Deep Feedforward Networks

• A robot wrote this entire article. Are you scared yet, human?
• A powerful language generator (GPT-3) was asked to write an essay from scratch.
• The assignment was to write an essay.

Lecture Overview

• Modularity in deep learning
• Deep learning nonlinearities
• Gradient-based learning
• Chain rule
• Backpropagation

Last Time

• Neural networks transform input to output values.

From Linear Functions to Non-linear

• Linear function f = ReLU(Ax), with A in R^nxm, x in R^m×1, and ReLU(x) = max(0, x).
• ReLU(3) = 3 and ReLU(-3) = 0
• Non-linearity is essential for shallow to deep networks.

From Linear Functions to Non-linear: What About y = ReLU(B f(Ax))?

• Transformations (non-linear) using ReLU functions are applied to input variables.

We've Learned XOR.

• Neural networks have been trained to learn XOR logic operations.

Deep Feedforward Networks

• Feedforward neural networks (MLPs) aim to approximate functions, defining a mapping y = f(x; θ).
• They learn parameters θ for best function approximation.
• No feedback connections. Including feedback creates recurrent networks (not common in contemporary NN design).
• Brains have many feedback connections.

Deep Feedforward Networks in a Formula

• y=f(x; θ) = a_l (x; θ_1,…,l) = h₁(h_l−1(...(h₁(x, θ₁),...). , θ_l−1), θ_l) where θ_l is the parameter in the l^th layer.
• Simplification: a_l = f(x; θ) = h₁°h_l−1°...°h₁°x, where each function hᵢ is parameterized by θᵢ

Neural Networks in Blocks

• Neural networks are visualized as a cascade of blocks, with input, multiple hidden layers and output modules/layers.

What is a Module?

• A module is a building block/transformation/function that receives either data x or another module's output, which produces an output based on an activation function, and may or may not have trainable parameters.
• Examples: f = Ax, and f = exp(x)

Requirements

• Activation functions must be 1st-order differentiable (almost) everywhere.
• Take special care when there are cycles in the architecture of blocks.
• No other requirements for use.

Feedforward Model

• The vast majority of models are feed forward architectures.
• Almost all CNNs/Transformers have feedforward architecture.

Non-linear Feature Learning Perspective

• Linear models (logistic regression and linear regression) are convex with closed-form solutions, fitting efficiently and reliably, but have limited capacity.
• To extend to non-linear models, apply a linear model to a transformed input q(x), using kernel tricks (e.g., RBF kernel).
• Use non-linear dimension reduction.

Non-linear Feature Learning Perspective: Strategy

• Strategy: learn a function q: y=f(x;θ) = φ(x;θ)τω, finding the θ that results in a good representation (linearly separable).
• Design different families q(x;θ) instead of a single function.
• Encode human knowledge to help generalization.

Non-linear Feature Learning Perspective: Learning XOR

• In the transformed space, a linear model can learn XOR operations.

Directed Acyclic Graph Models

• Hierarchies can be mixed.
• Combining different inputs and modalities is relevant for combining information from multiple sources, e.g., RGB and LIDAR.

Hierarchies of Modules

• Efficient modules and hierarchies trade off model complexity with efficiency.
• More training steps with a weaker model could perform better than less steps with more complex models.
• ReLUs are half linear functions that create state of the art results by training faster.
• Often, GPU memory limits the complexity of modules.
• Modules should be computed in the correct order.

Loopy Connections

• Modules' past outputs serve as future input.
• Cycles require unfolding the graph into recurrent neural networks, but is uncommon.

How to get w? Gradient Based Learning

• Non-linearity produces a non-convex loss function, therefore iterative gradient-based optimizers are needed, e.g., stochastic gradient descent.
• No guarantee of convergence, and initialization of parameters is important.

Cost Function

• Typically, the cost function is maximum likelihood on the training set.
• The parameters are found that maximize the likelihood of the model explaining the data.
• This corresponds to minimizing the negative log-likelihood or cross-entropy loss.
• This is equivalent to the mean squared error cost.

Cost Functions: Choices

• Euclidean loss. Suitable for regression but sensitive to outliers and magnifies errors quadratically.
• Other cost functions include cross-entropy and KL-divergence

Cost Functions: Considerations

• Cost function gradients should be large and predictable.
• Saturated functions (e.g., sigmoid activation) produce poor gradients and reduce learning efficiency.
• Negative log-likelihood helps by reversing output exponentiation (e.g., Softmax).

Activation Functions

• The transformation of weighted input into network output.
• Squashing functions have limited output ranges.
• Activation function choice significantly affects neural network capability and performance.

Linear Units

• Identity activation function.
• No activation saturation.
• Strong and stable gradients.
• Good for reliable learning in linear modules.

Rectified Linear Unit (ReLU)

• Activation function, h(x)= max(0,x).
• Advantages: sparse activation in randomly initialized networks, efficient computation, scale invariant.
• Problems: non differentiable at zero; unbounded.
• The derivative is 1 for x>0, and 0 if x≤0

Leaky ReLU

• Leaky ReLU allows positive gradients for non-active units.
• Parametric, with a learnable parameter for gradient when the unit is not active.

Exponential Linear Unit (ELU)

• Approximates rectifier in a smooth manner.
• Non-monotonic for x<0.
• Default activation for Models, such as BERT.

Gaussian Error Linear Unit (GELU)

• Similar to ELU but non-monotonic
• Often default activation for vision transformers.

Sigmoid and Tanh

• Sigmoid and Tanh saturate at extremes (0 or 1 decisions), leading to little gradient changes.
• Sigmoids can be good for output to model probabilities, but can have overconfidence problems.
• Tanh is better for middle layers because of its centered outputs.

Softmax

• Probability distribution output.
• Normalizing outputs to a probability distribution.
• Helps with stability by avoiding exploding exponentiations when dealing with extremely big or small numbers.

How to Choose Activation Function

• Hidden layers use ReLU or GELU. Recurrent networks often use Tanh or Sigmoid.
• Regression: linear.
• Binary classification: sigmoid.
• Multiclass classification: softmax.

New Modules

• Any differentiable function is a potential module.
• Modules of modules are easy to combine.

Architecture Design

• Neural networks organize into groups called layers.
• Layers form a chain with linear and non-linear operations.

Quiz

• Deep networks without non-linearity are essentially just a single layer.

Width and Depth

• Width (number of units in hidden layers): broader is not always better, because it could lead to increased memory use.
• Depth: deeper nets may require fewer units to generate accurate output representations, improving generalization.
• Could lead to overfitting with an exponentially increasing increase in training time.

Deeper Networks: Hierarchical Pattern Recognition

• Deep networks have a division of labor between layers.
• Early layers extract low-level input information.
• Deeper layers extract higher level information.

Width and Depth: Convolutions

• Increasing number of parameters in layers without increasing depth in convolutional networks is not nearly as effective for increasing test set performance.

A Neural Network Jungle

• Neural network models list included MLPs, RNNs, LSTMs, GRUs, autoencoders, convolutional nets, transformers.

Intermezzo: Chain Rule (Math Review)

• Mathematical chain rule review for computing derivatives of composed functions
• Chain rule review examples for multi variable functions.

Computational Graph

• Each node represents a variable.
• Operations are simple functions of one or more variables

Example

• Illustrate the concept of computing partial derivatives (dL/dx).

Example (à la Rosenblatt)

• Illustrate that if the neural network does not have differentiability, there is no gradient and learning cannot occur

Chain Rule of Calculus

• Review of the chain rule for calculus; a fundamental concept in the use of gradient propagation.

Jacobian

• Generalization of gradient for vector-valued functions h(x)
• Input and output dimensions contribute to the Jacobian.
• A Jacobian is the matrix of all the partial derivatives.

Taking Gradients

• "Vectorizing" matrix/tensors to find the effect of output w.r.t. the input
• Matrix and vector order (row-wise) are important for computations.

Jacobians Intuitively

• The Jacobian of h is its gradient.
• The Jacobian captures how the output changes with respect to changes in the input.
• Einstein notation can be used for complicated computations to prevent memory issues (np.einsum or torch.einsum)

Jacobian Geometrically

• The Jacobian represents the best local approximation of how space changes under a transformation.
• The Jacobian determinant measures the ratio of areas, similar to how the slope measures the ratio of change in 1d functions.

Basic Rules of Partial Differentiation

• Review of product and sum rules for partial differentiation.

Computing Gradients

• The chain rule is used to compute derivatives (gradients) of composed functions, making the computation highly efficient.

Chain Rule and Tensors Intuitively

• Chain rule's concept generalizes for high-dimensional tensors to efficiently compute and evaluate derivatives.
• Computation is often done over all possible inputs (sums), keeping tensor shapes in mind.

Example (Computation of Partial Derivatives)

• Illustrative example of how to calculate partial derivatives efficiently when combining functions of other variables using chain rule methods.

How Research Gets Done: Part II

• Step 2 of deep learning research: understand fundamentals and read research papers effectively.
• Stages like title, abstract, figures/tables, conclusion and introduction are key elements to effective comprehension.

Backpropagation

• Process of recursively computing the gradient of the loss function with respect to model parameters.

Backprop: Former Tesla AI Head Thoughts

• Former Tesla AI Head, Andrej Karpathy, highlights the importance of backpropagation in neural network training.

Backpropagation = Chain Rule

• Neural network loss is a composite function of modules (mathematical composition of functions).
• The goal during backprop is to calculate gradients w.r.t. the parameters of a specific model layer.
• Backpropagation is an algorithm for calculating the chain rule efficiency.

Backpropagation = Chain Rule!!

• Backprop calculates (repeats) gradient and Jacobian computations.

Backpropagation = Chain Rule, but as an algorithm

• Recursive computation that reuses computation results more efficiently during the training process.

But Why Do We Actually Use Backprop?

• Backprop is a sophisticated, highly optimized mathematical computation that allows machine learning model training. The brain does not implement this computation exactly like the machine algorithm.

Regarding Point 4.

• Illustrative demonstration that even for a small MLP, calculating gradients directly is simple.

Computational Feasibility

• Illustration that with large data sets, the Jacobian becomes huge and less computationally feasible.

What if the output is a scalar?

• The sizes of some derivatives (e.g., for Jacobian) decrease if the output is a scalar.

Chain Rule Visualized

• Visualization of the chain rule computations in a neural network, which can be extremely large for multiple nested variables.

But We Still Need the Jacobian?

• ReLU/sigmoid activations have sparse Jacobians, making efficient computation of gradients feasible.

Computational Graphs: Forward Graph

• Illustrates the steps for computing activations for all modules in a network recursively, preserving intermediate module variables for computations in the back propagation step.

Computational Graphs: Reverse Graph

• Algorithm computing activations backwards to compute gradients, from the output towards the input of the network.
• This is also known as reverse mode automatic differentiation.

Test your knowledge on deep learning concepts including backpropagation, ReLU activation functions, and the structure of neural networks. This quiz will cover essential topics necessary for understanding how neural networks are trained and function. Perfect for those studying machine learning or artificial intelligence.