Deep Learning: History and Fundamentals

Study Notes

Deep learning involves training very large neural networks.
The resurgence of neural networks began with the AlexNet breakthrough in 2012.
This resurgence was enabled by:
- Availability of extensive datasets.
- Algorithmic improvements like ReLU and Adam.
- Powerful hardware such as GPUs.

The Perceptron (1958) is an early algorithm designed to learn linear classifiers.
Given input data points ( x_{i} \in \mathbb{R}^{n} ) and corresponding labels ( y_{i} \in {-1,1} ), the aim is to learn a weight vector ( w \in \mathbb{R}^{n} ) such that ( \text{sign}(w^{T} x_{i}) = y_{i} ).
The perceptron update rule is: ( w \leftarrow w + \eta x_{i} y_{i} ), where ( \eta ) is the learning rate.
This algorithm is guaranteed to converge if the dataset is linearly separable.

Many functions are not linearly separable.
An example of a non-linearly separable function is ( XOR(x_{1}, x_{2}) = (x_{1} \land \neg x_{2}) \lor (\neg x_{1} \land x_{2}) ).
Linear classifiers can only learn linear decision boundaries.

MLPs are neural networks with one or more hidden layers.
An MLP with a single hidden layer can approximate any continuous function, according to the Universal Approximation Theorem.
The number of neurons required can increase exponentially with the input dimension.

An algorithm used to train MLPs.
It applies the chain rule to calculate the gradient of the loss function relative to the weights.
Weights are updated using gradient descent: ( w \leftarrow w - \eta \nabla_{w} \mathcal{L} ).
Backpropagation gained popularity in the 1980s but faced limitations due to insufficient data and computational power.

The AI winter refers to a period when there was less funding and interest in AI research.
This downturn resulted from AI systems failing to deliver on overblown promises:
- Expert systems were too inflexible.
- Neural networks were challenging to train and scale.

Key components include:
- Data: Large datasets are crucial.
- Algorithms: Algorithms like ReLU, Adam, and Batch Normalization facilitate the training of deep neural networks.
- Hardware: GPUs greatly reduce training times.
The AlexNet moment:
- AlexNet won the ImageNet competition in 2012 by a significant margin.
- It was a deep CNN with 60 million parameters.
- Training required five to six days on two GPUs and represented a pivotal advancement in deep learning.

Has achieved state-of-the-art results in fields such as:
- Computer vision
- Natural language processing
- Speech recognition
Is utilized in commercial applications, including:
- Image search
- Machine translation
- Voice assistants

Assignments involve implementing deep learning algorithms and applying them to real-world problems.
Grading criteria include:
- Correctness
- Clarity
- Efficiency

The final project requires implementing a novel deep learning algorithm or applying an existing algorithm to a new problem.
The project will be evaluated based on:
- Originality
- Technical depth
- Presentation