Deep Learning History and Perceptrons

Deep Learning History

• Deep learning started with perceptrons in 1958, from Rosenblatt
• Adaline and perceptron models followed around the same time, developed by Widrow and Hoff and Minsky and Papert respectively
• Backpropagation, a key learning algorithm, was introduced in various iterations throughout the 1970s and 80's
• LSTM networks were introduced in 1997
• The term "deep learning" appeared in a publication in 2006
• Image Recognition models, such as Alexnet, emerged in 2012
• Resnet, with 154 layers, emerged in 2015
• DeepMind's Go-playing algorithm emerged around 2015/2016

Perceptrons

• Perceptrons are a type of neural network that performs binary classifications
• Rosenblatt's perceptron model comprises one weight per input
• Calculation involves multiplying weights with input values and adding bias
• If the result is positive, the output is 1; otherwise, it's -1

Training a Perceptron

• Perceptron learning algorithm uses random weights as starting point
• Samples a new input-output pair (x, l)
• Calculates the output (y) using the weights and the input values
• Adjusts the weights based on the difference between the output (y) and the expected value (l) if the output and expected value differ
• The weights are adjusted according to a learning rate (η)
• Weights recalibrated to attain a desired outcome

Perceptron Limitations

• Limitations arise in simple tasks such as exclusive-or (XOR) problems that could not be solved using single layered perceptrons
• These perceptrons require multiple layers to perform more effectively.

Multi-Layer Perceptrons (MLPs)

• MLPs use multiple layers of perceptrons
• These layers work together to allow more complex problems to be solved

The "AI Winter"

• The period between 1969 - 1983 is considered the first AI winter
• The original expectation for solutions weren't realized.
• Significant discoveries in this period included Backpropagation, RNNs and CNNs

The Second "AI Winter" (1995-2006)

• This era saw the introduction of machine learning models comparable to those used earlier.
• Key advancements included support vector machines (SVMs) and various kernel methods.
• Further advancements in manifold learning, and sparse coding techniques were implemented.

The Rise of Deep Learning (2006-Present)

• Deep Learning came from Hinton and Salakhutdinov in 2006
• Deep Belief Nets (DBNs) emerged, based on Boltzmann Machines
• Deep Learning became substantially easier with deep learning algorithms capable of training multi-layered perceptrons simultaneously with one layer at a time.

Deep Learning's Data Needs

• The ImageNet dataset (2009) contributed significantly to driving the need for large datasets within deep learning
• A large number of images and the considerable processing power required are two significant necessities in contemporary deep learning models

AlexNet

• AlexNet from 2012 was a notable deep learning model that won the ImageNet competition.
• The increase in processing capacity was a driving force behind this advancement in deep learning

Scaling of Deep Learning Models

• The growth in costs of model training for large language models has increased substantially
• BERT, ROBERTa and GPT-3 are examples of notable large language models that have pushed deep learning scaling into new dimensions: these examples reflect the rise of model parameters, computational expenditures, and costs.

Deep Learning's Impact and Scope

• Deep learning's capabilities extend beyond the technological realm and into diverse sectors such as visual recognition, multi-modal language learning and robotics
• AI is capable of tasks beyond the abilities of humans, such as mastering Go, Protein folding and weather forecasting.

Deep Learning's Significance

• Deep Learning is a significant field with significant impact across various scientific and practical fields
• The ability of deep learning models to learn from a vast arraying of data is exceptional and continues to create significant advancements.