Deep Learning Models and Their Parameters

A Brief History of Deep Learning

• Deep learning is a field of artificial intelligence that has shown significant progress over time.
• Key figures and developments in deep learning, including perceptrons, Adaline, and backpropagation, are marked on a timeline.
• Milestones include Perceptrons by Rosenblatt (1958), Adaline by Widrow and Hoff (1960), Perceptrons by Minsky & Papert (1969), Backpropagation (1974), LSTM by Hochreiter and Schmidhuber (1998), Deep Learning (2006), Imagenet (2009), AlexNet (2012), Resnet & GO (2015).

First Appearance (Roughly)

• A timeline shows the approximate first appearance of various deep learning concepts.
• Perceptrons were introduced by Frank Rosenblatt in 1958.
• Adaline was developed by Widrow and Hoff in 1960.
• Perceptrons by Minsky and Papert in 1969.
• Backpropagation by several researchers, including Werbos and Rumelhart, Hinton, and Williams, emerged in the 1970s and 1980s.
• Milestones include LSTM, OCR, Deep Learning, Imagenet, AlexNet, Resnet, and GO.

Rosenblatt: The Design of an Intelligent Automaton (1958)

• Rosenblatt's work described a machine resembling a biological brain.
• The machine would recognize, remember, and respond like a human mind.
• Graphs and diagrams illustrate the organization of biological brains and perceptrons.

Perceptrons

• McCulloch and Pitts introduced binary inputs and outputs, but had no learning component.
• Rosenblatt proposed perceptrons as a model for binary classifications.
• A perceptron comprises weights for each input (xi), added to a bias (b = w0x0), to produce a weighted sum (y= Σ(wjxj) + b).
• The output is 1 if the weighted sum is positive and -1 otherwise.

Training a Perceptron

• The primary innovation was a learning algorithm for perceptrons.
• Weights (wj) assigned randomly
• Samples (xi, Li) used to compute a weighted sum (y)
• If the output is incorrect, the weights are adjusted using a learning rate (η\etaη)

From a Single Output to Many Outputs

• A perceptron was originally designed for binary decisions.
• To handle multiple decisions (e.g., digit classification), multiple outputs can be appended to create a neural network.
• The diagram shows a 4-way neural network with input and output layers.

From a Single Output to Many Outputs (Quiz)

• Calculating the number of weights necessary for a visual input and a large number of categories demonstrates the significant increase in complexity that large datasets represent for neural networks.
• The calculations show large-scale datasets are needed.

XOR & 1-layer Perceptrons

• Initially, perceptrons failed with simple non-linear tasks like XOR.
• XOR's input patterns cannot be separated by a single line.
• The weighted sum needs to evaluate combinations, beyond simple linear separation.

Multi-layer Perceptrons to the Rescue

• Minsky did not establish XOR as unsolvable by neural networks.
• Multi-layer perceptrons (MLPs) can be used to solve the XOR problem.
• MLPs use multiple layers and nonlinearities (like sigmoid functions) to improve the model's capacity.

Multi-layer Perceptrons to the Rescue (Why not Rosenblatt’s method)

• Rosenblatt's algorithm cannot train intermediate layers of an MLP.
• The absence of a clear learning target in hidden layers of perceptrons made training them very problematic.

The "AI winter" despite notable successes

• A timeline shows the approximate first appearance of various deep learning concepts.
• Progress was made through the period, but there was also a period of reduced funding and interest after initial promise.

The first "AI winter" (1969-1983)

• The prevalent view was that perceptrons could not solve basic logical problems, therefore, investment declined.
• However, significant discoveries occurred during this period, such as the development of backpropagation, recurrent networks and CNNs.

The second "AI winter" (1995-2006)

• New machine learning models were developed with similar accuracy but provided mathematical models and proof.
• Kernel methods (SVMs, decision trees, random forests) and manifold learning were developed.

The Rise of Deep Learning (2006-Present)

• Significant advancement and progress in the field.
• Large-scale datasets became available, allowing training of very deep neural networks.
• The increasing computing power of hardware (especially Graphics Processing Units or GPUs) made large-scale training possible.

The thaw of the "AI winter"

• The timeline shows the approximate first appearance of various deep learning concepts.
• Backpropagation, significant progress in various sub-fields such as RNNs, CNNs and more, helped overcome some prior limitations in the field.

The Rise of Deep learning

• Hinton and Salakhutdinov developed multilayer feed-forward networks that could be pretrained layer by layer and fine-tuned with backpropagation.
• Deep Belief Networks utilize Boltzmann machines.

Neural Networks: A decade ago

• Challenges in the field included limited processing power, small datasets, and difficulties in training deeply layered perceptrons effectively

Neural Networks: Today

• The limitations described in the previous point are mitigated by advancements in technology and hardware, thereby enabling effective training.

Deep Learning arrives

• Deep learning training became easier because of the layer-by-layer approach.
• Multiple layers of networks yield better performance, but the training process for single layers is comparatively less complex.

Deep Learning Renaissance

• Timeline, which marks the approximate first appearance of various deep learning concepts, showing significant growth in the field since its initial development.

Turns out: Deep Learning is Big Data Hungry!

• ImageNet dataset introduced in 2009 was a large-scale visual dataset that became crucial for advances in deep learning.
• The dataset consisted of 1 million images with 1000 categories, crucial to evaluation and enabling the training of deep learning models in various fields.

ImageNet 2012 Winner: AlexNet

• AlexNet was a significant architecture for image processing, showcasing the increased complexity and number of weights needed for training deep learning models with large datasets.
• More weights were needed for training deep learning models when training image recognition programs.

Why now?

• Advancements in hardware and datasets are key factors in recent deep learning successes.
• Processing power, availability of larger datasets, and improved algorithms have combined to enable significant progress.
• Graph demonstrating evolution of computer hardware, datasets, and algorithms, showcasing increasing power.
• Recent advances in deep learning models.

The current scaling of models

• Scaling of models and associated computational requirements for training deep learning models.
• The increasing cost of training large language models highlights the large computational power and data resources required for deep learning models.

Deep Learning: The "Field"

• Deep learning has made significant advancements in scientific study.
• Publication counts from Google Scholar highlight deep learning's increasing importance in different scientific fields.

Deep Learning Golden Era

• Deep learning's journey through various stages of development, highlighting key milestones like the development of perceptrons, backpropagation, and deep learning itself (with associated figures).
• Timeline of key developments from perceptrons to modern tools to assist in development.

How research gets done part I

• Deep learning research involves theoretical foundations and practical application.
• Begin by solidifying fundamentals and reading various research articles.

Deep Learning in practice

• Examples of practical applications of deep learning in various domains, such as image recognition and video classification, were used in the years 2013-2016.

Deep Learning even for the Arts

• Deep learning methods are used to create and edit images.
• Illustrations showcasing various types of art created by deep learning models.

The "wow" what Deep learning can do! - 2022 edition

• A summary of interesting and new applications for deep learning and its capabilities which were recently developed.

AI beyond human capacity

• Deep learning models have demonstrated accomplishments surpassing human capabilities in complex domains like Go.
• The computations required, far exceeding the number of atoms in the universe, are illustrative of the vast computational power needed.

Vision-text Multi-modal Learning

• Focuses on multimodal approaches, which utilize both visual and textual information, for a more complete and integrated representation of data.
• Research highlights the scale of training data necessary.

Generative Pretraining

• Deep learning models can produce new or unique generated data types such as texts.
• It involves pretraining massive amounts of data to generate new, original text.

Music from AI

• AI is being used to generate music.
• Demonstration through code and example prompts.

Deep learning in robotics too

• Recent development of deep learning in robotics.
• Limitations of prior approaches and how deep learning is helping.

There's a lot more

• Resources to continue learning about current research in deep learning.
• Links to relevant newsletters, relevant websites, courses and news sources.

Conclusion

• Deep learning has made significant progress across several domains, with datasets playing an increasingly important role.
• The field keeps evolving rapidly.

This quiz explores various deep learning models, their parameters, and the costs associated with training them. It covers important concepts such as weight calculations, neural network outputs, and energy consumption. Test your knowledge on the fundamentals of deep learning and its applications.