ML lecture 5

10 Questions

What is a limitation of Gradient Descent related to reaching the minimum?

GD only guaranteed to reach local minimum

Why might Gradient Descent not generalize well outside the training set?

Training-set minimum might over-fit

What is a potential issue with the computation of Gradient Descent over large datasets?

GD does not scale well with the size of the dataset

What is the purpose of weight decay in neural network training?

To achieve classification with smaller weights to prevent overfitting

Which method is used to prevent overfitting by stopping training before reaching the minimum on the training set?

Early stopping

What is the purpose of dropout in neural network training?

To approximate ensemble methods with exponentially many models and improve generalizability

What is the purpose of dropout in neural networks?

To avoid over-reliance on single nodes for generalization

What does adversarial training involve in neural networks?

Training the network on misclassified examples by adding imperceptibly small vectors to the input

What is the purpose of data augmentation in neural networks?

To create more data that is a noisy version of the original data to improve generalization to unseen samples

What does transfer learning enable in neural networks?

Transfer of knowledge from one model or problem to another, enabling faster adaptation to new domains

Study Notes

Neural Networks: Key Concepts and Techniques

Dropout is used to get rid of dominant nodes in each layer to avoid over-reliance on single nodes for generalization.
Dropout approximates ensemble methods by pasting a mask onto the network for each stochastic gradient-descent mini-batch, allowing prediction using only the sub-network.
Adversarial training involves training the network on examples that it misclassified by adding imperceptibly small vectors to the input, leading to better generalization to similar, unseen adversarial samples.
Small mini-batch stochastic gradient descent offers a regularization effect due to the noise they add to the learning process.
Data augmentation involves creating more data that is a noisy version of the original data to improve generalization to unseen samples, especially beneficial for object recognition in images.
Transfer learning allows for the transfer of knowledge from one model or problem to another, enabling faster adaptation to new domains and making models more robust.
Deep neural networks are powerful tools in data science, particularly effective in cases with large amounts of data and where combinations of basic units of information are useful.
Neural networks are constantly evolving with new results, and they are currently the best performing models on "big data" composed of complex combinations of local features.
The feature space of deep neural networks, including local minima, saddle points, and multiple global minima, is well understood, but the exact workings of the models remain unclear.
The algorithms described are over 20 years old, but advancements in technology, such as more data and stronger computers, have made previously intractable problems solvable.
The field of neural networks has a constant influx of new results, and it is considered a fast-moving field with ongoing developments.
Despite the evolution of neural networks, the underlying concepts and methods, such as regularization, have been around for a long time and are not theoretically complex, but advancements in technology have made them more effective.