Deep Learning Concepts Recap

Optimizers like Adam and RMSprop improve the gradient descent process to find optimal weights for a neural network more efficiently.

Techniques like dropout and batch normalization prevent overfitting by introducing randomness and normalization during training, leading to better generalization.

These libraries (TensorFlow, Theano, PyTorch, MxNet, Keras) provide building blocks and tools for constructing and training deep learning models.

Traditional ML focuses on handcrafted features for training, while DL extracts features automatically from data.

Traditional ML often requires smaller datasets, while Deep Learning thrives on massive amounts of data.

Deep Learning models, being more complex, often demand more computational resources than traditional ML models.

Traditional ML models are generally simpler and easier to understand, while Deep Learning models can be complex and opaque.

Deep Learning excels at handling unstructured data like images and text, while Traditional ML has limitations in this area.

An optimization algorithm used to minimize the loss function of a neural network by adjusting its weights

A type of neural network designed specifically for processing grid-like data like images.

A small matrix that slides over input data, performing dot products to detect patterns.

The output of the convolution operation, representing detected features like edges or textures.

A process that reduces the spatial size of feature maps, making computations more efficient and improving translation invariance.

An early CNN architecture designed for recognizing handwritten digits. It has multiple convolutional and fully connected layers.

A deeper and more powerful CNN architecture that won the ImageNet competition in 2012. It consists of multiple convolutional and fully connected layers.

A CNN architecture known for its simplicity and depth. It uses small convolution filters and is available in several variants like VGG16 and VGG19.

When a model is too complex and performs well on the training data but poorly on new data. This occurs when a model remembers the training data too well, limiting its ability to generalize to new, unseen examples.

When a model is too simple and cannot learn the underlying patterns in the training data. The model struggles to perform well on both training and new data.

A graph showing training and validation errors over epochs, which helps identify overfitting or underfitting. A large gap between training and validation errors suggests overfitting.

Using multiple subsets of the data to train and test a model, providing a more robust estimate of the model's generalization ability.

A technique that adds the sum of absolute weights to the loss function, encouraging sparsity and potentially improving generalization.

A technique that adds the sum of squared weights to the loss function. It helps prevent overfitting by discouraging large weight values.

A technique that involves setting a certain percentage of neurons to zero randomly during training, preventing the model from over-relying on specific neurons.

A technique that normalizes the values of hidden layers in a neural network, helping to improve the stability and performance of training.

Google developed this framework, offering high performance and a vast community.

Facebook's creation, known for its dynamic graphs and ease of use, often preferred for research.

A high-level library built on top of other frameworks like TensorFlow, providing a user-friendly interface for building neural networks.

The mathematical representation of data in deep learning, enabling efficient computation.

The sequence of computations performed in a deep learning model, representing the flow of information.

A powerful visualization tool that helps understand and debug deep learning models.

Pre-trained models ready for use, saving time and resources in development.

Refers to the limited computational resources, memory, and energy requirements of devices in an IoT network.

Challenges related to bandwidth, latency, and intermittent connectivity, affecting data exchange between IoT devices and the cloud.

Handling real-time data processing, ensuring data privacy and security, and managing data from various sources.

The ability to manage a large number of devices with diverse hardware and software specifications.

The ability of IoT systems to function in harsh or unpredictable environments.

A model compression technique involving removing unnecessary weights and neurons to reduce model size, making it more suitable for resource-constrained environments.

A model optimization technique for reducing the size of a deep learning model without losing significant accuracy, often used in resource-constrained environments.

The process of adjusting model parameters to reduce the error between predicted outputs and actual target values, enabling the model to learn from data.

Edge devices can make decisions based on local information, which may not be known to a central system.

Processing and storing data closer to the source, often on the device itself, minimizing network latency.

Processing and storing data in a centralized server, enabling large-scale analysis and resource sharing.

A machine learning technique that trains a model on decentralized data distributed across devices.

Bringing the code to the data, rather than the data to the code, in federated learning.

A central server initializes a global model, and device models are trained locally.

Local models send their updates to the central server, which updates the global model.

The global model is distributed back to the devices for further training and use.