Gradient Descent and Neural Networks Quiz

A mathematical formula that represents the change in the loss function (a measure of how inaccurate a model is) with respect to the model's parameters. It tells us how much the loss changes when we tweak a particular model parameter.

A parameter in machine learning algorithms that controls the step size taken during optimization. It determines how much we update the model's parameters in each iteration.

The process of adjusting the parameters of a model to minimize the loss function. This is done by repeatedly moving the parameters in the direction of the negative gradient.

A specific point in time during the training process where the model's parameters are updated. It represents one pass through the data.

The values of the model's parameters at the start of the training process.

A function that calculates the difference between a model's predictions and the true labels. It quantifies how well the model fits the data.

The process of finding the optimal parameters within a neural network by minimizing the total loss across the entire training set.

The sum of the individual losses calculated for each image in the training set.

The optimal set of parameters that minimize the total loss, thus making the model perform best.

A loss function specific for classification tasks. It measures the difference between the predicted probability distribution of classes and the true class labels.

A loss function used for regression tasks. It calculates the average squared difference between the predicted values and the true values.

A loss function used for regression tasks. It calculates the average absolute difference between the predicted values and the true values.

A numerical method used to adjust the parameters of a neural network by repeatedly moving in a direction that reduces the loss function.

The direction of the fastest increase of the loss function when parameters are changed.

The process of updating the parameters of a neural network by moving in the direction opposite to the gradient of the loss function.

The first step in the gradient descent algorithm. It involves choosing an initial set of parameters for the neural network.

The second step in the gradient descent algorithm. It involves calculating the gradient of the loss function with respect to the parameters.

The third step in the gradient descent algorithm. It involves updating the parameters by moving in the opposite direction of the gradient.

The fourth step in the gradient descent algorithm. It involves checking if the loss has converged to a satisfactory level. If not, repeat steps 2 and 3.

Deep Learning (DL) is a type of machine learning that excels at learning complex patterns from large datasets and is used in a wide range of applications like image recognition, speech processing, and natural language understanding. DL has surpassed traditional machine learning techniques, particularly in areas such as speech recognition and computer vision.

Deep Neural Networks (DNNs) are a type of artificial neural network with multiple layers of interconnected nodes, allowing them to learn complex representations from data. These layers enable DNNs to capture intricate patterns that traditional machine learning models might miss.

Universal approximators are models that can approximate any continuous function given enough parameters and complexity. Deep neural networks with at least one hidden layer are universal approximators, meaning they can approximate any continuous function with a desired accuracy.

Deep Learning (DL) allows models to learn not only the final output but also the intermediate representations of the data. This end-to-end learning approach simplifies the process of building and training models, enabling efficient learning directly from raw data with fewer assumptions.

Perceptrons are basic processing units in a neural network, taking inputs and applying a weighted sum to generate an output. Each perceptron receives inputs and outputs based on learned weights associated with each input. It represents the foundational building block for more complex neural networks.

A single-layer neural network consists of a single layer of perceptrons, where inputs are directly connected to the output layer. Each perceptron in the layer sums the weighted inputs and applies a non-linear activation function to determine the output.

Training a neural network involves adjusting the weights of its connections to minimize the error between predicted and target outputs. This adjustment is done iteratively by presenting training instances to the model, comparing the predicted outputs with the actual labels, and updating the weights based on the error.

A dataset is a collection of data instances used to train a machine learning model. Each instance represents a specific observation with features and a corresponding label. The model learns patterns from this dataset to make predictions on unseen data.

Training data is a subset of the dataset specifically used to train a machine learning model. This data is used to adjust the model's parameters and help it learn patterns and relationships in the data.

Feature is a specific attribute or characteristic of a data instance. For example, in an image dataset, features could be pixel values or colors. Features provide the model with information to make predictions about the target variable.

Class is a label or category assigned to a data instance. It represents the target variable that the model is trying to predict. For example, in a classification task, classes could be 'cat' or 'dog.'

Weights are adjustable parameters in a neural network that determine the strength of connections between neurons. During training, weights are adjusted to optimize the model's performance by reducing the error between predicted and actual outputs.

Activation function is a non-linear function applied to the weighted sum of inputs in a neuron. It introduces non-linearity to the model, allowing it to learn complex relationships in the data. Common activation functions include sigmoid, ReLU, and tanh.

The decision boundary is a line or surface that separates different classes in the data. It represents the model's ability to classify instances based on their features. The decision boundary is learned during training and adjusted to minimize prediction errors.

The decision boundary perspective helps visualize the learning process of a neural network by observing the evolution of the decision boundary as the model is trained. It shows how the model learns to differentiate between different classes by continually adjusting the decision boundary based on the training data.

A small region in the input image that a neuron in a convolutional layer processes.

A layer in convolutional neural networks (CNNs) that performs a dot product between a filter and a small region of the input, creating a new feature map.

A 2D representation of the outputs from neurons in a convolutional layer. It shows how the filter responds to different parts of the input image.

The parameters that define how the filter works, these are shared across all neurons in the convolutional layer that use the same filter.

A layer in CNNs that downsamples the feature maps, reducing the spatial size and preventing overfitting. It can be either max pooling or average pooling.

A pooling method that selects the maximum value within a rectangular neighborhood in the feature maps.

A pooling method that calculates the average value within a rectangular neighborhood in the feature maps.

A layer in a neural network where each neuron is connected to all the values in the previous layer, creating a fully connected network.

The process of training a model to learn patterns and make predictions. It involves adjusting the model's parameters iteratively using an optimization algorithm.

A measure of how inaccurate a model is on its predictions. It's an important metric to minimize during training.

The process of finding the minimum of a loss function, which measures the error between the prediction and the actual output.

The loss function may have multiple minima points, and gradient descent may converge to a local minimum, which is not necessarily the global minimum. This means it might not find the best possible solution to the problem.

The initial weights and biases of a neural network are typically randomly assigned. This means that with every run of training, the network might find a different minimum point on the loss surface, leading to inconsistent predictions.

The process of calculating the gradients of the loss function with respect to the model parameters. It involves computing the partial derivatives of the loss function using the chain rule, starting from the output layer towards the input layer.

The forward pass in a neural network involves passing the input data through the network layers to obtain the output prediction. The loss function is then calculated based on the difference between the prediction and the actual output.

A type of gradient descent that computes the loss over a subset of the training data, called a mini-batch, instead of the entire dataset. This makes the training process more efficient for large datasets.

A technique used to prevent the model from getting stuck in local minima or plateau regions. It involves adding small random noise to the gradients during training, allowing the model to escape these areas.

A process of adjusting the weights and biases of a neural network based on the gradients of the loss function. The idea is to update the parameters in the direction that minimizes the loss.

The process of repeatedly training a machine learning model using a subset of the data multiple times. This helps to reduce the chance of getting stuck in local minima and improve the generalization ability of the model.

A technique used to gradually reduce the learning rate during training, helping the model converge to a better solution and avoid getting stuck in local minima.

An approach to learning rate decay where the learning rate is reduced by a factor (e.g., half) after a certain number of epochs.

A method of adjusting the learning rate where it gradually decreases over time, often following an exponential or cosine function.

A method to adapt the learning rate where it is reduced when the validation loss stops improving. This helps prevent overfitting.

A strategy where the learning rate is gradually increased at the beginning of training and then reduced as training progresses.

A problem that can occur in neural networks where gradients become very small during training, causing the model to learn very slowly.

A situation where a model fails to capture the underlying relationship between features and targets. It struggles to learn from the training data.

A condition where a model learns the training data too well but performs poorly on new data. It fits the noise in the data instead of the true patterns.

A technique to reduce overfitting by adding a penalty to the loss function based on the magnitude of the model's weights.

A regularization method that adds a penalty to the loss function based on the absolute values of the weights.

A regularization technique where units in a neural network are randomly dropped during training, preventing the model from relying too much on any single unit.

A technique to stop training a neural network early when the performance on a validation set starts to decline.

A technique that normalizes the input data within each batch during training, helping to stabilize training and improve performance.

The process of finding the optimal values for the hyperparameters of a machine learning model.

A technique for evaluating models by dividing the training data into k folds and training the model on k-1 folds, then evaluating it on the remaining fold. This process is repeated k times.