Neural Networks and Activation Functions

The input layer receives real data values and transmits them to the hidden layer.

Hidden layers are layers between the input layer and the output layer.

The output layer produces the final estimation of the output.

The perceptron was the first form of neural network.

Weights determine the strength of the connection between neurons. They are initially set randomly.

A multi-layer perceptron is a network of interconnected perceptrons, with an input layer, output layer and possibly hidden layers.

The AI Winter was a period of reduced funding and research in Artificial intelligence.

A simple model of a biological neuron, consisting of dendrites, a nucleus, and an axon.

A mathematical function used in neural networks to determine the output of a node. It maps the resulting values into a specific range, often between 0 and 1 or -1 and 1.

A type of activation function that allows for a linear relationship between input and output. It doesn't have a fixed range.

A type of activation function that introduces non-linearity into the neural network, allowing it to learn complex patterns. They have a limited output range, often between 0 and 1 or -1 and 1.

A non-linear activation function with a S-shaped curve. It maps inputs to values between 0 and 1. It is commonly used for binary classification tasks.

A non-linear activation function with a range from -1 to 1. It is similar to the Sigmoid function but with a steeper slope. It is known for being more effective in certain scenarios.

A non-linear activation function that outputs the input directly if it is positive, and zero if it is negative. It is known for its simplicity and efficiency.

An activation function that converts a vector of numbers into a probability distribution. The output is a vector with probabilities that sum to 1.

The process of training a neural network by feeding it a large dataset with known answers. The network adjusts its internal weights to minimize errors and improve its accuracy.

A method of updating model parameters based on the difference between predicted and actual values. It involves iteratively adjusting weights to minimize the error. Common methods include batch, stochastic, and mini-batch gradient descent.

A type of gradient descent that uses the entire training dataset to calculate the gradient and update parameters in each iteration. It is computationally expensive but provides accurate updates.

A type of gradient descent that uses a single data point to calculate the gradient and update parameters in each iteration. It is computationally efficient but can lead to noisy gradients and oscillations.

A type of gradient descent that uses a small batch of data points to calculate the gradient and update parameters in each iteration. It balances efficiency and accuracy, making it a popular choice for training neural networks.

A measure of how well a model predicts the actual values. It quantifies the difference between predicted and actual values, with a higher value indicating a greater error.

A method for evaluating the performance of a machine learning model. It measures how well the model fits the training data.

An optimization algorithm for gradient descent that utilizes exponentially decaying averages of past gradients and squared gradients, allowing for adaptive learning rates for each parameter. It is one of the most commonly used optimization algorithms in deep learning.

A technique used in gradient descent optimization to adjust the step size (learning rate) during training. By starting with larger steps and gradually decreasing them, it helps to converge faster and avoid getting stuck in local minima.

The process of adjusting weights and biases in a neural network to minimize the error between predicted outputs and actual outputs.

One of the key components of a neural network, weights represent the strength of connections between neurons.

A set of nodes representing a level in a neural network where information is processed.

The process of passing input data through the neural network to generate outputs.

The error is calculated between the predicted output and actual output in the network.

The initial values assigned to weights in a neural network, often chosen randomly.

The gradient of a function provides information about the direction and magnitude of the steepest change in the function's output. It's a vector that points in the direction of the largest increase in the function's value.

A mathematical technique used to calculate the gradient of a composite function by chaining together the derivatives of its individual components.

A measure used for evaluating the performance of a classification machine learning model. It calculates the average difference between the probabilities predicted by the model and the actual probabilities of the true classes.

A measure used for evaluating the performance of a regression machine learning model. It calculates the average squared difference between the predicted values and the actual values.

A measure used for evaluating the performance of a regression machine learning model. It calculates the average absolute difference between the predicted values and the actual values.

The difference between the value predicted by the neural network (output) and the original value (target). For example, if the network predicts 0.56 and the target is 1, the error is 0.5 * (1 - 0.56)² = 0.0968.

The change by which the connecting weights in a neural network need to be updated to minimize the error and improve the network's accuracy.

Using a neural network's learned weights to make predictions on new, unseen data. It involves feeding input data into the trained network and obtaining an output.

A type of neural network architecture that can learn nonlinear relationships in the data, using one or more hidden layers. It allows for a complex mapping between input and output values.

A mathematical function that introduces non-linearity into a neural network, allowing it to learn complex patterns. It maps the resulting values into a specific range, often between 0 and 1 or -1 and 1.