Neural Networks Basics

A TWO-LAYER NEURAL NETWORK

• Inputs are represented as vectors
• The network has an input layer, a hidden layer, and an output layer
• Each node represents a neuron
• Neurons calculate an output based on a weighted sum of inputs and an activation function

NN REPRESENTATION

• Each node in the hidden layer performs a computation
• Weights (w) and biases (b) are applied to the inputs
• The output of each node is an activation value (a)
• The output layer predicts the output based on the activations of the hidden layer

COMPUTING THE OUTPUT

• Vectorized representation is used for efficiency
• Weights are stored in a matrix (W)
• Biases are stored in a vector (b)
• The output (a) is calculated by multiplying the input (x) with the weight matrix (W) and adding the bias vector (b)

VECTORIZING ACROSS MULTIPLE EXAMPLES

• The process can be vectorized for multiple examples
• This allows for parallel computations and faster training

OTHER ACTIVATION FUNCTIONS

• tanh (tangent hyperbolic) function:
  • Shifted version of the sigmoid function
  • Outputs values between +1 and -1
  • Helps with zero-centered outputs, making learning easier in subsequent layers
  • Exception: Binary classifiers use the sigmoid function
• ReLU (Rectified Linear Unit) function:
  • Outputs the input value if it's positive, otherwise outputs zero
  • Derivative is 1 for positive inputs and 0 for negative inputs
  • Prevents the vanishing gradient problem
  • Can lead to "dead" nodes if the input is always negative

VANISHING GRADIENT PROBLEM

• Occurs during backpropagation when the product of derivatives becomes close to zero
• This is caused by the use of sigmoid and tanh activation functions
• Results in slow learning or model failing to learn

OVERCOMING THE VANISHING GRADIENT PROBLEM

• Solutions:
  • ReLU activation function: Avoids small gradients
  • Weight initialization: Prevents biases towards zero gradients

DERIVATIVES OF ACTIVATION FUNCTIONS

• Sigmoid function derivative: The derivative is the product of the sigmoid function and its complement
• tanh function derivative: The derivative is 1 minus the square of the tanh function
• ReLU function derivative: The derivative is 1 for positive inputs and 0 for negative inputs

BACKPROPAGATION EQUATIONS FOR A TWO-LAYERED NN

• Calculate dz, dW, db for both the hidden layer and the output layer
• These derivatives are used to adjust the weights and biases during training

HOW TO INITIALIZE WEIGHTS?

• **Logistic Regression:` Weights can be initialized to 0.
• Neural Networks: Initializing weights to 0 creates symmetry and hinders learning
• Solution: Random Initialization
• To avoid this, weights are randomly initialized using the np.random.randn() function
• Random initialization does not affect biases

WHAT IS A DEEP NEURAL NETWORK

• Deep networks are neural networks with multiple hidden layers
• Shallow networks have only one hidden layer
• Deep networks allow for learning more complex patterns and relationships

NEURAL NETWORKS NOTATIONS

• Layer 0 represents the input layer
• Layers 1, 2, 3... represent hidden layers
• The final layer represents the output layer

FORWARD PROPAGATION IN DEEP NETS

• Forward propagation calculates the activations for each layer
• Input (a[l-1]) is multiplied by weights (w[l]) and biases (b[l])
• The result is passed through the activation function to produce the output (a[l])
• This process is repeated for all layers

MATRICES AND THEIR DIMENSIONS

• Matrices represent weights and activations
• The dimensions of these matrices are crucial for computations
• The dimensions of the matrices need to be compatible for multiplication and addition

DIMENSIONS FOR VECTORIZED IMPLEMENTATIONS

• The dimensions of matrices (W, b, x, Z, a) are important for vectorized implementation
• The dimensions should be consistent throughout the computations

WHY DEEP NETWORKS ARE BETTER - INTUITIONS

• Deep networks can learn more complex patterns and relationships
• They can approximate functions more efficiently than shallow networks

WHY DEEP NETWORKS ARE BETTER

• A deep network with a small number of layers can represent functions that would require much more units in shallower networks

FORWARD AND BACKWARD FUNCTIONS

• The forward function takes (a[l-1]) as input and outputs (a[l]), caching (z[l], w[l], b[l])
• The backward function takes (da[l]) as input and outputs (da[l-1], dW[l], db[l])

FORWARD AND BACKWARD FUNCTIONS LAYER L

• Forward Propagation:
  • Takes (a[l-1]) as input
  • Outputs (a[l])
  • Caches (z[l], w[l], b[l])
• Backward Propagation:
  • Takes (da[l]) as input
  • Outputs (da[l-1], dW[l], db[l])
  • Calculates (dz[l])

FORWARD AND BACKWARD FUNCTIONS

• Forward propagation progresses through all layers, calculating activation values
• Backward propagation traverses layers in reverse, calculating derivatives of the loss function

SUMMARIZING

• Forward propagation calculation for layer l involves input, output and caching
• Backward propagation for layer l involves input, output, and derivative calculations

SUMMARIZING

• Backward propagation uses chain rule to update weights
• ReLU activation function helps minimize the vanishing gradient problem

APPLIED DEEP LEARNING IS A VERY EMPIRICAL PROCESS

• Experimentation is essential to find optimal parameters for deep learning

WHAT DOES ALL THESE HAVE TO DO WITH THE HUMAN BRAIN???

• Neural networks are inspired by the structure and function of the human brain
• They are not a perfect model of the brain, but they share some similarities

Explore the fundamentals of a two-layer neural network, including the role of inputs, hidden layers, and output layers. This quiz covers the computation process, vectorization for efficiency, and various activation functions used in neural networks.