Deep Learning Basics and Architectures

Deep Learning Basics

• Deep learning is a subset of machine learning based on artificial neural networks.
• Artificial neural networks use interconnected nodes (neurons) to process input data.
• Neural networks have an input layer, one or more hidden layers, and an output layer.
• Neurons in each layer receive input from the previous layer.
• The output of one neuron becomes input for others in the next layer.
• Layers transform input data.
• The network learns complex representations of the input.
• Networks are trained by using large amounts of data through algorithms to progressively improve accuracy.

Types of Deep Learning Architectures

• Convolutional Neural Networks (CNNs): Used primarily for image processing, CNNs automatically and adaptively learn spatial features through layers.
• Recurrent Neural Networks (RNNs): Suitable for sequential data (time series, natural language). RNNs have loops which allow information to persist, making them well-suited for tasks like speech recognition and language modeling.
• Long Short-Term Memory Networks (LSTMs): A type of RNN designed to handle complex sequences (text and speech), LSTMs overcome the vanishing gradient problem.
• Generative Adversarial Networks (GANs): These networks consist of a generator and a discriminator, competing against each other to produce high-quality synthetic data like images.
• Transformers: Designed to manage long-range dependencies in data, Transformers are the basis for models like GPT and BERT, often used in natural language processing.

Biological Neuron

• Neurons are the fundamental units of the nervous system.
• Neurons transmit information to various parts of the body through electrical and chemical signals.
• Neurons have dendrites (branch-like structures that receive messages), a cell body, and an axon (tube-like structure that sends signals to other neurons).
• Synapses are the chemical junctions between neurons' axons and dendrites.

Neuron Structure

• Neurons have dendrites, a cell body (containing the nucleus), and an axon.

Linear Perceptron

• A simple type of artificial neural network.
• Used primarily for binary classification.
• It learns linearly separable patterns.
• Consists of a single layer of input nodes connected to a layer of output nodes.
• It maps input features to output predictions using weights and an activation function.
• Input features represent characteristics of input data.

Perceptron Learning Algorithm

• Iteratively updates weights to minimize misclassification errors.
• The weight update rule adjusts weights based on the learning rate and difference between predicted and actual values.

Convergence Theorem for Perceptron Learning Algorithm

• If data is linearly separable, the perceptron learning algorithm will converge to a solution in a finite number of iterations.
• This convergence is dependent on the rate of adjustment to the weights.
  • The maximum norm of input vectors
  • The margin of the separating hyperplane

Activation Functions

• Functions help determine neuron activation and the network's model output.
• Step function: Simple, computationally efficient, but not differentiable at 0.
• Sigmoid function: Outputs bounded between 0 and 1; smooth; prone to vanishing gradient.
• Tanh function: Similar to sigmoid, but zero-centered; also susceptible to vanishing gradient.
• ReLU function: Popular; computationally efficient; faster to converge.
• Softmax function: Used in the output layer for networks that require probabilities for multiple categories (classes).

Linear Separability

• Data points can be separated by a straight line or hyperplane.
• Linear separability is a crucial condition for the perceptron learning algorithm to converge.
• The algorithm may not converge if the data is not linearly separable or if there's noise or overlap in the data.