RNNs: Understanding Recurrent Neural Networks

• Recurrent Neural Networks (RNNs) are designed to process sequential data using internal memory to maintain information about previous inputs.
• RNNs excel in tasks needing context and temporal dependencies due to their recurrent connections.
• RNNs face challenges like vanishing and exploding gradients, which are addressed by LSTM and GRU architectures.
• Innovations like LSTM and GRU enhance the usability and performance of RNNs.
• RNN architecture involves interconnected layers, passing information between time steps, and updating the hidden state.
• Interconnected layers form a loop, thus making RNNs effective for learning temporal dependencies and patterns.

Mathematical Definition of Sequences

• Sequences are ordered lists where the order of elements matters, used for modeling temporal or ordered data.
• A sequence maps indices from integers or natural numbers to a set of values: x: N → X, with xt ∈ X.
• T represents the length of the sequence.
• Finite sequences contain a specific number of elements.
• Infinite sequences continue indefinitely.
• Discrete sequences take integer values.
• Continuous sequences span real numbers.
• Sequence elements can be scalars, vectors, or tensors.

Deterministic vs Stochastic Sequences

• Deterministic sequences have elements determined by a fixed rule, allowing exact prediction of the next element.
• Deterministic sequence follow a specific rule, are predictable, and lack randomness.
• Stochastic sequences have elements that are random variables, and governed by a probability distribution.
• Stochastic sequences are influenced by a random process and they involve uncertainty.

Sequence Examples

• Numerical Sequence (Finite, Scalar): X = {1, 3, 5, 7, 9}.
• Time Series Data (Infinite, Continuous): x(t) = Asin(2πft), t ∈ R.
• Vector Sequence (Finite, Vector): X = {x1, x2, x3}, xt ∈ Rd.

Mathematical Operations on Sequences

• Shift: Delays or advances the sequence: xt+k (shift by k).
• Concatenation: Combines two sequences x and y: z = {x, y}.
• Aggregation operations include Sum (∑t=1T xt) and Mean (1/T ∑t=1T xt).
• Temporal dependencies relate elements in a sequence, where the particular element value is depends on preceding elements.
• The Markov Chain model has future state dependent only on current state, called the Markov property. The equation is P(xt+1|xt, xt-1, ..., x1) = P(xt+1|xt).

Feedforward Networks Limitations

• Feedforward networks cannot capture temporal dependencies.
• Feedforward networks use fixed-size inputs.
• Feedforward networks lack memory of past inputs.
• Feedforward networks are inefficient with long sequences.
• Feedforward networks provide fixed input representations.

RNNs Advantages for Sequential Data

• RNNs preserve hidden states for past inputs which captures long-term dependencies.
• RNNs process sequences of variable lengths
• RNNs build contextual understanding by using information from previous time steps.

Neural Networks without Hidden States

• The forward propagation from the input layer to the output layer is given by: h = f(W1x + b1)
• x ∈ Rn is the input vector.
• W1 ∈ Rm×n is the weight matrix for the input-to-hidden layer.
• b1 ∈ Rm is the bias vector for the hidden layer.
• f(⋅) is the activation function (e.g., ReLU, sigmoid).
• The output layer is computed as: y = W2h + b2
• h ∈ Rm is the hidden layer activation vector.
• W2 ∈ Rp×m is the weight matrix for the hidden-to-output layer.
• b2 ∈ Rp is the bias vector for the output layer.
• y ∈ Rp is the output vector.

Neural Networks with Hidden States

• RNNs have a feedback mechanism that allows information to persist over time.
• The hidden state acts as a memory updated at each time step.
• The mathematical representation of the hidden state: ht = f(Wxxt + Whht-1 + b).
• ht is the hidden state at time step t
• xt is the input at time step t
• Wx and Wh are the weight matrices for the input and previous hidden state, respectively
• b is the bias term
• f(⋅) is the activation function (such as tanh or ReLU)
• The output of the RNN at time step t is computed: yt = Wyht + c
• Wy is the weight matrix connecting the hidden state to the output
• c is the bias term for the output

Forward Propagation in RNNs

• An RNN processes sequential data by maintaining a hidden state.
• The input xt combines with the previous hidden state ht-1 to compute the current hidden state ht.
• The hidden state is then used to calculate the ouput.
• Hidden state calculated as: ht = f(Wxxt + Whht−1 + b).
• Wx is the weight matrix for the input xt.
• Wh is the weight matrix for the previous hidden state ht-1.
• f(⋅) is often a tanh or ReLU activation function.
• The output at time step t is computed as: yt = Wyht + c

Backpropagation Through Time (BPTT)

• BPTT extends the backpropagation algorithm for RNNs by propagating error backward through time.
• The RNN performs the following computations at each time step t: ht = f(Whht-1 + Wxxt + b).
• yt = Wyht + by outputs at time t.
• To update weights, gradients of the loss C are computed in relation to the weights. Backpropagation requires unrolling the RNN across all time steps.
• Gradient of Loss formula with respect to the output.

Activation Functions in RNNs

• Activation functions are mathematical functions to introduce non-linearity, like:
• Sigmoid Function: σ(x) = 1 / (1 + e-x), Range: (0,1), Often used in LSTM gates
• Hyperbolic Tangent (tanh): Range: (-1,1), Commonly used in hidden states to ensure zero-centered data.
• ReLU (Rectified Linear Unit): ReLU(x) = max(0, x), Range: [0, ∞), Rarely with RNNs but in deep learning architectures.
• Softmax Function: Softmax(xi) = exi / ∑j exj , Range: (0, 1), sequence classification tasks

Loss Functions for Sequential Data

• Loss functions quantify the difference between network predictions and the true target values:
• Mean Squared Error (MSE); formula is MSE = 1/T ∑t=1T (yt - ŷt)2, tasks that use regression.
• Cross-Entropy Loss; formula is ℒ = − 1/T ∑t=1T ∑i yt, i log(ŷt,i), sequence classification
• CTC Loss; formula is CTC Loss = −log(P(y|x)), tasks where alignment between input and output sequences is unknown

Limitations and Challenges

• Computational complexity arises as models and datasets grow.
• Time complexity is the number of basic operations to solve a problem as a function of input size.
• Space complexity refers to the amount of memory required by an algorithm to store data structures.
• Long-term dependency is the challenge models face when learning patterns in data spreading over long sequence
• Vanishing and Exploding Gradients problems arise while training deep models, affecting weights of network.

Overfitting and Generalization

• Overfitting occurs when a machine learning model learns the noise or random fluctuations in the training data rather than what is underlying.

Solutions

LSTM networks, Gated Recurrent Units (GRU) networks, and Deep Recurrent Neural Networks (RNNs).

Long Short-Term Memory (LSTM) Networks

• LSTMs address the vanishing gradient problem.
• They introduce memory cells to enable learning of long-term dependencies.
• An LSTM unit consists of input, forget, and output gates, and a memory cell.
• Benefits include retainment of important information over many time steps, effective addressing the vanishing gradient problem.

Gated Recurrent Unit (GRU) Networks

• GRUs simplify LSTMs by combining the input and forget gates into a single update gate, resulting in fewer parameters.

Bidirectional RNNs

• BiRNNs process input sequences in both directions to capture information from past and future time steps useful for making predictions.

Deep RNNs

• Deep RNNs include stacking multiple layers of RNNs for learning complex representations and improve the ability to capture temporal patterns.

RNN Applications

• RNNs are suited for the tasks where input is sequential.

Sentiment Analysis Example

• Input consists of the sequence of words encoded as vectors.
• Initial hidden state and a formula allows the hidden state to be updated
• By processing steps a hidden state is generated containing learning representation to make output predictions.

Other Domains

• RNNs are used in time series analysis like financial forecasting, stock price prediction
• RNNs are used in audio processing like speech recognition.
• RNNs are in video analytics like action recognition using video frames.

Future of RNNs

• RNNs will likely remain a valuable tool, as they have good memory efficiency.
• As research continues there will be improvements that combine the strengths of RNNs.

Explore recurrent neural networks (RNNs) and their key characteristics. Understand how RNNs differ from feedforward networks and process sequential data using hidden states. Learn about applications, challenges like vanishing gradients, and the relevance of RNNs alongside Transformers.