Recurrent Neural Networks (RNNs)

Recap

• Machine learning is used when tasks can't be mathematically formulated or are too complex for closed-form solutions. Mathematical models, used in linear regression, approximate the relationship between predicted variables and predictors.
• A classifier with a sigmoid function underlies logistic regression forming the basis of feedforward neural networks (FFNNs).
• Convolutional neural networks (CNNs), designed for image processing, extract spatial hierarchies with fewer parameters than fully connected neural networks.
• Overfitting happens if a model is trained too well on training data.

Learning Outcomes

• Standard FFNNs and CNNs aren't effective for modeling sequential data.
• Recurrent neural networks' (RNNs) architecture is capable of sequential learning.
• Vanilla RNNs struggle with the vanishing gradient problem, impacting performance.
• State-of-the-art vehicle trajectory prediction models use transformers to outperform common models.

Outline of Topics

• Sequence learning fundamentals
• Shortcomings of standard neural networks for sequence learning
• Importance of weight sharing in RNNs
• Forward pass, Back-propagation through time (BPTT), and the challenges of vanishing and exploding gradients in RNNs
• LSTM and GRU architectures
• Utilisation of Attention Mechanisms & Transformers
• Real-world application examples of RNNs
• Using RNNs for vehicle trajectory prediction

Sequential Learning

• Focuses on machine learning algorithms designed for sequential data.
• Transforms input sequential data into an output sequence in a different domain.
• Speech recognition (audio to text) is an example of many-to-many learning.
• Sentiment analysis and abuse language identification are applications of sequential learning.
• Many-to-one learning is when an input is a sequence, but not the actual output.
• Machine translation is a many-to-many paradigm, which allows input and output sequences of variable lengths.
• Frame-level video classification is another example of a many-to-many model.
• Image captioning: one-to-many learning approach

Video Prediction with ANNs

• By inputting a certain time of frames predict the next frame.
• Lots of parameters are needed to train the model.
• Scalability and accuracy are compromised when capturing temporal correlations for sequential learning.

Features of Sequential Learning

• Sequential learning involves input and output sequences, whose lengths can vary across tasks.
• Traditional networks treat inputs as independent, unlike sequential data where strong correlations exist.
• The ordering of inputs matters significantly as changes to the order will impact the meaning.
• Long-term dependencies are a key factor.

Weight Sharing in RNN

• Sequences of frames are entered as vectors in standard neural networks.
• RNNs take into account pixels shared among frames to find temporal relations.
• Weight sharing allows generalisation to sequences of variable lengths.
• Weights W_ax are distributed to frames in the same sequence.
• Recursion is added at hidden layer with a new set of weights W_aa shared within the training sequence.
• RNN architecture extends to cover many-to-many models.
• Weight matrices such as W_aa, W_ax, and W_ya, are shared across time.
• On the computation graph, that's the RNN representation at the left-hand-side.

Forward Propagation in RNN

• Forward propagation is similar to standard FFNNs.
• Internal state relies on inputs from all sequence.

Forward Propagation in Many-to-Many RNN

• A loss must happen in each time step, while the the total loss L is derived from each loss at time step L_t.

Back-Propagation Through Time (BPTT)

• An RNN with a few cells and single output is used to illustrate.

BPTT Details

• During training, loss function gradient must be calculated re all weights: W_ya, W_ax, W_aa.
• The loss gradient must be calculated over previous time steps in sequence.

BPTT calculations

• α^<3> = tanh(W_ax x^<3> + W_aa α^<2>)

Vanishing Gradients

• Gradients vanish when a large sequence with long-term dependencies is used.
• The chain rule dictates repeated multiplications occur in calculating the loss gradient relative to the weights W_ax or W_aa.
• Weights are not updated if each partial derivative is below one and n is large.
• The vanishing gradient problem occurs with both the tanh() and sigmoid functions.
• RNNs tend to capture short-term memory due to vanishing gradients.
• Solutions include different activation functions such as ReLU, using weight initialisation, or using gates.

Vanishing and Exploding Gradients

• The Rectified Linear Unit (ReLU) heads towards one direction.
• Gradients might also explode.
• Gradient clipping mitigates the exploding gradient problem.

Gated Recurrent Unit (GRU)

• A GRU modifies the hidden layer of the basic RNN.
• Improves in capturing long-term dependencies and mitigating the vanishing gradient problem.
• Hidden state formula: c = tanh(W_xc x + W_cc c)

GRU cont'd

• GRU adds a sigmoid gate Γu alongside extra weights.
• It calculates a new cell state c different to that of standard RNNs.
• Parameter č has the standard RNN calculation. č = tanh(W[c, x] + b_c)
• The cell state is a weighted sum of previous state and č.
• c* = Γ_u č + (1 – Γ_u)c
• GRU adds more memory calculating by taking a fraction from the other state c
• The parameter's value will determine info that is irrelevant and the significance to store č
• The GRU and RNNs compute the hidden state given the previous, but in different ways.

Long Short-Term Memory (LSTM)

• LSTM was proposed in 1997 and has 3 sigmoids beyond the tanh().
• Gates will work in tandem to identify what is let through.
• A value of the forget gate f_t near 1 mostly passes state c_t-1 to the next c_t
• An gate where the new value c~ is not passed onto the next state c_t when gate i_t is close to zero.
• The current plus previous cell state is known as additive.
• The output gate determines how much gets output to the next hidden layer.
• Hidden is a filtered version of the filter c_t.
• The memory h_t will be used to readout as y_t.
• LSTM capable of of forgetting, storing, outputting and updating data.
• Relationship between the current and previous is additive and helps with the vanishing and exploding gradients.

LSTM vs Attention Mechanism

• Sequence-to-sequence (Seq2Seq) consist of an encoder/decoder model.
• An LSTM encodes the input sentence and another LSTM receives the translation.
• The last hidden and cell states' encoder may not be good to use translation.
• The attention mechanism makes predictions at the current time-step at the decoder by attending all hidden states of the encoder.
• Each time step has different states, to define what is important at each time step.

Attention Mechanism

• Weight at,j shows the relevance of hidden state h_j of said encoder.
• Softmax shows the decoder's inputs xj to state s_t-1
• Context vector includes these parameters. c_t = Σ^T_j=1 a_tj h_j.
• The predicted state s_t at the decoder is trained via a small feedforward neural network with inputs via context c_t and previous output s_t−1.
• Instead of encoding to the hidden state, it leverags the hidden state to select them adaptively until termination.

Transformers

• Transformers outperform models such as LSTMs.
• Employs self and cross-attention to find relationship between input and output sequences.

Typical Applications

• Includes Video, Audio and Text.
• Video Action Recognition
• Speech Recognition
• Sentiment Analysis
• Video classification
• Video prediction
• Language modeling
• Image captioning
• Machine translation

Video Prediction

• Can pre-trained CNNs be extracted like this?
• Can a pro-trained CNN be used?
• CNN is used during inferences stage with the existing training.

Vehicle Trajectory Predications

• Detecting the surrounding vehicles' location is insufficient for safe decision making.
• The problem involves estimating the sequence of trajectory TV's future positions via a prediction window YTv = {(X1,Y1), ..., (XTpred, YT pred)} from info of the TV during the observation window.
• Observe with Tobs
• There are 3 pipelines: Inputs, Encoder, and a Decoder.
• Baseline models: Constant Velocity Model (CV) when they contain the same speed in the observation window.
• Vanilla LSTM is known as V-LSTM which makes predictions with the same trajectory.

Model

• Highway Drone Dataset (highd) consists of more than 16 hours of recording.
• Root Mean Squared Error
• Final Displacement Error Final Displacement Error (FDE). This metric computes the distance between the predicted and ground truth position of the TV at the final time-step of the prediction window.

Qualitative Results

• Show results in lane change detection with a 3 second window.

Reflection Questions

• Why are standard FFNN and CNN models inefficient for sequential data?
• What are the key features?
• Can you explain how vanilla RNNs learn sequentially?
• Primary reason for the RNN's poor learning for long sequences?
• How does Attention mechanisms improve learning via LSTMs?
• Why do attention mechanisms and transformers outperform algorithms for trajectory prediction in autonomous driving applications?

Summary

• MLPs and FFNN are inadequate due to scalability.
• RNNs have long term issues, and use exploding gradients.
• LSTM provides solutions to the memory issues. LSTM/GRUs use pointwise multiplication and addition for gated info.
• RNNs, GRUs, LSTMs used high memory and computation.
• Difficulty occurs due to the nature of computations that happen.
• Very long encodings might miss certain temporal dependencies and inputs in the sequence.

Summary

• Attention mechanism allows a subset of inputs which will enhance the vanilla LSTM.
• Transformers will perform well in driving simulations.
• Independece can explain long horizon results.
• models are enough for driving but not those more difficult.
• Implemented predictions and motion planning should be sensory.

Explore recurrent neural networks (RNNs) for sequential data modeling, addressing the limitations of FFNNs and CNNs. Understand the challenges of vanishing gradients in vanilla RNNs. Learn how transformers enhance vehicle trajectory prediction, surpassing traditional models.