AIE332 - Deep Learning - Lecture 6 - RNN Architectures

• AIE332 Deep Learning, Lecture 6 covers special RNN architectures, focusing on LSTM, Encoder/Decoder models, and Attention Mechanisms, by Prof. Mohamed Abdel Rahman.

Common RNN-NLP Architectures

• Sequence labeling involves training a model to assign a label to each input word or token, useful for part-of-speech tagging.
• Sequence classification ignores the output for each token, taking one value from the end of the sequence, as seen in sentiment analysis.
• Language modeling (sentence completion) trains the model to predict the next word at each token step.
• Encoder-decoder models are used for machine translation.
• A significant problem with standard RNNs is their inability to support long dependencies or long sequences.

RNN Limitations

• High accuracy is hard to achieve in long sentences using RNNs.
• Training RNNs to use distant information is challenging, especially when relationships between words are separated by intermediate words.
• Example: Predicting "were" in "The flights the airline was canceling were full" is difficult due to intermediate words and context.
• The aim is to retain distant information while processing intermediate sequence parts correctly.

Overcoming RNN Limitations

• Conventional RNNs struggle because hidden layers must perform dual tasks: providing information for current decisions and updating information for future decisions.
• Backpropagation with longer RNNs leads to vanishing gradient problems.
• Long Short-Term Memory (LSTM) networks, introduced by Hochreiter and Schmidhuber, are a special RNN extension to address these issues.

LSTM Networks

• LSTMs manage context by dividing the problem into removing unnecessary information and adding information needed for later decisions.
• LSTMs add a context layer in addition to the RNN hidden layer to achieve this.
• LSTMs compute three filters plus one conventional hidden layer output.
• All four are trained with different (U, W) matrices due to differing tasks.
• The three filters share a design pattern: a feed-forward RNN with a sigmoid activation function, followed by pointwise multiplication.
• The sigmoid function outputs either 0 or 1.
• Pointwise multiplication acts as a binary mask.

LSTM Gates: Forget & Add

• Both gates use a weighted sum of ht-1 and the current input.
• Each gate is trained with different weights (U, W) to perform different functions.
• This is similar to an RNN but uses a sigmoid activation function (0 or 1).
• f_t and i_t represent filter masks.
• The first gate is the forget gate for short-term memory, deleting information from the context that is no longer needed.
• The forget gate mask multiplies element-wise with the previous context vector, retaining only the fittest values/fractions and deleting unnecessary zeros.
• The second gate is the add gate for long-term memory, retaining long dependency information.
• The add gate mask operates similarly to the forget gate but is multiplied element-wise by the conventional RNN hidden layer output.

RNN vs LSTM algorithmic differences

• LSTM uses two versions of h(t): one with sigmoid activation as a filter and one with tanh as the conventional RNN-h(t).
• LSTM employs two similar filters trained with different weights: a forget gate (f) and an add gate (i).
• The context vector comes from both gates, conventional-RNN h(t) is multiplied with a filter, summing both gates.
• The resultant new LSTM h(t) is a long-term plus short-term context masked with a filter.

LSTM Computation

• LSTM inputs include the current input x_t, the previous hidden state h_t-1, and the previous context c_t-1.
• Outputs are a new hidden state h_t and an updated context c_t.
• Inclusion of a context vector means the model uses current fittest information and long dependencies.
• Tanh adds non-linearity to the context vector.
• Multiplication with another filter strengthens h(t).

The Role of Forget and Add Gates

• Previous context usage in forget gate makes it function similar to short-term memory in conventional RNNs.
• The effect of each previous word is retained and emphasized through filtration, such as deleting less relevant values.
• The model will disregard irrelevant past information.
• A positive attribute is keeping h(t) values increasing, preventing the gradient from vanishing.
• Current context allows the add gate to merge new, relevant information into the cell state.
• The model includes new information about current words.
• Context vectors from add and forget gates are summed.
• Can be repeated to study long dependencies without a vanishing gradient.

Visual Applications of RNNs

• RNNs can be combined with CNNs for image applications.
• ConvLSTM is used for video sequences.
• Input, hidden, and cell states are higher-order tensors (images).
• Gates have CNN instead of fully connected (FC) layers.
• Images coupled with sentences can be input to RNNs as a language model.

Encoder-Decoder Model

• Sequence-to-sequence networks generate contextually appropriate output sequences.
• The model transforms an input sequence into an output sequence of a different length.
• Encoder-decoder models are used for summarization, question answering, dialogue, and particularly for machine translation.
• The encoder creates a contextualized representation (context) from the input sequence, which is then used by the decoder to generate a task-specific output sequence.

Conceptual Components of Encoder-Decoder Models

• Encoder converts the input sequence, x₁:n, into contextualized representations, h₁:m.
• Context vector, c, captures the essence of the input for the decoder.
• Decoder uses c to generate an arbitrary length sequence of hidden states, h₁:m, to get an output sequence, y₁:m.
• Encoders and decoders can be realized using any kind of sequence architecture.

RNN Encoder-Decoder Training

• The model uses paired strings, an input (X) and its corresponding output (y).
• Input words (X) to the encoder are treated to get a contextual output, C, representing the effect of the last word token such that C = h_n.
• Both the output dataset and the contextual output from the encoder feed into the decoder stage.

RNN Encoder-Decoder Inference

• This is when the decoder has only context from the encoder as its input.
• The estimated output y^t acts as a additional input for the time step Xt+1.
• The decoder tends to diverge, so teacher forcing is used.
• Teacher forcing uses correct labels from training as the input Xt+1.

Drawbacks of Encoder-Decoders

• The context vector h_n, which is the final hidden state, may not fully represent the entire source text's meaning.
• It acts as a bottleneck
• Decoder knows the context but essentially loses context of the last word
• More suitable for short sentences.
• Information at the beginning of a long sentence may not be well-seen.

Attention Mechanism

• It is a solution to the bottleneck problem in sequence processing.
• The decoder can retrieve data from all hidden states of the encoder; not just the last state.
• The attention layer provides importance to tokens in the input sequence.