Untitled Quiz

Transformer Architecture

• The Transformer excels at handling sequential text data, such as translating English to Spanish.
• It consists of an Encoder stack and a Decoder stack, each with corresponding Embedding layers.
• The Encoder stack and Decoder stack each have their corresponding Embedding layers for their respective inputs.
• An Output layer generates the final output.
• All Encoders and Decoders in the stack are identical to one another.
• The Encoder consists of a Self-attention layer and a Feed-forward layer.
• The Decoder consists of a Self-attention layer, a Feed-forward layer, and an Encoder-Decoder attention layer.
• Each Encoder and Decoder has its own set of weights.
• The Encoder is a reusable module, a defining component of all Transformer architectures.
• It also has Residual skip connections around both layers along with LayerNorm layers.
• Variations of the Transformer architecture exist, some with no Decoder and rely solely on the Encoder.

Attention Mechanism

• The Transformer's groundbreaking performance is due to its use of Attention.
• Attention enables the model to focus on closely related words in the input when processing a word.
• For example, "ball" is related to "blue" and "holding," but "blue" is not related to "boy."
• The Transformer uses self-attention by relating every word in the sequence to every other word.
• This allows the model to understand the context of words, even when they are separated by other words.
• This is particularly useful for understanding the meaning of pronouns, which can refer to different entities in the sentence.

Attention Layers

• Self-attention in the Encoder: The input sequence pays attention to itself.
• Self-attention in the Decoder: The target sequence pays attention to itself.
• Encoder-Decoder-attention in the Decoder: The target sequence pays attention to the input sequence.
• The Attention layer takes three inputs: Query, Key, and Value.

Multi-Head Attention

• The Transformer uses multiple Attention heads, running in parallel, enhancing discrimination.
• Query, Key, and Value are passed through separate Linear layers, each with its own weights, resulting in Q, K, and V.
• These are combined using the Attention formula to produce the Attention Score.
• Q, K, and V values carry encoded representations of words in the sequence.
• The Attention calculations combine each word with every other word in the sequence, encoding a score for each word.

Attention Masks

• The Attention module implements a masking step while computing the Attention Score.
• Masking serves two purposes:
  • To zero attention outputs where there is padding in input sentences, ensuring that padding doesn’t contribute to self-attention in the Encoder Self-attention and Encoder-Decoder-attention.
  • To prevent the Decoder from attending to future positions in the sequence, ensuring that the output does not contain information about the future.

Encoder

• The first Encoder receives its input from the Embedding and Position Encoding.
• Subsequent Encoders receive input from the previous Encoder.
• The Encoder passes its input into a Multi-head Self-attention layer.
• The Self-attention output is passed into a Feed-forward layer.
• The output of the last Encoder is fed into each Decoder in the Decoder Stack.

Decoder

• The Decoder's structure is similar to the Encoder, with a couple of differences.
• The first Decoder receives its input from the Output Embedding and Position Encoding.
• Subsequent Decoders receive input from the previous Decoder.
• The Decoder passes its input into a Multi-head Self-attention layer.
• This Self-attention layer only attends to earlier positions in the sequence, preventing it from seeing future information.
• The Decoder has a second Multi-head attention layer, called the Encoder-Decoder attention layer.
• The Encoder-Decoder attention layer combines two sources of inputs:
  • The Self-attention layer below.
  • The output of the Encoder stack.
• The Self-attention output is passed into a Feed-forward layer.