Attention Mechanism in Transformers

Attention Mechanism in Transformers

• Goal of Transformer: Predict the next word in a piece of text.
• Input: Text broken into tokens (words or parts of words).
• First step: Assign each token a high-dimensional embedding vector.
• Embedding Space: Directions represent semantic meaning.
• Transformer's Role: Adjust embeddings to capture contextual meaning.
• Attention Mechanism: Understands and manipulates word relationships in a sentence.

Key Components of Attention

• Query Matrix (WQ): Transforms embeddings to query vectors (smaller, for searching).
• Key Matrix (WK): Transforms embeddings to key vectors (representing potential answers).
• Value Matrix (WV): Transforms embeddings to value vectors (representing meaning).

Attention Pattern

• Dot Product: Measures key-query relevance, creating a grid of alignment weights.
• Softmax: Normalizes the relevance grid, creating an attention pattern (weights for word relevance).
• Masking: Prevents later words from influencing earlier ones, avoiding future knowledge.
• Size: The attention pattern size equals the context size squared.

Updating Embeddings

• Value Map: Calculates information to add to embeddings based on relevance.
• Value Down Matrix: Maps embedding vectors to a smaller space.
• Value Up Matrix: Maps vectors from smaller space back to embedding space, providing updates.

Multi-Headed Attention

• Multiple Heads: Multiple attention heads operate in parallel, extracting different aspects of meaning.
• Computational Efficiency: Value map is factored to decrease parameters and increase efficiency.
• Self-Attention: Standard attention for internal word relationships within a text.
• Cross-Attention: A variation where different parts of the text interact; less frequently discussed.

Cross-Attention

• Models process two distinct differing data types (e.g., different languages).
• Key and query matrices operate on separate data sets.
• Example: Translation – keys from one language, queries from another.
• No masking; no notion of later tokens affecting earlier tokens.

Self-Attention

• Context significantly impacts a word's meaning.
• Examples: Adjectives modifying nouns, or grammatical/non-grammatical associations.
• Key and query matrix parameters capture various attention patterns based on the context type.
• Value map parameters determine embedding updates.
• Complex and intricate behavior of maps in practice.

Multi-Headed Attention

• Each head has its own key, query, and value matrices.
• GPT-3 uses 96 attention heads per block.
• Each head generates a unique attention pattern and value vectors.
• Value vectors are combined with attention patterns as weights.
• Each head proposes embedding changes at each context position.
• Proposed changes are summed to refine the overall embedding.

Value Matrix Nuances

• Value map factors into value down and value up matrices.
• Practical implementation differs from the concept.
• Value up matrices combine into a large output matrix for the attention block.
• "Value matrix" often refers to the value down projection in practice.

Transformer Architecture

• Data flows through multiple attention blocks and multi-layer perceptrons (MLPs).
• Embeddings gain contextual information from surrounding embeddings.
• Higher-level meanings like sentiment, tone, and underlying truths are encoded.

GPT-3 Parameters

• GPT-3 has 96 layers, leading to 58 billion parameters for attention heads.
• Represents about one-third of GPT-3's parameters.

Advantages of Attention

• Highly parallelizable, enabling effective GPU processing.
• Increased scale improves model quality.

Resources

• Additional resources available in video descriptions (Andrej Karpathy, Chris Ola).
• Videos on attention history and large language models by Vivek and Britt Cruz.