Transformer Architecture Overview

Transformer Architecture

• Transformers are a core technology in large language models and advanced AI systems.
• Introduced in the 2017 paper "Attention is All You Need."
• The primary function is predicting the next word in a text sequence.
• Text is broken down into tokens, often words or parts of words.

Embeddings

• Each token is represented by a high-dimensional vector called an embedding.
• Embeddings capture the semantic meaning of a word.
• Different directions in the embedding space correspond to distinct semantic aspects.
• Transformers adjust embeddings to reflect contextual meaning beyond individual words.

Attention Mechanism

• The attention mechanism is foundational to transformers.
• It allows the model to understand word relationships in context, refining word meaning.
• Attention enables nuanced understanding of context (e.g., the word "mole" in different contexts).

Attention Head

• A single attention head focuses on a particular relationship between words.
• This focuses on updating word embeddings based on the surrounding context.
• An example is how an attention head might analyze how adjectives modify nouns.

Query, Key, and Value Matrices

• Attention heads employ three matrices: query, key, and value.
• The query matrix maps embeddings to query vectors, representing questions about the context.
• The key matrix maps embeddings to key vectors, potentially answering the queries.
• The value matrix maps embeddings to value vectors, providing updates to embeddings.

Attention Pattern

• The attention pattern reveals how each word interacts with others.
• Calculated as a weighted sum of key vectors, weighted by their similarity to the query vector.
• Softmax normalizes these weights between 0 and 1.
• This pattern is a square matrix, correlating to the context size.

Masking

• Masking prevents future words from influencing earlier ones.
• Weights for influencing earlier words (future words) are set to negative infinity.
• After softmax is applied, these weights become zero, while maintaining overall column normalization.

Updating Embeddings

• The attention pattern updates word embeddings.
• Value vectors are multiplied by corresponding attention pattern weights.
• These weighted sums are added to the original embeddings, modifying their meaning.

Multi-Headed Attention

• Multiple attention heads run concurrently focusing on distinct context aspects.
• Outputs of multiple heads are combined to create a single refined representation.

Parameter Efficiency

• The value matrix is decomposed into smaller matrices, reducing parameters.
• The "value down" matrix maps embeddings to a smaller space.
• The "value up" matrix maps back to the original embedding space.

Self-Attention vs. Cross-Attention

• Self-attention analyzes relationships between words within the same sequence.
• Cross-attention examines relationships between words in different sequences.

Attention Mechanism

• Cross-Attention: Used for processing different data types (e.g., text in different languages, correlating audio and transcription).
• Key and Query Maps: Operating on disparate datasets in cross-attention, defining connections between elements/words.
• Self-Attention: Key and query maps on a single dataset, enabling models to understand input relationships and dependencies.
• Contextual Updating: The attention mechanism transforms word embeddings based on context, refining their meaning.
• Example: The presence of "they crashed the" before "car" significantly alters the embedding of "car," suggesting a specific scenario.

Multi-Headed Attention

• Multiple Attention Heads: GPT-3 uses 96 heads per block, each with unique key, query, value matrices.
• Parallel Operations: Each head independently analyzes the input, enabling 96 distinct attention patterns and value vectors.
• Combined Output: The outputs from each head are combined to refine the embedding for each position in the context.

Parameters and Design

• Parameter Estimate: A single multi-headed attention block in GPT-3 has about 600 million parameters.
• Value Map: In practice, the value map is represented as a single output matrix, simplifying the process.
• Large-Scale Training: Transformers use many attention blocks and other operations, resulting in a cascade of contextual refinements, boosting comprehension of complex relationships and concepts.

Attention and GPT-3

• GPT-3 Parameters: Approximately 58 billion parameters are dedicated to attention heads in GPT-3.
• Parallelism: The attention mechanism's highly parallelizable nature is well-suited to efficient GPU computation, contributing to its success.
• Continuous Learning: Transformers' layered design enhances understanding via multiple rounds of contextual refinement, capturing complex language semantics.