Seq2Seq Models and Encoder-Decoder Architecture

Web and Text Analytics 2024-25, Week 11

• Course material covers Web and Text Analytics for the academic year 2024-2025, specifically week 11.
• Instructor: Evangelos Kalampokis, with website link: https://kalampokishub.io
• Information Systems Lab website: http://islab.uom.gr

RNN and Encoder-Decoder Architecture

• Recurrent Neural Networks (RNNs) have seen significant innovation, leading to complex architectures.
• Sequence-to-sequence problems, like machine translation, often involve unaligned input and output sequences of varying lengths.
• Encoder-decoder architecture is the standard for handling such data.
• This architecture consists of two primary components:
  • An encoder that processes variable-length input sequences into a fixed-length hidden state.
  • A decoder that uses the encoded input and preceding tokens in the output sequence to predict the next token in the target sequence using a conditional language model.

Machine Translation

• Machine translation serves as a specific example of encoder-decoder architecture.
• English sentences, such as "They are watching.", are encoded into a state, and then decoded to produce the French translation, "Ils regardent.".
• The encoder-decoder architecture is fundamental to various sequence-to-sequence models.

Sequence-to-Sequence (Seq2Seq) Models

• Seq2Seq models, a type of neural network, are particularly suited for tasks involving input and output sequences of varying lengths.
• Examples of such tasks include machine translation, question answering, chatbot creation, and text summarization.

Use Cases of Seq2Seq Models

• Machine Translation: Translating text between languages.
• Text Summarization: Generating concise summaries of longer texts.
• Speech Recognition: Converting spoken language into written text.
• Chatbots and Conversational AI: Creating human-like conversational agents.
• Image Captioning: Describing the content of images in natural language.
• Video Captioning: Creating descriptions of videos.
• Time Series Prediction: Forecasting future values in a sequence based on past observations.
• Code Generation: Generating code snippets or full programs from natural language descriptions.

Encoder-Decoder Architecture (Details)

• The most common architecture for building Seq2Seq models is encoder-decoder architecture.
• RNNs implement the encoder and decoder functions.
• The encoder RNN transforms variable-length input into a fixed-length hidden state.
• Attention mechanisms enable access to encoded inputs without full compression to a fixed length.
• The model incorporates a special "" token to signal the end of the sequence.

Encoder-Decoder Architecture (Initial Time Step)

• The initial time step of the RNN decoder uses a special beginning-of-sequence token "".
• The final hidden state of the encoder is used to initialize the decoder's hidden state, either only at the first step or at every step during decoding.

Encoder-Decoder Architecture (Training and Testing)

• During training, the decoder is conditioned on the preceding tokens in the ground truth.
• During testing, the decoder output is conditioned on previously predicted tokens.

Transformers

• The Transformer architecture was introduced in June 2017 and focused initially on translation tasks.
• Key models followed, including:
  • GPT (2018): First pre-trained Transformer model; used for fine-tuning on various NLP tasks (Natural Language Processing).
  • BERT (2018): Large pre-trained model specifically designed for sentence summarization tasks.
  • GPT-2 (2019): Improved and larger version of GPT.
  • DistilBERT (2019): Distilled version of BERT; faster and lighter.
  • BART/T5 (2019): Large pre-trained models employing the same architecture as the original Transformer model–the first such pre-trained models of this architecture.
  • GPT-3 (2020): Even larger version of GPT-2; exhibits zero-shot learning capabilities. (no need for fine-tuning).

Transformer Models (Categories)

• Broadly, Transformer models fall into three categories:
  • GPT-like (auto-regressive)
  • BERT-like (auto-encoding)
  • BART/T5-like (sequence-to-sequence)

General Architecture (Encoder-Decoder)

• The model consists primarily of an encoder and a decoder block.
• The encoder receives input and creates a representation (features) of the input.
• The decoder utilises the encoder's representation and other inputs to generate the target sequence.

Transformer Models (Independent Components)

• Each component (encoder-only, decoder-only, encoder-decoder) can be applied depending on the task.
• Encoder-only models: suited for tasks requiring input comprehension (e.g. sentence classification, named entity recognition).
• Decoder-only models: suited for generative tasks (e.g. text generation).
• Encoder-decoder (Seq2Seq): suited for tasks needing input in order to generate the output (e.g. translation, summarization).

Attention Layers

• A key feature of transformer models is attention layers.
• The attention mechanism selectively focuses on specific words within the input to inform the representation of each word. This consideration of context is important.

Attention Mechanism (Translation)

• Context is crucial in translation tasks.
• Models need to attend to related words to appropriately translate specific words.
• Applying attention to words that might be further away, but are contextually important, is vital in complex sentences and grammatical structures.

Attention Mechanism (General)

• Attention mechanisms are crucial for tasks in natural language.
• Word meaning is deeply influenced by surrounding context. Attention allows the model to give weighted attention to critical words in the context to achieve proper meaning.

Model Examples and Tasks

• Specific Transformer models excel at different tasks (listed below in a table structure)

  Model	Examples	Tasks
  Encoder	ALBERT, BERT, DistilBERT, ELECTRA, ROBERTa	Sentence classification, named entity recognition, question answering
  Decoder	CTRL, GPT, GPT-2, Transformer XL	Text generation
  Encoder-decoder	BART, T5, Marian, mBART	Summarization, translation, generative question answering

Architectures vs. Checkpoints

• Architecture: The structure and operations within a model(e.g. the layers and their connections)
• Checkpoints: The specific weights of the model at a certain point in training.
• Models: A broader term to refer to the overall configuration of the architecture.
• Understanding the difference between these terms will be crucial for ambiguity reduction when discussing models.

Encoder Models

• Encoder models utilize only the encoder portion of a Transformer, accessing all words in the initial sentence.
• They are typically described as "bi-directional," and the pretraining process commonly involves corrupting sentences to task the model to reconstruct them.
• Tasks these models excel at include sentence classification, named entity recognition and extractive question answering.

Decoder Models

• Decoder models use only the decoder portion of a Transformer architecture, limiting attention layers to words before the current word position in the sentence.
• They are often referred to as "auto-regressive", and their training typically focuses on predicting the next word.
• They are most suitable for tasks like text generation.

Sequence-to-Sequence Models

• Encoder-decoder models, also known as sequence-to-sequence models, combine both encoder and decoder parts in the Transformer.
• The encoder accesses all initial words, while the decoder only accesses those before the target word.
• Pretraining often involves replacing elements of the text with masked words, requiring the model to predict them.
• This model type is suitable for tasks centered on transforming input sequences into output sequences, such as summarisation, translation, or generative question answering.

Types of Attention Mechanisms

• Soft Attention: A continuous and differentiable manner of focus on input, providing varying weights to different parts of the input.
• Hard Attention: Makes a discrete choice to focus on a specific part of the input, like highlighting with a spotlight.
• Self-Attention: Allows every element in a sequence to attend to all other elements within the same sequence. Useful for understanding relationships between different parts of a sentence, for example.
• Multi-Head Attention: Multiple instances of self-attention are applied in parallel, providing a richer understanding.

How Does Attention Mechanism Work?

• Involves three main components: queries, keys, and values, which can represent words or parts of the input or output sentence.
• The query is related to the current word of the output sequence.
• The key is the representation of an input element that the model should attend to.
• The value is what the model should focus on if an associated key is deemed important.
• The model calculates attention scores by comparing the query and key, yielding an alignment score.
• A softmax function transforms the alignment scores into probabilities, determining the weight of each corresponding value.
• The model computes a weighted combination of values as a context vector.

Attention Scores

• The model calculates an attention score by comparing the query with each key to obtain an alignment score.
• This alignment score determines how much attention to pay to the corresponding value using a dot product as a means of measuring similarity between vectors.
• The alignment scores are processed through a softmax function to yield probabilities (between 0 and 1) that sum up to 1. These probabilities then determine the weight of each value in the final output sequence.
• Using the softmax probabilities as weights, the model produces a context vector by combining the weighted values. This represents the output of the attention mechanism.
• The resulting context vector focuses on contextually important parts of the input, thus improving accuracy and understanding.

The World Wide Web (WWW)

• The WWW is a system, not synonymous with the Internet.
• Web documents are identified by Uniform Resource Locators (URLs) and interlinked by hyperlinks.
• Resources are transferred via the Hypertext Transfer Protocol (HTTP).
• Web browsers access and web servers publish these resources on the Internet.
• The WWW is an application of the larger communication infrastructure of the Internet.

W3C

• The W3C is an international organization that develops Web standards.
• The standards taught in this course are developed by the W3C.

The Web as Originally Envisioned

• The initial conceptualisation of the web is outlined in a document from March 1989 by Tim Berners-Lee.
• This document details how different data and information are linked together in web documents.

The Web of Documents

• The web is composed of two crucial elements: web clients (browsers) and web servers.
• The web client's request is processed by the web server, fulfilling the client's request.
• The web architecture includes vital components like URLs for addressing documents, HTTP for communication, and HTML for content representation.

The Web of Documents (Detailed)

• Web documents are based on simple standards:
  • Uniform Resource Identifiers (URIs) for unique identification.
  • Hypertext Transfer Protocol (HTTP) for universal access.
  • Hypertext Markup Language (HTML) for formatting.
• Hyperlinks connect documents on different servers, facilitating navigation.

Linked Data

• Introduced in 2006 by Tim Berners-Lee, linked data aims to connect related data resources on the web.
• Principles for Linked Data:
  • Unique identification of resources using URIs
    • Using HTTP URIs for looking up resource names
    • Provide useful information based on URI standard
    • Provide links to other related resources
• The Semantic Web principles, and specifically linked data, build on previous implementations of the internet.

Knowledge Graphs

• Knowledge graphs (often referred to as Linked Data): are a collection of interconnected descriptions describing various entities, objects, events and concepts.
• Knowledge graphs utilise data context, link information based on semantics, and integrate information for effective interpretation and analysis.
• This allows for data to be interpreted and correlated for purposes such as data integration, unification, analytics and sharing.

Knowledge Graph (Structure)

• A knowledge graph can be represented as a directed labelled graph with:
  • Nodes (entities).
  • Edges (links, relationships between entities).
    • Edge direction defines the relationship type.
  • Labels (semantic concepts) that describe the meaning of the relationship.

Knowledge Graph (Further details)

• The knowledge graph captures interlinked descriptions of entities (objects, events, concepts).
• Semantic metadata enables efficient processing and unambiguous interpretation.
• Entities in a knowledge graph are interconnected, forming a descriptive network with each entity contextualizing the related entities in the entire network.

Examples of Knowledge Graphs

• Google Knowledge Graph: Became widely known via its 2012 announcement, containing a significant amount of linked data but with limited public usage outside of Google's applications.
• DBpedia: Uses Wikipedia's infobox structure to create a large dataset (4.58 things); covers numerous encyclopedic entities like people, places, films and much more.

Graph

• Graphs represent relations between various entities, known as "nodes" or "vertices".
• Relations between entities are denoted by "edges".
• Graph attributes provide information including edge identity, edge weight, node identity, and number of neighbours.

Graph Example: Social Networks

• Social Networks demonstrate relationships between individuals and organisations based on their interactions (represented by edges).

Graph Example: Images as Graphs

• Images can be represented as graphs: Pixels are nodes; Adjacent pixels are interconnected using edges.

Graph Example: Natural Language Processing

• Graphs are used for NLP for correlated information extraction to generate a knowledge graph.
• Entity and relationship extraction from text are tasks benefitting from graph representations.

Graph Example: Computer Vision

• Image understanding systems use graph representations to capture relationships between detected objects.
• Identifying relationships such as a man holding a bucket and a horse feeding from it are examples of object detection in computer vision.

Graphs as Input to Machine Learning

• Graph Neural Networks (GNNs), a subfield of machine learning, deal with graph structures and data.
• GNNs are becoming increasingly important in research.

Open Graph Benchmark (OGB)

• OGB contains datasets of realistic graphs for machine learning.
• Task categories exist for graph learning tasks including, predicting attributes of nodes, links and also entire graphs.

Node Property Prediction

• Node property prediction involves predicting properties of individual nodes, such as in an Amazon product network where edges link products purchased together, using bag-of-words extracted from the product descriptions to derive the features and use these characteristics to discover the appropriate category the corresponding product falls in.

Link Property Prediction

• Link prediction tasks utilize edges to identify relationships between entities.
• The DrugBank database provides an example where edges represent interactions between drugs, allowing for prediction based on known interactions.
• Extracting and predicting relations based on known triples of relations is one approach for predicting relationships given training edges.

Graph Property Prediction

• Graph property prediction is used for determining properties related to the entire graph or subgraph.
• In moleculenet, nodes represent atoms in a molecule and edges indicate chemical bonds.
• Input features include atomic number, chirality, and other additional features, to determine molecular properties.