Seq2Seq Models and Encoder-Decoder Architecture

Podcast

Play an AI-generated podcast conversation about this lesson

Questions and Answers

Which of the following scenarios is best suited for a Seq2Seq model?

Predicting the sentiment of a single sentence.

Detecting anomalies in a single numerical dataset.

Classifying images into predefined categories.

Generating a summary of a lengthy document. (correct)

In the context of the Encoder-Decoder architecture, what is the primary function of the encoder?

To generate the final output sequence.

To provide the special beginning-of-sequence token.

To transform input sequences into a fixed-shape hidden state. (correct)

To predict the next token in the sequence.

What is the purpose of the special `<END>` token in a sequence-to-sequence model?

To mark the beginning of the input sequence.

To represent a missing or unknown word.

To indicate the end of the decoding process. (correct)

To initialize the decoder's hidden state.

When does the decoder RNN usually begin its process?

After the entire input sequence is processed by encoder. (A) Signup and view all the answers

What is the function of the special `<BEGIN>` token in the decoder's input?

It marks the start of the decoding sequence. (A) Signup and view all the answers

Besides the encoded input, what else may be fed into the decoder at each time step?

The final hidden state of the encoder. (D) Signup and view all the answers

In which of the following applications would an encoder-decoder architecture be the least suitable approach?

Image classification. (C) Signup and view all the answers

How does the attention mechanism enhance the Encoder-Decoder architecture?

It allows access to encoded inputs without compressing all into fixed length. (D) Signup and view all the answers

What is the primary role of the World Wide Web Consortium (W3C)?

To develop and maintain web standards through collaboration. (C) Signup and view all the answers

According to the content, what is considered an application of the internet's infrastructure?

The World Wide Web. (B) Signup and view all the answers

What are the three core components of the web architecture that enable communication between client and server, according to the provided content?

URIs, HTTP, HTML. (C) Signup and view all the answers

What is the primary function of Uniform Resource Identifiers (URIs) in the web architecture?

To uniquely identify and locate resources on the web. (D) Signup and view all the answers

What does the content identify as the universal access mechanism for the web?

Hypertext Transfer Protocol (HTTP). (C) Signup and view all the answers

Which of these represents the content format used for web documents, based on the provided material?

Hypertext Markup Language (HTML). (D) Signup and view all the answers

Besides the listed standards, what foundational concept is identified as contributing to the structure of the Web?

Hyperlinks between documents on different servers. (C) Signup and view all the answers

Who is credited with the initial vision of the Web as described in the content?

Tim Berners-Lee. (C) Signup and view all the answers

What is the primary distinction between 'architecture' and 'checkpoint' in the context of machine learning models?

Architecture denotes the framework of layers and operations within a model, while a checkpoint signifies the model's learned weights. (D) Signup and view all the answers

Which of the following best describes the attention mechanism in encoder models?

Bidirectional, accessing all words in the input sentence. (B) Signup and view all the answers

What is a common pre-training task used for encoder models?

Reconstructing a corrupted sentence, such as masking some words (D) Signup and view all the answers

For tasks such as named entity recognition, which type of model would be most suitable?

Encoder models (B) Signup and view all the answers

How does the attention mechanism in decoder models differ from that of encoder models?

Decoder models are limited to accessing only preceding words, while encoder models access all words. (C) Signup and view all the answers

What is a common pre-training objective for decoder models?

Predicting the next word in a sentence. (C) Signup and view all the answers

Which of the following types of task are decoder models best suited for?

Text Generation (C) Signup and view all the answers

What is another term for encoder-decoder models?

Sequence-to-sequence models (D) Signup and view all the answers

What is the primary function of the links between bubbles in a linked data representation?

To indicate the relationships between data points. (C) Signup and view all the answers

Which term has recently gained wider acceptance as a descriptor for Linked Data?

Knowledge Graphs (D) Signup and view all the answers

What does a knowledge graph primarily represent?

A network of interlinked entity descriptions. (C) Signup and view all the answers

How do knowledge graphs provide context for data?

Via linking and semantic metadata. (C) Signup and view all the answers

In a knowledge graph, what does an edge between two nodes represent?

The relationship of interest between the two nodes. (D) Signup and view all the answers

Which of the following best describes the nature of labels in a knowledge graph?

Definitions of the relationships between nodes. (D) Signup and view all the answers

What is a key characteristic of entity descriptions in a knowledge graph?

They form a network by referencing each other. (A) Signup and view all the answers

According to the information provided, what is one limitation of the Google Knowledge Graph?

There are limited ways to use it outside Google’s own projects. (C) Signup and view all the answers

In graph machine learning, what is the primary focus of node property prediction?

Determining attributes specific to individual nodes (A) Signup and view all the answers

In the context of the Amazon product co-purchasing network, what do the edges signify?

Products that are purchased together. (D) Signup and view all the answers

What type of features are used in node property prediction for the Amazon product co-purchasing network?

Bag-of-words extracted from product descriptions. (C) Signup and view all the answers

What is the core objective of link property prediction?

To determine new or hidden relationships that were previously not declared. (B) Signup and view all the answers

In the DrugBank database, what is the meaning of an edge between two drug nodes?

They have a synergistic interaction when taken together. (D) Signup and view all the answers

How is the concept of link property prediction applied in Wikidata?

Forecasting new relations between entities i.e., (entity, relation, entity). (D) Signup and view all the answers

In Moleculenet, what do nodes and edges represent respectively?

Atoms and chemical bonds (D) Signup and view all the answers

What kind of property is being predicted in the MoluculeNet Graph Property Prediction task.

A binary outcome, such as whether the molecule will inhibit the HIV virus. (C) Signup and view all the answers

Which of the following best describes the core idea behind the Semantic Web?

To enable computers and people to work more cooperatively through well-defined information meaning. (C) Signup and view all the answers

According to Tim Berners-Lee, what is the primary function of HTTP URIs in Linked Data?

To allow people to lookup the names of things and access associated information. (C) Signup and view all the answers

What was the primary reason for the introduction of the Linked Data principles in 2006?

To provide clear guidelines for connecting and publishing data using web infrastructure. (C) Signup and view all the answers

In the context of Linked Data principles, what role do URIs play in defining 'things'?

They act as unique names for identifying resources including real-world objects and abstract concepts. (B) Signup and view all the answers

What does the 'Semantic Web Stack' generally represent?

A range of technologies and concepts that aim for greater computer-human cooperation. (D) Signup and view all the answers

The vision of the Semantic Web was to fulfill the idea of 'linked information'. When was this vision initially proposed?

In 1989 with the initial idea of interconnected data. (C) Signup and view all the answers

What is a critical factor that hindered the wide adoption of Semantic Web technologies?

The inherent complexity of the technologies. (C) Signup and view all the answers

According to the Linked Data principles, what should be provided when a URI is looked up?

Useful information using the standard formats. (D) Signup and view all the answers

Study Notes

Web and Text Analytics 2024-25, Week 11

Course material covers Web and Text Analytics for the academic year 2024-2025, specifically week 11.
Instructor: Evangelos Kalampokis, with website link: https://kalampokishub.io
Information Systems Lab website: http://islab.uom.gr

RNN and Encoder-Decoder Architecture

Recurrent Neural Networks (RNNs) have seen significant innovation, leading to complex architectures.
Sequence-to-sequence problems, like machine translation, often involve unaligned input and output sequences of varying lengths.
Encoder-decoder architecture is the standard for handling such data.
This architecture consists of two primary components:
- An encoder that processes variable-length input sequences into a fixed-length hidden state.
- A decoder that uses the encoded input and preceding tokens in the output sequence to predict the next token in the target sequence using a conditional language model.

Machine Translation

Machine translation serves as a specific example of encoder-decoder architecture.
English sentences, such as "They are watching.", are encoded into a state, and then decoded to produce the French translation, "Ils regardent.".
The encoder-decoder architecture is fundamental to various sequence-to-sequence models.

Sequence-to-Sequence (Seq2Seq) Models

Seq2Seq models, a type of neural network, are particularly suited for tasks involving input and output sequences of varying lengths.
Examples of such tasks include machine translation, question answering, chatbot creation, and text summarization.

Use Cases of Seq2Seq Models

Machine Translation: Translating text between languages.
Text Summarization: Generating concise summaries of longer texts.
Speech Recognition: Converting spoken language into written text.
Chatbots and Conversational AI: Creating human-like conversational agents.
Image Captioning: Describing the content of images in natural language.
Video Captioning: Creating descriptions of videos.
Time Series Prediction: Forecasting future values in a sequence based on past observations.
Code Generation: Generating code snippets or full programs from natural language descriptions.

Encoder-Decoder Architecture (Details)

The most common architecture for building Seq2Seq models is encoder-decoder architecture.
RNNs implement the encoder and decoder functions.
The encoder RNN transforms variable-length input into a fixed-length hidden state.
Attention mechanisms enable access to encoded inputs without full compression to a fixed length.
The model incorporates a special "" token to signal the end of the sequence.

Encoder-Decoder Architecture (Initial Time Step)

The initial time step of the RNN decoder uses a special beginning-of-sequence token "".
The final hidden state of the encoder is used to initialize the decoder's hidden state, either only at the first step or at every step during decoding.

Encoder-Decoder Architecture (Training and Testing)

During training, the decoder is conditioned on the preceding tokens in the ground truth.
During testing, the decoder output is conditioned on previously predicted tokens.

Transformers

The Transformer architecture was introduced in June 2017 and focused initially on translation tasks.
Key models followed, including:
- GPT (2018): First pre-trained Transformer model; used for fine-tuning on various NLP tasks (Natural Language Processing).
- BERT (2018): Large pre-trained model specifically designed for sentence summarization tasks.
- GPT-2 (2019): Improved and larger version of GPT.
- DistilBERT (2019): Distilled version of BERT; faster and lighter.
- BART/T5 (2019): Large pre-trained models employing the same architecture as the original Transformer model–the first such pre-trained models of this architecture.
- GPT-3 (2020): Even larger version of GPT-2; exhibits zero-shot learning capabilities. (no need for fine-tuning).

Transformer Models (Categories)

Broadly, Transformer models fall into three categories:
- GPT-like (auto-regressive)
- BERT-like (auto-encoding)
- BART/T5-like (sequence-to-sequence)

General Architecture (Encoder-Decoder)

The model consists primarily of an encoder and a decoder block.
The encoder receives input and creates a representation (features) of the input.
The decoder utilises the encoder's representation and other inputs to generate the target sequence.

Transformer Models (Independent Components)

Each component (encoder-only, decoder-only, encoder-decoder) can be applied depending on the task.
Encoder-only models: suited for tasks requiring input comprehension (e.g. sentence classification, named entity recognition).
Decoder-only models: suited for generative tasks (e.g. text generation).
Encoder-decoder (Seq2Seq): suited for tasks needing input in order to generate the output (e.g. translation, summarization).

Attention Layers

A key feature of transformer models is attention layers.
The attention mechanism selectively focuses on specific words within the input to inform the representation of each word. This consideration of context is important.

Attention Mechanism (Translation)

Context is crucial in translation tasks.
Models need to attend to related words to appropriately translate specific words.
Applying attention to words that might be further away, but are contextually important, is vital in complex sentences and grammatical structures.

Attention Mechanism (General)

Attention mechanisms are crucial for tasks in natural language.
Word meaning is deeply influenced by surrounding context. Attention allows the model to give weighted attention to critical words in the context to achieve proper meaning.

Model Examples and Tasks

Specific Transformer models excel at different tasks (listed below in a table structure)

Model	Examples	Tasks
Encoder	ALBERT, BERT, DistilBERT, ELECTRA, ROBERTa	Sentence classification, named entity recognition, question answering
Decoder	CTRL, GPT, GPT-2, Transformer XL	Text generation
Encoder-decoder	BART, T5, Marian, mBART	Summarization, translation, generative question answering

Architectures vs. Checkpoints

Architecture: The structure and operations within a model(e.g. the layers and their connections)
Checkpoints: The specific weights of the model at a certain point in training.
Models: A broader term to refer to the overall configuration of the architecture.
Understanding the difference between these terms will be crucial for ambiguity reduction when discussing models.

Encoder Models

Encoder models utilize only the encoder portion of a Transformer, accessing all words in the initial sentence.
They are typically described as "bi-directional," and the pretraining process commonly involves corrupting sentences to task the model to reconstruct them.
Tasks these models excel at include sentence classification, named entity recognition and extractive question answering.

Decoder Models

Decoder models use only the decoder portion of a Transformer architecture, limiting attention layers to words before the current word position in the sentence.
They are often referred to as "auto-regressive", and their training typically focuses on predicting the next word.
They are most suitable for tasks like text generation.

Sequence-to-Sequence Models

Encoder-decoder models, also known as sequence-to-sequence models, combine both encoder and decoder parts in the Transformer.
The encoder accesses all initial words, while the decoder only accesses those before the target word.
Pretraining often involves replacing elements of the text with masked words, requiring the model to predict them.
This model type is suitable for tasks centered on transforming input sequences into output sequences, such as summarisation, translation, or generative question answering.

Types of Attention Mechanisms

Soft Attention: A continuous and differentiable manner of focus on input, providing varying weights to different parts of the input.
Hard Attention: Makes a discrete choice to focus on a specific part of the input, like highlighting with a spotlight.
Self-Attention: Allows every element in a sequence to attend to all other elements within the same sequence. Useful for understanding relationships between different parts of a sentence, for example.
Multi-Head Attention: Multiple instances of self-attention are applied in parallel, providing a richer understanding.

How Does Attention Mechanism Work?

Involves three main components: queries, keys, and values, which can represent words or parts of the input or output sentence.
The query is related to the current word of the output sequence.
The key is the representation of an input element that the model should attend to.
The value is what the model should focus on if an associated key is deemed important.
The model calculates attention scores by comparing the query and key, yielding an alignment score.
A softmax function transforms the alignment scores into probabilities, determining the weight of each corresponding value.
The model computes a weighted combination of values as a context vector.

Attention Scores

The model calculates an attention score by comparing the query with each key to obtain an alignment score.
This alignment score determines how much attention to pay to the corresponding value using a dot product as a means of measuring similarity between vectors.
The alignment scores are processed through a softmax function to yield probabilities (between 0 and 1) that sum up to 1. These probabilities then determine the weight of each value in the final output sequence.
Using the softmax probabilities as weights, the model produces a context vector by combining the weighted values. This represents the output of the attention mechanism.
The resulting context vector focuses on contextually important parts of the input, thus improving accuracy and understanding.

The World Wide Web (WWW)

The WWW is a system, not synonymous with the Internet.
Web documents are identified by Uniform Resource Locators (URLs) and interlinked by hyperlinks.
Resources are transferred via the Hypertext Transfer Protocol (HTTP).
Web browsers access and web servers publish these resources on the Internet.
The WWW is an application of the larger communication infrastructure of the Internet.

W3C

The W3C is an international organization that develops Web standards.
The standards taught in this course are developed by the W3C.

The Web as Originally Envisioned

The initial conceptualisation of the web is outlined in a document from March 1989 by Tim Berners-Lee.
This document details how different data and information are linked together in web documents.

The Web of Documents

The web is composed of two crucial elements: web clients (browsers) and web servers.
The web client's request is processed by the web server, fulfilling the client's request.
The web architecture includes vital components like URLs for addressing documents, HTTP for communication, and HTML for content representation.

The Web of Documents (Detailed)

Web documents are based on simple standards:
- Uniform Resource Identifiers (URIs) for unique identification.
- Hypertext Transfer Protocol (HTTP) for universal access.
- Hypertext Markup Language (HTML) for formatting.
Hyperlinks connect documents on different servers, facilitating navigation.

Linked Data

Introduced in 2006 by Tim Berners-Lee, linked data aims to connect related data resources on the web.
Principles for Linked Data:
- Unique identification of resources using URIs
  - Using HTTP URIs for looking up resource names
  - Provide useful information based on URI standard
  - Provide links to other related resources
The Semantic Web principles, and specifically linked data, build on previous implementations of the internet.

Knowledge Graphs

Knowledge graphs (often referred to as Linked Data): are a collection of interconnected descriptions describing various entities, objects, events and concepts.
Knowledge graphs utilise data context, link information based on semantics, and integrate information for effective interpretation and analysis.
This allows for data to be interpreted and correlated for purposes such as data integration, unification, analytics and sharing.

Knowledge Graph (Structure)

A knowledge graph can be represented as a directed labelled graph with:
- Nodes (entities).
- Edges (links, relationships between entities).
  - Edge direction defines the relationship type.
- Labels (semantic concepts) that describe the meaning of the relationship.

Knowledge Graph (Further details)

The knowledge graph captures interlinked descriptions of entities (objects, events, concepts).
Semantic metadata enables efficient processing and unambiguous interpretation.
Entities in a knowledge graph are interconnected, forming a descriptive network with each entity contextualizing the related entities in the entire network.

Examples of Knowledge Graphs

Google Knowledge Graph: Became widely known via its 2012 announcement, containing a significant amount of linked data but with limited public usage outside of Google's applications.
DBpedia: Uses Wikipedia's infobox structure to create a large dataset (4.58 things); covers numerous encyclopedic entities like people, places, films and much more.

Graph

Graphs represent relations between various entities, known as "nodes" or "vertices".
Relations between entities are denoted by "edges".
Graph attributes provide information including edge identity, edge weight, node identity, and number of neighbours.

Social Networks demonstrate relationships between individuals and organisations based on their interactions (represented by edges).

Graph Example: Images as Graphs

Images can be represented as graphs: Pixels are nodes; Adjacent pixels are interconnected using edges.

Graph Example: Natural Language Processing

Graphs are used for NLP for correlated information extraction to generate a knowledge graph.
Entity and relationship extraction from text are tasks benefitting from graph representations.

Graph Example: Computer Vision

Image understanding systems use graph representations to capture relationships between detected objects.
Identifying relationships such as a man holding a bucket and a horse feeding from it are examples of object detection in computer vision.

Graphs as Input to Machine Learning

Graph Neural Networks (GNNs), a subfield of machine learning, deal with graph structures and data.
GNNs are becoming increasingly important in research.

Open Graph Benchmark (OGB)

OGB contains datasets of realistic graphs for machine learning.
Task categories exist for graph learning tasks including, predicting attributes of nodes, links and also entire graphs.

Node Property Prediction

Node property prediction involves predicting properties of individual nodes, such as in an Amazon product network where edges link products purchased together, using bag-of-words extracted from the product descriptions to derive the features and use these characteristics to discover the appropriate category the corresponding product falls in.

Link Property Prediction

Link prediction tasks utilize edges to identify relationships between entities.
The DrugBank database provides an example where edges represent interactions between drugs, allowing for prediction based on known interactions.
Extracting and predicting relations based on known triples of relations is one approach for predicting relationships given training edges.

Graph Property Prediction

Graph property prediction is used for determining properties related to the entire graph or subgraph.
In moleculenet, nodes represent atoms in a molecule and edges indicate chemical bonds.
Input features include atomic number, chirality, and other additional features, to determine molecular properties.

Studying That Suits You

Use AI to generate personalized quizzes and flashcards to suit your learning preferences.

Description

This quiz explores the intricacies of Seq2Seq models and the Encoder-Decoder architecture. It addresses key concepts such as the roles of the encoder and decoder, the importance of special tokens, and the enhancement provided by the attention mechanism. Test your understanding of the applications and limitations of these models in various contexts.