Language-Based Learning in Robotics 2024

Language-Based Learning: A Short Overview of Contemporary Language Use in Robotics

• This presentation covers language-based learning, specifically its use in robotics.
• The speaker, Dr. Alex Mitrevski, delivered this presentation in the winter semester of 2024/25.

Structure

• (Large) Language models
• Robot learning and language

(Large) Language Models

• Language models are computational models for language processing, understanding, and generation.
• Natural language tasks were previously performed using classical machine learning, such as Naive Bayes for text classification.
• Large language models (LLMs) are neural networks trained on massive datasets, featuring a large number of parameters. Example: GPT-3 has 175 billion parameters.
• LLMs are computational models that enable language processing, understanding, and sometimes generation.

Language Models

• Language models enable language processing, understanding, and sometimes generation.
• Previously, natural language tasks like text classification relied on classical machine learning.

Tokenization

• When processing text, a variety of preprocessing steps are applied before language processing.
• This includes removing stop words (e.g., "a," "the") and punctuation, as they are often irrelevant.
• Tokenization is the process of converting text into constituent entities (typically words), a fundamental step in pre-processing.
• Common tokenization strategies involve converting text into individual words, with subsequent processing performed at the word level.
• Hierarchical token representations are also possible, initiating tokenisation at the character or sub-word level.

Word Embeddings

• Numerical representations of tokens are necessary for computations using models like neural networks.
• Bag-of-words representation and term frequency-inverse document frequency (TF-IDF) are examples of classical methods.
• Word embeddings numerically represent tokens, typically using a fixed-size vector space (k) learned by a neural network.
• Embeddings in models are frequently represented as one-hot encoded vectors.
• Examples of common word embeddings include word2vec, BERT, and ELMo.
• Word embeddings encode tokens and put similar meanings in close proximity.

Transformer Architecture

• Most LLMs use transformer architecture.
• The core component is the attention layer, which calculates token importance factors by considering the context of surrounding tokens.
• Multi-head attention layers combine multiple attention layers' outputs.

Why Does Language Matter for Robotics?

• The use of language improves the ability of human-robot communication.
• Language reduces the reliance on specialized, less intuitive, and natural communication interfaces.
• Language improves task description, enabling simplified explanations.
• Language acts as a data source (written format), providing relevant data for human-centered environments.

Foundation Models

• Foundation models are large neural network-based models trained on diverse data.
• They can be trained on a single data modality or multimodal data (e.g., text, audio, images).
• Foundation models are pretrained and can be further refined for specific tasks (transfer learning).
• GPT family models are examples of foundation models.

Vision Transformers

• Transformers were initially used for language processing and have recently been adapted for images.
• A vision transformer splits an image into patches, creates an embedding for each, and processes them sequentially.
• The patches are considered as image tokens to allow processing through the transformer architecture.
• The attention layers operate independently of the modality type if appropriately embedded input modality.

Vision-Language Models (VLMs)

• VLMs are models that combine visual and language inputs for making predictions.
• They ground language to real-world concepts and entities.
• VLMs are trained using contrastive learning techniques.
• VLMs align visual and language data.

Contrastive Learning

• Contrastive learning focuses on learning distance functions between similar and dissimilar inputs, encouraging similar inputs to be closer in the embedding space.
• The method works in both single-modality and multimodal embeddings.
• The focus is on producing a better representation of similar inputs.

Vision-Language-Action Models (VLAs)

• VLMs are not trained for or designed for robot control, instead, they are used for visual question-answer tasks.
• VLAs represent robot actions as discrete tokens, with predictions being de-tokenised to align with the action.
• Action prediction relies on end-effector delta actions and gripper positioning.

RT-X: Robot-Agnostic Foundation Models

• RT-X is a collection of foundation models for robotics trained on the Open X-Embodiment dataset.
• Each model architecture has two variations.
• The focus of RT-X is on generating robot actions for open-source research and adaptability to multiple robots and environments.

OpenVLA

- OpenVLA is a vision-language-action model pretrained on a subset of the Open X-Embodiment dataset.
- Includes additional training datasets.
- Uses a predefined visual input view (third-person).
- Architecturally based on a pretrained VLM.

Octo

• Octo is a transformer-based model trained on a subset of Open X-Embodiment.
• It's a goal-conditioned policy applicable to images or text.
• Uses a diffusion model for generating robot actions.

Diffusion Policy

• Octo applies diffusion policy, a visuomotor policy that generates actions through a probabilistic diffusion process.
• The denoising process (governed by learned network) creates a gradient-based solution.

Summary of Observations

• Vision-language models are actively developed, with recurring new models.
• Open X-Embodiment is a central dataset for training
• Models are often generalized and typically trained using multiple GPUs over multiple days.
• Further research is needed on safety and generalisation to maintain efficiency, and to improve robot reliability and performance.

Next Lecture: Explainable Robotics