Overview of Learning for Robot Manipulation

Learning for Robot Manipulation Overview

• The presentation is an overview of learning for robot manipulation, focusing on techniques for enabling robots to perform manipulation tasks.
• The presentation covers why learning is important for robot manipulation, different approaches to learning for manipulation, state representation methods, manipulation policy learning, and transition model learning.
• Practical examples of manipulation skills illustrate the need for adaptable and flexible methods in everyday environments.

Structure and Why Learning for Robot Manipulation

• Learning for robot manipulation is useful due to the dexterity required, the variety of skills, and the complexity of modeling those skills.
• Manipulation tasks involve a broad range of skills, from simple to complex, making direct programming impractical in many cases.
• Current programming techniques are inflexible, while learning methods adapt to the task at hand.

Learning for Contact-Heavy Interactions

• Contact-heavy interactions, such as prolonged or precise contacts, are difficult to model explicitly.
• Robots must learn appropriate interaction policies to handle contact-heavy tasks effectively.

Learning and Robot Control

• Learning for robot manipulation is conceptually distinct from classical control theory, but related to it.
• Control theory often models the system and controller explicitly whereas learning enables robots to optimize controllers through experience.
• The combination of control theory and learning is a useful approach depending on the learning problem.

Lessons from Natural Systems

• Biological creatures learn most of their skills through developmental experience.
• Learning and adaptive capabilities in robots are crucial for success in dynamic environments.
• Robots capable of learning and adapting similar to biological creatures are useful in complex and evolving environments.

Overview of Learning for Manipulation

• The process of learning for manipulation involves multiple aspects, such as object models, policy parameters, skill models, and skill hierarchies.

What to Learn for Manipulation

• Object models help to understand and handle objects, often involving aspects like visual recognition.
• Policy parameters are used when we want a robot to control its actions through well-defined parameters.
• Skill models help develop specific skills complete with initiation and termination conditions.
• Skill hierarchies help combine different skills to accomplish complex tasks. (e.g. combining two different skills that have been learned independently).

Learning for Manipulation Overview - Diagrams

• Diagrams depict a comprehensive overview of learning for manipulation, illustrating the interconnectedness between different elements.
• The elements covered include object and environment representations, transition models, skill policies, and learning aspects.

State Representation

• The overall objective of robot manipulation is to enable purposeful action that changes the environment to a desired state.
• The state representation captures the changes in the environment, based on the robot's actions.
• An appropriate state representation can make complex learning problems more tractable.
• A robust state representation should encompass both the robot's internal state and its external environment state when dealing with robot manipulation.

Robot and Environment State - Equation

• The presentation uses an equation (S = Sr U Se) to represent the state as a combination of robot's internal state (Sr) and task environment state (Se).

Object-Centric Environment Representation - Equation

• The representation of a task environment can be constructed (n/i=1)- summation of states of individual objects of interest.
• A composite general environment state is also helpful and can be used to augment individual object states to construct the complete environment state (Se).
• The formula helps build robust environment states.

Generalisation over Contexts

• Robot learning often results in an execution policy that is specific to certain environmental parameters, thus not generalizable.
• The execution context vector (TC) can explicitly represent these parameters, allowing the execution policy to be varied based on the context.

Task Family

• When modelling complex tasks, a "task family" is a collection of tasks (Ti) with similar characteristics.
• Shared characteristics include state spaces, action spaces, transition functions, and reward functions that can be modeled in different ways.
• Tasks in the same family can use a shared "cost function" (E) to account for similarities and variations.

Object Representations

• Objects are often essential parts of robot manipulation state representations.
• Several hierarchical levels exist for object representation (point, part, or object level).
• Each object level has advantages based on the tasks being done.
• Hierarchical levels can be used together to solve more complex skills, such as task-oriented grasping.

Passive vs. Interactive Perception

• Robots can passively observe the environment or actively interact to get more information.
• Passive methods rely on sensors for data, while interactive methods involve actions to collect that data.
• Passive observation is limited and often needs to be augmented with interactive methods.

Manipulation Policy Learning

• A policy ( π ) is a function that selects actions ( a ) based on the current state (s).
• A key aspect of robotic learning lies in obtaining such a policy, for which multiple approaches exist.

Execution Policies Revisited

• Key approaches to learning a policy include reinforcement learning, imitation learning (from expert demonstrations), and transfer learning.

Action Spaces

• Policies might dictate a variety of actions, such as Cartesian velocity, Cartesian force, joint torques, and joint velocities.
• The outputs from a policy might not always be used directly for actuators but are processed through a low-level controller.

Policy Representations

• Policy representations use different approaches to represent the function, such as neural networks, lookup tables, local weighted regression, or decision trees.

Deterministic vs. Stochastic Policies

• Policies can be deterministic or stochastic when choosing an action given a certain state.

Parameterised Policies and Trajectories

• Robot policies can be characterized by parameters.
• Policies define trajectories that involve sequential states and actions.
• Probability calculation of trajectories are possible given a policy.

Reinforcement Learning Objective

• The goal in reinforcement learning is to find a policy that optimizes an expected return, reflecting accumulated rewards over time.

Exploration vs. Exploitation

• Balancing between exploiting the best known action versus exploring alternative actions is important during learning.
• Exploration and exploitation tradeoffs involve exploring the range of possibilities, avoiding suboptimal solutions that come from too much exploitation or quick convergence.

Model-Free Learning

• Model-free reinforcement learning avoids relying on a model of the environment.
• Learning occurs via trial-and-error, exploring the environment and building rewards for actions.

Temporal Difference — TD(λ) — Learning and Q-Learning

• Temporal difference (TD) learning methods bring the value function close to the reward function in an iterative learning process.
• Q-learning estimates the state-action value function, a crucial component in RL.
• The Q-learning update rule provides iterative approximations based on rewards and actions.

Deep Q-Learning

• Learning using deep neural networks can be applied in Q-learning for continuous spaces, not discrete ones as frequently used.
• The deep Q-learning framework updates the parameters of the network with an objective involving discrepancies in the current state or action from the ideal.

Policy Search

• Policy search directly optimizes the policy in the policy space rather than the value function.
• Policy search methods are useful in robotics because they allow incorporating prior knowledge about the policy directly.

Policy Gradients

• Policy gradient methods form a family of policy search methods that estimate gradients of expected returns.
• Gradient calculations are a key component to finding optimal policies and updating parameters.

REINFORCE Algorithm

• REINFORCE is a policy gradient algorithm that forms the backbone of many policy gradient algorithms.

Actor-Critic Learning

• Actor-critic algorithms use both critic (value function) and actor (policy) in learning, for better results than doing one or the other.
• The use of a baseline helps lower the variance in policy updates.

Proximal Policy Optimisation (PPO)

• Proximal Policy Optimisation (PPO) is a policy gradient algorithm.
• It aims to find optimal parameters in the policy function via updating policies according to a gradient calculation.
• PPO is a popular baseline due to its stability and applicability in robotic settings.

Imitation Learning

• Imitation learning uses expert demonstrations to train new policies.
• Imitation learning can be used in different methods such as behavior cloning or inverse reinforcement learning.

Behaviour Cloning

• Behavior cloning learns by directly copying demonstrations from an expert.

Inverse Reinforcement Learning

• Inverse reinforcement learning figures out a possible reward function based upon expert demonstrations.

Policy Transfer

• Policy transfer helps transfer policies between different tasks (T_i and T_j) to avoid relearning everything.

Skill Learning

• This section focuses on learning a policy, not skills that involve pre- and post-condition behaviors.

Transition Model Learning (Transition Models for State Prediction)

• Transition models predict the effect of actions on the environment's internal and external states.
• These models allow both discrete and continuous state predictions.
• Transition models help understand the outcomes of actions for tasks that operate in complex environments.

Model Uncertainty

• Probabilistic models account for uncertainties in the environment.
• Aleatoric uncertainty reflects inherent unpredictability, while epistemic uncertainty stems from incomplete knowledge.
• Training data or feedback help improve epistemic uncertainty in models.

Inverse Models

• Inverse models predict the action that produces a certain state transition.
• This is the opposite of forward models, in that forward models predict a state given an action, while inverse models attempt to identify an action that produces a desired state.
• Inverse models are useful for inferring actions to move to a desired state.

Next Lecture: Learning-Based Robot Navigation

• The following lecture will focus on learning-based robot navigation techniques.