Learning-Based Object Grasping Overview

Learning-Based Object Grasping: An Overview

• Presentation by Dr. Alex Mitrevski, Master of Autonomous Systems, Winter semester 2023/24.
• Focuses on learning-based robotic grasping.

Structure

• Object grasping primer: Introduces basic concepts and definitions of object grasping.
• Learning-based grasping: Explanation of learning-based methods for robotic grasping.
• Closer look at concrete learning-based grasping frameworks: Detailed analysis of specific learning approaches.

What is Object Grasping?

• Informally, object grasping is the process of a robotic hand picking up an object.
• Formally, defined by finding an end-effector pose (p = (t*, R*)) that ensures a stable object grasp.
  • t* is the grasp center point.
  • R* is the gripper's orientation.
• Can be additionally parameterized by an approach vector (a).
• More generally, involves determining positions and applied forces for each gripper finger.

Grasp Synthesis

• An optimization process to generate grasp candidates that satisfy desired quality metrics.
• Formally, given an object model (O) and gripper description, it generates a grasp candidate (C*) that optimizes the quality metrics.
• Often uses a sampling-based approach – generating multiple grasp candidates (C_i), evaluating them, and selecting the best one.

Grasp Properties

• Dexterity: Finger configuration should avoid singularities and meet dexterity measures.
• Stability: Grasped object should return to equilibrium after disturbances.
• Equilibrium: Forces and moments acting on the grasped object must sum to zero.
• Dynamic behavior: How the grasped object responds to applied forces.

Grasp Quality Metrics

• Various metrics used to evaluate grasps.
  • Researchers study compliance, connectivity, dexterity, disturbance resistance, dynamic behavior, equilibrium, force applicability, force closure, form closure, isotropy, internal forces, manipulability, robustness, slip resistance, and stability.
• Used in evaluating grasp candidates generated by grasp synthesis procedures.

Factors Affecting Grasp Synthesis

• Knowledge of an object: if known, enhances synthesis efficiency.
• Input modality: method for identifying objects.
• Robotic hand type: gripper characteristics affect grasp candidate generation.
• Task information: Constrains valid grasp candidates.

Challenges With Analytical Grasp Synthesis

• Reliance on object models: Challenges in adjusting to unknown object types.
• Reliance on simulations: Simulation metrics might not reflect real-world conditions.
• Slow synthesis: Computationally expensive grasp quality metric evaluations can lead to slower synthesis processes.
• Inability to use prior experiences: Each instance is treated independently, limiting learning from past instances (less efficient than analytical methods).

Learning-Based Grasping

• Aims to replace, partially or completely, analytical methods for grasping.
• Offers versatile approaches depending on learning outcomes.

Learning for Object Grasping

• Goal is to replace/supplement analytical methods with learned approaches.
• Methods employed depend on the desired outcome (classification, model, policy).

Grasp Candidate Classification

• Learns a model to score grasp candidates based on quality.
• Requires a labeled dataset with ground truth metrics.
• Online application needs to generate a set of grasp candidates that the model evaluates.

Grasp Policies

• An alternative to candidate generation/evaluation is learning a policy (visuomotor) to directly execute the grasp.
• The policy may use a sparse or shaped reward.
• Sparse reward: reward only for successful grasping.
• Shaped reward: integrates analytical grasp metrics in reward.

Techniques Used in Learning-Based Grasping

• Supervised learning is prevalent.
• Extracts features from PointNet++ and DGCNN (used on point cloud data).
• Utilizes existing datasets like ShapeNet and Semantic3D.

Learning Data Sources

• Learning by trial and error: Robot collects its own data.
• Learning from demonstrations: Uses demonstrations for grasp learning.
• Learning from labelled data: Labelled data is essential for learning appropriate grasp models.

Next Lecture

• Focuses on Sim-to-Real transfer.

Dexterity Network (Dex-Net)

• Convolutional neural network-based grasp quality evaluation model.
  • Earlier version for parallel-jaw grippers.
  • Latest version for suction grippers with pixel coordinates, gripper depth (height), and orientation for parameterization.

Grasp Quality Convolutional Neural Network (GQ-CNN)

• Network to gauge grasp success.
• Input is a grasp candidate and gripper depth as aligned depth image.
  • Output is the grasp success probability estimate.
• Learns from synthetic dataset of 3D objects and grasps.
• Success metric (probability of force closure) is used to label candidates.

Context-Aware Grasping (CAGE)

• A deep neural network model evaluates grasp candidates.
  • Features task context (semantic task information, point material information) and potential grasp candidate features.
  • Architecture is wide-and-deep.
  • Training uses a negative log-likelihood loss.