Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Tasks

Albert Yu, Raymond J. MooneyICLR 2023

How do you teach a robot a new task? Usually, you have two choices: show it a video (demonstration) or tell it what to do (language). But as any human trainee knows, a video without context can be confusing, and a manual without pictures is a nightmare.

In their ICLR 2023 paper, researchers from UT Austin introduce DeL-TaCo (Joint Demo-Language Task Conditioning), a framework that proves robots, much like humans, learn best when they have both a visual guide and a verbal instruction.

DeL-TaCo Training and Testing Framework

TL;DR

DeL-TaCo is a multi-task robotic policy that conditions on both visual demonstrations and natural language instructions simultaneously. By allowing these two modalities to "mutually disambiguate" each other, the system achieves significantly better generalization to novel objects and reduces the human effort required to teach new tasks by a factor of 50.

What Problem Does This Paper Address?

The "Ambiguity Gap" is a major hurdle in robotics.

  • Language is vague: If you tell a robot to "put the cup in the red bin," but there are two red bins, the robot is stuck.
  • Demonstrations are context-dependent: If you show a video of a robot picking up a vase in a kitchen, but the actual robot is in a lab with different lighting and distractor objects, it might fail to understand the "true intention" of the demo.

Previous state-of-the-art models usually picked one modality at test time. This paper argues that unimodal representations are inefficient. The authors set out to build a system where language grounds the demonstration (explaining the intent) and the demonstration grounds the language (showing the physical target).

Methods

The authors propose a multi-task imitation learning policy. The architecture consists of three main components:

DeL-TaCo Architecture Diagram
  1. The Policy ($\pi$): A ResNet-18 backbone that processes live camera feeds.
  2. Language Encoder ($f_{lang}$): A frozen, pre-trained DistilBERT model that converts text instructions into embeddings.
  3. Demonstration Encoder ($f_{demo}$): A CNN trained from scratch that takes a few frames from a video demonstration (typically the first and last frames) and produces a visual task embedding.

The "Secret Sauce": Integration and Loss

  • Conditioning: The visual demo embedding is fed into the policy's ResNet backbone via FiLM (Feature-wise Linear Modulation) layers, which apply an affine transformation to intermediate image features. Meanwhile, the language embedding and the robot's proprioceptive state (joint angles, etc.) are concatenated later in the network.
  • Contrastive Learning: To ensure the two modalities actually help each other, the authors use a CLIP-style contrastive loss. This forces the demonstration embeddings and language embeddings for the same task to be close to each other in a shared latent space, while pushing different tasks apart.

Datasets

The researchers developed a custom Pybullet simulation environment featuring a WidowX 250 robot arm.

  • The Tasks: 300 unique pick-and-place tasks involving 32 objects (various colors/shapes) and 2 containers.
  • Adversarial Distractors: To make it hard, the scene always includes "distractor" objects that match the target's color or shape, forcing the robot to actually understand the instruction rather than just guessing.
  • Evaluation Splits: They tested two scenarios:
  • Scenario A: Generalization to entirely novel objects, colors, and shapes.
  • Scenario B: Generalization to novel combinations of colors and shapes for known objects.

Key Insights and Ideas

  • Mutual Disambiguation: This is the core "aha moment." The demo might show which object to grab (resolving noun ambiguity), while the language specifies which bin to use (resolving spatial/color ambiguity in the demo).
  • Language as a Force Multiplier: The most striking finding is that adding a single language instruction to a single demonstration is equivalent to providing 50 additional expert demonstrations for a demo-only policy.
  • The Importance of Alignment: Simply "having" both modalities isn't enough. The contrastive loss is crucial; without it, the model fails to leverage the semantic relationship between what it sees in the demo and what it reads in the text.

Key Results

  • Superior Generalization: In the toughest test (Scenario A), DeL-TaCo achieved a 19.9% success rate, significantly outperforming demo-only (14.6%) and language-only (10.4%) baselines.
  • Beating the SOTA: DeL-TaCo outperformed existing multi-modal methods like BC-Z (which regresses demo embeddings to language) and MCIL (which averages losses across modalities).
  • Efficiency: As noted, the "Value of Language" experiment showed that DeL-TaCo with 0 fine-tuning demos performed nearly as well as a demo-only policy fine-tuned on 50 expert demonstrations per task.

Limitations and Future Work

  • Task Complexity: The current work is limited to pick-and-place. Future iterations would need to handle more complex, temporally extended tasks (e.g., "open the drawer, then put the fork inside").
  • Rigid Language: The instructions used were template-based (e.g., "Put [object] in [bin]"). Real-world users use diverse paraphrases, which the current model might struggle to ground.
  • Simulation vs. Reality: The model was trained and tested in simulation. Bridging the "Sim-to-Real" gap remains a standard but significant challenge for this type of end-to-end policy.
  • Pre-trained Vision-Language Models: While the authors tried using CLIP, it didn't provide a massive boost due to the domain gap between real-world training data and the simulation. Better ways to leverage massive pre-trained models for specific robotic domains is a clear next step.

Summary updated

BibTeX
@inproceedings{yu2023using,
  author = {Albert Yu and Raymond J. Mooney},
  title = {{Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Tasks}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
}
Figure