Abstract
Developing agents that can follow multimodal instructions remains a fundamental challenge in robotics and AI. Although large-scale pre-training on unlabeled datasets (no language instruction) has enabled agents to learn diverse behaviors, these agents often struggle with following instructions. While augmenting the dataset with instruction labels can mitigate this issue, acquiring such high-quality annotations at scale is impractical. To address this issue, we frame the problem as a semi-supervised learning task and introduce GROOT-2, a multimodal instructable agent trained using a novel approach that combines weak supervision with latent variable models. Our method consists of two key components: constrained self-imitating, which utilizes large amounts of unlabeled demonstrations to enable the policy to learn diverse behaviors, and human intention alignment, which uses a smaller set of labeled demonstrations to ensure the latent space reflects human intentions. GROOT-2's effectiveness is validated across four diverse environments, ranging from video games to robotic manipulation, demonstrating its robust multimodal instruction-following capabilities.
Introduction
While large-scale pre-training has allowed AI agents to learn diverse behaviors from massive unlabeled datasets—such as YouTube videos or game replays—these agents often fail at the critical step of instruction following. The core challenge lies in "latent space ambiguity." When an agent learns from video alone, it typically falls into one of two traps: Mechanical Imitation, where the agent copies exact pixel-level movements rather than high-level intent, or Posterior Collapse, where the agent ignores instructions entirely.
GROOT-2 addresses this by framing instruction following as a semi-supervised latent variable task. By combining massive amounts of unlabeled "play" data with a small fraction of human-annotated labels, the authors created an agent capable of understanding instructions across text, video, and even target reward levels.
Methodology
GROOT-2 utilizes a Variational Autoencoder (VAE) architecture to treat instruction following as a semi-supervised learning task. The training process consists of two interleaved components:
- Constrained Self-Imitating: The model trains on a massive set of unlabeled demonstrations. It learns to encode a trajectory into a latent variable $z$ and then uses a policy (decoder) to reconstruct the actions. This phase teaches the agent how to behave and interact with the world.
- Human Intention Alignment: Using a much smaller set of labeled data (e.g., a video paired with the text "mine a diamond"), the model forces the latent representation of the text to match the latent representation of the video. This ensures the latent space $Z$ reflects human-understandable goals rather than just raw motion.
Architecturally, GROOT-2 employs a TransformerXL backbone for the policy to handle long-term dependencies and a Vision Transformer (ViT) for processing visual observations. A "pre-fusion" cross-attention mechanism is used to compress visual tokens before they reach the main policy network, significantly boosting efficiency.
Key Insights: The Intent vs. Motion Spectrum
A central contribution of this work is the definition of a ratio ($R$) that measures the balance between mechanical imitation and posterior collapse. The authors argue that the "sweet spot" for a useful agent lies in the middle of this spectrum, where the instruction provides high-level guidance, but the agent uses its own learned "common sense" to navigate environmental obstacles.
Because the model maps text, video, and rewards into the same latent space, it enables Cross-Modal Knowledge Transfer. An agent trained on video instructions can "zero-shot" transfer that knowledge to text instructions because both modalities share the same "intent" coordinate in the latent space. Furthermore, the paper demonstrates favorable scaling laws: in robotics benchmarks, the agent achieved 82% success with only 50% of the labels, compared to 84% with full labeling, suggesting that massive annotation is not strictly necessary for high performance.
Experimental Setup
The researchers evaluated GROOT-2 across four distinct domains:
- Minecraft: Utilizing the "Contractor" dataset (160M frames) and an expanded SkillForge benchmark of 100 tasks.
- Atari: 35 different games using 350 million frames of DQN replay buffer data.
- Language Table: A robotics simulator featuring 100M trajectories.
- Simpler Env: A simulation of the Google Robot environment using the OpenX dataset.
Results and Qualitative Success
GROOT-2 significantly outperformed previous state-of-the-art models like STEVE-1 and GROOT-1. In Minecraft's "Climb Mountain" task, scaling up unlabeled data led to a 5x performance gain over the baseline. In robotics environments, GROOT-2 (trained on 50% labels) achieved performance comparable to RT-1-X, a much larger and more computationally expensive model.
Qualitatively, the model demonstrates true goal-oriented behavior. When given a video of a player mining diamonds, GROOT-2 does not just copy the path; it successfully navigates around new obstacles, such as lava pits, that were not present in the original reference video.
Conclusion and Future Work
GROOT-2 represents a significant step toward scalable, instructable agents. However, some limitations remain. The model still relies heavily on trajectory data (state-action pairs) and cannot yet learn from "action-less" data like standard YouTube videos. Additionally, while it excels at single-task instructions, long-horizon multi-step reasoning remains an area for future exploration. The authors suggest that future iterations could leverage inverse dynamics models to "guess" actions in raw video data, potentially unlocking the vast amount of unlabeled human activity available online.
Summary updated