Abstract
Training instruction-following agents via imitation learning requires large-scale labeled datasets, which are often costly to acquire. This paper proposes Play Segmentation (PS), a probabilistic model that extracts high-quality task segments from unannotated "play" data. By leveraging a small seed of labeled data and a global optimization approach via dynamic programming, PS identifies precise start and end points for specific instructions within long trajectories. Experiments on BabyAI and CALVIN benchmarks demonstrate that augmenting training sets with PS-extracted segments significantly improves policy performance, effectively doubling the utility of human-provided labels compared to standard baselines.
Introduction
To enable robots or virtual agents to follow natural language instructions, researchers typically rely on Imitation Learning (IL). IL requires paired datasets of instructions and corresponding trajectories. However, high-quality labeled data is scarce and expensive to produce. Conversely, "play data"—unannotated sequences of an agent interacting freely with its environment—is often abundant, found in sources like teleoperated robot logs or gameplay videos.
The primary challenge lies in the "messy" nature of play data. These trajectories contain continuous behaviors without clear boundaries for specific tasks. Simple random sampling or heuristic-based labeling often introduces noise that degrades agent performance. This paper addresses the problem by introducing a semi-supervised pipeline to "harvest" precise, labeled segments from large pools of unlabeled play data.
Methodology: Play Segmentation (PS)
The core contribution is the Play Segmentation (PS) model, a probabilistic framework designed to identify task-aligned segments within long trajectories. Unlike traditional action segmentation models that use fixed windows, PS treats segmentation as a global optimization problem.
The PS pipeline involves three key components:
- Segment-Level Learning: The model is trained on a small set of successful task segments and synthetic "negative" samples (e.g., shifted or incorrectly sized segments) to learn the characteristics of valid task execution.
- Probabilistic Factorization: The model factorizes the probability of a segment into a labeling component (identifying the action) and a segmentation component (identifying the boundaries).
- Global Optimization: At inference time, the model employs Dynamic Programming (DP) to find the maximum likelihood segmentation of an entire long trajectory. This ensures that the extracted pieces are mathematically the most probable way to partition the sequence into valid, instruction-aligned tasks.
Experimental Setup
The authors evaluated PS against adapted versions of state-of-the-art video models, including UnLoc and TriDet, across two environments:
- BabyAI: A grid-world navigation environment with high variance in instruction length.
- CALVIN: A complex robotic manipulation benchmark featuring a 7-DOF Franka Emika Panda arm. This dataset includes massive play trajectories (up to 30,000 timesteps) covering 34 distinct task types.
Results and Discussion
The experimental results highlight the critical importance of segmentation quality in data augmentation.
Performance Gains
In the CALVIN environment, augmenting a 25% labeled dataset with PS-extracted segments produced a policy that matched or exceeded the performance of a policy trained on 50% ground-truth labeled data. This effectively doubles the efficiency of the initial human labeling effort. In BabyAI, PS achieved 70.2% task accuracy, significantly outperforming UnLoc (58.3%) and TriDet (26.1%).
The Importance of Boundaries
A key insight from the BabyAI experiments was that random sampling of unlabeled data actually decreased policy performance. This confirms that "more data" is only beneficial if the task boundaries are accurate. Standard video models like TriDet struggled to generalize, often predicting segments that were too short to capture full task execution. In contrast, the global DP approach of PS maintained a segment length distribution close to the ground truth.
Limitations
Despite its success, PS faces computational challenges. The DP algorithm has a cubic complexity ($O(T^3)$) relative to trajectory length. While windowing techniques can mitigate this, it remains a bottleneck for extremely long sequences. Additionally, the model tends to over-extract "easy" tasks, suggesting a need for active learning strategies to balance the augmented datasets.
Conclusion
Play Segmentation provides a robust method for leveraging unannotated data to train instruction-following agents. By treating segmentation as a global probabilistic optimization problem, PS extracts high-quality training samples that significantly improve data efficiency in imitation learning. Future work will focus on improving computational efficiency and addressing label imbalance in extracted datasets.
Summary updated