Abstract
Procedure planning in instructional videos involves predicting a sequence of intermediate actions to transition from a starting visual state to a desired goal state. Traditional approaches attempt to plan within visual latent spaces, which are often plagued by high variance and sensitivity to environmental noise. This paper presents a "language-first" framework that abstracts visual observations into text, utilizing pre-trained language models (PLMs) as reasoning engines. By mapping pixels to prose, the proposed method achieves significant performance gains, including a 98.9% relative improvement in success rates over previous state-of-the-art visual planning models.
Introduction
Imagine an autonomous agent tasked with making a salad. Given a start state (uncut vegetables) and a goal state (a finished bowl of greens), the agent must determine the necessary intermediate steps: cut ingredients, put ingredients in bowl, and stir. This task, known as procedure planning, is traditionally approached by learning a latent visual space. However, pixels are inherently messy; the visual representation of "cutting ingredients" varies wildly across different kitchens, lighting conditions, and actors.
As shown in Figure 1, planning in visual latent space often leads to inaccurate predictions due to high variance. In contrast, language provides a robust abstraction layer that is invariant to visual noise. A "cut" action remains semantically identical regardless of the background, making text a superior medium for high-level planning.
Methodology
The proposed framework operates in two distinct phases: visual-to-text alignment and text-to-text reasoning.
Phase 1: Image-to-Text Double Retrieval
To bridge the gap between pixels and prose, the authors developed a Double Retrieval Model. Unlike standard image captioning, which can be overly generic, this model jointly retrieves the most likely start and goal actions from a predefined pool. By considering the start image, goal image, and the task name (e.g., "Make Salad") simultaneously, the model resolves visual ambiguities that independent captioning might miss.
Phase 2: Text-to-Text Generation
Once the visual states are converted into text, they are processed by a pre-trained BART model. BART leverages the linguistic priors and "common sense" sequences it learned during pre-training to fill in the intermediate steps. To prevent the model from generating invalid actions, the authors employ constrained decoding via a Trie structure, ensuring every predicted step exists within the dataset's action space.
Experimental Results
The model was evaluated on two major instructional video benchmarks: COIN (11,000+ videos across 180 tasks) and CrossTask.
Key findings include:
- State-of-the-Art Performance: On the COIN dataset for a 3-step horizon, the model achieved a 30.64% success rate (SR), nearly doubling the 15.40% achieved by previous visual-only methods like P3IV.
- Relative Gains: Across various benchmarks, the language-first approach demonstrated relative improvements ranging from 19.2% to 98.9%.
- Robustness of Abstraction: Even a simple baseline using off-the-shelf image captioning (BLIP) outperformed specialized visual planning models, confirming that language is a more effective bottleneck for this task than visual features.
Discussion and Conclusion
While the language-first approach offers a significant leap in performance, challenges remain. Pre-trained language models may struggle with highly specialized domains (e.g., surgery) where general web-based "common sense" is insufficient. Furthermore, as planning horizons lengthen, the complexity of the sequence increases, leading to a drop in accuracy.
Future research should explore dynamic, closed-loop planning where the agent can re-evaluate and re-plan in real-time based on visual feedback, moving beyond the current offline generation paradigm. Nevertheless, this work demonstrates that for complex procedure planning, language is not just a descriptive tool but a powerful substrate for reasoning.
Summary updated