A Language-First Approach for Procedure Planning

Jiateng Liu, Sha Li, Zhenhailong Wang, Manling Li, Heng JiACL 2023 · OpenReview

Abstract

Procedure planning in instructional videos involves predicting a sequence of intermediate actions to transition from a starting visual state to a desired goal state. Traditional approaches attempt to plan within visual latent spaces, which are often plagued by high variance and sensitivity to environmental noise. This paper presents a "language-first" framework that abstracts visual observations into text, utilizing pre-trained language models (PLMs) as reasoning engines. By mapping pixels to prose, the proposed method achieves significant performance gains, including a 98.9% relative improvement in success rates over previous state-of-the-art visual planning models.

Introduction

Imagine an autonomous agent tasked with making a salad. Given a start state (uncut vegetables) and a goal state (a finished bowl of greens), the agent must determine the necessary intermediate steps: cut ingredients, put ingredients in bowl, and stir. This task, known as procedure planning, is traditionally approached by learning a latent visual space. However, pixels are inherently messy; the visual representation of "cutting ingredients" varies wildly across different kitchens, lighting conditions, and actors.

Comparison between planning in visual latent space versus the proposed language-first approach.

As shown in Figure 1, planning in visual latent space often leads to inaccurate predictions due to high variance. In contrast, language provides a robust abstraction layer that is invariant to visual noise. A "cut" action remains semantically identical regardless of the background, making text a superior medium for high-level planning.

Methodology

The proposed framework operates in two distinct phases: visual-to-text alignment and text-to-text reasoning.

Phase 1: Image-to-Text Double Retrieval

To bridge the gap between pixels and prose, the authors developed a Double Retrieval Model. Unlike standard image captioning, which can be overly generic, this model jointly retrieves the most likely start and goal actions from a predefined pool. By considering the start image, goal image, and the task name (e.g., "Make Salad") simultaneously, the model resolves visual ambiguities that independent captioning might miss.

Phase 2: Text-to-Text Generation

Once the visual states are converted into text, they are processed by a pre-trained BART model. BART leverages the linguistic priors and "common sense" sequences it learned during pre-training to fill in the intermediate steps. To prevent the model from generating invalid actions, the authors employ constrained decoding via a Trie structure, ensuring every predicted step exists within the dataset's action space.

The modularized framework consisting of Image-to-Text Double Retrieval and Text-to-Text Generation.

Experimental Results

The model was evaluated on two major instructional video benchmarks: COIN (11,000+ videos across 180 tasks) and CrossTask.

Key findings include:

  • State-of-the-Art Performance: On the COIN dataset for a 3-step horizon, the model achieved a 30.64% success rate (SR), nearly doubling the 15.40% achieved by previous visual-only methods like P3IV.
  • Relative Gains: Across various benchmarks, the language-first approach demonstrated relative improvements ranging from 19.2% to 98.9%.
  • Robustness of Abstraction: Even a simple baseline using off-the-shelf image captioning (BLIP) outperformed specialized visual planning models, confirming that language is a more effective bottleneck for this task than visual features.

Discussion and Conclusion

While the language-first approach offers a significant leap in performance, challenges remain. Pre-trained language models may struggle with highly specialized domains (e.g., surgery) where general web-based "common sense" is insufficient. Furthermore, as planning horizons lengthen, the complexity of the sequence increases, leading to a drop in accuracy.

Future research should explore dynamic, closed-loop planning where the agent can re-evaluate and re-plan in real-time based on visual feedback, moving beyond the current offline generation paradigm. Nevertheless, this work demonstrates that for complex procedure planning, language is not just a descriptive tool but a powerful substrate for reasoning.

Summary updated

BibTeX
@inproceedings{liu2023language,
  author = {Jiateng Liu and Sha Li and Zhenhailong Wang and Manling Li and Heng Ji},
  title = {{A Language-First Approach for Procedure Planning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=x78fJO4VUYj},
}
Figure