Learning Procedural-aware Video Representations through State-Grounded Hierarchy Unfolding

Jinghan Zhao, Yifei Huang, Feng Lü

TL;DR

Researchers have developed a new framework called Task-Step-State (TSS) that helps AI better understand instructional videos by bridging the gap between abstract commands and raw pixels. By introducing a "State" layer—textual descriptions of how objects actually look before and after an action—and using a progressive "U-shaped" training strategy, the model learns to ground high-level tasks in concrete visual evidence, significantly outperforming previous state-of-the-art methods.

What Problem Does This Paper Address?

Teaching machines to understand procedural activities (like "changing a tire" or "making a latte") usually involves aligning video clips with text descriptions at two levels: the Task (the goal) and the Step (the action).

However, there is a massive "semantic gap" here. An abstract step like "cut oranges" doesn't tell a model what to look for in the pixels. Does it look for a knife? A hand? The orange's internal flesh? Because "tasks" and "steps" are so abstract, models often struggle to form a robust connection to the actual visual details. This paper argues that we are missing a crucial middle layer: the State. A state describes the observable configuration of objects (e.g., "the orange is now in two pieces with juice flowing out"). Without this grounding, models are essentially trying to learn a recipe without knowing what the ingredients look like.

Methods

The authors introduce the Task-Step-State (TSS) framework and a specific training recipe to make it work:

  1. State Generation via LLMs: Since manual annotation is impossible at scale, they used GPT-4o-mini to generate "before," "mid," and "after" state descriptions for over 10,000 steps found in the WikiHow database.
  2. Pseudo-Labeling: They aligned these text descriptions with a large-scale video dataset (HowTo100M) using frozen pre-trained encoders. This created a massive "silver-standard" dataset where video clips are linked to tasks, steps, and specific object states.
  3. Progressive Hierarchy Unfolding: Instead of training on everything at once (joint training), they proposed a U-shaped pre-training pathway:
  4. Top-Down Analysis: Start with Task $\rightarrow$ Step $\rightarrow$ State. This forces the model to use high-level context to predict fine-grained details.
  5. Bottom-Up Synthesis: Then go State $\rightarrow$ Step $\rightarrow$ Task. This uses the newly learned visual grounding to refine and "verify" the higher-level concepts.
  6. Lightweight Adapters: To keep training efficient, they kept the main video backbone frozen and only trained a small "bottleneck adapter" that learns to project generic visual features into the procedural-aware space.

Datasets

  • Pre-training: A subset of HowTo100M (85K videos) for visual data and WikiHow (1,053 tasks, 10,588 steps) for procedural knowledge.
  • Evaluation:
    • COIN: A large-scale dataset for instructional video analysis (11.8K videos, 180 tasks).
  • CrossTask: A dataset focused on cross-task step alignment (83 tasks).

Key Insights and Ideas

  • States as the "Skeleton": The authors view states as the foundation of any procedure. A task is just a macro-transition from an initial state to a final state, driven by a series of steps. By focusing on these transitions, the model learns causality, not just pattern matching.
  • The Failure of Joint Training: Interestingly, the paper found that training on all levels simultaneously (Task, Step, and State) is actually less effective than the staged, progressive approach. Joint training acts like a "black box" that doesn't properly enforce the hierarchical logic.
  • The "U-Shape" is Essential: Moving directly from the concrete State layer back to the abstract Task layer (skipping the Step layer) hurts performance. This confirms that the "Step" layer acts as an essential intermediate bridge between raw visual states and high-level goals.

Key Results

The proposed method consistently outperformed existing baselines like MIL-NCE and Paprika across multiple downstream tasks:

  • Task Recognition: On the COIN dataset, the model achieved a +2.24% improvement in Top-1 accuracy over the previous best method (Paprika) when using a simple MLP head.
  • Step Recognition & Forecasting: The model saw gains of +2.15% in recognition and +3.97% in forecasting (predicting the next step).
  • Robustness: The performance gains held true whether using a simple MLP or a more complex Transformer as the evaluation head, suggesting the learned representations are inherently more "procedural-aware."

Limitations and Future Work

  • LLM Dependency: The quality of the "State" layer depends on the LLM's ability to describe visual changes accurately. If the LLM generates "hallucinated" or non-visual descriptions, the grounding will be noisy.
  • Temporal Resolution: The current method uses fixed-length video clips (9.6 seconds). Real-world steps vary wildly in duration, and future work could involve more flexible temporal modeling.
  • Complex Relations: While the model learns "last step" and "next step" relations, it doesn't yet capture more complex, non-linear procedural graphs (e.g., "you can do step A or step B, but both must happen before C").

Summary updated

Abstract

Learning procedural-aware video representations is a key step towards building agents that can reason about and execute complex tasks. Existing methods typically address this problem by aligning visual content with textual descriptions at the task and step levels to inject procedural semantics into video representations. However, due to their high level of abstraction, 'task' and 'step' descriptions fail to form a robust alignment with the concrete, observable details in visual data. To address this, we introduce 'states', i.e., textual snapshots of object configurations, as a visually-grounded semantic layer that anchors abstract procedures to what a model can actually see. We formalize this insight in a novel Task-Step-State (TSS) framework, where tasks are achieved via steps that drive transitions between observable states. To enforce this structure, we propose a progressive pre-training strategy that unfolds the TSS hierarchy, forcing the model to ground representations in states while associating them with steps and high-level tasks. Extensive experiments on the COIN and CrossTask datasets show that our method outperforms baseline models on multiple downstream tasks, including task recognition, step recognition, and next step prediction. Ablation studies show that introducing state supervision is a key driver of performance gains across all tasks. Additionally, our progressive pretraining strategy proves more effective than standard joint training, as it better enforces the intended hierarchical structure.
BibTeX
@misc{zhao2025learning,
  author = {Jinghan Zhao and Yifei Huang and Feng Lü},
  title = {{Learning Procedural-aware Video Representations through State-Grounded Hierarchy Unfolding}},
  year = {2025},
  eprint = {2511.20073},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.20073},
}

Loading related content...

Figure