VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning

Han Lin, Tushar Nagarajan, Nicolas Ballas, Mido Assran, Mojtaba Komeili, Mohit Bansal, Koustuv Sinha

TL;DR

VEDIT (Video Embedding Diffusion Transformer) demonstrates that expensive sequence-level pretraining and noisy text supervision are unnecessary for learning procedural video representations. By performing iterative denoising via flow matching entirely within the latent space of frozen, off-the-shelf visual encoders, VEDIT achieves state-of-the-art performance across video forecasting, planning, and classification tasks.

What Problem Does This Paper Address?

To build AI systems capable of assisting humans in complex, multi-step activities (like cooking a recipe or assembling furniture), models must be able to reason about time, anticipate future steps, and plan intermediate actions.

Historically, state-of-the-art methods have relied on compute-intensive pretraining over video sequences, heavily dependent on language supervision like noisy Automatic Speech Recognition (ASR) transcripts or external knowledge bases. However, sequence-level data is scarce, and poorly aligned text annotations often introduce noise. The authors of VEDIT tackle a fundamental question: Can we skip sequence-level pretraining entirely and still learn an effective temporal transition function for procedural videos?

Methods

Instead of learning representations from raw pixel space or relying on text-to-video alignment losses, VEDIT operates entirely within the latent embedding space of publicly available, frozen vision encoders (such as SigLIP or TimeSformer).

Figure 1: Overview of our VEDIT training pipeline. Model architecture (left): We introduce masked clip-level latent prediction as our training objective, where we train a Vector Embedding DiT (VEDiT) to iteratively denoise T steps from random gaussian noise with flow matching noise scheduler. Downstream tasks (right): We train VEDiT with a light-weight attentive classifier (Bardes et al., 2024) with cross-entropy loss for the following tasks. (a) Step forecasting / action anticipation task: predict the embeddings of next unseen clip from observed clips with VEDiT. (b) Procedure planning task: predict the embeddings of intermediate unseen clips from observed starting and goal clips with VEDiT. (c) Procedural activity classification task: given a sequence of observed video clips, predict the label of the procedural video.

The core architecture consists of a Vector Embedding Diffusion Transformer (VEDIT) that treats future or missing video steps as random Gaussian noise and iteratively denoises them using Flow Matching.

Key architectural highlights include:

  • Two-Branch Joint Attention: The model processes observed ("seen") clips and target ("unseen/noisy") clips in separate branches, fusing their information via joint attention blocks rather than standard cross-attention.
  • Rotary Positional Embeddings (RoPE): Applied right before attention to explicitly model the strict temporal order of procedural steps.
  • Direct Downstream Training: The prediction model is trained directly on downstream tasks using a simple cross-entropy loss paired with a lightweight attentive classifier, completely bypassing any sequence-level pretraining phase.
Detailed VEDIT Block Architecture

Datasets

The authors evaluated VEDIT across five distinct procedural learning tasks using four widely recognized benchmark datasets:

  • COIN: 476 hours of YouTube videos covering 180 daily tasks; used for step forecasting and task classification.
  • Ego4D-v2: An egocentric video dataset used specifically to evaluate the long-term action anticipation benchmark.
  • NIV & CrossTask: Instructional video datasets used alongside COIN to evaluate procedure planning (predicting intermediate steps given a start and goal state).

Key Insights and Ideas

  • The Power of Frozen Latents: Strong, language-aligned image/video encoders (like SigLIP) already capture rich semantic features. A model only needs to learn the temporal transitions between these abstract states rather than re-learning visual features from scratch.
  • Latent Space Efficiency: Generative modeling via diffusion in the latent space is significantly more computationally efficient than generating raw pixels, allowing the model to easily scale to long video horizons.
  • Architectural Stability over Vanilla Transformers: When scaling up standard transformers for sequence-modeling tasks, performance often degrades due to overfitting. VEDIT’s diffusion-style architecture paired with adaptive normalization layers remains highly stable as the number of transformer blocks increases.
Figure 3: SCHEMA w/ VEDIT is more stable than SCHEMA w/ vanilla transformer as we increase the number of transformer blocks.

Key Results

VEDIT consistently outperformed existing state-of-the-art (SoTA) methods—even those heavily pretrained on massive sequence datasets like HowTo100M:

  • Step Forecasting & Task Classification (COIN): VEDIT achieved a +5.0% absolute improvement in step forecasting and a +3.8% boost in task classification compared to previous top-performing models.
Step forecasting and task classification results on the COIN dataset
  • Procedure Planning: When substituted into the state-of-the-art SCHEMA framework, VEDIT improved performance across all metrics, yielding up to a +2.28% increase in success rate, +3.39% in mean accuracy (mAcc), and +0.90% in mean Intersection over Union (mIoU).
  • Long-Horizon Action Anticipation (Ego4D-v2): VEDIT advanced the strong baseline by +2.6% in Verb Edit Distance (ED@20) and +3.1% in Noun ED@20.

Limitations and Future Work

Interestingly, the authors experimented with adding large-scale pretraining on 1.16 million videos from the HowTo100M dataset to VEDIT, but it only provided a marginal +0.3% boost in accuracy. This highlights a limitation in current sequence-level pretraining datasets and objectives, which are often too noisy to be effective.

The authors suggest that future work should focus on developing better pretraining objectives that generalize more effectively across diverse downstream tasks. Additionally, they note that combining VEDIT's visual transition capabilities with the high-level reasoning and planning strengths of Large Language Models (LLMs) represents a promising future direction.

Summary updated

Abstract

Procedural video representation learning is an active research area where the objective is to learn an agent which can anticipate and forecast the future given the present video input, typically in conjunction with textual annotations. Prior works often rely on large-scale pretraining of visual encoders and prediction models with language supervision. However, the necessity and effectiveness of extending compute intensive pretraining to learn video clip sequences with noisy text supervision have not yet been fully validated by previous works. In this work, we show that a strong off-the-shelf frozen pretrained visual encoder, along with a well designed prediction model, can achieve state-of-the-art (SoTA) performance in forecasting and procedural planning without the need for pretraining the prediction model, nor requiring additional supervision from language or ASR. Instead of learning representations from pixel space, our method utilizes the latent embedding space of publicly available vision encoders. By conditioning on frozen clip-level embeddings from observed steps to predict the actions of unseen steps, our prediction model is able to learn robust representations for forecasting through iterative denoising - leveraging the recent advances in diffusion transformers (Peebles & Xie, 2023). Empirical studies over a total of five procedural learning tasks across four datasets (NIV, CrossTask, COIN and Ego4D-v2) show that our model advances the strong baselines in long-horizon action anticipation (+2.6% in Verb ED@20, +3.1% in Noun ED@20), and significantly improves the SoTA in step forecasting (+5.0%), task classification (+3.8%), and procedure planning tasks (up to +2.28% in success rate, +3.39% in mAcc, and +0.90% in mIoU).
BibTeX
@inproceedings{lin2025vedit,
  author = {Han Lin and Tushar Nagarajan and Nicolas Ballas and Mido Assran and Mojtaba Komeili and Mohit Bansal and Koustuv Sinha},
  title = {{VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=LDAj4UJ4aL},
}
Figure