TL;DR
VEDIT (Video Embedding Diffusion Transformer) demonstrates that expensive sequence-level pretraining and noisy text supervision are unnecessary for learning procedural video representations. By performing iterative denoising via flow matching entirely within the latent space of frozen, off-the-shelf visual encoders, VEDIT achieves state-of-the-art performance across video forecasting, planning, and classification tasks.
What Problem Does This Paper Address?
To build AI systems capable of assisting humans in complex, multi-step activities (like cooking a recipe or assembling furniture), models must be able to reason about time, anticipate future steps, and plan intermediate actions.
Historically, state-of-the-art methods have relied on compute-intensive pretraining over video sequences, heavily dependent on language supervision like noisy Automatic Speech Recognition (ASR) transcripts or external knowledge bases. However, sequence-level data is scarce, and poorly aligned text annotations often introduce noise. The authors of VEDIT tackle a fundamental question: Can we skip sequence-level pretraining entirely and still learn an effective temporal transition function for procedural videos?
Methods
Instead of learning representations from raw pixel space or relying on text-to-video alignment losses, VEDIT operates entirely within the latent embedding space of publicly available, frozen vision encoders (such as SigLIP or TimeSformer).
The core architecture consists of a Vector Embedding Diffusion Transformer (VEDIT) that treats future or missing video steps as random Gaussian noise and iteratively denoises them using Flow Matching.
Key architectural highlights include:
- Two-Branch Joint Attention: The model processes observed ("seen") clips and target ("unseen/noisy") clips in separate branches, fusing their information via joint attention blocks rather than standard cross-attention.
- Rotary Positional Embeddings (RoPE): Applied right before attention to explicitly model the strict temporal order of procedural steps.
- Direct Downstream Training: The prediction model is trained directly on downstream tasks using a simple cross-entropy loss paired with a lightweight attentive classifier, completely bypassing any sequence-level pretraining phase.
Datasets
The authors evaluated VEDIT across five distinct procedural learning tasks using four widely recognized benchmark datasets:
- COIN: 476 hours of YouTube videos covering 180 daily tasks; used for step forecasting and task classification.
- Ego4D-v2: An egocentric video dataset used specifically to evaluate the long-term action anticipation benchmark.
- NIV & CrossTask: Instructional video datasets used alongside COIN to evaluate procedure planning (predicting intermediate steps given a start and goal state).
Key Insights and Ideas
- The Power of Frozen Latents: Strong, language-aligned image/video encoders (like SigLIP) already capture rich semantic features. A model only needs to learn the temporal transitions between these abstract states rather than re-learning visual features from scratch.
- Latent Space Efficiency: Generative modeling via diffusion in the latent space is significantly more computationally efficient than generating raw pixels, allowing the model to easily scale to long video horizons.
- Architectural Stability over Vanilla Transformers: When scaling up standard transformers for sequence-modeling tasks, performance often degrades due to overfitting. VEDIT’s diffusion-style architecture paired with adaptive normalization layers remains highly stable as the number of transformer blocks increases.
Key Results
VEDIT consistently outperformed existing state-of-the-art (SoTA) methods—even those heavily pretrained on massive sequence datasets like HowTo100M:
- Step Forecasting & Task Classification (COIN): VEDIT achieved a +5.0% absolute improvement in step forecasting and a +3.8% boost in task classification compared to previous top-performing models.
- Procedure Planning: When substituted into the state-of-the-art SCHEMA framework, VEDIT improved performance across all metrics, yielding up to a +2.28% increase in success rate, +3.39% in mean accuracy (mAcc), and +0.90% in mean Intersection over Union (mIoU).
- Long-Horizon Action Anticipation (Ego4D-v2): VEDIT advanced the strong baseline by +2.6% in Verb Edit Distance (ED@20) and +3.1% in Noun ED@20.
Limitations and Future Work
Interestingly, the authors experimented with adding large-scale pretraining on 1.16 million videos from the HowTo100M dataset to VEDIT, but it only provided a marginal +0.3% boost in accuracy. This highlights a limitation in current sequence-level pretraining datasets and objectives, which are often too noisy to be effective.
The authors suggest that future work should focus on developing better pretraining objectives that generalize more effectively across diverse downstream tasks. Additionally, they note that combining VEDIT's visual transition capabilities with the high-level reasoning and planning strengths of Large Language Models (LLMs) represents a promising future direction.
Summary updated