Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity

Yucheng Zhou, Jianbing Shen

TL;DR

Autoregressive models excel at image generation but suffer from massive computational overhead when extended to video due to long token sequences. This paper introduces Local Optimization (Local-Opt.) and Representation Continuity (ReCo) to accelerate training by optimizing over localized windows while enforcing temporal smoothness via a Lipschitz-inspired loss. The resulting framework cuts training costs in half while matching or exceeding baseline video quality and consistency.

What Problem Does This Paper Address?

Autoregressive (AR) language models have recently shown great promise in visual generation, often outperforming diffusion models in inference speed and seamlessly integrating with Large Language Models (LLMs). However, moving from images to videos causes a sequence length explosion. A standard 17-frame video can easily require over 5,000 tokens after VQ-VAE discretization. Training AR models on such long sequences is computationally grueling and slow.

A naive fix is training on fewer frames (e.g., short 2-frame chunks), but this introduces severe exposure bias and error accumulation during inference. Because the model only looks at immediate history, tiny generation errors compound exponentially over time, causing the video's content and appearance to drift drastically.

Visual artifacts caused by error accumulation in fewer-frame training

Methods

To solve the training bottleneck without sacrificing video quality, the authors propose two core strategies:

  1. Local Optimization (Local-Opt.): Instead of training on short, isolated frame sequences, the model is exposed to the full video sequence but only computes the cross-entropy loss within a randomly sampled local window of length $W$. Crucially, the preceding tokens act as a frozen context via a stop-gradient operation. This prevents expensive full-sequence backpropagation while ensuring the model learns to generate tokens based on accurate, error-free history. They also introduce a balanced sampling strategy, oversampling the initial video frames since early errors catastrophically impact later generation.
  2. Representation Continuity (ReCo): To enforce temporal consistency, the authors view the AR model as a discrete-time dynamical system where hidden states evolve over time. Inspired by Lipschitz continuity, they introduce a regularization loss ($\mathcal{L}_{ReCo}$) that penalizes large jumps in the model's hidden states between consecutive time steps. This forces the model to learn smooth latent transitions, bounding error propagation during inference.
ReCo constrains sharp variations in hidden states to enforce smooth transitions

Datasets

The approach was evaluated on several standard video generation benchmarks:

  • FaceForensics (FFS): A dataset of human facial videos, highly sensitive to temporal inconsistencies in appearance.
  • SkyTimelapse (SKY): A dataset featuring continuous motion dynamics of clouds and skies.
  • UCF101: A diverse dataset of human action videos used to test performance across low-, medium-, and high-motion regimes.
  • Taichi-HD: A dataset of high-definition martial arts movements.
  • Vimeo & MSR-VTT: For text-to-video evaluation, the model was trained on the Vimeo dataset (~300K video-text pairs) and evaluated zero-shot on MSR-VTT.

Key Insights and Ideas

  • The Naive Speedup Trap: Training on fewer frames reduces GPU memory and training time, but it destroys global context. During inference, the model is forced to generate block-by-block, causing tiny errors to compound exponentially (exposure bias) and leading to severe visual artifacts.
  • Stop-Gradients as a Speed Booster: Local-Opt demonstrates that you don't need to backpropagate through the entire video history to learn long-range coherence. Freezing the historical context provides the necessary conditioning at a fraction of the computational cost.
  • First Frames Matter Most: The authors discovered a significant loss imbalance: token generation difficulty drops as the video progresses. Poor quality in the initial frames dooms the rest of the video, making balanced sampling of the first frame essential for closing the generalization gap.
  • Smooth Latent Spaces = Stable Videos: By penalizing abrupt hidden state changes via the ReCo loss, the model behaves like a stable dynamical system, preventing the chaotic drift typical of standard AR generation.

Key Results

  • 2x Training Speedup: ReCo cuts the training cost in half (0.5x cost) while achieving competitive or superior performance compared to full-context baselines.
  • Superior Video Quality: On the FFS dataset, the large ReCo* model achieved a Fréchet Video Distance (FVD) score of 42.5 (compared to the baseline's 46.1). When combined with advanced tokenizers like LARP, it achieved a state-of-the-art FVD of 46.2 on FFS and 56.1 on UCF101.
  • Enhanced Consistency: Quantitative evaluations using PSNR and Optical Flow confirmed that ReCo effectively eliminates the motion inconsistencies and abrupt visual shifts plaguing alternative acceleration methods.
  • Text-to-Video Scaling: In a 7B parameter Loong-based architecture evaluated on MSR-VTT, ReCo improved the FVD from 274 to 212 while maintaining strong text-semantic alignment (CLIP score).

Limitations and Future Work

The authors note that their study primarily focuses on algorithmic innovations and validation on standard academic benchmarks rather than commercial-scale, multi-billion parameter large language models. Future work includes scaling these methods to massive commercial production models to explore their full potential in large-scale multimodal systems.

Summary updated

Abstract

Autoregressive models have shown superior performance and efficiency in image generation, but remain constrained by high computational costs and prolonged training times in video generation. In this study, we explore methods to accelerate training for autoregressive video generation models through empirical analyses. Our results reveal that while training on fewer video frames significantly reduces training time, it also exacerbates error accumulation and introduces inconsistencies in the generated videos. To address these issues, we propose a Local Optimization (Local Opt.) method, which optimizes tokens within localized windows while leveraging contextual information to reduce error propagation. Inspired by Lipschitz continuity, we propose a Representation Continuity (ReCo) strategy to improve the consistency of generated videos. ReCo utilizes continuity loss to constrain representation changes, improving model robustness and reducing error accumulation. Extensive experiments on class- and text-to-video datasets demonstrate that our approach achieves superior performance to the baseline while halving the training cost without sacrificing quality.
BibTeX
@inproceedings{zhou2026accelerating,
  author = {Yucheng Zhou and Jianbing Shen},
  title = {{Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1501},
}
Figure