BlockAlign: Preference Optimization for Trajectory-Distilled Video Generation

Jingyuan Zhu, Biaolong Chen, Yangyang Qin, Shengyang Shen, Yaohua Wang, Weiming Bai, Miao Lu, Aixi Zhang

TL;DR

BlockAlign is a new preference optimization framework designed specifically for "few-step" distilled video generators. By resolving a fundamental mismatch between how these models predict motion and how they are typically fine-tuned, BlockAlign achieves significant gains in visual quality and structural plausibility while being 3x faster to train than traditional alignment methods.

What Problem Does This Paper Address?

The current frontier of video generation is moving toward trajectory distillation—techniques like Parallel Decoding Distillation (PDD) and Parametric Trajectory Distillation (PTD) that compress a 50-step denoising process into just 10 steps. While fast, these models still struggle with "human" details: extra limbs, physics-defying movements, and inconsistent identities.

To fix these, researchers usually turn to Preference Optimization (like DPO), where a model is fine-tuned on pairs of "good" vs. "bad" videos. However, there’s a catch. Standard alignment tools assume the model predicts "instantaneous velocity" at any given moment. Distilled models don't do that; they predict an average velocity over a finite "block" or "hop"—essentially a macro-jump through time.

If you try to align these models using standard methods (a naive approach the authors call DirectAlign), you only update a fraction of the model's internal heads. This creates a "training-inference mismatch" that actually makes the video quality worse than the original unaligned model.

Methods

The authors introduce BlockAlign, an interface-consistent framework that respects the "blocky" nature of distilled generators.

  1. Boundary-Aligned Supervision: Instead of supervising the model at arbitrary points, BlockAlign only looks at valid block boundaries. It queries the fused output of all velocity heads, ensuring the gradients flow back through the entire prediction mechanism used during actual inference.
  2. Multi-Regime Pipeline: The framework isn't just one algorithm; it’s a pipeline that combines:
  3. Offline Alignment: Using fixed pairs of videos ranked by humans.
  4. Semi-Online Alignment: A clever "reward-free" method that uses partial noising/denoising (SDEdit-style) to create pairs that specifically target either texture refinement or compositional layout.
  5. Online Alignment: Real-time rollouts where a Large Vision-Language Model (VLM) like Gemini-4-Argon acts as a judge.
  6. Hierarchical Preference Selection: To prevent "reward hacking" (where a model makes a pretty video that is physically impossible), they use a gatekeeper approach: a video must pass a structural/physical plausibility check before it is even ranked for "softer" qualities like color or style.
  7. Objective-Aware Scheduling: They recognize that different blocks in the denoising process handle different things. Early blocks control global motion and layout, while late blocks handle fine textures. BlockAlign prioritizes updates to specific blocks based on the training goal.

Datasets

The researchers evaluated BlockAlign across two massive backbones:

  • RefVid-V4: An in-house 30B parameter Diffusion Transformer (DiT) trained on 5,000 human-annotated pairs and 10,000 curated reference videos.
  • MiniMax-H3: A 33B parameter open-source DiT.
  • Public Benchmarks: They also tested zero-shot generalization on OpenS2V-Eval (360 test cases) and OmniVBench (813 test cases), which cover everything from subject consistency to complex narrative following.

Key Insights and Ideas

  • The "Aha" Moment: The realization that the "fused-head" interface of distilled models is a first-class citizen. You cannot treat a 10-step distilled model as if it were just a 10-step version of a 100-step model; the underlying math of the "macro-jump" is different.
  • Efficiency as an Enabler: Because the distilled models are 5x faster to run, "online" training (which requires generating thousands of candidate videos during the training loop) finally becomes computationally practical.
  • Structure First: By using a hierarchical judge, the authors found they could eliminate "nightmare fuel" artifacts (like three-legged people) much more effectively than by using a single scalar reward score.

Key Results

  • Human Preference: On RefVid-V4, BlockAlign achieved a 61.8% non-tie win rate over the base distilled model. Against the naive DirectAlign method, it reached a staggering 82.7% win rate.
  • Training Speed: The approach unlocked a 3x end-to-end training speedup, reducing a 1,500-step training run from an estimated 600 hours down to 200 hours on 96 H200 GPUs.
  • Benchmarks: On OmniVBench, BlockAlign ranked second only to much more expensive models, outperforming Kling 3.0 Omni and Gemini Omni in several structural and content-related categories.
  • Diversity: Crucially, the authors proved that this alignment doesn't lead to "mode collapse." The models still produce a diverse range of videos for the same prompt, just with higher quality.

Limitations and Future Work

  • Compute Requirements: While 3x faster, it still requires a massive footprint (96 H200 GPUs), which is out of reach for most academic labs.
  • Reward Model Bottleneck: The authors noted that current off-the-shelf video reward models aren't quite "precise" enough for fine-grained human-object interactions, necessitating the use of expensive VLM judges.
  • Future Directions: They suggest exploring more advanced RL paradigms like GRPO (Group Relative Policy Optimization) or differentiable reward backpropagation specifically tailored for these block-based architectures.

For more details, see the full paper: "BlockAlign: Preference Optimization for Trajectory-Distilled Video Generation" (arXiv:2611.20351v1).

Summary updated

Abstract

Directly aligning trajectory-distilled video generators slashes rollout costs and optimizes the deployed few-step model without teacher fine-tuning or re-distillation. However, conventional flow-matching preference objectives assume single-head local velocity supervision, mismatching the finite-interval, block-average transition interface of trajectory-distilled generators such as Parallel Decoding Distillation (PDD) and Parametric Trajectory Distillation (PTD). Naively supervising only the primary initial-subinterval velocity sub-head rather than the fused block-average velocity at block starts (DirectAlign) induces a severe training--inference mismatch that degrades quality below the unaligned baseline. We introduce BlockAlign, an interface-consistent preference optimization framework that constructs states at valid block boundaries, queries the fused block-level velocity predictor, and supervises span-normalized transition residuals-seamlessly supporting DPO-, NFT-, and rollout-style objectives while preserving the native few-step sampler. We further build a unified post-training pipeline integrating offline pairs, reward-free semi-online continuations, and online rollouts with hierarchical preference selection and objective-aware block scheduling. Experiments on multi-reference video generation across MiniMax-H3 Ref2VA and an in-house model RefVid-V4 show that BlockAlign consistently outperforms unaligned distilled models and DirectAlign in structural plausibility, reference consistency, and visual quality. Notably, our approach unlocks PDD's 5x few-step rollout speedup for 3x faster end-to-end online training, achieving statistically significant human preference gains over PDD baselines (54.0% vs. 33.3% wins/losses, or 61.8% non-tie win rate on RefVid-V4; 51.3% vs. 34.7%, or 59.7% non-tie win rate on MiniMax-H3) and up to an 82.7% win rate against DirectAlign.
BibTeX
@misc{zhu2026blockalign,
  author = {Jingyuan Zhu and Biaolong Chen and Yangyang Qin and Shengyang Shen and Yaohua Wang and Weiming Bai and Miao Lu and Aixi Zhang},
  title = {{BlockAlign: Preference Optimization for Trajectory-Distilled Video Generation}},
  year = {2026},
  eprint = {2611.20351},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20351},
}
Figure