OmniForcing: Unleashing Real-time Joint Audio-Visual Generation

Yaofeng Su, Yuming Li, Zeyue Xue, Jie Huang, Siming Fu, Haoran Li, Ying Li, Zezhong Qian, Haoyang Huang, Nan Duan

Abstract

Recent joint audio-visual diffusion models achieve remarkable generation quality but suffer from high latency due to their bidirectional attention dependencies, hindering real-time applications. We propose OmniForcing, the first framework to distill an offline, dual-stream bidirectional diffusion model into a high-fidelity streaming autoregressive generator. However, naively applying causal distillation to such dual-stream architectures triggers severe training instability, due to the extreme temporal asymmetry between modalities and the resulting token sparsity. We address the inherent information density gap by introducing an Asymmetric Block-Causal Alignment with a zero-truncation Global Prefix that prevents multi-modal synchronization drift. The gradient explosion caused by extreme audio token sparsity during the causal shift is further resolved through an Audio Sink Token mechanism equipped with an Identity RoPE constraint. Finally, a Joint Self-Forcing Distillation paradigm enables the model to dynamically self-correct cumulative cross-modal errors from exposure bias during long rollouts. Empowered by a modality-independent rolling KV-cache inference scheme, OmniForcing achieves state-of-the-art streaming generation at $\sim$25 FPS on a single GPU, maintaining multi-modal synchronization and visual quality on par with the bidirectional teacher.

From Offline Quality to Real-Time Streaming

State-of-the-art generative models like LTX-2 and Sora produce stunning video and audio, but they are "offline" by nature. They rely on bidirectional attention, meaning the model needs to "see" the entire timeline to generate even the first frame. This leads to massive Time-To-First-Chunk (TTFC) latency—sometimes minutes—making them unsuitable for interactive or real-time streaming applications.

OmniForcing addresses this by transforming heavy, offline audio-visual diffusion models into high-fidelity, real-time streaming generators. By distilling the bidirectional LTX-2 model into a causal autoregressive framework, it achieves a ~35x speedup, delivering synchronized 480p video and audio at ~25 FPS on a single GPU with a latency of just 0.7 seconds.

Overcoming Temporal Asymmetry

Simply forcing these models to be "causal" (predicting the next step based only on the past) usually fails for multi-modal data. The core challenge is temporal asymmetry: in the latent space, audio tokens are much denser than video tokens (e.g., 25 audio frames vs. 3 video frames per second). This mismatch causes "Softmax collapse" and gradient explosions during training, as the sparse audio stream lacks enough historical context to stabilize the attention mechanism.

The Asymmetric Block-Causal Alignment mechanism utilizes a specific mask to manage multi-modal synchronization.

To solve this, OmniForcing introduces Audio Sink Tokens—learnable tokens at the start of the sequence that act as a "global memory buffer." These are paired with an Identity RoPE (Rotary Position Embedding) constraint to ensure they remain position-agnostic anchors. Furthermore, the Asymmetric Block-Causal Alignment (shown above) syncs modalities at 1-second "macro-block" boundaries, while a Modality-Independent Rolling KV-Cache keeps memory usage constant regardless of video length.

The Three-Stage Distillation Pipeline

The framework employs a robust three-stage pipeline to migrate the bidirectional teacher into a streaming student:

  1. Stage I: Bidirectional DMD: Uses Distribution Matching Distillation to compress the teacher's multi-step denoising into a few-step process while maintaining the original bidirectional attention.
  2. Stage II: Causal ODE Regression: Introduces the causal mask and the Audio Sink Token mechanism to stabilize training during the shift to autoregressive generation.
  3. Stage III: Joint Self-Forcing Distillation: Combats "exposure bias" (where small errors accumulate during long generations) by training the model to unroll its own predictions and self-correct against the teacher's distribution.

Experimental Results

OmniForcing was evaluated on JavisBench (for joint quality and synchrony) and VBench (for visual fidelity). The results demonstrate that the student model preserves the high quality of the teacher while drastically reducing latency.

Comparison between OmniForcing (Ours) and the LTX-2 teacher, showing a reduction in TTFC from 197s to 0.7s.
  • Speed & Latency: OmniForcing achieves ~25 FPS on a single NVIDIA A100/H100 GPU, reducing TTFC from ~197 seconds to just ~0.7 seconds.
  • Quality: On JavisBench, OmniForcing outperformed all non-teacher baselines in cross-modal coherence (AV-IB score of 0.269).
  • Fidelity: VBench results show that the distilled student actually slightly outperforms the teacher in aesthetic quality and subject consistency, likely due to the student learning a "smoother" version of the teacher's manifold.

Conclusion

OmniForcing represents a significant step toward real-time, interactive multi-modal AI. By identifying and solving the mathematical instabilities inherent in causal multi-modal attention, it enables high-fidelity video and audio generation with sub-second latency, opening the door for applications in live streaming, gaming, and interactive digital humans.

Summary updated

Abstract

Recent joint audio-visual diffusion models achieve remarkable generation quality but suffer from high latency due to their bidirectional attention dependencies, hindering real-time applications. We propose OmniForcing, the first framework to distill an offline, dual-stream bidirectional diffusion model into a high-fidelity streaming autoregressive generator. However, naively applying causal distillation to such dual-stream architectures triggers severe training instability, due to the extreme temporal asymmetry between modalities and the resulting token sparsity. We address the inherent information density gap by introducing an Asymmetric Block-Causal Alignment with a zero-truncation Global Prefix that prevents multi-modal synchronization drift. The gradient explosion caused by extreme audio token sparsity during the causal shift is further resolved through an Audio Sink Token mechanism equipped with an Identity RoPE constraint. Finally, a Joint Self-Forcing Distillation paradigm enables the model to dynamically self-correct cumulative cross-modal errors from exposure bias during long rollouts. Empowered by a modality-independent rolling KV-cache inference scheme, OmniForcing achieves state-of-the-art streaming generation at $\sim$25 FPS on a single GPU, maintaining multi-modal synchronization and visual quality on par with the bidirectional teacher.\textbf{Project Page:} \href{https://omniforcing.com}{https://omniforcing.com}
BibTeX
@misc{su2026omniforcing,
  author = {Yaofeng Su and Yuming Li and Zeyue Xue and Jie Huang and Siming Fu and Haoran Li and Ying Li and Zezhong Qian and Haoyang Huang and Nan Duan},
  title = {{OmniForcing: Unleashing Real-time Joint Audio-Visual Generation}},
  year = {2026},
  eprint = {2603.11647},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.11647},
}
Figure