AURA: AUdio-dRiven streaming Avatar

Nathaniel Cohen, Nicolas Dufour, Amélie Royer, Alasdair Newson, Patrick PérezarXiv 2026 · arXiv

TL;DR

AURA is a new family of audio-driven talking-head models designed for real-time streaming on a single GPU. By distilling massive bidirectional diffusion models into efficient causal students, the authors achieve up to 23x speedups, enabling high-quality, low-latency virtual avatars that respond to live audio without needing to "look ahead" at future speech.

What Problem Does This Paper Address?

State-of-the-art talking-head models, like TalkVerse or Wan-S2V, produce stunningly realistic video but are fundamentally ill-suited for live interaction. They typically rely on bidirectional attention (needing to see the entire audio clip at once) and require dozens of diffusion denoising steps to generate a single frame. On a high-end GPU, these models can take minutes to generate just a few seconds of video.

For applications like voice assistants or interactive NPCs, we need "streaming" generation: the model must produce video chunk-by-chunk as the audio arrives, with minimal latency and a constant memory footprint regardless of how long the conversation lasts.

Methods

The authors introduce AURA (AUdio-dRiven streaming Avatar), which transforms a 50-step bidirectional "teacher" model into a 4-step causal "student." Their approach hinges on three main innovations:

  1. Two-Stage Distillation:
    • Stage 1 (ODE Teacher Forcing): The student is trained to match the teacher’s sampling trajectories, aligning the new causal architecture with the teacher's knowledge.
  2. Stage 2 (DMD with Backward Simulation): Using Distribution Matching Distillation (DMD), the authors close the quality gap. Crucially, they use "backward simulation," training the student on its own generated (and potentially noisy) outputs to ensure it remains stable during long-form inference.
  3. Per-Chunk Mixed Timesteps: In standard diffusion, a KV cache would need to store data for every denoising step, which is memory-prohibitive. AURA trains the model to handle chunks at different noise levels simultaneously. This allows for a single, shared KV cache across all denoising steps, drastically reducing memory overhead.
  4. Streaming Architecture: AURA uses a rolling KV cache with a "first-chunk anchor" (an attention sink) to maintain long-term consistency. To prevent the model from "forgetting" the reference identity over time, they implement a RoPE (Rotary Positional Embedding) re-indexing scheme that keeps the reference image at a stable relative position to the generation front.
The two-stage distillation pipeline: Stage 1 aligns trajectories, while Stage 2 uses DMD and backward simulation to refine quality.

Datasets

The authors found existing datasets (like Hallo3 or CelebV-HQ) were either too short or difficult to access for distillation. Consequently, they released a new Synthetic Talking-Head Dataset:

  • 30,000 clips (38.7 hours) of high-resolution video.
  • Generated using the TalkVerse teacher model.
  • Includes precomputed latent ODE trajectories to help other researchers train their own distilled models.

For evaluation, they used the HALLO3 TEST and CELEBV-HQ TEST sets, re-pairing images with longer audio tracks to test streaming stability.

Key Insights and Ideas

  • The Efficiency of Causality: AURA proves that you don't need to see the future of an audio signal to generate realistic lip-sync. By enforcing causality during training, the model learns to react to audio features as they happen.
  • Decoupling Memory from Denoising: The "per-chunk mixed timesteps" idea is a major "aha" moment. It solves the primary bottleneck of streaming diffusion by ensuring that the memory cost of the KV cache does not scale with the number of denoising steps.
  • Stability via Anchoring: Long-form generation in autoregressive models often suffers from "identity drift," where the face slowly morphs into someone else. AURA's use of a fixed first-chunk anchor and periodic RoPE re-indexing effectively tethers the generation to the original reference image.

Key Results

AURA achieves a remarkable balance between visual quality and generation speed, as shown in the quality-speed trade-off plot.

Figure 1: AURA streams talking-head video chunk by chunk on a single GPU: AURA 5B at 720p and 9.7 fps, AURA 1.3B at 576p in real time (20.7 fps for 16 Hz playback). (Left) Two reference identities and a frame generated by each model at t = 4 s, from the same ref- erence image and audio. (Right) Quality–speed trade-off: FVD averaged over the HALLO3 TEST and CELEBV-HQ TEST sets versus end-to-end time to generate 10 s of video on one H100 (log scale, lower is better). Circle areas are proportional to parameter counts, and the shaded band marks real-time generation. Video results can be found at kyutai.org/aura.
  • AURA 1.3B: Streams 576p video at 20.7 fps on a single H100 GPU. This is faster than its 16 Hz playback rate, making it truly real-time. It represents a 23x speedup over its teacher.
  • AURA 5B: Streams 720p video at 9.7 fps, a 17x speedup over TalkVerse.
  • Quality: Despite using only 4 steps, AURA matches or exceeds the visual quality (FVD/JEDi metrics) of its 50-step teacher. In human preference studies, AURA was consistently preferred over other open-source baselines like Hallo3.
Qualitative results showing AURA 5B and 1.3B maintaining high fidelity across different identities and lighting conditions.

Limitations and Future Work

  • Real-time 720p: While the 1.3B model is real-time, the higher-quality 5B model currently runs at ~10 fps. Further optimization is needed to hit 25+ fps at 720p on a single GPU.
  • Extreme Long-form Drift: While stable for a minute, the 5B model still shows slight identity drift in very long sequences (CSIM dropping from 0.76 to 0.69 over 60 seconds).
  • Synthetic Reliance: The model is distilled from a specific teacher. Improving the teacher or training on larger-scale real-world video trajectories could further push the quality ceiling.

AURA represents a significant step toward making realistic, AI-driven digital humans a standard part of real-time interactive applications.

Summary updated

Abstract

Current audio-driven video diffusion models can synthesize realistic talking-head videos but generally require dozens of diffusion denoising steps over multi-second video chunks, making them unsuitable for streaming applications such as voice assistants and virtual avatars, where low-latency and online causal generation are required properties. AURA generates video chunk by chunk with a rolling KV cache, requiring only the audio of the current chunk to start generating, rather than a multi-second audio look-ahead like bidirectional models. It comes in two sizes: AURA 1.3B streams 576p video in real time on a single GPU (20.7 fps for a 16 Hz playback rate), making it suited to interactive avatar applications, and AURA 5B streams 720p video at 9.7 fps. Our two-stage distillation pipeline first aligns the causal student with the teacher, by training it to match the teacher's trajectories with fewer diffusion steps and a chunk-wise sliding window causal attention mask, then applies Distribution Matching Distillation (DMD) with backward simulation to close the remaining quality gap. On standard talking-head benchmarks, AURA 1.3B runs 23x faster than its own bidirectional teacher and reaches a quality comparable to TalkVerse at a lower resolution, and AURA 5B matches its TalkVerse teacher at 720p while running 17x faster. To support reproducibility and further research on streaming avatar generation, we release the weights of both models along with a dataset of 30K synthetic videos and their precomputed teacher trajectories.
BibTeX
@misc{cohen2026aura,
  author = {Nathaniel Cohen and Nicolas Dufour and Amélie Royer and Alasdair Newson and Patrick Pérez},
  title = {{AURA: AUdio-dRiven streaming Avatar}},
  year = {2026},
  eprint = {2611.26540},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26540},
}
Figure