Beyond Co-Generation: Rethinking Jointness in Joint Audio-Video Generation

Jihoo Jung, Ji-Hoon Kim, Suyeon Lee, Gia Thinh Pham, Joon Son ChungarXiv 2026 · arXiv

TL;DR

Current text-to-audio-video (T2AV) models often appear consistent only because they independently follow highly detailed prompts, rather than genuinely coordinating between modalities. This paper introduces intrinsic jointness—the ability to align on details left unsaid—and proposes JointREPA, a representation alignment method that forces audio and video features to coordinate, significantly improving semantic consistency across multiple benchmarks.

What Problem Does This Paper Address?

When a model generates a video of a man playing an electric guitar accompanied by the correct audio, is it because the audio and video components are "talking" to each other? Or is it just because both components are independently following the detailed instruction "electric guitar"?

The authors argue that current T2AV evaluation is flawed because detailed prompts mask a lack of genuine cross-modal coordination. They define a new property called intrinsic jointness: the ability of audio and video to converge on the same realization for details left underspecified by a prompt. For example, if the prompt is simply "a man playing a musical instrument," a truly joint model should pick one instrument (e.g., a violin) and reflect it in both sight and sound. Currently, many models fail this test, generating a guitar in the video and a violin in the audio.

Methods

The researchers first diagnosed the problem using "attention knockout" (disabling cross-modal attention at inference) and internal probing. They found that even when models have the machinery for cross-modal interaction, they often underutilize it for semantic coordination.

To fix this, they introduced JointREPA, a lightweight post-training method based on representation alignment:

  • Representation Alignment (REPA): Traditionally, this involves aligning a generative model's internal features with those of a high-quality pretrained encoder (like CLIP or CLAP) to improve semantic richness.
  • The "Joint" Twist: JointREPA goes further by aligning each modality’s intermediate features with pretrained representations from both its own modality and the other modality.
  • Implementation: They use LoRA (Low-Rank Adaptation) for efficient training, projecting hidden states from Diffusion Transformer (DiT) blocks to match targets from frozen encoders—CLAP for audio and V-JEPA for video. This forces the video tower to "know" what the audio should sound like and vice versa.
Figure 1. Intrinsic jointness. Under detailed prompts, output consistency can arise from each modality independently following the text prompt alone, without audio–visual interaction. In contrast, underspecified prompts enable disentangling whether this consis- tency stems from mere prompt following or genuine audio–visual coordination. We define intrinsic jointness as the ability to maintain consistency under such underspecified prompts.

Datasets

  • Training: A balanced subset of VGGSound, covering 52 classes.
  • Evaluation Benchmarks: The authors tested on standard benchmarks like JavisBench-mini and Verse-Bench.
  • UNSAIDBench: Most importantly, they introduced this new benchmark specifically designed to evaluate intrinsic jointness. It takes real-world prompts and abstracts specific entities into broader categories (e.g., changing "electric guitar" to "string instrument" or "musical instrument") to see if the model can still maintain consistency.

Key Insights and Ideas

  • The Independence Trap: As prompts become more abstract (moving from specific "L3" labels like guitar to broad "L1" labels like musical instrument), semantic consistency in current models drops precipitously. At the L1 level, models often perform no better than random chance (shuffled pairs).
  • Internal Disagreement: Probing shows that this inconsistency isn't just a final-layer fluke; it persists throughout the internal representations of the Transformer blocks.
  • AV-Attention is Underutilized: In some model families (like JavisDiT), the pathway for audio-visual interaction exists but is largely ignored for semantic tasks, acting more like two independent models running in parallel.
  • Cross-Modal Alignment as a Bridge: By explicitly training the video features to align with audio encoder targets (and vice versa), JointREPA provides the "signal" that the existing attention mechanisms were missing.
Figure 2. Q1. Semantic jointness under varying prompt specificity. We evaluate semantic consistency using three metrics (Agreement, AV-IB, and AVHScore) across different levels of prompt specificity. All metrics generally decrease as prompt specificity decreases (L3 →L2 →L1). At L1, Agreement scores for most models are close to the shuffled-pair baseline (marked by ×), obtained by randomly pairing audio and video generated with different seeds under the same prompt. This suggests that current T2AV models exhibit limited semantic jointness.

Key Results

JointREPA demonstrated consistent improvements in semantic consistency (measured by AV-IB and AVHScore) across different model architectures (LTX-2.3 and JavisDiT++):

  • Quantitative Gains: On the new UNSAIDBench, JointREPA improved the "Agreement" score (how often audio and video classifiers agree on the generated content) from 0.11 to 0.17 at the most abstract prompt level.
  • Qualitative Success: In one striking example, the base model generated a video of cattle but audio of a chicken clucking. After applying JointREPA, the model correctly synchronized cattle visuals with cattle vocalizations under the same underspecified "farm animal" prompt.
  • Efficiency: Because it uses LoRA and aligns intermediate layers, JointREPA achieves results comparable to much more expensive reinforcement learning methods (like AV-GRPO) without needing complex reward functions.
Figure 6. Qualitative comparison showing JointREPA converging on a single identity (Cattle) vs. the base model's inconsistent output (Cattle video, Chicken audio).

Limitations and Future Work

The authors acknowledge that their current study focuses primarily on explicit semantic identities (like instrument types or animal species). They suggest that future work should extend the concept of intrinsic jointness to physical attributes—such as an object's material, size, or weight—which should naturally and jointly influence both its visual appearance and the sound it makes when struck or moved.

Ultimately, they envision "intrinsic jointness" becoming a core design principle, moving the field from "co-generation" (two models following one script) toward truly unified audio-video synthesis.

Summary updated

Abstract

Our aim is to advance the understanding of genuine audio-visual coordination in joint text-to-audio-video (T2AV) generation, with particular emphasis on semantic consistency. Evaluating audio-visual consistency in T2AV generation is nontrivial, as detailed text prompts obscure whether such consistency stems from genuine cross-modal coordination or merely from independent text following. To isolate the former, we introduce intrinsic jointness: the ability of audio and video to produce the same realization for details left underspecified by the prompt. Using semantically underspecified prompts (e.g., "musical instrument playing" rather than "electric guitar playing"), we investigate four recent models and reveal their limited intrinsic jointness. To address this limitation, we propose JointREPA, a simple representation alignment-based post-training method to facilitate cross-modal coordination. Since existing benchmarks use semantically detailed prompts, we introduce UNSAIDBench, a benchmark designed to evaluate intrinsic jointness under semantic underspecification. Experiments across two distinct models and three benchmarks, including UNSAIDBench, demonstrate that JointREPA consistently improves semantic consistency.
BibTeX
@misc{jung2026beyond,
  author = {Jihoo Jung and Ji-Hoon Kim and Suyeon Lee and Gia Thinh Pham and Joon Son Chung},
  title = {{Beyond Co-Generation: Rethinking Jointness in Joint Audio-Video Generation}},
  year = {2026},
  eprint = {2611.22857},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22857},
}
Figure