TL;DR
Current text-to-audio-video (T2AV) models often appear consistent only because they independently follow highly detailed prompts, rather than genuinely coordinating between modalities. This paper introduces intrinsic jointness—the ability to align on details left unsaid—and proposes JointREPA, a representation alignment method that forces audio and video features to coordinate, significantly improving semantic consistency across multiple benchmarks.
What Problem Does This Paper Address?
When a model generates a video of a man playing an electric guitar accompanied by the correct audio, is it because the audio and video components are "talking" to each other? Or is it just because both components are independently following the detailed instruction "electric guitar"?
The authors argue that current T2AV evaluation is flawed because detailed prompts mask a lack of genuine cross-modal coordination. They define a new property called intrinsic jointness: the ability of audio and video to converge on the same realization for details left underspecified by a prompt. For example, if the prompt is simply "a man playing a musical instrument," a truly joint model should pick one instrument (e.g., a violin) and reflect it in both sight and sound. Currently, many models fail this test, generating a guitar in the video and a violin in the audio.
Methods
The researchers first diagnosed the problem using "attention knockout" (disabling cross-modal attention at inference) and internal probing. They found that even when models have the machinery for cross-modal interaction, they often underutilize it for semantic coordination.
To fix this, they introduced JointREPA, a lightweight post-training method based on representation alignment:
- Representation Alignment (REPA): Traditionally, this involves aligning a generative model's internal features with those of a high-quality pretrained encoder (like CLIP or CLAP) to improve semantic richness.
- The "Joint" Twist: JointREPA goes further by aligning each modality’s intermediate features with pretrained representations from both its own modality and the other modality.
- Implementation: They use LoRA (Low-Rank Adaptation) for efficient training, projecting hidden states from Diffusion Transformer (DiT) blocks to match targets from frozen encoders—CLAP for audio and V-JEPA for video. This forces the video tower to "know" what the audio should sound like and vice versa.
Datasets
- Training: A balanced subset of VGGSound, covering 52 classes.
- Evaluation Benchmarks: The authors tested on standard benchmarks like JavisBench-mini and Verse-Bench.
- UNSAIDBench: Most importantly, they introduced this new benchmark specifically designed to evaluate intrinsic jointness. It takes real-world prompts and abstracts specific entities into broader categories (e.g., changing "electric guitar" to "string instrument" or "musical instrument") to see if the model can still maintain consistency.
Key Insights and Ideas
- The Independence Trap: As prompts become more abstract (moving from specific "L3" labels like guitar to broad "L1" labels like musical instrument), semantic consistency in current models drops precipitously. At the L1 level, models often perform no better than random chance (shuffled pairs).
- Internal Disagreement: Probing shows that this inconsistency isn't just a final-layer fluke; it persists throughout the internal representations of the Transformer blocks.
- AV-Attention is Underutilized: In some model families (like JavisDiT), the pathway for audio-visual interaction exists but is largely ignored for semantic tasks, acting more like two independent models running in parallel.
- Cross-Modal Alignment as a Bridge: By explicitly training the video features to align with audio encoder targets (and vice versa), JointREPA provides the "signal" that the existing attention mechanisms were missing.
Key Results
JointREPA demonstrated consistent improvements in semantic consistency (measured by AV-IB and AVHScore) across different model architectures (LTX-2.3 and JavisDiT++):
- Quantitative Gains: On the new UNSAIDBench, JointREPA improved the "Agreement" score (how often audio and video classifiers agree on the generated content) from 0.11 to 0.17 at the most abstract prompt level.
- Qualitative Success: In one striking example, the base model generated a video of cattle but audio of a chicken clucking. After applying JointREPA, the model correctly synchronized cattle visuals with cattle vocalizations under the same underspecified "farm animal" prompt.
- Efficiency: Because it uses LoRA and aligns intermediate layers, JointREPA achieves results comparable to much more expensive reinforcement learning methods (like AV-GRPO) without needing complex reward functions.
Limitations and Future Work
The authors acknowledge that their current study focuses primarily on explicit semantic identities (like instrument types or animal species). They suggest that future work should extend the concept of intrinsic jointness to physical attributes—such as an object's material, size, or weight—which should naturally and jointly influence both its visual appearance and the sound it makes when struck or moved.
Ultimately, they envision "intrinsic jointness" becoming a core design principle, moving the field from "co-generation" (two models following one script) toward truly unified audio-video synthesis.
Summary updated