CIViC: Evidence Gates for Audio-Conditioned Visual Prediction on Proxy Tasks

Mengzhe Geng, Max W. Y. Lam, Yuewen Cao, Shoukang Hu, Jing Xu, Marc Tessier, Xu Li

TL;DR

CIViC (Conditional Input and Visual Interpretation Checks) is a new diagnostic framework designed to verify if audio-conditioned visual models—like digital avatars—are actually using audio information as intended. By applying a rigorous four-gate protocol, the authors demonstrate that models often pass simple "audio removal" tests while failing to prove that the audio actually improves performance or provides specific control.

What Problem Does This Paper Address?

In the world of AI-generated video and talking heads, researchers often claim their models are "audio-driven." The standard proof is usually a simple ablation: remove the audio, show the output changes, and conclude the audio is "controlling" the visual.

However, the authors argue this is a low bar that leaves several "alternative explanations" open. A model might react to the absence of a signal without actually understanding the content of the audio. Or, it might be using "shortcuts"—non-audio cues like speaker identity or dialogue metadata—to predict motion, making the audio input redundant or even detrimental. The paper addresses the lack of a formal, falsifiable framework to distinguish genuine audio-conditioned prediction from these superficial activations.

Methods

The authors introduce CIViC, a protocol consisting of four "evidence gates" that a model must pass to support a specific interpretation:

  1. Activation: Does removing the input change the output? (The baseline check).
  2. Substitution: Does swapping the audio for a different, matched sample change the outcome in a specific way?
  3. Benefit: Does including the audio actually improve performance on a held-out task compared to a model without it?
  4. Endpoint Validity: Does a "shortcut" (like a condition-only model) perform better than the audio-informed model?
Figure 1: Four interpretation gates (A) and two selected diagnostic summaries (B). CREMA-D meets its registered fixed-pair detection rule but has no practical threshold. In the synthetic feature-mixture proxy, the condition changes predictions while early held-out error is higher; that gap shrinks with training. Its matched endpoint control is untested. The newer SpeakerVid proxy comparison is described in Section 5, not shown here. Color and text distinguish the gates.

Crucially, CIViC requires researchers to "register" their interpretation rules—fixing the data split, the intervention, and the success criteria—before scoring. This prevents "p-hacking" or cherry-picking metrics that happen to look favorable.

Datasets

The study evaluates the CIViC framework across several proxy tasks and datasets:

  • CREMA-D & RAVDESS: Standard actor-based audiovisual datasets used for "face-raster" (pixel-level) prediction checks.
  • AVA Active Speaker: Used for external boundary audits.
  • Synthetic Feature Mixture: A private dataset of 819 synthetic mixtures designed to test how models handle combined audio and "nuisance" signals.
  • SpeakerVid-5M: A large-scale dataset used for a new source-video-disjoint proxy study, focusing on dyadic (two-person) interactive human generation.

Key Insights and Ideas

  • Sensitivity $\neq$ Benefit: A model can be highly sensitive to audio (passing the Activation gate) but actually perform worse when audio is included compared to a "constant-audio" baseline (failing the Benefit gate).
  • The "Constant-Audio" Trap: In several tests on RAVDESS, models trained with a constant zero-audio signal actually achieved lower error than models trained with real audio. This suggests the audio was acting as noise rather than a helpful signal for that specific task.
  • Metric Conflict: In the SpeakerVid study, the authors found that zeroing out audio features actually reduced pixel-level error (RMSE) but increased "segment-drift" error. This highlights that "better" performance depends entirely on which "endpoint" or metric you choose to value.
  • Shortcut Warnings: The authors show that a simple "condition-only" model (using only metadata like who is speaking) can sometimes outperform complex audio-informed models, raising a red flag about whether the audio is doing any heavy lifting at all.
Table 2: Interpretation rules. A detection result excludes only the named alternative; practical use needs a task-scale rule.

Key Results

  • CREMA-D: While the model passed the activation gate, the actual effect of swapping audio was tiny (+0.000109 Mean Absolute Error), calling into question its practical relevance.
  • Synthetic Mixture: At 1,000 training steps, adding a structured condition actually increased error (0.1807 vs 0.1707 for audio-only). While this gap shrank with more training, it initially failed the benefit check.
  • SpeakerVid: A "five-feature-zero" model (where low-level audio was removed but identity/dialogue remained) achieved lower grayscale RMSE than the model with aligned audio across all tested hardware platforms (H100, A100, V100). The differences ranged from -0.0142 to -0.0216 RMSE.

Limitations and Future Work

The authors are careful to note that CIViC is a diagnostic framework for proxy tasks, not a universal evaluator for all video generators.

  • Proxy vs. Reality: The tests use "proxies" like grayscale rasters rather than full-resolution video. A model might fail a proxy check but still produce visually pleasing video (or vice versa).
  • Human Perception: The framework does not currently account for human perceptual quality or "naturalness," which remains the gold standard for generative media.
  • Scope: The paper does not establish if these four gates are sufficient for all possible audiovisual tasks, only that they provide necessary friction against over-interpreting simple ablation tests.

Future work will likely involve extending these "gates" to more complex, high-fidelity generative architectures and incorporating human-in-the-loop evaluation.

Summary updated

Abstract

Removing audio can shift a visual predictor's output without showing whether the task uses the intended audio information. We introduce Conditional Input and Visual Interpretation Checks (CIViC), a four-gate protocol that links an interpretation to an intervention, held-out task comparison, or endpoint control. We demonstrate CIViC on actor-disjoint face-raster tests, a synthetic audio-feature mixture, and source-video-disjoint SpeakerVid proxy tasks. In the static face-raster test, cross-actor audio replacement meets the registered detection rule with a small effect; on a separate temporal endpoint, constant-audio fits score better than audio-trained fits in four matched seeds. In the synthetic mixture, the condition-augmented fit has higher 1,000-step RMSE than the audio-only fit (0.1807 vs. 0.1707), with the gap shrinking during training. In the SpeakerVid proxy, the five-feature-zero fit (five low-level audio summaries zeroed; identity, dialogue, and condition inputs retained) has lower whole-frame grayscale RMSE than aligned audio across four held-out conditions and five paired seeds, while aligned audio has lower segment-drift error. The results show that output activation, held-out error, and endpoint controls support different interpretations across these proxy tasks.
BibTeX
@misc{geng2026civic,
  author = {Mengzhe Geng and Max W. Y. Lam and Yuewen Cao and Shoukang Hu and Jing Xu and Marc Tessier and Xu Li},
  title = {{CIViC: Evidence Gates for Audio-Conditioned Visual Prediction on Proxy Tasks}},
  year = {2026},
  eprint = {2610.15327},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.15327},
}
Figure