Systems that track how a person's emotion changes over a video usually run a face model on every frame. Each prediction is two numbers: valence (how pleasant or unpleasant the feeling is) and arousal (how energized it is). Running a face model on every frame is expensive, and it rules out long or streaming video. Video question-answering already handles this cost by picking a small set of informative frames. Emotion tracking has a property those frame pickers ignore: the cheap audio stream is dense and reliable for arousal, while the costly visual stream matters mostly for valence and mostly at the moments when the expression changes. We propose to let the audio, which is processed at every instant anyway, decide when the expensive face model should look. Between the frames it does look at, the emotion trajectory is filled in by moving along the circle of emotion categories rather than cutting straight across it.
The proposal makes two claims. The engineering claim is that, at a fixed small budget of visual frames, an audio-scheduled selector recovers most of the accuracy of dense multimodal tracking, especially on rare intense episodes. The scientific claim is that part of a reported contradiction in the field comes from how sparsely the video was sampled. On controlled data, audio dominates arousal and the face dominates valence. On in-the-wild Aff-Wild2, the reverse has been reported. Varying visual frame rate in a controlled way can test whether that reversal is about modality or about sampling.
Core Claim
The prior work splits into three groups, and each lacks a piece. Frame selectors such as Adaptive Keyframe Sampling score frames by similarity to a text query and balance that against temporal coverage. Flexible Frame Selection learns which frames to keep through a differentiable discrete choice. Both are visual-only and query-driven. Continuous-emotion models such as Multi-modal Conditional Attention Fusion and Multi-modal Emotion Estimation for in-the-wild Videos exploit the split between channels but process every frame densely. Circumplex-aware models treat emotion as an angle (which emotion) plus an intensity (how strong): MAVEN predicts emotion this way, and CAGE uses the circumplex to guide expression recognition on still images. Neither handles sparse sampling.
Our contribution is the combination these groups lack. A selector driven by the dense channel spends visual frames where the dense channel cannot settle the answer. A gap-filling model then uses the geometry of the emotion space to decide what happens between the chosen frames.
The geometric part is narrow and concrete. Write the emotional state as $z_t = I_t(\cos\theta_t, \sin\theta_t)$, where $I_t$ is intensity and $\theta_t$ is the angle on the circumplex. Suppose two selected keyframes $a$ and $b$ disagree in valence. Linear interpolation in valence–arousal coordinates draws a straight line between them. That line passes near the origin, so the filled-in frames are predicted close to neutral even when the audio says arousal never dropped.
The neutral pull matters because Aff-Wild2 labels are already heavily skewed toward neutral, and the ABAW2 technical report identifies that skew as a main failure mode. Straight-line gap filling makes the skew worse. We instead interpolate angle and intensity separately:
$$\theta(t) = \theta_a + w(t)\,\mathrm{Log}_{\theta_a}(\theta_b), \qquad I(t) = g_\phi(\text{audio}_{a:t}).$$
Each term is defined as follows. $\mathrm{Log}_{\theta_a}(\theta_b)$ is the signed shortest-arc angle difference on the circle, which is the circle's logarithm map. $w(t)\in[0,1]$ is a monotone warp computed from the cumulative audio change since keyframe $a$, so the angle moves when the voice changes and not uniformly in time. $g_\phi$ is a small audio network that predicts intensity, consistent with the evidence that acoustic cues set arousal (Chen & Jin report audio-only arousal CCC of 0.787 versus 0.432 for video on AVEC 2015).
We do not claim that heavy Riemannian optimizer machinery is needed for a one-dimensional circle. Projecting gradients onto the circle's tangent line, of the kind formalized in Geometry Aware Constrained Optimization, is used for the angle head and compared directly against plain Euclidean regression. We expect the benefit, if any, to come from interpolation geometry rather than from the optimizer, and the experiments are designed to tell those two sources apart.
Research Program
The system has three parts.
The first is a dense, cheap audio encoder. It is a pretrained speech model of the HuBERT or wav2vec 2.0 type, as used in MAVEN and in Meng et al. It runs at every timestep and outputs arousal, an audio-only valence guess, and an uncertainty for each.
The second is the scheduler. Given a budget of $K$ visual frames per window, it picks frames according to three signals: the audio model's valence uncertainty, the magnitude of audio change, and a very low-resolution motion score from a pooled fast pathway in the style of SlowFast-LLaVA. It is trained end-to-end with the same straight-through discrete selection as Flexible Frame Selection. That choice is deliberate: the only difference from the strongest visual-only learned selector is what the scheduler conditions on.
The third is a full-resolution face model and fusion head, run only on the selected frames. Gap filling between selected frames uses the equation above.
Two analyses accompany the system. The first is a frame-rate sweep for modality dominance. Unimodal visual models are retrained on Aff-Wild2 and AVEC 2015 at visual rates from full frame rate down to well under one frame per second. For each dimension and rate we record audio-only and visual-only CCC (Concordance Correlation Coefficient, which rewards both correlation and correct scale and offset). The question is whether the arousal/valence dominance ordering on Aff-Wild2 moves toward the controlled-data ordering as visual sampling becomes denser.
The audio pipeline is a confound here. MAVEN's audio-only CCC of 0.0299 on Aff-Wild2 is far below what the AVEC literature would suggest, so its "inversion" may partly reflect a weak audio encoder. We therefore hold one strong audio encoder fixed across all runs.
The second analysis evaluates on the tail. Mean CCC is dominated by near-neutral frames, so we also report CCC and error on segments where labeled intensity is in the top decile or where the angle changes sharply.
Baselines
The naive reference is uniform sparse sampling: $K$ evenly spaced face frames per window, fused with the same dense audio, with linear interpolation in valence–arousal space between them. It uses exactly our encoders and budget, so any gain over it comes from scheduling and gap filling.
The upper reference is dense processing. The strongest number in the file is the ensemble of Multi-modal Emotion Estimation for in-the-wild Videos, which reports validation CCC of 65.55% for valence and 70.88% for arousal on Aff-Wild2. A single-model reference is the ABAW2 technical report, with validation CCC of 0.591 for valence and 0.617 for arousal. We will reproduce a dense single model with our encoders, because the ensemble's number is not compute-matched. Success means coming within a small margin of our own dense model while running the face encoder on a small fraction of frames.
The strongest competitor on the core question is a visual-only learned selector in the style of Flexible Frame Selection. It is trained on the same data, with the same face encoder, the same budget, and the same tuning sweep as ours. The only change is that its selector sees low-resolution frames instead of audio features. The file contains no reported result on Lacuna for any frame selector on Aff-Wild2. The reported FFS figure is 62.2% versus 61.6% for 32-frame uniform sampling on NExT-QA, a different task. Beating this baseline means higher CCC at equal $K$, and especially higher tail CCC.
We also run Adaptive Keyframe Sampling with a fixed emotion-describing text query. This tests whether a training-free query-relevance selector is already enough.
Two ablations isolate the ingredients. The first swaps audio-driven gap filling on the circle for linear valence–arousal interpolation while keeping the scheduler. The second swaps tangent-projected angle training for plain Euclidean regression of valence and arousal while keeping the interpolation. If only the first ablation hurts, the geometry earns its place through interpolation and not optimization, and the paper will say so.
Why not just…?
Why not just let audio predict arousal and run a uniform face sampler for valence? This is the real threat, and it is simply our system with a trivial scheduler. The uniform-sparse baseline is exactly this design. If audio-driven scheduling does not beat it on valence CCC and tail CCC at equal budget, the scheduling idea fails, and we would report the result as evidence that valence changes are not predictable from audio.
Why not just use a bigger visual backbone on uniform frames? In video QA, Adaptive Keyframe Sampling reports a 7B model with adaptive selection reaching 62.7% on LongVideoBench, above a uniformly sampled 72B model. SPORTU and UAL-Bench show that missing transient frames is not fixed by capacity or more data. Brief expression changes are the emotional analogue. We test it directly by including a larger face encoder at uniform $K$ in the sweep.
Why not just learn a visual-only selector, since FFS already learns what to drop? A visual-only selector must look at frames, even low-resolution ones, to decide which to drop. Audio is already paid for and carries the arousal signal. The matched FFS-style baseline settles whether the audio conditioning matters.
Why not just interpolate linearly and smooth afterwards? Meng et al. note that moving-average smoothing cannot recover fast shifts. Straight-line interpolation also systematically drags gap frames toward neutral. The interpolation ablation measures this, and the effect should appear specifically on high-intensity segments with valence reversals. If it does not appear there, the geometric component is dropped.
Proposed Experiments
The main experiment is on Aff-Wild2. Using the official splits, we sweep visual budgets equivalent to roughly 8, 4, 2, 1 and 0.5 face frames per second against our dense model and against all baselines above. We report valence CCC, arousal CCC, tail-segment CCC, and the number of face-encoder calls.
The modality-dominance sweep runs on Aff-Wild2 and AVEC 2015 with a fixed audio encoder. We also train on AffectNet with the CAGE-style multi-task loss only to pretrain the face encoder, so that face quality is not a variable.
Three outcomes would decide the proposal. The first is whether audio-scheduled selection beats both uniform-sparse and the visual-only learned selector at budgets of two frames per second or fewer. The second is whether the Aff-Wild2 modality inversion shrinks as visual rate rises; a flat curve would mean the inversion is about the data, not about sampling. The third is whether circle-based interpolation beats linear interpolation specifically on intense valence-reversal segments. A positive first result with negative second and third results would still be a useful and honest finding: cheap audio scheduling works, but neither sampling nor geometry explains the dominance contradiction.
References
- Adaptive Keyframe Sampling for Long Video Understanding
- Flexible Frame Selection for Efficient Video Reasoning
- Multi-modal Conditional Attention Fusion for Dimensional Emotion Prediction
- Multi-modal Emotion Estimation for in-the-wild Videos
- MAVEN: Multi-modal Attention for Valence-Arousal Emotion Network
- CAGE: Circumplex Affect Guided Expression Inference
- Technical Report for Valence-Arousal Estimation in ABAW2 Challenge
- Geometry Aware Constrained Optimization Techniques for Deep Learning
- SlowFast-LLaVA: A strong training-free baseline for video large language models
- SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models
- UAL-Bench: The First Comprehensive Unusual Activity Localization Benchmark