Emotion recognition systems that look at the whole scene, not only the face, are more accurate. A smile at a funeral and a smile at a party mean different things, and roughly a quarter of people in real photographs have faces that are not clearly visible. Scene context also creates a known weakness: models learn that "outdoor wedding" means "happy" and then ignore a crying guest. The current fix trains a second branch that sees only the background and subtracts its prediction at inference, treating everything the scene alone predicts as bias. This note argues that this framing is wrong for deployed systems. What the scene predicts by itself is neither pure bias nor a fixed truth. It is a prior over emotions specific to an environment, and it changes when the system moves from a sitcom set to a hospital corridor or a classroom. The parts of the model that read faces and bodies should stay fixed. The scene prior should be re-estimated without labels for each environment the system meets, and stored so that it can be retrieved when that environment comes back.
The resulting method is small. A source-trained subject branch outputs a posterior over emotions given the person. A scene-embedding memory groups incoming frames into environments, and each environment keeps an emotion prior that is updated online by expectation-maximization on unlabeled predictions. Because no network weights change at test time, the model cannot forget earlier environments through gradient overwriting. New environments add entries to the memory rather than taking space from old ones, which avoids the shrinking-capacity problem of orthogonal-subspace continual learning. The research question is whether this prior-only adaptation beats feature-level continual test-time adaptation on situational affect streams. It should, if domain shift in context-aware emotion recognition is mostly a shift in which emotions occur where, and much less a shift in how faces look.
Core Claim
Two results in the literature pull against each other. Robust Emotion Recognition in Context Debiasing (CLEF) trains a context-only branch on images with the subject blacked out and subtracts its "direct effect" at inference, so all context-only predictive power counts as harmful. VEATIC, a video benchmark for continuous valence and arousal tracking, reports that context-only models often outperform character-only models on valence, so context is frequently the main signal. Context De-confounded Emotion Recognition (CCIM) takes a middle position. It averages subject features over a fixed dictionary of scene "confounder" atoms, clustered offline from Places365 features, through backdoor adjustment. That dictionary cannot grow, and the authors report over-correction on small subsets.
Our claim is that both findings hold, and both point to one deployment problem. Within a single environment, the scene-conditional emotion distribution is real, useful information. Across environments that distribution changes, and a prior learned on one set of environments becomes a shortcut on another. The right operation is therefore to replace the source prior with one estimated for the current environment, not to subtract it and not to average over a frozen dictionary. Formally, let $q_s(y\mid s)$ be the subject branch's posterior over emotion $y$ given subject features $s$, trained under source label distribution $\pi_0(y)$. Let $e(c)$ be the environment that scene features $c$ are assigned to, and let $\pi_{e}(y)$ be that environment's current prior. The adapted prediction is
$$ p(y \mid s, c) \;\propto\; q_s(y \mid s)\,\frac{\pi_{e(c)}(y)}{\pi_0(y)} . $$
This is the standard correction for a change in class proportions. It is exact when the shift between environments acts only through which emotions occur, while the appearance of an emotion given its label stays the same. How far real deployments depart from that assumption is the empirical question this proposal is built to answer. For multi-label EMOTIC, the same correction is applied per category with Bernoulli priors.
Research Program
The source model is a two-stream network in the style of Context-Aware Emotion Recognition Networks (CAER-Net). CAER-Net encodes the face and, separately, the scene with the face masked out, then fuses them with learned attention weights. We reuse its face masking so that the subject branch and the scene branch carry separable information. We train the subject branch to produce $q_s$ and use the scene branch only as an embedding for routing.
At deployment, each frame's scene embedding is matched to an online environment memory. We use Places365-initialized features, which CCIM showed capture emotionally relevant scene structure. A frame joins the nearest existing environment if its distance falls under a threshold calibrated on source data; otherwise it opens a new entry initialized at $\pi_0$. Each entry's prior is updated by streaming expectation-maximization. The adapted posteriors of recent frames re-estimate $\pi_e$, which is smoothed toward $\pi_0$ by a Dirichlet pseudo-count $\alpha$, so that a handful of frames cannot drive a prior to a single class.
This guards against the class collapse that CLIPTTA documents for entropy-based test-time adaptation. It also removes the feedback loop behind that collapse: the features never change, so a wrong pseudo-label distorts only one environment's prior and can be undone by later evidence. Memory grows linearly with the number of environments and holds a single probability vector per entry. This is far cheaper than the 32-augmentation teacher-student pipeline of Continual Test-Time Domain Adaptation (CoTTA).
Three analyses go with the method. The first is a diagnostic that decides whether the premise holds. Using ground-truth labels, we decompose the source-to-target accuracy drop into the part recovered by oracle per-environment priors and the part that remains, which reflects changes in how subjects look. The second studies sensitivity to the routing threshold and to $\alpha$, the two quantities that trade adaptation speed against stability. The third asks whether adding feature adaptation on top of prior re-estimation helps or hurts. For this we stack vision-side low-rank adapters trained at test time, as in LoRA-TTT, which leaves continual, non-reset test-time training as an open problem.
Baselines
The naive reference is the source-trained two-stream model applied with no adaptation. It shows how much environment shift actually costs. We also report the subject-only branch with $\pi_0$ to show what context contributes before adaptation.
The strongest offline debiasing competitors are CLEF and CCIM, each applied to the same source model. On static CAER-S, Robust Emotion Recognition in Context Debiasing reports 88.41% accuracy for CD-Net with CLEF. Context De-confounded Emotion Recognition reports up to 91.17% on CAER-S and 39.13% mAP on EMOTIC. These figures come from standard in-distribution splits and not from our held-out-environment streams, so they serve to check that our reimplementations are faithful. They are not targets under shift.
The strongest continual test-time adaptation competitor is CoTTA. It combines a weight-averaged teacher, augmentation-averaged pseudo-labels, and stochastic restoration of 1% of weights to source values. It reports 16.2% error on CIFAR10-C against 20.4% for batch-norm adaptation and 20.7% for continual TENT, which updates normalization parameters by minimizing prediction entropy. No reported result was found on Lacuna for CoTTA on any emotion recognition benchmark. We also run TENT-continual as the lightweight gradient baseline.
The strongest training-free competitor is the dual-cache adapter of Efficient Test-Time Adaptation of Vision-Language Models (TDA). TDA keeps positive and negative feature caches of confident test samples, and we apply it to the fused feature. No reported result was found on Lacuna for TDA on affect data.
All methods share one source checkpoint, one backbone, one stream order, and one unlabeled validation stream built from source-held-out shows. Each gets the same number of hyperparameter configurations, chosen on that stream. Per-frame compute is reported alongside accuracy.
Two ablations isolate the key ingredient. The first, global prior re-estimation, uses a single $\pi$ for the whole stream. The second, random environment routing, keeps the per-environment bank but assigns frames to environments arbitrarily. The method beats the field if three things hold at once. Its online accuracy on held-out environments must exceed CoTTA and TDA. Its accuracy on revisited environments must not fall below its first-visit accuracy. It must beat global re-estimation by a margin larger than the seed-to-seed spread.
Why not just…?
Why not just run CoTTA on the two-stream network? CoTTA was built for streams of synthetic corruptions and driving segmentation, where shift lives in pixel statistics. In context-aware emotion recognition, much of the shift plausibly lives in which emotions co-occur with which scenes. Entropy-driven updates to scene features then risk turning the dominant class of the current environment into a shortcut, which is the collapse described in CLIPTTA. CoTTA also needs a teacher model and many augmented passes per frame. This is a genuine threat if environment shift turns out to be mainly about appearance. The oracle decomposition in our diagnostic, together with the feature-plus-prior stacking experiment, settles it directly.
Why not just use a training-free cache like TDA? TDA stores features but never corrects the label prior, and its authors report degradation once caches exceed roughly three to six samples per class because wrong pseudo-labels accumulate. A single global FIFO cache also evicts old environments, so revisits start from scratch. Our revisit metric tests exactly this.
Why not just debias offline with CLEF or CCIM and skip adaptation? Offline debiasing fixes how much the model trusts context at training time. VEATIC's result that context often carries the valence signal suggests that subtracting context throws information away in environments where the prior is informative. CCIM's frozen dictionary cannot represent scenes it has not seen. If CLEF matches our method on held-out-environment streams, the premise that deployment priors differ is false, and we would report that.
Why not just re-estimate one global label prior? This is the most serious alternative, since it is our method without the environment memory. It will win if each emotion's frequency drifts uniformly across the stream rather than per environment. The global-prior ablation and the per-show label-distribution diagnostic decide this. If per-environment priors do not beat the global one, we will narrow the contribution to the claim that affect shift is mostly label shift.
Why not just use orthogonal low-rank continual learning such as InfLoRA? InfLoRA reaches 71.01% on 20-task ImageNet-R. It does so by projecting updates away from earlier tasks' features, but it needs labels and task boundaries. Its authors also note that the free subspace shrinks as tasks accumulate, which would quickly run out under the open-ended variety of real scenes. Our memory costs one vector per environment and needs no labels.
Proposed Experiments
The main testbed is CAER (13,201 clips from 79 TV shows) and its static version CAER-S. Each show is a natural environment with its own sets, lighting, and emotional register. We train on a subset of shows and stream the held-out shows in sequence, with each show recurring several times. This mirrors the ten-cycle recurring protocol that CoTTA used on Cityscapes-to-ACDC.
For EMOTIC we form environments from the dataset's source collections (MSCOCO, ADE20k, web) and from Places365 scene clusters, scoring with mean average precision. VEATIC provides a continuous-valence stream across movies, documentaries, and home videos, scored by concordance correlation coefficient. Its fully informed model reports a valence coefficient of 0.6678 and an arousal coefficient of 0.6296 in-distribution. For the continuous case, the prior becomes an environment-specific shift in mean and variance of the predicted ratings.
The metrics are online accuracy (or mAP or concordance) over the stream, accuracy on each revisit compared with the first visit, accuracy on source shows after the stream ends to measure forgetting, and per-frame latency. Accuracy is also reported on a "bias-conflict" subset, frames whose label disagrees with the environment's majority emotion. This subset shows whether adaptation reinforces shortcuts.
The decisive checks are three, run in order. The first comes before any method comparison. On CAER we measure, with ground-truth labels, how much the per-show emotion distributions differ and how much of the shift gap oracle per-show priors recover. If that fraction is small, the core claim is false and the project turns to subject-side shift. The second test is whether our method beats global prior re-estimation and random routing on online and revisit accuracy, which is what separates environment memory from simple label-shift correction. The third test is whether its bias-conflict accuracy stays at or above CLEF's. This shows that substituting a local prior did not reintroduce the shortcut that CLEF and CCIM were designed to remove.
References
- Robust Emotion Recognition in Context Debiasing
- VEATIC: Video-based Emotion and Affect Tracking in Context Dataset
- Context De-Confounded Emotion Recognition
- Context-Aware Emotion Recognition Networks
- CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation
- Continual Test-Time Domain Adaptation
- LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Models
- Efficient Test-Time Adaptation of Vision-Language Models
- InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning