Emotion recognition models are usually asked a single question: what does a viewer feel when looking at this photograph or painting? They are rarely asked a second one: which part of the image causes that feeling? The standard way to answer the second question is to cover a region and see how the prediction changes. For emotion, this test is flawed at its root. A black, gray, or blurred patch is not emotionally neutral. Darkness and murk are themselves cues that the affective literature links to fear and sadness, so the occluder adds a feeling while it removes content. We propose to fill covered regions with what a masked autoencoder predicts should be there instead. A masked autoencoder is a Vision Transformer (a network that splits an image into square patches and lets every patch attend to every other) that was pre-trained to redraw hidden patches from the visible ones. Its fills are plausible, match the surrounding palette and lighting, and regress toward typical content. That makes them a far better stand-in for "this region, but without its specific content."
With a neutral fill in hand, we can measure patch-level emotional triggers and test them against human explanations. We can also use them as a training signal. A classifier should keep its prediction when non-trigger regions are re-imagined, and should change it when trigger regions are. This targets two known failures in visual emotion recognition: models that lean on dataset artifacts instead of content (Towards Unbiased Visual Emotion Recognition via Causal Intervention), and multimodal models that invent visual evidence to justify an emotion (EmotionHallucer, ProFocus).
Core Claim
There are two claims, and the first matters more. First, masked-autoencoder inpainting is a better counterfactual than conventional occlusion for attributing evoked emotion to image regions. It should produce trigger maps that agree more closely with the objects humans name when they explain their feelings. It should also produce smaller spurious emotion shifts when irrelevant regions are masked.
Second, a consistency penalty built on these counterfactuals reduces shortcut learning. It should improve cross-dataset transfer of emotion classifiers without costing in-domain accuracy.
The mechanism also yields a useful side effect. Because the fill inherits global color and lighting from visible patches, masking a region removes local content while largely preserving global atmosphere. Attribution by inpainting therefore separates two kinds of cause: object-bound triggers (a "scary dog") and atmospheric ones (a gloomy palette). The latter can be probed separately with global color edits.
Relation to Prior Work
Masked image modeling supplies the method. The standard pipeline masks most patches and has a light decoder reconstruct them from a Vision Transformer encoder (Masked Image Modeling: A Survey). Work building on it has so far treated reconstruction as a pre-training pretext. Two examples are pixel targets (Masked Vision-language Transformer in Fashion) and discrete-token targets (BEiT). We instead use the trained reconstructor at analysis and fine-tuning time as a counterfactual generator.
The closest work to ours is a group of three papers that adapt masked modeling to affect or perception:
- MART masks the most emotional temporal segments of subtitled videos.
- Multi-modal Facial Affective Analysis based on Masked Autoencoder pre-trains an MAE (masked autoencoder) on cropped faces for expressed emotion.
- QPT-V2 designs degradations so that masked modeling becomes sensitive to quality and aesthetics.
All three change the pre-training objective. None uses inpainting to ask which spatial region of a static scene causes an evoked emotion. The obvious spatial analogue of MART, "mask the emotional patches during pre-training," needs a way to know which patches are emotional. Our measurement step supplies that. We deliberately do not claim a new pre-training recipe.
The affective literature motivates the problem. Two surveys (Affective Image Content Analysis: Two Decades Review and New Perspectives, Affective Image Content Analysis: A Comprehensive Survey) and EmoSet argue that emotion is carried by object–attribute combinations and by low-level atmosphere such as brightness and colorfulness. That is exactly why gray or black occlusion is a biased probe.
The causal and grounding lines constrain what we must beat. IERN removes context confounders by backdoor adjustment over learned confounder clusters on a ResNet-50. We attack the same shortcut problem with per-image counterfactuals instead of dataset-level strata. ProFocus and EEmo-Logic localize affect through language-guided attention or chain-of-thought in large multimodal models. Both inherit hallucination risk from language priors. Our attribution never consults a language model.
Research Program
The measurement. Let $p(x)$ be a frozen emotion classifier's distribution over emotion classes for image $x$. Let $\tilde{x}_S$ be the image with patch set $S$ replaced by a pre-trained MAE's reconstruction, with visible patches left untouched. The affective effect of $S$ is
$$\Delta(S) = \mathrm{KL}\big(p(x)\,\|\,p(\tilde{x}_S)\big),$$
which measures how much the predicted emotion distribution moves when the content of $S$ is re-imagined.
Computing $\Delta$ for every single patch is expensive, and single patches are too small to hold objects. We therefore sample a few hundred random masks at about 25% coverage and compute $\Delta$ for each. We then regress the effects onto patch indicators to obtain a per-patch trigger map. This is the same logic as randomized-mask saliency, with inpainting replacing the black fill. The same procedure run with gray, black, blur, and mean-color fills gives the comparison occluders.
We separately apply global color-transfer and brightness edits that preserve layout. These give an atmosphere score, which EmoSet's brightness and colorfulness attributes let us sanity-check.
The neutrality check. Before trusting any map, we test whether the MAE fill is actually closer to neutral than the alternatives. The test uses masks placed on regions that human explanations never mention. Under each fill, we measure the average emotion shift and, more tellingly, the direction of that shift along valence (pleasant versus unpleasant).
Our prediction is that dark fills push predictions toward negative emotions and MAE fills do not. Blurry reconstructions could themselves read as gloomy. To guard against that, we repeat the analysis with an MAE whose reconstructions are sharpened by a feature-level target, in the style of A Unified View of Masked Image Modeling. We also report the residual shift of each.
The training signal. During fine-tuning on an emotion dataset, we compute trigger maps with the current model every few epochs. We add a counterfactual consistency term to the cross-entropy loss:
$$\mathcal{L} = \mathcal{L}_{\mathrm{CE}} + \lambda\, \mathrm{KL}\big(p(x)\,\|\,p(\tilde{x}_{\bar{T}})\big),$$
where $\bar{T}$ is a sampled set of low-trigger patches and $\lambda$ weights the penalty. The term tells the model that re-imagining irrelevant content should not change its answer. This is the pressure that discourages latching onto background artifacts. We do not add a term forcing change on trigger regions in the main method, because it risks rewarding brittle behavior. It appears only as an ablation.
Baselines
The naive reference is a standard ImageNet-pretrained ViT-Base fine-tuned with cross-entropy on each emotion dataset. For classification, PACL reports 62.08% for this backbone on FI, and we treat that as the floor. For attribution, the naive reference is gray-patch occlusion on the same classifier.
The strongest accuracy competitor on in-the-wild photos is PACL-enhanced ViT-Base, reported at 64.77% eight-class accuracy on FI in PACL. On EmoSet, the attribute-module model reaches 78.40% eight-class accuracy (EmoSet). We will start from the released PACL-style initialization where available, or reproduce it on TumEmo otherwise. We add our consistency term with the same data, epochs, and a matched hyperparameter search of equal trial count.
Our accuracy claim is modest. We expect to match within noise in-domain. We claim a win only on cross-dataset transfer.
For cross-dataset robustness, the reference is IERN. It reports an 8.18% gain over curriculum learning when training on Deep Sentiment and testing on Emotion-6, but the file gives no absolute accuracy for that setting. We will re-run IERN's backdoor adjustment on the same ViT backbone so that the only difference is confounder clustering versus inpainting consistency. Beating it means higher Emotion-6 and EmoSet accuracy when trained on FI, averaged over three seeds with non-overlapping intervals.
For attribution, the competitors are four methods on the identical classifier:
- attention rollout over the Vision Transformer's patch attention;
- gradient-weighted class activation maps;
- occlusion with black, gray, blur, and mean-color fills;
- trigger regions named by a multimodal model in the style of EEmo-Logic, prompted to list evidence and mapped to boxes.
We found no reported result on Lacuna for patch-level affective attribution on Affection or ArtEmis. All attribution numbers will therefore be produced in this study under a fixed protocol. We count a win as a pointing-game improvement over the best occlusion fill that holds across both positive-valence and negative-valence images, not only on negative ones.
The key ablation isolates the fill. We run the identical pipeline, same masks and same regression, with only the fill swapped. We do this for measurement and for the consistency loss alike. If gray-fill consistency training works as well, the inpainting is not the active ingredient.
Why not just…?
Why not just read the attention maps? Attention weights show where information flows, not what changes the output. To Sink or Not to Sink documents that patch and special tokens act as attention sinks that soak up weight without carrying content. We include attention rollout as a baseline. If it matches our pointing-game scores at a fraction of the cost, we will say the counterfactual machinery is unnecessary for localization, and the remaining contribution is the neutrality finding.
Why not just occlude with a blur or the mean color, which is cheaper? This is the real threat. The whole proposal rests on the premise that conventional fills inject affect. The neutrality check tests that premise directly by measuring the valence direction of shifts under each fill on unmentioned regions. If the mean-color fill turns out to be as neutral as the MAE fill, we will adopt it and report that the concern was overstated.
Why not just ask a multimodal language model which objects cause the emotion? Such models generate fluent justifications. Yet EmotionHallucer and ProFocus's own error analysis show them fabricating evidence, such as a "crying" baby that is not present. EEmo-Logic notes difficulty separating direct from environmental triggers. Our comparison measures this directly: the model's named regions are scored on the same pointing game, and we count how often it names objects absent from the image's annotations.
Why not just fine-tune a larger CLIP backbone with PACL and accept its accuracy? For accuracy on FI, that may well win. PACL reports 96.06% on the two-class FI variant with a CLIP ViT. We do not claim to beat it in-domain. Our claims concern where the model looks and whether it transfers. We therefore also apply our consistency term on top of the PACL backbone. If PACL alone already transfers as well and grounds as well, the training half of this proposal fails, and we will report that outcome.
Proposed Experiments
Human-grounded trigger evaluation. The main evaluation uses the Affection dataset. Its roughly 85,000 images come from MS-COCO and Visual Genome, which provide object annotations. Its 500,000 explanations name what caused the feeling. We match explanation nouns to annotated objects to obtain human-mentioned regions. We then score each attribution method with two measures: a pointing game (does the peak of the map land in a mentioned object?) and a deletion curve (how fast does the emotion prediction fall as top-ranked patches are inpainted?).
Two stratifications guard against confounds. Affection is 71.3% positive, so we report results separately by valence. Human majority agreement is 67.5%, so we also split results by annotator agreement. On ArtEmis, where there are no boxes and majority agreement is only 45.6%, we run a smaller human study. Raters choose which of two maps better matches their own stated reason.
The neutrality test. We measure the valence shift under each fill on regions no explanation mentions, using FI, EmoSet, and Affection. The decisive outcome is that dark and blur fills push predictions measurably toward negative emotions while the MAE fill stays near zero. If this fails, the central premise fails.
Atmosphere versus object. Using EmoSet's brightness and colorfulness attributes, we test the separation claim. For images whose annotated emotion correlates with low-level attributes, the global color edits should explain more of the prediction than local inpainting does. For images dominated by object and action attributes, the reverse should hold.
Transfer. We fine-tune on FI with and without the consistency term, then test zero-shot on Emotion-6 and EmoSet. We compare against IERN and PACL as described in the Baselines section. We also test the reverse direction, EmoSet to FI.
Two decisive checks. Together these settle whether the idea earns its place. The first is whether MAE-fill attribution beats every conventional fill on the Affection pointing game in both valence strata. The second is whether gray-fill consistency training falls short of MAE-fill consistency training on cross-dataset accuracy. If both hold, inpainting is established as the right counterfactual for affect. If neither holds, the result is a clean negative finding about a tempting idea.
References
- Towards Unbiased Visual Emotion Recognition via Causal Intervention
- EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Models
- ProFocus: Interpreting Affective Experience in Artistic Images with Progressive Visual Focusing
- Masked Image Modeling: A Survey
- Masked Vision-language Transformer in Fashion
- BEiT: BERT Pre-Training of Image Transformers
- MART: Masked Affective RepresenTation Learning via Masked Temporal Distribution Distillation
- Multi-modal Facial Affective Analysis based on Masked Autoencoder
- QPT-V2: Masked Image Modeling Advances Visual Scoring
- Affective Image Content Analysis: Two Decades Review and New Perspectives
- Affective Image Content Analysis: A Comprehensive Survey
- EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributes
- EEmo-Logic: A Unified Dataset and Multi-Stage Framework for Comprehensive Image-Evoked Emotion Assessment
- A Unified View of Masked Image Modeling
- Bridging Visual Affective Gap: Borrowing Textual Knowledge by Learning from Noisy Image-Text Pairs
- To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models
- Affection: Learning Affective Explanations for Real-World Visual Data
- ArtEmis: Affective Language for Visual Art