Emotion recognition models increasingly predict a probability distribution over emotions for each image rather than a single label. The target distribution is built from many annotators, because people really do disagree about what a picture makes them feel. Real affective data is also imbalanced: a few emotions, such as joy, dominate the training mass, and others, such as fear or disgust, are rarely the main emotion. The standard fix for imbalance is to push predictions toward rare classes. That fix assumes a model's hesitation about a rare emotion comes from having seen too few examples. In affective data that assumption is often false. Many images near the boundary between, say, fear and sadness sit there because humans themselves split on them. Pushing every prediction toward the rare emotion repairs the first kind of error and creates a new one on the second: the model reports more fear than human observers actually feel.
This note proposes to measure that confusion directly and to apply rarity correction per image, in proportion to how much of the model's uncertainty is due to data scarcity rather than human ambiguity. Put briefly, a model should correct for the rarity of fear on a clearly frightening image it is unsure about, and should leave alone an image that people themselves rate as half fear, half sadness. The deliverables are a controlled diagnostic that tells these two error sources apart on real multi-annotator data, and a small per-image gate on prior correction. The gate is only worth keeping if it beats the much simpler uniform correction.
Core Claim
Closely related work has already framed imbalance in label distribution learning. Label distribution learning means training on, and predicting, a probability vector over labels instead of a single class. Xingyu Zhao et al. (2023) define imbalanced label distribution learning and show that training features cluster around majority labels. Their fix aligns feature and label latents (RDA) and adapts focal and class-balanced losses to soft targets. Xingyu Zhao et al. (DILDL) argue that inside each target vector, the large "dominant" entries drown the gradients of the small "non-dominant" entries, and they learn the two parts in separate branches. Both papers treat every gap between prediction and target as error that rebalancing should close. Neither asks whether that gap comes from scarcity or from genuine ambiguity.
Our claim is narrower and testable. On imbalanced emotion distributions, uniform rarity correction helps tail-emotion images that humans rate unambiguously. On the same tail emotions, it hurts images that humans rate as ambiguous, because it moves probability mass beyond the human distribution. A per-image gate driven by the ratio of scarcity-driven to ambiguity-driven uncertainty should recover both groups.
The motivating evidence is spread across the two literatures. Chi Zhang et al. (2025) and Haotian Deng et al. (2025) find that network decision boundaries for facial expressions coincide with regions of maximal human perceptual variance. Work on imbalance and neural collapse, the tendency of class features to shrink onto evenly spaced class means, finds a different effect: head classes compress tail regions, so boundaries end up too close to tail clusters for optimization reasons alone (Sun et al., Yao et al.). The same "near the boundary" observation therefore has two incompatible readings. Nobody has measured how much of each is present.
The constraining evidence is Bolian Li et al. (TLC). TLC uses evidential (Dirichlet-based) uncertainty to recruit extra experts for tail samples. It treats all uncertainty as missing knowledge, which is the conflation we target. Its evidential machinery is still a reasonable way to estimate the scarcity component, so we reuse that idea rather than reject it.
Research Program
The method has three parts. It is built on a standard backbone that outputs logits $z(x)$ over $K$ emotions.
The first part is prior removal with soft targets. Let $\pi$ be the soft label prior, the average of all training target distributions $d$. This prior measures each emotion's continuous mass, not a count of argmax labels. We train with a soft-target version of logit-adjusted cross-entropy, $\mathrm{KL}(d \,\|\, \mathrm{softmax}(z(x) + \log \pi))$. The idea of correcting through the class prior inside the softmax is the same one PaCo pairs with Balanced Softmax in Jiequan Cui et al. (2021). Training this way makes $z$ approximately free of the training prior.
The second part is two uncertainty estimates per image. Ambiguity, written $a(x)$, comes from a small head trained to predict the entropy of the human target distribution. That entropy is observed for every training image, so this is ordinary regression. The finding of De'Aira Bryant et al. (2022) that 5–6 annotators already give a stable distribution suggests the regression target is reliable on datasets with more raters than that. Scarcity, written $s(x)$, is the disagreement among a small ensemble of heads, measured by mutual information. We also try a density alternative: distance to the nearest training images, weighted by each emotion's mass.
The third part is the gated prediction. The test-time distribution is
$$ \hat p(x) = \mathrm{softmax}\big(z(x) + (1-\alpha(x))\,\log \pi\big), \qquad \alpha(x) = \sigma\big(w \log \tfrac{s(x)}{a(x)} + b\big), $$
where $\sigma$ is the logistic function and $w, b$ are two scalars fit on a validation split. When $\alpha \to 1$, the prior is fully removed, which is uniform correction. When $\alpha \to 0$, the training prior is restored, which is plain training. The gate has two parameters on purpose. If a two-parameter gate cannot beat uniform correction, a larger one is not the answer.
Alongside the method, the analysis is a two-axis controlled study on Emotion6, which has 15 ratings per image. One axis thins the training images whose dominant emotion is a chosen tail emotion, which varies scarcity. The other axis keeps the image count fixed and thins the annotators on test targets, which varies how noisily ambiguity is measured. Every metric is broken down by bins of human target entropy. This grid separates the two readings of boundary proximity. Under the optimization reading, error on tail images should fall as tail images are added, regardless of entropy bin. Under the ambiguity reading, error in the high-entropy bins should stay flat as tail images are added.
Baselines
The naive reference is a backbone trained with plain KL divergence on soft targets, with no imbalance handling. The naive-but-common reference is the argmax collapse: each target becomes its dominant emotion, and a strong imbalance method is trained on those hard labels, namely Seesaw loss from Jiaqi Wang et al. (2021). The hard-label model's softmax is scored as a distribution. This tests whether distributions are worth modelling at all under imbalance.
The strongest existing method is DILDL (Xingyu Zhao et al.). It is the only method in the file built for within-sample imbalance in label distributions, and it reports a KL divergence of 0.7393±0.0254 on its imbalanced Emotion6 split and a Chebyshev distance of 0.1752±0.0091 on Movie. RDA (Xingyu Zhao et al., 2023) reports 0.1962 Chebyshev on Movie. We use DILDL's split protocol if it can be reproduced, and otherwise rebuild the split and re-run DILDL ourselves rather than quote its number. All methods share the same backbone, the same epoch budget, and the same number of hyperparameter trials, chosen on one validation split. Beating DILDL means a lower mean KL over five seeds, with non-overlapping standard deviations, on at least four of six ILDL benchmarks, and no loss in the head-emotion entropy bins.
The key ablation is uniform soft-prior correction, the $\alpha \equiv 1$ case above. It isolates the gate. A second ablation sets $\alpha(x)$ from scarcity alone, ignoring ambiguity. This is what a TLC-style system would effectively do, and it tests whether the ambiguity term carries the gain. A third ablation feeds the true human entropy in place of $a(x)$, which gives an upper bound on what the gate can achieve.
Why not just…?
Why not just apply uniform logit adjustment with the soft prior? This is the real threat, and it is built into the design as the main ablation. If the claim is wrong, uniform correction will match the gated version in every entropy bin. In that case we drop the gating claim and report the simpler finding: a one-line soft-prior correction competes with DILDL. That would itself be useful, since the file contains no such comparison. The gate earns its place only if uniform correction measurably worsens the high-entropy tail bins while the gate does not.
Why not just reweight or resample tail emotions? For distributions, resampling an image chosen for a tail emotion also duplicates the head emotions it contains. Xingyu Zhao et al. (2023) report that soft-target focal and class-balanced losses still leave features skewed toward majority labels. We include class-balanced reweighting as a baseline, but evidence and mechanism both suggest it will behave like uniform correction applied less precisely.
Why not just collapse to dominant labels and use Seesaw or PaCo? On Emotion6, annotators disagree on the dominant emotion for more than half of the images (Kuan-Chuan Peng et al., 2015). The collapse therefore throws away exactly the signal the gate relies on. The argmax-plus-Seesaw baseline measures what that loss costs in distributional error.
Why not just tune a global softmax temperature after training? A single temperature spreads or sharpens every prediction equally. It cannot spread an ambiguous fear image while sharpening a clear one. The per-bin analysis will show whether a global temperature, fit on the same validation split, reaches the gated model's error. If it does, the per-image story fails.
Proposed Experiments
The main benchmarks are the six ILDL datasets used by DILDL: SCUT-FBP, Flickr-LDL, Movie, Emotion6, Natural Scene, and RAF-ML. We report KL divergence, Chebyshev distance, and intersection similarity. We also report the per-emotion-bin metrics described above, and the Spearman correlation between predicted and human target entropy, split into head and tail emotions. The controlled grid runs on Emotion6.
There are two secondary checks. The first uses the 1,000-image ExpW subset with Beta-fitted intensities from De'Aira Bryant et al. (2022) and asks whether the gate transfers to faces. The second uses ArtELingo (Youssef Mohamed et al., 2022) and asks whether ambiguity estimated from pooled annotators is really disagreement between cultures. We compute per-language target entropies and test whether the gate under-corrects images that are split across cultures but agreed upon within each culture. No reported distributional result on ArtELingo was found on Lacuna; the file gives only the 72.4% classification accuracy of XLM-ACE, so this check is exploratory.
The decisive check is the breakdown by entropy bin on Emotion6 and Flickr-LDL. The prediction is specific: uniform correction lowers KL in the lowest-entropy tail bin relative to plain training and raises it in the highest-entropy tail bin, while the gate is no worse than the better of the two in every bin. If the high-entropy bins show no degradation under uniform correction, the premise is false and the gate is dropped. If the gate driven by true human entropy helps but the learned $a(x)$ does not, then predicting ambiguity is the bottleneck, and that becomes the follow-up problem.
References
- Imbalanced Label Distribution Learning
- Decoupled Imbalanced Label Distribution Learning
- Synthesizing Images on Perceptual Boundaries of ANNs for Uncovering Human Perceptual Variability on Facial Expressions
- Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective
- Exploration of Class Center for Fine-Grained Visual Classification
- Trustworthy Long-Tailed Classification
- Parametric Contrastive Learning
- Multi-Dimensional, Nuanced and Subjective - Measuring the Perception of Facial Expressions
- Seesaw Loss for Long-Tailed Instance Segmentation
- A mixed bag of emotions: Model, predict, and transfer emotion distributions
- ArtELingo: A Million Emotion Annotations of WikiArt with Emphasis on Diversity over Language and Culture