Robust Emotion Recognition Under Severe Class Imbalance

228 concepts177 papers2 proposals

Human emotion in daily life is not evenly distributed. In natural interactions, people spend the vast majority of their time in neutral or positive states; high-intensity emotional expressions like fear, disgust, or rage are remarkably rare. When researchers collect affective data "in the wild," this natural asymmetry creates a severe long-tail distribution.

As Attila Schulc et al. observe in Analysing Affective Behavior in the First ABAW 2020 Competition, common expressions like neutrality or happiness vastly outnumber infrequent states. This pattern repeats across modalities: it appears in large-scale speech corpora like LSSED (LSSED: a large-scale dataset and benchmark for speech emotion recognition), video datasets like Aff-Wild2 (Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace), and multi-party conversational datasets like MELD (MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations).

For machine learning models, this imbalance is catastrophic. Standard classification objectives are designed to maximize average accuracy across an entire dataset. If 90% of the training samples are neutral, a model can achieve 90% accuracy by simply guessing "neutral" every time. Consequently, the decision boundaries collapse around the majority classes, leaving the model blind to rare but critical emotional states.

In safety-critical or high-stakes applications, this failure mode is unacceptable. For instance, in virtual classrooms, the system developed by Javid Iqbal in Online Classroom Student Engagement Analysis using Enhanced YOLOv5 must accurately capture rare indicators of distress or anger to prevent cyberbullying. Similarly, in assistive driving systems (Upper-Body Hierarchical Graph for Skeleton Based Emotion Recognition in Assistive Driving), failing to identify a driver's sudden anxiety or rage compromises the entire utility of the safety intervention.

To overcome this bottleneck, recent research focuses on three core strategies: synthesizing the tail of the distribution, modifying the algorithmic loss landscape, and anchoring rare emotions with contextual scaffolding.


Synthesizing and Mixing the Tail

If the physical world does not provide enough examples of rare emotions, one intuitive solution is to engineer them. Rather than relying purely on scarce real-world data, researchers use generative models to populate the underrepresented tail of the distribution.

The authors of Synthetic Expressions are Better Than Real for Learning to Detect Facial Actions demonstrate that synthetic facial expressions generated via a 3D-aware Generative Adversarial Network (GAN) pipeline can actually outperform real images during training. By generating perfectly balanced datasets, they eliminate the rare-class bottleneck entirely. Similarly, Guoxian Song et al. bypass direct pixel generation in Conditional adversarial synthesis of 3D facial action units, choosing instead to synthesize 3D Morphable Model parameters. This allows the model to generate high-resolution, identity-preserving training data for rare muscle configurations (Action Units) that human actors struggle to replicate naturally.

[Imbalanced Real Data] ──> [Isolate Rare Action Units] ──> [3D Morphable Model / GAN]
                                                                   │
                                                                   ▼
[Downstream Classifier] <── [Balanced Training Set] <── [High-Res Synthetic Faces]

Data synthesis can also happen at the label level. In dynamic video recognition, the authors of MIDAS: Mixing Ambiguous Data with Soft Labels for Dynamic Facial Expression Recognition interpolate pairs of video clips and their corresponding labels. Instead of forcing a hard categorical choice, they generate "soft labels"—probabilistic distributions over multiple emotions. This smooths out the sharp, imbalanced boundaries between common and rare emotions, reflecting the natural ambiguity of human affect.

For text-based models, Xianda Zhou leverages millions of Twitter emoticons and emojis as "natural labels" in MojiTalk: Generating Emotional Responses at Scale. This massive, self-labeled corpus provides the volume of fine-grained, long-tail emotional expressions required to train conversational agents without relying on slow, expensive manual annotation.


Algorithmic Rebalancing and Feature Mining

When modifying the dataset is impossible, models must change how they learn from the existing skew. This requires adjusting the optimization landscape so that minority classes exert a stronger pull on the model's gradients.

One approach is to penalize the model more heavily when it misclassifies a rare emotion. The authors of MultiEMO: An Attention-Based Correlation-Aware Multimodal Fusion Framework for Emotion Recognition in Conversations introduce a Sample-Weighted Focal Contrastive loss. This loss dynamically scales based on class frequency and prediction confidence, forcing the network to prioritize easily confused and rare emotional states over the easily mastered majority classes.

A deeper intuition is that the features of rare emotions are often buried inside images of common ones. A face labeled "happy" might still contain subtle muscle contractions or micro-expressions relevant to surprise or fear.

Standard Learning:
[Happy Face Image] ───> [Cross-Entropy Loss] ───> Optimizes ONLY for "Happy"
                                                  (Rare features are ignored)

Deep Feature Mining:
                        ┌───> Majority Attention ───> Optimizes for "Happy"
[Happy Face Image] ────┤
                        └───> Re-balanced Attention ─> Mines hidden features
                                                      (e.g., subtle muscle tension 
                                                       shared with "Fear")

To exploit this hidden signal, Yaqi Li introduces a framework in Leave No Stone Unturned: Mine Extra Knowledge for Imbalanced Facial Expression Recognition. The proposed method uses a re-balanced attention consistency mechanism to actively mine features of rare emotions from images belonging to majority classes. By preventing the majority-class gradients from completely wiping out minority feature detectors, the model learns robust representations of the tail without requiring more tail examples.


Anchoring the Tail with Context and Self-Supervision

Isolated expressions of rare emotions are highly ambiguous. A scream can signify fear, surprise, or intense joy. However, when an expression is framed by conversational history, acoustic pitch, and semantic context, its true identity becomes clear. Providing models with structural context acts as a scaffolding that helps them identify rare emotions with fewer labeled examples.

Self-supervised pre-training provides a powerful baseline context. By masking and reconstructing audio-visual streams without using any emotion labels, frameworks like HiCMAE (HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition) and the work by Wei Zhang 0219 et al. in Multi-modal Facial Affective Analysis based on Masked Autoencoder learn the fundamental geometry of human faces and voices. When the model is later fine-tuned on an imbalanced dataset, it does not need to learn what a face looks like from scratch; it merely maps its rich, pre-existing structural representations to the rare labels.

External symbolic knowledge can also anchor these representations. The authors of Contextual Information and Commonsense Based Prompt for Emotion Recognition in Conversation use prompt-based learning to inject inferential commonsense knowledge directly into language models. Instead of relying on raw data frequency to learn emotional associations, the model uses explicit textual prompts to deduce rare emotional states based on the speaker's implied intent and situation.

This concept is expanded to multimodal settings by the authors of Multiple Knowledge-Enhanced Interactive Graph Network for Multimodal Conversational Emotion Recognition. They construct relational graphs where textual and visual "if-then" commonsense knowledge serve as explicit edges. This structured graph guides the model's attention, allowing it to successfully navigate the long tail of conversational affect by relying on logical human behavior patterns rather than raw label counts.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_affective_class_imbalance_cbfdb6d7,
  title = {affective class imbalance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/affective-class-imbalance/rdr_cbfdb6d729564b3a89accc3eadba591e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_affective_class_imbalance_cbfdb6d7_v1,
  title = {affective class imbalance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/affective-class-imbalance/rdr_cbfdb6d729564b3a89accc3eadba591e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake