ASM: Adaptive Sample Mining for In-The-Wild Facial Expression Recognition

Ziyang Zhang, Xiao Sun, Liuwei An, Meng Wang

Abstract

Recognizing facial expressions in real-world, "in-the-wild" datasets is challenging due to inherent label ambiguity and noise. This paper introduces Adaptive Sample Mining (ASM), a framework that dynamically partitions datasets into clean, ambiguous, and noisy subsets. By applying class-adaptive thresholds and a tri-regularization strategy, ASM effectively leverages useful information from difficult samples while mitigating the impact of incorrect labels. The method achieves state-of-the-art performance on major benchmarks, including RAF-DB, FERPlus, and AffectNet.

Introduction

Facial expression recognition (FER) in uncontrolled environments suffers from two primary issues: ambiguity and noisy labels. Ambiguity arises when a face displays a mixture of emotions (e.g., fear and surprise), while noisy labels occur due to human error or disagreement during annotation.

Existing models often treat all "hard-to-learn" samples as noise, which is suboptimal. Ambiguous samples, though difficult, contain critical information about decision boundaries. Conversely, truly noisy samples (incorrectly labeled) can mislead the model if not handled properly. Furthermore, the difficulty of learning varies across categories; for instance, "Happiness" is typically easier for models to learn than "Disgust." A uniform filtering approach fails to account for this class-wise variance.

Proposed Method: Adaptive Sample Mining

The ASM framework utilizes a dual-network co-training architecture to categorize and process samples through three integrated modules:

  1. Adaptive Threshold Learning: Instead of a fixed global threshold, ASM calculates a "clean" threshold ($T_c$) and a "noisy" threshold ($T_n$) for each specific expression category. These thresholds adapt based on the model's confidence levels, accounting for the varying difficulty of different emotions.
  2. Sample Mining: By averaging confidence scores from two independent networks, the framework partitions the dataset into three distinct subsets: Clean (high confidence), Ambiguous (mid-confidence, likely near decision boundaries), and Noisy (low confidence, likely mislabeled).
  3. Tri-Regularization: Tailored loss functions are applied to each subset to maximize learning efficiency:
  4. Supervised Learning: Standard cross-entropy is used for clean samples.
  5. Mutual Learning: For ambiguous samples, the two networks guide each other via a symmetric KL-divergence loss, preventing over-memorization of potential noise while capturing complex features.
  6. Unsupervised Consistency Learning: For noisy samples, labels are ignored. The model instead enforces consistency between different augmentations of the same image, learning robust features without being misled by incorrect labels.
The ASM pipeline illustrates the categorization of samples into clean, ambiguous, and noisy subsets, followed by the application of specific loss functions in the Tri-regularization module.

Experimental Setup

The researchers evaluated ASM on three prominent "in-the-wild" FER benchmarks:

  • RAF-DB: 12,271 training images across seven basic expressions.
  • FERPlus: An extension of FER2013 containing 28,709 training images.
  • AffectNet: The largest available FER dataset, with over 280,000 training images.

To test robustness, the authors also conducted experiments with synthetic noise levels of 10%, 20%, and 30% injected into these datasets.

Key Insights and Analysis

A core contribution of this work is the distinction between ambiguity and noise. As visualized in the t-SNE plot below, clean samples form tight clusters, while ambiguous samples reside near the boundaries between different emotion clusters. Noisy samples are often found far from their assigned clusters, justifying the need for unsupervised treatment.

t-SNE visualization of facial expression features, demonstrating how clean samples cluster centrally while ambiguous samples occupy boundary regions and noisy samples appear as outliers.

By implementing class-adaptive thresholds, the model ensures fairness across categories. It avoids unfairly discarding samples from harder-to-learn classes, which improves the overall balance and accuracy of the system.

Results

ASM consistently outperformed previous state-of-the-art (SOTA) methods across all tested benchmarks:

  • RAF-DB: Achieved a top accuracy of 90.58%, surpassing methods like EAC (89.99%) and RUL (88.98%).
  • FERPlus: Reached 90.21% accuracy.
  • AffectNet: Achieved 62.36% on the 8-class task and 65.68% on the 7-class task.
  • Noise Robustness: In scenarios with 30% synthetic noise, ASM outperformed the baseline by over 10% on RAF-DB, demonstrating its superior ability to filter harmful data while retaining valuable information from difficult samples.

Limitations and Future Work

Despite its success, the ASM framework has some limitations. The dual-network co-training approach doubles the computational and memory requirements during the training phase. Additionally, the model's performance depends on several hyper-parameters that balance the tri-regularization losses, which may require tuning for new datasets. Future research could explore scaling ASM with larger architectures, such as Vision Transformers (ViTs), to further enhance feature representation.

Summary updated

Abstract

Given the similarity between facial expression categories, the presence of compound facial expressions, and the subjectivity of annotators, facial expression recognition (FER) datasets often suffer from ambiguity and noisy labels. Ambiguous expressions are challenging to differentiate from expressions with noisy labels, which hurt the robustness of FER models. Furthermore, the difficulty of recognition varies across different expression categories, rendering a uniform approach unfair for all expressions. In this paper, we introduce a novel approach called Adaptive Sample Mining (ASM) to dynamically address ambiguity and noise within each expression category. First, the Adaptive Threshold Learning module generates two thresholds, namely the clean and noisy thresholds, for each category. These thresholds are based on the mean class probabilities at each training epoch. Next, the Sample Mining module partitions the dataset into three subsets: clean, ambiguity, and noise, by comparing the sample confidence with the clean and noisy thresholds. Finally, the Tri-Regularization module employs a mutual learning strategy for the ambiguity subset to enhance discrimination ability, and an unsupervised learning strategy for the noise subset to mitigate the impact of noisy labels. Extensive experiments prove that our method can effectively mine both ambiguity and noise, and outperform SOTA methods on both synthetic noisy and original datasets. The supplement material is available at https://github.com/zzzzzzyang/ASM.
BibTeX
@inproceedings{zhang2023asm,
  author = {Ziyang Zhang and Xiao Sun and Liuwei An and Meng Wang},
  title = {{ASM: Adaptive Sample Mining for In-The-Wild Facial Expression Recognition}},
  booktitle = {PRCV},
  year = {2023},
  url = {https://openreview.net/forum?id=To7Oy5H7h8},
}

Loading related content...

Figure