LION: Label Disambiguation for Semi-supervised Facial Expression Recognition with Progressive Negative Learning

Zhongjing Du, Xu Jiang, Peng Wang, Qizheng Zhou, Xi Wu, Jiliu Zhou, Yan WangIJCAI 2023 · DOI

Introduction

Facial Expression Recognition (FER) in real-world scenarios is a complex challenge for computer vision. While deep learning has significantly advanced the field, these models typically require vast amounts of accurately labeled data. In "in-the-wild" datasets, semi-supervised learning (SSL) faces two primary hurdles: label ambiguity and the underutilization of unlabeled data.

An example of label ambiguity where the given label (Surprise) differs from the latent truth (Fear), leading to low model confidence.

Human emotions are inherently subjective; a labeler might categorize a face as "Surprise," while others see "Fear." Standard SSL methods often treat initial labels as absolute ground truth, which can propagate errors. Furthermore, most SSL techniques, such as pseudo-labeling, only utilize unlabeled images when the model is highly confident (e.g., >95%). This results in a significant loss of information from "difficult" or low-confidence images that are typically discarded.

The LION Framework

The authors propose LION (Label DIsambiguation and PrOgressive Negative Learning) to address these issues by cleaning labeled data and extracting insights from uncertain unlabeled samples.

Label DIsambiguation (LD) Module

The LD module audits the labeled dataset to distinguish between "clear" and "ambiguous" data. Clear data—where the model's prediction aligns with the provided label with high confidence—is used to calculate "prototypes," which serve as feature-space anchors for each emotion.

A high-confidence 'clear' sample where the model correctly identifies 'Anger', providing a reliable anchor for feature prototypes.

Ambiguous images are assigned a candidate label set (a multi-hot vector). By leveraging the established prototypes and a moving-average weight strategy, the model iteratively "re-labels" these images toward their most likely latent truth, effectively correcting the noise in the original supervision.

PrOgressive Negative Learning (PNL) Module

For unlabeled images where the model remains uncertain, LION employs Negative Learning. Rather than forcing a potentially incorrect "positive" label (e.g., "This is Happy"), the PNL module identifies "complementary" labels (e.g., "This is definitely NOT Disgust").

Through iterative mining, the model progressively rules out classes with the lowest probabilities. To ensure robustness, a Negative Consistency Loss (NC-Loss) penalizes the model if its negative predictions for different augmented views of the same image are inconsistent. This allows the model to learn from low-confidence data without the risk of introducing incorrect pseudo-labels.

Experimental Results

LION was evaluated on three benchmark datasets: RAF-DB, AffectNet, and SFEW. The framework achieved state-of-the-art (SOTA) results, demonstrating particular strength in low-data regimes:

  • Minimal Data Performance: On RAF-DB with only 100 labeled images, LION achieved 67.83% accuracy, outperforming the previous best method (Ada-CM) by 5.47%.
  • Challenging Environments: On the difficult SFEW dataset (400 labels), LION reached 54.18%, a 3.73% improvement over the runner-up.
  • Baseline Comparison: In several instances, LION's semi-supervised approach outperformed fully-supervised baselines, suggesting that its method of handling ambiguity is more effective than simply increasing the volume of potentially noisy labels.

Conclusion and Future Work

The core insight of LION is the "Power of No." In emotion recognition, boundaries between similar expressions like "Anger" and "Disgust" are often blurry, but it is much easier for a model to determine what an expression is not. By combining dynamic relabeling with progressive negative learning, LION creates a mutually reinforcing system where better features lead to more accurate prototypes and vice versa.

Future research may address the computational complexity introduced by the iterative negative learning loops and the sensitivity of the model to its various thresholds (τ, μ, γ, δ). However, the success of LION suggests that this framework could be broadly applicable to other domains characterized by high label noise and ambiguity.

Summary updated

BibTeX
@inproceedings{du2023lion,
  author = {Zhongjing Du and Xu Jiang and Peng Wang and Qizheng Zhou and Xi Wu and Jiliu Zhou and Yan Wang},
  title = {{LION: Label Disambiguation for Semi-supervised Facial Expression Recognition with Progressive Negative Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/78},
}
Figure