Aggregation-Induced Calibration Failure in Sum-Based Label Expansion for Multi-Label Facial Expression Recognition: A Diagnostic Study

Weihang YuanarXiv 2026 · arXiv

TL;DR

This diagnostic study reveals that "label-space expansion"—a popular strategy for multi-label facial expression recognition—suffers from a severe calibration failure during its aggregation step. While the models learn good internal representations, the common practice of summing branch outputs leads to wildly inconsistent predictions that vary drastically with training data scale, often performing no better than simple baselines.

What Problem Does This Paper Address?

Facial Expression Recognition (FER) in the wild is notoriously difficult due to two main factors: image quality degradation (blur, poor lighting) and coexisting emotions (e.g., a face showing both "surprise" and "fear").

A natural architectural response is "label-space expansion." Instead of just predicting 7 emotions, you predict 7 emotions across $N$ different conditions (like "low resolution" or "strong light"), creating a larger joint space. To get a final multi-label prediction, you sum the scores back together. While intuitive, the failure modes of this "sum-back" approach haven't been rigorously studied. This paper investigates Condition-Aware Label Expansion (CALE) to find out if this complexity actually pays off.

Methods

The authors evaluate CALE, a framework that encodes image quality directly into the label space.

  1. Architecture: Images pass through a shared backbone (like ResNet or EfficientNet) followed by a Squeeze-and-Excitation (SE) module to recalibrate channel features (see Figure 1).
  2. Expanded Output: The final layer produces a 28-dimensional vector (7 emotions $\times$ 4 quality conditions).
  3. Condition Classifier: A simple rule-based classifier uses brightness and Laplacian variance to identify the "main" condition of an image.
  4. Aggregation Rule: During inference, the scores are summed using a specific rule: the "main" condition branch gets full weight, while others get half weight. $$p_e = \sigma(x_{e,c^*}) + 0.5 \sum_{c \neq c^*} \sigma(x_{e,c})$$
  5. Diagnostic Focus: Rather than chasing a new State-of-the-Art (SOTA), the authors systematically vary the training data scale (10% to 100%) and backbones to see where the system breaks.
Fig. 1: Overall framework of the CALE method. After MTCNN alignment, the image is processed by a shared backbone with shallow layers frozen. A global average pooling layer and an SE module recalibrate channel features, followed by flatten and dropout. The 28-dimensional output is organized as four condition-specific 7-dimensional branches. Each branch is sigmoid-activated, and the four branches are aggregated per emotion to produce seven scores. A fixed threshold yields the final multi-label prediction. This figure was drawn by the author with TikZ.

Datasets

  • Self-built Dataset: 30,240 training images (7 emotions augmented across 4 conditions) and 3,460 independently collected multi-label test images.
  • RAF-DB: A standard public benchmark for single-label FER.
  • RAF-ML: A public benchmark for multi-label (blended) emotion recognition.

Key Insights and Ideas

  • The "Aggregation vs. Representation" Paradox: The most striking finding is that the models actually learn to rank emotions quite well, but they fail at making decisions. At 10% training data, the average number of predicted labels varies by 28$\times$ across different backbones when using a fixed threshold. However, the threshold-free mean Average Precision (mAP) only varies by 1.28$\times$. This localizes the fault entirely to the summation step, not the learned features.
  • Data-Scale Sensitivity: The aggregation rule is "miscalibrated" by design. Under limited data, the model over-predicts (scores are too high); as data increases, it collapses into systematic under-prediction.
  • The SE Module Works: Visualization of the SE scale factors (Figure 2) confirms that the model successfully learns to selectively enhance specific channels for expression recognition while suppressing noise.
  • Baselines are Hard to Beat: Simple 7-dimensional sigmoid baselines (using standard Binary Cross-Entropy or Asymmetric Loss) often match or exceed the performance of the much more complex CALE setup.
Fig. 2: Visualization of the 512-dimensional SE scale factors for a test image. Higher values indicate channels more strongly enhanced by the SE module.

Key Results

  • Single-Label Gains: On RAF-DB, CALE showed a nominal improvement (+1.29 Macro-F1), but this did not survive rigorous statistical significance testing (Benjamini–Hochberg correction).
  • Multi-Label Failure: In multi-label tests, CALE's performance is highly unstable. Figure 3 shows how sensitive the model is to the choice of decision threshold; the "optimal" threshold shifts significantly depending on the backbone and data scale.
  • Calibration Collapse: At full training data, the aggregate scores for all backbones collapsed to a narrow range, leading to near-zero label predictions at the standard 0.5 threshold.
Fig. 3: Threshold sensitivity on the multi-label test set. (a) Macro-F1 vs threshold. (b) UAR vs threshold. The CALE curve spans a wider threshold range because the aggregation score is not normalized, not because the model predicts more labels correctly.

Limitations and Future Work

The authors are refreshingly honest about the "negative" nature of their results:

  • Rule-Based Fragility: The condition classifier is extremely sensitive. A 20% change in its internal thresholds can shift the dominant condition class by 29 percentage points, fundamentally changing the training objective.
  • Structural Mismatch: The models were trained on single-label data but evaluated on multi-label data, which likely contributed to the calibration issues.
  • Future Directions: The paper suggests moving away from fixed summation rules toward learned aggregation modules or performing post-hoc threshold calibration on held-out multi-label validation sets. They also recommend replacing rule-based condition classifiers with learnable, soft-label schemes.

Summary updated

Abstract

Label-space expansion - predicting a larger joint space and summing back to the original labels - is a natural design for multi-label facial expression recognition, but its failure modes have not been characterised. We evaluate Condition-Aware Label Expansion (CALE) on a self-built dataset (30,240 training images, 3,460 multi-label test images), on seven backbones, and on the public RAF-DB and RAF-ML benchmarks. The central finding is that the aggregation rule, not the representation, decides whether the method works. With a fixed threshold of 0.5, the aggregate score spans 28x across five backbones at 10% of the training data (0.24 to 6.79) and collapses to a 1.9x range at full data, while threshold-free mAP stays within 1.28x over the same backbones, which localises the fault to the summation step. The mechanism is the uncalibrated aggregation rule, which adds the main-branch score to half the sum of the three non-main branches; we verify it by measuring main- and non-main-branch activations. Two plain seven-dimensional sigmoid baselines (BCE, ASL) reach mAP 0.5055 and 0.5175; ASL exceeds three of the five CALE configurations and falls within 0.03 of the best. On single-label FER the gain over a seven-dimensional baseline is nominally significant (paired t-test, p = 0.0436) but does not survive Benjamini-Hochberg correction (p_BH = 0.2055) or a Wilcoxon test (p = 0.1250). We conclude that sum-based label expansion requires threshold calibration at the deployment data scale, or a learned aggregation, before it can support multi-label decisions.
BibTeX
@misc{yuan2026aggregation,
  author = {Weihang Yuan},
  title = {{Aggregation-Induced Calibration Failure in Sum-Based Label Expansion for Multi-Label Facial Expression Recognition: A Diagnostic Study}},
  year = {2026},
  eprint = {2611.11843},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11843},
}
Figure