TL;DR
This diagnostic study reveals that "label-space expansion"—a popular strategy for multi-label facial expression recognition—suffers from a severe calibration failure during its aggregation step. While the models learn good internal representations, the common practice of summing branch outputs leads to wildly inconsistent predictions that vary drastically with training data scale, often performing no better than simple baselines.
What Problem Does This Paper Address?
Facial Expression Recognition (FER) in the wild is notoriously difficult due to two main factors: image quality degradation (blur, poor lighting) and coexisting emotions (e.g., a face showing both "surprise" and "fear").
A natural architectural response is "label-space expansion." Instead of just predicting 7 emotions, you predict 7 emotions across $N$ different conditions (like "low resolution" or "strong light"), creating a larger joint space. To get a final multi-label prediction, you sum the scores back together. While intuitive, the failure modes of this "sum-back" approach haven't been rigorously studied. This paper investigates Condition-Aware Label Expansion (CALE) to find out if this complexity actually pays off.
Methods
The authors evaluate CALE, a framework that encodes image quality directly into the label space.
- Architecture: Images pass through a shared backbone (like ResNet or EfficientNet) followed by a Squeeze-and-Excitation (SE) module to recalibrate channel features (see Figure 1).
- Expanded Output: The final layer produces a 28-dimensional vector (7 emotions $\times$ 4 quality conditions).
- Condition Classifier: A simple rule-based classifier uses brightness and Laplacian variance to identify the "main" condition of an image.
- Aggregation Rule: During inference, the scores are summed using a specific rule: the "main" condition branch gets full weight, while others get half weight. $$p_e = \sigma(x_{e,c^*}) + 0.5 \sum_{c \neq c^*} \sigma(x_{e,c})$$
- Diagnostic Focus: Rather than chasing a new State-of-the-Art (SOTA), the authors systematically vary the training data scale (10% to 100%) and backbones to see where the system breaks.
Datasets
- Self-built Dataset: 30,240 training images (7 emotions augmented across 4 conditions) and 3,460 independently collected multi-label test images.
- RAF-DB: A standard public benchmark for single-label FER.
- RAF-ML: A public benchmark for multi-label (blended) emotion recognition.
Key Insights and Ideas
- The "Aggregation vs. Representation" Paradox: The most striking finding is that the models actually learn to rank emotions quite well, but they fail at making decisions. At 10% training data, the average number of predicted labels varies by 28$\times$ across different backbones when using a fixed threshold. However, the threshold-free mean Average Precision (mAP) only varies by 1.28$\times$. This localizes the fault entirely to the summation step, not the learned features.
- Data-Scale Sensitivity: The aggregation rule is "miscalibrated" by design. Under limited data, the model over-predicts (scores are too high); as data increases, it collapses into systematic under-prediction.
- The SE Module Works: Visualization of the SE scale factors (Figure 2) confirms that the model successfully learns to selectively enhance specific channels for expression recognition while suppressing noise.
- Baselines are Hard to Beat: Simple 7-dimensional sigmoid baselines (using standard Binary Cross-Entropy or Asymmetric Loss) often match or exceed the performance of the much more complex CALE setup.
Key Results
- Single-Label Gains: On RAF-DB, CALE showed a nominal improvement (+1.29 Macro-F1), but this did not survive rigorous statistical significance testing (Benjamini–Hochberg correction).
- Multi-Label Failure: In multi-label tests, CALE's performance is highly unstable. Figure 3 shows how sensitive the model is to the choice of decision threshold; the "optimal" threshold shifts significantly depending on the backbone and data scale.
- Calibration Collapse: At full training data, the aggregate scores for all backbones collapsed to a narrow range, leading to near-zero label predictions at the standard 0.5 threshold.
Limitations and Future Work
The authors are refreshingly honest about the "negative" nature of their results:
- Rule-Based Fragility: The condition classifier is extremely sensitive. A 20% change in its internal thresholds can shift the dominant condition class by 29 percentage points, fundamentally changing the training objective.
- Structural Mismatch: The models were trained on single-label data but evaluated on multi-label data, which likely contributed to the calibration issues.
- Future Directions: The paper suggests moving away from fixed summation rules toward learned aggregation modules or performing post-hoc threshold calibration on held-out multi-label validation sets. They also recommend replacing rule-based condition classifiers with learnable, soft-label schemes.
Summary updated