Hard Sample-Aware Consistency for Low-Resolution Facial Expression Recognition

Bokyeung Lee, Kyungdeuk Ko, Jonghwan Hong, Hanseok KoWACV 2024

TL;DR

This paper tackles the challenge of recognizing facial expressions from low-resolution images, a common issue in real-world scenarios. They introduce a novel "hard sample-aware consistency" (HSAC) framework that combines attention consistency and label distribution learning to improve robustness, achieving state-of-the-art performance on the RAF-DB dataset.

What Problem Does This Paper Address?

Facial Expression Recognition (FER) is crucial for human-computer interaction, but its performance significantly degrades when dealing with low-resolution (LR) facial images. These LR images, often encountered due to camera limitations or distance, lack the fine details (like wrinkles) that are vital for accurate expression interpretation. While consistency constraints have been used to improve model robustness to input variations, they struggle with LR images because augmented samples can become too distorted for the network to extract meaningful features. This leads to an "overfitting problem" where the network learns to match identity features rather than expression features, especially for "hard samples" – those that are difficult to classify across different resolutions.

Methods

The core of the proposed solution is the Hard Sample-Aware Consistency (HSAC) framework. It builds upon existing consistency techniques but introduces two key innovations:

  • Combined Attention Consistency: Instead of directly aligning attention maps from LR images with high-resolution (HR) images, HSAC creates a "combined attention map." This map is a weighted average of the HR attention map and a horizontally flipped LR attention map. The LR attention map is then aligned with this combined map. This approach aims to provide a more balanced and robust target for attention alignment, preventing the network from overfitting to identity features by allowing a controlled distance between LR and HR attention maps, reflecting the degree of distortion.
  • Label Distribution Learning (LDL) for Easy Samples: LDL is used to handle ambiguity and noisy labels. In HSAC, LDL is applied only to "easy samples" (those not classified as hard). For these easy samples, the target distribution for the LR image is generated by combining the original one-hot label with the HR prediction, weighted by a similarity measure between the LR and HR predictions. This helps to refine the predictions for less challenging samples.
  • Hard Sample Identification: A crucial component is the method for identifying "hard samples." A sample is deemed hard if its predicted class from the LR image differs from both the predicted class from the HR image and the ground truth label. This criterion helps to selectively apply the loss functions, preventing the network from being overly constrained by these difficult samples.

The overall loss function combines a standard cross-entropy loss for HR images with the proposed combined attention consistency loss and label distribution learning loss, with the latter two being conditionally applied based on the hard sample identification.

Datasets

The paper evaluates their method on several datasets:

  • Synthetic Multi-Scale LR Datasets:
    • RAF-DB: A widely used "in-the-wild" dataset for FER.
  • AffectNet: Another large-scale "in-the-wild" dataset.
  • SFEW2.0: A dataset for spontaneous facial expression analysis.
  • Real LR Datasets:
    • CAER-S: Used to evaluate performance on real-world LR images.
  • FER+: Another dataset specifically for real LR scenarios.

The authors also mention using the MS-Celeb-1M dataset for pre-training the ResNet50 backbone.

Key Insights and Ideas

  • Hard samples are the bottleneck: The paper convincingly argues that "hard samples" are the primary reason for performance degradation in LR FER when using standard consistency methods. These samples confuse the network, leading it to learn identity rather than expression.
  • Balanced attention is key: Directly aligning LR attention maps to HR can be too aggressive. Creating a "combined" attention target that balances HR and flipped LR attention provides a more nuanced and effective learning signal.
  • Selective learning for hard samples: Not all samples are created equal. Applying sophisticated loss functions only to "easy" samples, while using a simpler approach for "hard" ones, prevents overfitting and improves generalization.
  • Hard samples reside at the periphery: Visualizations (t-SNE) show that their method successfully pushes hard samples to the outskirts of class distributions, leading to more discriminative feature representations.

Key Results

  • State-of-the-art performance: The proposed HSAC method achieved a new state-of-the-art accuracy of 90.97% on the original RAF-DB dataset.
  • Robustness across resolutions: HSAC demonstrated consistently high performance across various down-sampling factors (×2, ×4, ×8), outperforming baseline methods that showed significant performance drops at higher down-sampling ratios.
  • Superiority over baselines: HSAC consistently outperformed models using previous consistency constraints (like EAC+La) and other state-of-the-art methods on both synthetic and real LR datasets. For instance, it achieved a significant improvement over EAC+La on RAF-DB, AffectNet, and SFEW2.0.
  • Effective hard sample selection: The proposed hard sample selection criterion was shown to be more effective than simpler alternatives, leading to better accuracy.

Limitations and Future Work

  • Computational cost: While not explicitly detailed as a limitation, the multi-scale processing and attention mechanisms likely increase computational demands.
  • Hyperparameter sensitivity: The paper mentions empirically setting hyperparameters (Aca, λidl, λadv), suggesting that optimal performance might depend on careful tuning.
  • Future work: The authors imply that their framework's success in creating discriminative feature distributions and generalized models for multi-scale LR FER opens avenues for further research in this area. The focus on hard samples suggests potential for more targeted data augmentation or curriculum learning strategies.

Summary updated

BibTeX
@inproceedings{lee2024hard,
  author = {Bokyeung Lee and Kyungdeuk Ko and Jonghwan Hong and Hanseok Ko},
  title = {{Hard Sample-Aware Consistency for Low-Resolution Facial Expression Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure