Uncertainty-Aware Label Refinement on Hypergraphs for Personalized Federated Facial Expression Recognition

Hu Ding, Yan Yan, Yang Lu, Jing-Hao Xue, Hanzi Wang

Abstract

Facial Expression Recognition (FER) in real-world scenarios is often hampered by ambiguous expressions, low-quality images, and noisy labels. While Federated Learning (FL) offers a privacy-preserving solution, standard approaches struggle with heterogeneous sample uncertainty across different clients. We propose AMY (uncertainty-Aware label refineMent on hYpergraphs), a framework that leverages hypergraph neural networks to capture high-order relationships between facial images. AMY identifies unreliable samples and refines their labels locally using hypergraph label propagation. Experimental results on RAF-DB and FERPlus demonstrate that AMY significantly outperforms state-of-the-art decentralized models, particularly in highly heterogeneous data environments.

Introduction

Facial Expression Recognition is a complex task due to the inherent subjectivity of human emotions and the prevalence of noisy data. Traditional centralized training poses significant privacy risks regarding sensitive facial data. Federated Learning (FL) addresses this by training models locally on user devices. However, a significant "gap" remains: heterogeneous sample uncertainty. In a federated network, data quality varies wildly between clients—one may have clear, well-labeled images, while another has blurry photos or inconsistent labeling. Existing methods often rely on pairwise image relationships, failing to capture the complex, high-order clusters of expressions found in real-world datasets.

Methodology: The AMY Framework

The AMY framework introduces a personalized federated learning setup where each client maintains a local model designed to handle uncertainty through three primary components:

  1. Uncertainty Estimation (UE) Block: This block employs a Hypergraph Neural Network (HGNN). Unlike standard graphs that model pairwise connections, hypergraphs use "hyperedges" to connect multiple vertices, allowing the model to capture high-order relationships and more effectively identify outlier (uncertain) samples.
  2. Personalized Uncertainty Estimator (PUE): A local module that assigns uncertainty weights to each image. The PUE remains private to the client, allowing it to learn specific noise patterns unique to that user's data without compromising privacy.
  3. Expression Classification (EC) Block: For samples identified as uncertain, the model utilizes Hypergraph Label Propagation (HLPN). By examining "neighboring" clusters in the hypergraph, the model can self-correct noisy labels before the final classification step.
The AMY framework architecture, illustrating the Uncertainty Estimation (UE) and Expression Classification (EC) blocks within a client-server federated setup.

To ensure global alignment without sharing raw data, the server aggregates class prototypes (feature centroids) using the FedProto algorithm, rather than raw model weights.

Experimental Setup

The researchers evaluated AMY on two major benchmarks: RAF-DB (~30,000 images, 7 categories) and FERPlus (8 categories with crowd-sourced labels). To simulate a realistic federated environment, the data was partitioned among 10 clients using a Dirichlet distribution ($\alpha$), allowing for controlled levels of data heterogeneity.

Results and Discussion

AMY consistently outperformed traditional FL methods (e.g., FedAvg) and specialized FER uncertainty-suppression methods like SCN and RUL.

Performance comparison of AMY against state-of-the-art methods across varying levels of data heterogeneity (alpha).

Key Findings:

  • Accuracy Improvements: On the RAF-DB dataset with high heterogeneity ($\alpha=0.1$), AMY achieved 90.97% accuracy, significantly higher than the baseline (87.07%) and FedAvg (86.10%).
  • Robustness to Skewed Data: AMY maintained superior performance even when data distributions were highly unbalanced across clients, proving the effectiveness of prototype-based collaboration.
  • Qualitative Validation: Visualization confirmed that AMY correctly assigns high uncertainty to images with occlusions or extreme lighting, which typically degrade standard FER performance.

Conclusion

The AMY framework demonstrates that modeling high-order relationships via hypergraphs is a powerful tool for identifying and refining uncertain data in federated FER. By keeping uncertainty estimation local and personalized, the system turns data heterogeneity into a manageable characteristic rather than a performance bottleneck. Future work will explore making uncertainty thresholds learnable and optimizing the computational efficiency of hypergraph operations for edge devices.

Summary updated

Abstract

Most facial expression recognition (FER) models are trained on large-scale expression data with centralized learning. Unfortunately, collecting a large amount of centralized expression data is difficult in practice due to privacy concerns of facial images. In this paper, we investigate FER under the framework of personalized federated learning, which is a valuable and practical decentralized setting for real-world applications. To this end, we develop a novel uncertainty-Aware label refineMent on hYpergraphs (AMY) method. For local training, each local model consists of a backbone, an uncertainty estimation (UE) block, and an expression classification (EC) block. In the UE block, we leverage a hypergraph to model complex high-order relationships between expression samples and incorporate these relationships into uncertainty features. A personalized uncertainty estimator is then introduced to estimate reliable uncertainty weights of samples in the local client. In the EC block, we perform label propagation on the hypergraph, obtaining high-quality refined labels for retraining an expression classifier. Based on the above, we effectively alleviate heterogeneous sample uncertainty across clients and learn a robust personalized FER model in each client. Experimental results on two challenging real-world facial expression databases show that our proposed method consistently outperforms several state-of-the-art methods. This indicates the superiority of hypergraph modeling for uncertainty estimation and label refinement on the personalized federated FER task. The source code will be released at https://github.com/mobei1006/AMY.
BibTeX
@article{ding2025uncertainty,
  author = {Hu Ding and Yan Yan and Yang Lu and Jing-Hao Xue and Hanzi Wang},
  title = {{Uncertainty-Aware Label Refinement on Hypergraphs for Personalized Federated Facial Expression Recognition}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2025},
}
Figure