Combining Fixed-Weight ArcFace Loss and Vision Transformer for Facial Expression Recognition

Yunhao Xu, Xinran Duan, Pingyi Fan, Zengshun Zhao, Xiaoyu GuoSensors 2025 · DOI

TL;DR

This paper introduces a novel "fixed-weight ArcFace loss" that improves facial expression recognition by explicitly constraining the angular distribution of class centers. Combined with a Vision Transformer, this approach achieves higher accuracy and computational efficiency, particularly for challenging datasets with imbalanced classes and subtle expression variations.

What Problem Does This Paper Address?

Facial expression recognition (FER) is a crucial task with applications ranging from affective computing to human-computer interaction. However, it's plagued by significant challenges: expressions within the same category can vary greatly (intra-class variance), while different expressions can look very similar (inter-class similarity). Traditional methods like Softmax loss struggle with this. While advanced loss functions like ArcFace improve inter-class separability by adding angular margins, they don't explicitly enforce a uniform distribution of class centers. This is problematic for FER datasets, which are often small, imbalanced, and contain highly correlated expression categories, leading to unstable training and biased predictions.

Methods

The core innovation is a fixed-weight ArcFace loss. This method builds upon the standard ArcFace loss by adding two key components:

  • Fixed Classifier Weights: The weights of the final classification layer are initialized to be uniformly distributed on a hypersphere and are frozen during training. This prevents gradient imbalances from distorting the classifier's geometry and significantly reduces computational overhead by eliminating backpropagation through this layer.
  • Angular Diversity Regularization (L4): A new regularization term is introduced to explicitly minimize the pairwise cosine similarity between class centers (weights). This encourages a more uniform angular distribution of these centers, enhancing global angular separation.

The proposed loss function, $L = L_3 + qL_4$, combines the original ArcFace loss ($L_3$) with this new angular diversity term ($L_4$), controlled by a hyperparameter $q$. This approach aims to achieve both intra-class compactness and inter-class separability, especially for minority classes.

The authors integrate this loss function with a Vision Transformer (ViT). They opt for a lightweight, custom-designed ViT architecture to effectively capture long-range dependencies and structural details crucial for FER, while also being suitable for smaller datasets and reducing computational load.

Datasets

The paper evaluates its proposed method on two widely used facial expression recognition datasets:

  • FER2013: A large dataset with 35,886 grayscale images across seven basic expression categories.
  • RAF-DB: A dataset with 15,339 facial expression images, also featuring seven basic emotion categories, collected under diverse real-world conditions.

Key Insights and Ideas

  • The importance of class center distribution: The paper strongly argues that simply enforcing margins between individual features and their class centers (like ArcFace) isn't enough. Explicitly controlling the global angular relationships between all class centers is crucial for robust FER, especially with imbalanced data.
  • Fixed weights for efficiency and stability: Freezing the classifier weights not only reduces computation but also stabilizes training. This is a clever way to leverage the benefits of margin-based losses without the full backpropagation cost.
  • ViT's suitability for FER: The paper highlights how ViT's self-attention mechanism is well-suited for capturing global facial context, which is vital for understanding nuanced expressions, outperforming traditional CNNs in this regard.
  • Architecture-agnostic loss: The proposed loss function demonstrates strong performance across different architectures (ViT, ResNet, VGG), suggesting its general applicability.

Key Results

  • Accuracy Improvements: The proposed fixed-weight ArcFace loss consistently outperforms Softmax, Island loss, and standard ArcFace across all tested models and datasets. For instance, on RAF-DB, it achieves accuracies of 72.197% (ViT), 84.485% (ResNet18), and 82.562% (VGG19), outperforming standard ArcFace by up to 0.522%. On FER2013, it reaches 70.828% (Private) and 71.942% (Public), exceeding ArcFace by over 1.1%.
  • Minority Class Performance: The method shows significant gains for minority and highly confusable classes, such as "disgust" and "fear," which are often misclassified.
  • Computational Efficiency: The fixed-weight strategy reduces training time per epoch. On Transformer models, it achieves an average training time of 36.50s per epoch, which is faster than standard ArcFace (36.84s) and Island loss (39.62s).
  • Robustness: The approach demonstrates consistent performance across different datasets and architectures, highlighting its generalization capability.

Limitations and Future Work

  • Weight Space Overload: The authors acknowledge that as the number of expression categories increases, the available weight space might become overwhelmed.
  • Future Directions: They suggest exploring dynamic weight allocation or alternative optimization strategies to address this challenge in future research.

Summary updated

Abstract

In recent years, deep learning has demonstrated remarkable capability in the broad domain of feature learning. Although the facial expression recognition task has been extensively studied, it still faces many challenges such as significant variations within the same category of expressions. ArcFace loss is a widely adopted function designed to enhance inter-class separability and improve recognition performance. However, it does not explicitly constrain the angular distribution between class centers. Therefore, this study introduces a weight-constrained ArcFace loss and integrates it into the Vision Transformer (ViT) framework. This approach not only alleviates implicit biases induced by imbalanced data distributions but also significantly reduces computational overhead by stabilizing weight optimization. In the experiments, the authors evaluated the proposed approach against standard ArcFace loss, classical loss functions, and various network structures on the RAF-DB and FER2013 datasets. Comprehensive experimental results demonstrate that the proposed approach not only improves recognition accuracy but also achieves higher computational efficiency.
BibTeX
@inproceedings{xu2025combining,
  author = {Yunhao Xu and Xinran Duan and Pingyi Fan and Zengshun Zhao and Xiaoyu Guo},
  title = {{Combining Fixed-Weight ArcFace Loss and Vision Transformer for Facial Expression Recognition}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25237166},
}
Figure