QCS: Feature Refining from Quadruplet Cross Similarity for Facial Expression Recognition

Chengpeng Wang, Li Chen, Lili Wang, Zhaofan Li, Xuebin LvAAAI 2025 · DOI

Abstract

Facial Expression Recognition (FER) is often hindered by redundant "noise" such as identity, pose, and illumination, which can overshadow the actual emotional signal. This paper introduces the Quadruplet Cross Similarity (QCS) framework, a novel approach designed to isolate pure expression features. By leveraging a Cross Similarity Attention (CSA) mechanism within a centrally symmetric four-branch network, the model learns to disentangle expressions from identity-related features. Through contrastive residual distillation, the complex multi-branch insights are transferred to a single-branch model for efficient inference. QCS achieves state-of-the-art results on major benchmarks, including 93.02% accuracy on RAF-DB.

The Challenge of Facial Expression Recognition

The primary difficulty in FER lies in the fact that intra-class variation (e.g., different people expressing the same emotion) can be greater than inter-class variation (e.g., the same person expressing different emotions). Redundant features like bone structure, age, and lighting are often non-randomly distributed in datasets, leading models to learn identity as a proxy for emotion.

The challenge of FER: high similarity between different classes and variation within the same class.

Existing solutions often rely on facial landmarks or computationally expensive Generative Adversarial Networks (GANs). QCS aims to solve this without auxiliary labels by explicitly modeling the similarities and differences across multiple samples to isolate the pure expression signal.

The QCS Framework and Cross Similarity Attention

The core of the proposed method is the Quadruplet Cross Similarity (QCS) network. Unlike standard Vision Transformers (ViT) that focus on self-attention within a single image, QCS introduces Cross Similarity Attention (CSA). CSA calculates fine-grained similarity between an "anchor" image and other samples in a quadruplet: a Positive sample (same expression) and two Negative samples (different expressions).

Overview of the Quadruplet Cross Similarity (QCS) architecture and its internal feature refining modules.

The network uses two specialized modules to refine features:

  1. Module S (Similarity): Identifies shared features between the anchor and positive samples to reinforce the expression signal.
  2. Module D (Difference): Identifies similarities between the anchor and negative samples. Since these samples have different expressions, any shared features are likely "noise" (e.g., identity or lighting) that the model should learn to ignore.

Contrastive Residual Distillation

To maintain high inference speeds, the authors employ a distillation process. The complex cross-similarity modules and extra branches are only active during training. Their knowledge is distilled into a "base" branch via KL divergence and specific loss functions. During deployment, the auxiliary branches are discarded, allowing the model to run with the same overhead as a standard backbone (e.g., IR50), making it suitable for real-time applications.

Experimental Results

The QCS framework was evaluated on three major FER benchmarks: RAF-DB, FERPlus, and AffectNet.

  • RAF-DB: The model set a new state-of-the-art with 93.02% accuracy.
  • AffectNet-7: QCS reached 67.94%, outperforming recent transformer-based architectures like POSTER++ and S2D.
  • Efficiency: The model maintains the inference speed of its backbone, proving that sophisticated feature refining does not have to come at a computational cost.

Conclusion and Future Work

The QCS framework effectively addresses the problem of redundant features in FER through its quadruplet-based cross-similarity mechanism. While care must be taken to avoid overfitting in the cross-modules—where the model might memorize specific image pairs rather than generalizable features—the distillation approach ensures that these advanced learning techniques result in a fast, deployable model. Future research may explore the point of diminishing returns when adding more branches beyond the quadruplet setup.

Summary updated

BibTeX
@inproceedings{wang2025qcs,
  author = {Chengpeng Wang and Li Chen and Lili Wang and Zhaofan Li and Xuebin Lv},
  title = {{QCS: Feature Refining from Quadruplet Cross Similarity for Facial Expression Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i7.32814},
}
Figure