Facial Expression Recognition by Inter-Class Relational Learning

Yizhen Chen, Haifeng HuIEEE Access 2019 · OpenReview

Abstract

Human facial expressions often share overlapping physical traits, making discrete categorization challenging for traditional neural networks. This paper introduces Inter-class Relational Learning (IcRL), a novel training procedure that recognizes expressions by extracting independent expressive representations. By training on hybrid features blended from pairs of images with random ratios, the model learns the subtle relationships between different emotional classes. This approach enhances the discriminative power of learned features, significantly improving classification performance across five major benchmarks, including CK+, JAFFE, and FER2013.

The Challenge of Overlapping Emotions

Traditional Facial Expression Recognition (FER) systems typically treat emotions as isolated, discrete categories using "one-hot" labels. However, real-world expressions are rarely so distinct. For instance, "Fear" and "Surprise" often share physical characteristics such as widened eyes and an open mouth.

Visual similarity between 'fear' and 'surprise' expressions, illustrating the challenge of discrete classification.

Because standard models ignore these intrinsic connections, the mathematical representations (feature clusters) they create often overlap, leading to misclassification of similar-looking expressions. The authors aim to solve this by forcing the network to learn the relationships between classes rather than just looking at them in isolation.

Inter-class Relational Learning (IcRL)

To address this, the proposed IcRL method functions like a "digital mixology" for features. The architecture utilizes a CNN (Inception-ResNet-v1) to extract features from pairs of different expression images. These features are then integrated using a random ratio ($\lambda$) to create a hybrid feature. An attention module assigns weights to pixels most relevant to the expressions, typically focusing on the eyes and mouth.

The network is trained to predict the exact mixing ratio of the two expressions rather than a single category, using Kullback-Leibler (KL) divergence to calculate loss. By accurately discriminating these hybrid features, the model learns to define the boundaries between classes more precisely. During inference, a single image is processed as a special case (1:0 ratio), leveraging the model's deep understanding of inter-class boundaries to make a more accurate single-class prediction.

Key Insights and Feature Separation

The primary advantage of IcRL is its ability to enlarge Fisher's criterion—the ratio of inter-class distance to intra-class distance. By forcing the model to distinguish precise ratios in hybrid features, the training procedure naturally "pushes" the feature clusters of different emotions further apart while making individual clusters more compact.

Visualization of learned feature clusters. The IcRL method helps in separating similar expressions in the feature space by enlarging the inter-class distance.

Experimental Results

The IcRL method was evaluated on five public databases: CK+, JAFFE, TFEID, BAUM-2i, and FER2013. It consistently outperformed baseline models and state-of-the-art approaches:

  • CK+: Achieved 97.04% accuracy, outperforming the baseline by over 3%.
  • JAFFE: Reached an impressive 99.09%, a significant jump from the 94.37% baseline.
  • FER2013: Achieved 72.36% accuracy, which is highly competitive for this noisy, unconstrained dataset.

Furthermore, the model demonstrated superior cross-dataset generalization, proving that the learned features are more "universal" and less overfitted to specific dataset quirks.

Conclusion

By shifting the focus from isolated category recognition to inter-class relational learning, IcRL provides a more robust framework for facial expression recognition. It effectively handles the inherent ambiguity in human emotions, leading to more discriminative features and higher accuracy across diverse real-world scenarios.

Summary updated

Abstract

Certain correlation exists between various expressions of the human face, yet few works have explored it. In this paper, we propose to recognize facial expressions by extracting independent expressive representation through a novel learning procedure, called Inter-class Relational Learning (IcRL). First, an elaborately designed CNN architecture is utilized for simultaneously extracting features from two different expression images. Then, two extracted features are integrated with a random ratio so that a hybrid feature can be attained. An attention module is proposed to assign a weight for each pixel of the hybrid feature. At last, we feed the weighted hybrid feature to the subsequent classification module. The goal of the whole network training is to output the correct ratio of each expression. Different from previous work, our novel method is capable of learning the mutual relations between different classes of expression and enlarging Fisher's criterion between any two classes, i.e., the ratio of inter-class distance to intra-class distance, by accurately discriminating the hybrid feature. In this way, we will be able to enhance the discriminative power of learned features of each expression and, therefore, can improve the classification performance. The IcRL method has been evaluated on five public expression databases: CK+, JAFFE, TFEID, BAUM-2i, and FER2013. The experimental results demonstrate the superiority of the proposed method over many state-of-the-art approaches.
BibTeX
@inproceedings{chen2019facial,
  author = {Yizhen Chen and Haifeng Hu},
  title = {{Facial Expression Recognition by Inter-Class Relational Learning}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=S6w7ERDP8T},
}
Figure