Point Adversarial Self Mining: A Simple Method for Facial Expression Recognition in the Wild

Ping Liu, Yuewei Lin, Zibo Meng, Weihong Deng, Joey Tianyi Zhou, Yi Yang

Abstract

Facial Expression Recognition (FER) models often struggle with "in-the-wild" conditions such as occlusions and varied poses. This paper introduces Point Adversarial Self Mining (PASM), a novel training framework that enhances model robustness without increasing architectural complexity. PASM iteratively identifies the most informative facial regions via point adversarial attacks and masks them, generating progressively harder training samples. A teacher-student distillation setup then forces the network to learn secondary, more robust features. PASM achieves state-of-the-art results on major FER datasets like RAF-DB and FER2013 while maintaining standard inference speeds.

Introduction

While deep learning has advanced Facial Expression Recognition (FER), performance often degrades in real-world scenarios due to unpredictable occlusions (e.g., masks, glasses) and extreme head poses. Traditional solutions involve complex, heavy architectures with attention mechanisms or landmark detectors. PASM takes a different approach: improving robustness through a smarter training strategy using standard CNN architectures like ResNet or VGG.

Methodology: Point Adversarial Self Mining

PASM operates as an iterative training loop designed to simulate a human learning process with progressively difficult materials.

The Iterative Loop

The framework follows a four-step cycle:

  1. Teacher Initialization: A standard CNN is trained on the base dataset to establish initial feature recognition.
  2. Adversarial Material Generation: Instead of random erasing, PASM employs a point adversarial attack based on differential evolution. This attack identifies the single most sensitive pixel—the feature the network relies on most—and masks a small patch around it.
  3. Student Training: A student network with the same architecture is trained on these masked images. It uses a dual loss: cross-entropy for ground truth and Mean Squared Error (MSE) to match the teacher's logits (knowledge distillation).
  4. Self-Mining Iteration: The student becomes the next iteration's teacher, and the process repeats, uncovering new vulnerabilities.
The iterative PASM framework showing how masking and refinement evolve across iterations.

Teacher-Student Distillation

The core of the refinement process is the teacher-student setup. By matching the teacher's output distribution while viewing "harder" (masked) images, the student is forced to find alternative facial cues that the teacher might have overlooked or found less critical initially.

The teacher-student architecture showing the distillation process and loss functions.

Key Insights

  • Adaptive vs. Random Erasing: Randomly dropping pixels is a generic regularizer. PASM's adversarial approach specifically targets the network's "favorite" features, ensuring the model cannot rely on a single sparse set of regions (like just the mouth corners).
  • Efficiency at Inference: Since PASM is a training-time augmentation and distillation strategy, the final model remains a standard, lightweight CNN. It achieves the performance of heavy, attention-based models with zero additional inference cost.
  • Self-Correcting Curriculum: Each iteration creates a harder "curriculum" as the network becomes more robust, finding deeper and more subtle facial expressions.

Experimental Results

PASM was evaluated on several "in-the-wild" datasets:

  • RAF-DB: Achieved 88.68% accuracy, outperforming complex custom architectures.
  • FER2013: Reached 73.59% accuracy.
  • Robustness Subsets: On Occlusion-RAF-DB and Pose-RAF-DB, PASM scored 83.27% and 89.66% respectively, demonstrating superior resilience to real-world corruptions.
  • Ablation Study: PASM (86.86%) outperformed both standard Random Erasing (85.93%) and traditional Adversarial Erasing (86.57%) in single-round tests.

Limitations and Future Work

The primary drawback of PASM is the computational cost of the point adversarial attack during training, adding significant time per image. Future research will focus on optimizing these attacks. Additionally, determining the optimal number of iterations currently requires manual tuning; automating this process remains an open challenge.

Summary updated

Abstract

In this article, we propose a simple yet effective approach, called point adversarial self mining (PASM), to improve the recognition accuracy in facial expression recognition (FER). Unlike previous works focusing on designing specific architectures or loss functions to solve this problem, PASM boosts the network capability by simulating human learning processes: providing updated learning materials and guidance from more capable teachers. Specifically, to generate new learning materials, PASM leverages a point adversarial attack method and a trained teacher network to locate the most informative position related to the target task, generating harder learning samples to refine the network. The searched position is highly adaptive since it considers both the statistical information of each sample and the teacher network capability. Other than being provided new learning materials, the student network also receives guidance from the teacher network. After the student network finishes training, the student network changes its role and acts as a teacher, generating new learning materials and providing stronger guidance to train a better student network. The adaptive learning materials generation and teacher/student update can be conducted more than one time, improving the network capability iteratively. Extensive experimental results validate the efficacy of our method over the existing state of the arts for FER.
BibTeX
@article{liu2022point,
  author = {Ping Liu and Yuewei Lin and Zibo Meng and Weihong Deng and Joey Tianyi Zhou and Yi Yang},
  title = {{Point Adversarial Self Mining: A Simple Method for Facial Expression Recognition in the Wild}},
  journal = {IEEE Transactions on Cybernetics},
  year = {2022},
  url = {https://openreview.net/forum?id=GlX2T8NI70},
}
Figure