Abstract
Facial Expression Recognition (FER) models often struggle with "in-the-wild" conditions such as occlusions and varied poses. This paper introduces Point Adversarial Self Mining (PASM), a novel training framework that enhances model robustness without increasing architectural complexity. PASM iteratively identifies the most informative facial regions via point adversarial attacks and masks them, generating progressively harder training samples. A teacher-student distillation setup then forces the network to learn secondary, more robust features. PASM achieves state-of-the-art results on major FER datasets like RAF-DB and FER2013 while maintaining standard inference speeds.
Introduction
While deep learning has advanced Facial Expression Recognition (FER), performance often degrades in real-world scenarios due to unpredictable occlusions (e.g., masks, glasses) and extreme head poses. Traditional solutions involve complex, heavy architectures with attention mechanisms or landmark detectors. PASM takes a different approach: improving robustness through a smarter training strategy using standard CNN architectures like ResNet or VGG.
Methodology: Point Adversarial Self Mining
PASM operates as an iterative training loop designed to simulate a human learning process with progressively difficult materials.
The Iterative Loop
The framework follows a four-step cycle:
- Teacher Initialization: A standard CNN is trained on the base dataset to establish initial feature recognition.
- Adversarial Material Generation: Instead of random erasing, PASM employs a point adversarial attack based on differential evolution. This attack identifies the single most sensitive pixel—the feature the network relies on most—and masks a small patch around it.
- Student Training: A student network with the same architecture is trained on these masked images. It uses a dual loss: cross-entropy for ground truth and Mean Squared Error (MSE) to match the teacher's logits (knowledge distillation).
- Self-Mining Iteration: The student becomes the next iteration's teacher, and the process repeats, uncovering new vulnerabilities.
Teacher-Student Distillation
The core of the refinement process is the teacher-student setup. By matching the teacher's output distribution while viewing "harder" (masked) images, the student is forced to find alternative facial cues that the teacher might have overlooked or found less critical initially.
Key Insights
- Adaptive vs. Random Erasing: Randomly dropping pixels is a generic regularizer. PASM's adversarial approach specifically targets the network's "favorite" features, ensuring the model cannot rely on a single sparse set of regions (like just the mouth corners).
- Efficiency at Inference: Since PASM is a training-time augmentation and distillation strategy, the final model remains a standard, lightweight CNN. It achieves the performance of heavy, attention-based models with zero additional inference cost.
- Self-Correcting Curriculum: Each iteration creates a harder "curriculum" as the network becomes more robust, finding deeper and more subtle facial expressions.
Experimental Results
PASM was evaluated on several "in-the-wild" datasets:
- RAF-DB: Achieved 88.68% accuracy, outperforming complex custom architectures.
- FER2013: Reached 73.59% accuracy.
- Robustness Subsets: On Occlusion-RAF-DB and Pose-RAF-DB, PASM scored 83.27% and 89.66% respectively, demonstrating superior resilience to real-world corruptions.
- Ablation Study: PASM (86.86%) outperformed both standard Random Erasing (85.93%) and traditional Adversarial Erasing (86.57%) in single-round tests.
Limitations and Future Work
The primary drawback of PASM is the computational cost of the point adversarial attack during training, adding significant time per image. Future research will focus on optimizing these attacks. Additionally, determining the optimal number of iterations currently requires manual tuning; automating this process remains an open challenge.
Summary updated