Facial Expression Recognition Using Local Sliding Window Attention

Shuang Qiu, Guangzhe Zhao, Xiao Li, Xueping WangSensors 2023 · DOI

TL;DR

This paper introduces SWA-Net, a novel approach to facial expression recognition (FER) that tackles the challenges of occlusions and head pose variations in real-world images. By employing a "sliding window" strategy for feature extraction and an adaptive attention mechanism, SWA-Net effectively captures fine-grained local features, leading to state-of-the-art performance on several challenging datasets.

What Problem Does This Paper Address?

Facial Expression Recognition (FER) is crucial for human-computer interaction, but real-world scenarios present significant hurdles. Images often suffer from occlusions (e.g., hands, masks) and varying head poses, which obscure key facial features and make accurate expression detection difficult. Traditional methods often rely on prior knowledge of facial landmarks or fixed-size patches for feature extraction. However, prior knowledge can be error-prone due to occlusions, and fixed-size patches can fragment essential facial features, destroying their integrity. This paper aims to overcome these limitations by developing a method that can robustly extract discriminative local features even from incomplete or distorted facial images.

Methods

The core of the proposed SWA-Net lies in its innovative approach to local feature extraction and selection:

  • Sliding Window Cropping (SWC) Module: Instead of relying on predefined landmarks or fixed grids, SWC employs a sliding window at the feature level. This means it crops coarse-grained features extracted by a backbone network (ResNet-18 in this case). This feature-level cropping preserves the integrity of local features and avoids the need for complex preprocessing or the risk of fragmenting crucial information.
  • Local Feature Enhancement (LFE) Module: To further refine the extracted local features, LFE uses a Multi-Branch Enhancement (MBE) block. This block incorporates convolutions of different scales to capture multiscale features and a CBAM (Convolutional Block Attention Module) for channel and spatial attention. An "enhancement loss" is also introduced to suppress noise learning within these local features.
  • Adaptive Feature Selection (AFS) Module: Recognizing that not all local features are equally important, AFS assigns adaptive weights to each sub-feature block. It calculates weights based on the average and maximum feature values within a block, allowing the model to amplify important features and suppress less relevant ones. This ensures that the model focuses on the most discriminative local information.
  • Fusion Strategy: SWA-Net combines global features (extracted by the backbone) and the enhanced, selected local features. A decision-level fusion strategy is employed, where the prediction results from global and local branches are weighted and combined.

Datasets

The SWA-Net model was evaluated on several widely used facial expression datasets:

  • RAF-DB (Real-world Affective Faces Database): Contains approximately 30,000 images with multiple occlusions and varying head poses, labeled with seven expression categories. The authors selected 12,271 training and 3068 testing images.
  • FERPlus: An extension of the FER2013 dataset, with improved labeling for 8 expression categories. Images are low-resolution (48x48).
  • AffectNet: A very large-scale dataset with over a million images, but with a highly unbalanced class distribution. The authors used an undersampling strategy, employing 60,000 training and 3500 testing images for 7 expressions.
  • Occlusion & Pose Datasets: The authors also used specialized datasets from Wang et al. [17] to specifically test robustness against occlusions and multiangle head poses.

Key Insights and Ideas

  • Feature-level cropping is key: The paper convincingly argues that cropping at the feature map level, rather than the raw image level, is more efficient and better preserves the integrity of local information, especially under challenging conditions.
  • Attention is not just global: While global attention mechanisms are common, this work highlights the power of localized attention and adaptive feature weighting for FER, allowing the model to pinpoint critical facial regions.
  • Noise suppression is crucial for local features: Local features are inherently more susceptible to noise. The introduction of an enhancement loss to specifically combat this noise is a significant contribution.
  • Adaptive weighting unlocks discriminative power: The AFS module's ability to dynamically assign importance to local features is critical for distinguishing subtle expressions, especially when global information is compromised.

Key Results

SWA-Net achieved state-of-the-art performance across the evaluated datasets:

  • RAF-DB: 90.03% accuracy.
  • FERPlus: 89.22% accuracy.
  • AffectNet: 63.97% accuracy.

The ablation studies demonstrated the effectiveness of each proposed module:

  • SWC module improved accuracy by 1.79% over the baseline.
  • LFE module further improved accuracy by 0.45%.
  • AFS module provided an additional 0.66% improvement.

The model also showed superior performance on occlusion and pose datasets compared to existing methods. Grad-CAM visualizations confirmed that SWA-Net effectively focuses attention on relevant facial regions, even in challenging scenarios.

Limitations and Future Work

The authors acknowledge that their method struggles with extremely distorted faces due to severe occlusions and pose variations. In such cases, while the model can focus on facial regions, it finds it difficult to learn discriminative and detailed features. For future work, they aim to address the problem of larger facial occlusions and extremely distorted heads to further boost performance.

Summary updated

BibTeX
@article{qiu2023facial,
  author = {Shuang Qiu and Guangzhe Zhao and Xiao Li and Xueping Wang},
  title = {{Facial Expression Recognition Using Local Sliding Window Attention}},
  journal = {Sensors},
  year = {2023},
  doi = {10.3390/s23073424},
}
Figure