Dynamic Resolution Guidance for Facial Expression Recognition

Jie Ou, Xu Li, Tianxiang Jiang, Yuanlun XiearXiv 2024 · DOI · arXiv

Abstract

Facial expression recognition (FER) is vital for human-computer interaction and emotion analysis, yet recognizing expressions in low-resolution images remains challenging. This paper introduces a practical method called Dynamic Resolution Guidance for Facial Expression Recognition (DRGFER) to effectively recognize facial expressions in images with varying resolutions without compromising FER model accuracy. Our framework comprises two main components: the Resolution Recognition Network (RRN) and the Multi-Resolution Adaptation Facial Expression Recognition Network (MRAFER). The RRN determines image resolution, outputs a binary vector, and the MRAFER assigns images to suitable facial expression recognition networks based on resolution. We evaluated DRGFER on widely-used datasets RAFDB and FERPlus, demonstrating that our method retains optimal model performance at each resolution and outperforms alternative resolution approaches. The proposed framework exhibits robustness against resolution variations and facial expressions, offering a promising solution for real-world applications.

Introduction: The Resolution Challenge in FER

Facial Expression Recognition (FER) has seen significant advancements, yet it struggles with the "resolution problem." While most models excel on high-quality, standard-sized datasets, real-world scenarios—such as surveillance footage or group photos—present faces at varying distances and resolutions.

Real-world crowd scenes often contain faces at varying resolutions, complicating expression recognition.

When an image is downsampled, two primary issues arise:

  1. Information Loss: Subtle micro-expressions necessary for accurate classification disappear.
  2. Feature Shift: The statistical distribution of features changes significantly, causing models trained on high-resolution images to fail.

Previous solutions like "Multi-Scale Training" or "Domain Adaptation" often result in a "jack of all trades, master of none" scenario, where peak accuracy on high-resolution images is sacrificed for average performance across the board.

Methodology: Dynamic Resolution Guidance

The proposed Dynamic Resolution Guidance for FER (DRGFER) framework avoids these trade-offs by acting as a smart switchboard. Instead of one universal model, it routes images to specialized "expert" blocks.

The framework consists of two primary stages:

  1. Resolution Recognition Network (RRN): Utilizing a ResNet18-based architecture, this network categorizes input images into specific resolution "bins" (e.g., x1, x2, x4, x6, or x8 downsampling).
  2. Multi-Resolution Adaptation FER Network (MRAFER): The RRN's output is converted into a one-hot binary vector. A specialized module then groups images by their detected resolution and sends them to a corresponding FER_Block—an expert model trained specifically for that resolution level.
The proposed architecture utilizes a resolution predictor to guide the feature extraction process.

This expert-based approach allows each block to optimize for specific artifacts and feature distributions without interference from other resolution scales.

Experimental Setup and Results

The researchers evaluated DRGFER on two benchmark datasets: RAF-DB (15,000+ images, 7 emotions) and FERPlus (28,000+ images, 8 emotions). To simulate real-world conditions, they created downsampled versions at x2, x4, x6, and x8 scales using bicubic interpolation.

Performance Comparison

DRGFER consistently outperformed alternative methods such as Resolution-Aware Batch Normalization (RA-BN) and standard Domain Adaptation (DA).

  • Overall Accuracy: On RAF-DB, DRGFER achieved a mean accuracy of 84.41%, compared to 82.94% for RA-BN and 73.89% for DA.
  • High-Res Integrity: At original resolution (x1), DRGFER maintained 89.24% accuracy, significantly higher than the 85.88% achieved by standard multi-scale training.
  • Low-Res Robustness: Even at extreme x8 downsampling, the model maintained 77.35% accuracy, whereas basic models dropped to 66.85%.

Discussion: Why Resolution Guidance Works

A key insight from the study is that the "Resolution Gap" can be wider than the "Identity Gap." Beyond x7 downsampling, the feature difference between a high-res and low-res version of the same face is greater than the difference between two different individuals. By treating resolution as a known "prior" rather than noise, DRGFER allows the network to bridge this gap effectively.

Limitations and Future Outlook

Despite its success, the framework has areas for growth:

  • Discrete Bins: Currently, the model uses discrete resolution categories. Future iterations could explore handling continuous resolution variations.
  • Computational Efficiency: While only one expert block is active per image during inference, the need to store multiple blocks may be a constraint for memory-limited edge devices.
  • Super-Resolution Integration: Integrating RRN guidance directly into Image Super-Resolution (ISR) preprocessing could further enhance clarity before classification.

Summary updated

Abstract

Facial expression recognition (FER) is vital for human-computer interaction and emotion analysis, yet recognizing expressions in low-resolution images remains challenging. This paper introduces a practical method called Dynamic Resolution Guidance for Facial Expression Recognition (DRGFER) to effectively recognize facial expressions in images with varying resolutions without compromising FER model accuracy. Our framework comprises two main components: the Resolution Recognition Network (RRN) and the Multi-Resolution Adaptation Facial Expression Recognition Network (MRAFER). The RRN determines image resolution, outputs a binary vector, and the MRAFER assigns images to suitable facial expression recognition networks based on resolution. We evaluated DRGFER on widely-used datasets RAFDB and FERPlus, demonstrating that our method retains optimal model performance at each resolution and outperforms alternative resolution approaches. The proposed framework exhibits robustness against resolution variations and facial expressions, offering a promising solution for real-world applications.
BibTeX
@misc{ou2024dynamic,
  author = {Jie Ou and Xu Li and Tianxiang Jiang and Yuanlun Xie},
  title = {{Dynamic Resolution Guidance for Facial Expression Recognition}},
  year = {2024},
  eprint = {2404.06365},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.06365},
}
Figure