TL;DR
This paper introduces RNAS-MER, a novel neural architecture search (NAS) framework specifically designed for micro-expression recognition (MER). By proposing a refined hybrid module (RHM) for cell-level search and an optimal path explore network (OPEN) for architecture-level search, RNAS-MER efficiently discovers compact and effective spatiotemporal CNNs, achieving state-of-the-art performance on challenging MER benchmarks.
What Problem Does This Paper Address?
Micro-expressions (MEs) are subtle, involuntary facial movements that reveal genuine human emotions, making them valuable for applications like lie detection and psychological analysis. However, their transient nature and low intensity make them incredibly difficult to detect and recognize, both for humans and machines. While Convolutional Neural Networks (CNNs) have shown promise in MER, manually designing optimal CNN architectures is a tedious, trial-and-error process requiring significant domain expertise. Existing Neural Architecture Search (NAS) methods, while automating this process, often suffer from high computational costs and may not be optimized for the specific challenges of MER, such as limited training data and the need for fine-grained spatiotemporal feature extraction. This paper aims to bridge this gap by developing an efficient and effective NAS approach tailored for MER.
Methods
The core of RNAS-MER lies in its hierarchical, two-level search strategy:
- Inner-Level Search (Refined Hybrid Module - RHM): This module focuses on discovering optimal "cell" structures. It introduces a hybrid operator space with fourteen spatiotemporal operations (more than standard approaches) to capture multi-scale and complementary features. Crucially, it incorporates spatiotemporal attention blocks to refine feature aggregation and uses partial channel connections (inspired by PC-DARTS) to reduce memory footprints and computational burden during the search.
- Outer-Level Search (Optimal Path Explore Network - OPEN): Instead of exploring all possible connections between cells, OPEN aims to find an "optimal path" to construct a compact, shallow, and lightweight spatiotemporal CNN architecture. This is achieved by shrinking the search space and prioritizing efficient pathways, which is beneficial for MER tasks that often have limited training data.
Both the inner and outer search spaces are made continuously differentiable, allowing for the use of stochastic gradient descent for optimization.
Datasets
The paper evaluates RNAS-MER on four widely used micro-expression recognition datasets:
- COMPOSITE: A composite dataset.
- SMIC: Spontaneous, Micro-expression, Induced, and C-Rare.
- CASME-II: Chinese Academy of Sciences Micro-Expression II.
- SAMM: Spontaneous Actions and Micro-Movements.
For fair comparison, the authors merged classes from CASME-II into "positive," "negative," and "surprise" categories, aligning with the MEGC-19 challenge. They also employed a leave-one-subject-out cross-validation strategy and used a temporal interpolation model (TIM) to normalize video sequence lengths.
Key Insights and Ideas
- Tailoring NAS for MER: The paper highlights that generic NAS approaches are not optimal for MER. Specific considerations like limited data and the need for fine-grained spatiotemporal features necessitate a specialized NAS strategy.
- Hybrid Operations and Attention: The combination of a richer set of spatiotemporal operations and spatiotemporal attention mechanisms within the RHM is key to capturing the subtle nuances of micro-expressions.
- Efficient Architecture Exploration: The OPEN module's focus on finding an "optimal path" rather than exhaustive search is crucial for discovering lightweight architectures that are less prone to overfitting on small MER datasets.
- Computational Efficiency: By incorporating techniques like partial connections and an optimized search strategy, RNAS-MER achieves high performance with significantly fewer parameters and lower memory footprints compared to existing NAS and MER methods.
Key Results
RNAS-MER achieves state-of-the-art performance across the evaluated datasets. Specifically, it reports:
- COMPOSITE: 0.8511% UAR and 0.8302% UF1.
- SMIC: 0.7620% UAR and 0.7443% UF1.
- CASME-II: 0.9078% UAR and 0.8985% UF1.
- SAMM: 0.8235% UAR and 0.7880% UF1.
These results represent significant improvements over contemporary NAS-based methods like AutoDeepLab and AutoMER. Furthermore, RNAS-MER is computationally efficient, requiring only 1.91 million parameters and 15.60 MB of memory, substantially less than comparable methods.
Limitations and Future Work
The authors acknowledge that the inner/cell level search space can still be memory-intensive due to the number of convolution operations. While their approach mitigates this, it remains an area for potential improvement.
For future work, the authors plan to focus on designing NAS-based algorithms for other computer vision applications that require less computational time, suggesting a broader applicability of their NAS framework.
Summary updated