TL;DR
This paper introduces OF-PCANet+, a novel, lightweight method for micro-expression (ME) recognition that combines dense optical flow with a shallow PCANet+ network. By stacking optical flow sequences, the method effectively captures subtle spatiotemporal facial movements, achieving competitive performance against more complex deep learning models on standard ME datasets.
What Problem Does This Paper Address?
Micro-expressions (MEs) are fleeting, involuntary facial movements that are incredibly difficult to detect and recognize, even for trained human observers. Their subtlety and short duration (1/25 to 1/2 second) pose a significant challenge for automated recognition systems. Existing methods often struggle due to the limited availability of large ME datasets, which can lead to overfitting in deep neural networks. This paper aims to address this by proposing a computationally efficient yet effective approach that can learn discriminative spatiotemporal features from limited data.
Methods
The proposed OF-PCANet+ method consists of three main steps:
- Dense Optical Flow Calculation and Multi-Channel Stacking:
- Optical Flow: The method first computes dense optical flow between consecutive frames of an ME video clip. This is done using a subspace trajectory model that aims to maintain temporal consistency and enhance motion information. The optical flow field (u, v) captures pixel movements, effectively highlighting areas of facial change.
- Multi-Channel Stacking: To capture spatiotemporal dynamics, the computed optical flow sequences (u and v components) are stacked into multi-channel images. This is achieved by using a sliding window of a fixed number of frames (T) and concatenating the optical flow fields of adjacent frames. This creates a richer input representation that encodes motion over time.
- Feature Extraction with PCANet+:
- PCANet+: The stacked multi-channel optical flow images are then fed into a two-layer PCANet+ network. PCANet+ is a shallow network architecture that combines principal component analysis (PCA) with a convolutional structure. It's designed to be computationally efficient and avoid the overfitting issues common with deeper networks on small datasets. The "+" in PCANet+ signifies the addition of a mean pooling layer between convolutional layers, which helps reduce feature dimensionality and introduces some non-linearity.
- Feature Encoding: Each PCANet+ layer's output is subjected to a feature encoding process involving hashing and histogramming, which generates a compact feature representation. These features from both layers are then concatenated to form the final feature vector.
- Classification with Linear SVM:
- A linear Support Vector Machine (SVM) is used to classify the extracted features and recognize the micro-expression.
Datasets
The proposed method was evaluated on two publicly available micro-expression datasets:
- SMIC (Spontaneous Micro-expression Corpus): Specifically, the SMIC-HS (High-Speed) subset was used, recorded at 100 fps, which is crucial for capturing rapid ME movements.
- CASME2 (Chinese Academy of Sciences Micro-Expression II): This dataset was recorded at 200 fps, providing high temporal resolution.
The paper does not introduce a new dataset but rather focuses on evaluating its method on existing benchmarks.
Key Insights and Ideas
- Optical Flow as a Powerful Feature Enhancer: The paper demonstrates that dense optical flow, when used as input to a feature extractor, can significantly enhance the ability to capture subtle facial movements characteristic of micro-expressions. It effectively highlights motion and can filter out identity-specific facial information.
- Shallow Networks Can Be Effective: Contrary to the trend of increasingly deep neural networks, this work shows that a shallow architecture like PCANet+, when combined with appropriate feature engineering (optical flow stacking), can achieve competitive performance. This is particularly important for ME recognition due to limited data.
- Spatiotemporal Feature Learning is Crucial: The success of the method hinges on its ability to learn features that capture both spatial information (facial appearance) and temporal information (movement over time). The multi-channel stacking of optical flow sequences is key to achieving this.
- Computational Efficiency: The use of PCANet+ and optical flow, rather than complex end-to-end deep learning, results in a computationally lighter model, making it more practical for certain applications.
Key Results
- The proposed OF-PCANet+ method achieved competitive performance on both the SMIC and CASME2 datasets.
- On the SMIC dataset, the best accuracy achieved was 0.6280 with Macro-F1 of 0.6309 and Macro-Recall of 0.6369 using a frame stacking number of T=5.
- On the CASME2 dataset, the best accuracy was 0.5325 with Macro-F1 of 0.5493 and Macro-Recall of 0.5241 using specific filter configurations.
- The method outperformed popular handcrafted methods like LBP-TOP, STLBP-IP, and KGSL.
- It also achieved comparable performance to some deep learning methods, such as ELRCN and 3D-FCNN, despite being a shallower model.
- Experiments showed that frame stacking (T > 1) generally improved performance compared to using single frames (T=1), with T=5 yielding the best results.
Limitations and Future Work
- Dataset Size: The paper acknowledges that the small size of ME datasets remains a limitation for training deep models and contributes to the choice of a shallow PCANet+ architecture.
- Parameter Tuning: While the paper explores the impact of parameters like frame stacking number and filter sizes, optimal configurations might still be dataset-dependent.
- Future Work: The authors suggest applying this method to other related tasks such as behavior recognition and general video classification, indicating its potential broader applicability. They also imply that further exploration of feature learning strategies could yield even better results.
Summary updated