TL;DR
This paper introduces GazeNAS-ETH, a novel, highly efficient neural architecture for appearance-based gaze estimation. By combining Neural Architecture Search (NAS) with a multi-resolution fusion transformer, GazeNAS-ETH achieves state-of-the-art accuracy on multiple benchmarks while drastically reducing computational cost, making real-time gaze estimation on edge devices feasible.
What Problem Does This Paper Address?
Estimating human gaze from images (appearance-based gaze estimation) is crucial for numerous applications, from human-computer interaction to driver monitoring. While recent methods using transformers and high-resolution networks have achieved impressive accuracy, they are often computationally too expensive for real-time deployment on resource-constrained devices like those found in edge computing. This paper aims to bridge this gap by developing a gaze estimation model that is both highly accurate and computationally efficient.
Methods
The core of the proposed method, GazeNAS-ETH, lies in two main components:
- NAS-based Multi-Resolution Feature Extractor: The authors employ Neural Architecture Search (NAS) to discover an efficient feature extractor. This extractor is designed to capture both global and local information from face images, which is essential for robust gaze estimation. The NAS process searches for optimal combinations of convolutional operations (MixConv) and lightweight transformers within a "search block." This block is inspired by HR-Net but adapted for efficiency.
- Multi-Resolution Fusion Transformer (MRFT) Head: Instead of simply concatenating features from different resolutions, the paper proposes a novel transformer-based head. This MRFT fuses features from multiple resolutions by allowing them to interact through self-attention mechanisms. Specifically, query matrices from one resolution are enriched with information from other resolutions, enabling a more comprehensive understanding of the visual cues relevant to gaze.
The overall pipeline involves searching for the optimal feature extractor on the ETH-XGaze dataset and then using this architecture with the MRFT head for gaze estimation.
Datasets
- Training and NAS Search: ETH-XGaze [45] is used for both searching the neural architecture and pre-training the model. This dataset is chosen for its large size (1.1M images) and wide range of gaze values and head poses.
- Evaluation: The performance of the proposed GazeNAS-ETH is evaluated on four public datasets:
- MPIIFaceGaze [48]
- Gaze360 [23]
- RT-GENE [18]
- EYEDIAP [27]
Key Insights and Ideas
- NAS for Gaze Estimation is Viable and Efficient: The paper demonstrates that NAS can be effectively applied to gaze estimation to find architectures that are both accurate and computationally lean, a combination not previously achieved.
- Multi-Resolution Fusion is Key: Simply concatenating features from different resolutions is suboptimal. The proposed MRFT, which allows cross-resolution feature interaction via self-attention, significantly improves performance by leveraging global context from low-resolution features and local details from high-resolution features.
- ETH-XGaze as a Generalizable Search Dataset: The authors show that an architecture searched on ETH-XGaze can generalize well to other datasets, even with minimal fine-tuning. This significantly reduces the computational burden of NAS, as it doesn't need to be performed independently on every dataset.
- Transformer Efficiency: By integrating lightweight transformers within the NAS search space and using a specialized MRFT head, the authors achieve transformer-based performance with significantly lower computational costs.
Key Results
GazeNAS-ETH achieves state-of-the-art results on all four evaluation datasets (MPIIFaceGaze, Gaze360, RT-GENE, and EYEDIAP). Crucially, it does so with remarkable efficiency:
- Parameters: Approximately 1.027 million parameters.
- FLOPs: Only 0.28 GFLOPs.
- Running Time: 22 ms per image.
This is a significant improvement over previous state-of-the-art methods, which often require hundreds of millions of parameters and tens of GFLOPs. For instance, compared to GazeTR-Hybrid, GazeNAS-ETH achieves comparable or slightly better accuracy with drastically fewer parameters and FLOPs.
Limitations and Future Work
- Cross-Dataset Performance: The authors acknowledge that NAS-based methods can exhibit limitations in cross-dataset performance without fine-tuning. While GazeNAS-ETH performs well after fine-tuning, its direct cross-dataset generalization could be improved. They suggest that increasing the computational budget for NAS might enhance this.
- Resource-Awareness: While FLOPs are used as a proxy for computational cost, the authors note that other metrics like latency and energy consumption could also be considered for more comprehensive resource-aware optimization.
- Future Work: The paper implicitly suggests further exploration into optimizing NAS for gaze estimation, potentially with more advanced search strategies or by investigating other efficiency metrics. The MRFT architecture itself could also be further explored and adapted for other vision tasks requiring multi-resolution feature fusion.
Summary updated