TL;DR
This paper introduces SalM², a novel, extremely lightweight Mamba-based model for real-time driver attention prediction. By integrating "top-down" semantic information with "bottom-up" visual features, SalM² achieves state-of-the-art performance with a fraction of the parameters of existing methods, making it highly suitable for deployment in intelligent vehicles.
What Problem Does This Paper Address?
Understanding and predicting driver attention is crucial for road safety and the development of advanced driver-assistance systems (ADAS). Current approaches often struggle with the dynamic and semantically rich nature of driving scenes. Many existing models rely solely on visual features, neglecting the "top-down" semantic information related to the driving task (e.g., a pedestrian about to cross). Furthermore, these models tend to be computationally expensive and have large parameter counts, hindering their real-time application in resource-constrained environments like vehicles. This paper aims to bridge this gap by proposing a lightweight model that effectively fuses visual and semantic information for accurate and efficient driver attention prediction.
Methods
The core of SalM² is a dual-branch architecture:
- "Bottom-up" Branch: This branch utilizes an efficient Mamba-based backbone network for extracting visual features from the driving scene. To address the parameter explosion issue common with Mamba on high-dimensional data, they introduce a Selective Channel Parallel Mamba (SCPM) layer. This layer processes features in parallel through convolutional blocks and Mamba layers, correcting information loss from fixed channel partitioning.
- "Top-down" Branch: This branch extracts semantic information from the driving scene. They leverage the CLIP model to obtain semantic representations, which are then fused with the visual features.
- Cross-Modal Attention (CMA) Module: This novel module is designed to integrate the "top-down" semantic information with the "bottom-up" visual features. It addresses the challenge of feature dimension mismatch between the two modalities by projecting semantic information into the same feature channel space as the image features. This fusion is guided by a channel attention mechanism.
The overall framework integrates these two branches, with the semantic information guiding the visual feature extraction at the deepest layer, and the fused features are then decoded to produce the driver attention map.
Datasets
The model was trained and evaluated on three representative driver attention datasets:
- TrafficGaze: Offers clear-weather driving data.
- DrFixD-rainy: Focuses on adverse weather conditions (rainy and night).
- BDD-A (Berkeley DeepDrive-A): Provides urban driving scenarios and emergency events, with richer semantic information.
The authors resized images from these datasets to 256x256 for training to reduce computational load.
Key Insights and Ideas
- Semantic Information is Crucial: The paper strongly emphasizes that driver attention is not solely driven by visual saliency but is heavily influenced by the current driving task and semantic context. Ignoring this "top-down" information leads to less accurate and less generalizable attention predictions.
- Mamba for Lightweight Architectures: The adoption of the Mamba architecture, particularly with their proposed SCPM layer, is key to achieving an extremely lightweight model with high performance. This demonstrates Mamba's potential beyond natural language processing for efficient computer vision tasks.
- Cross-Modal Attention for Fusion: The CMA module effectively tackles the challenge of fusing information from different modalities (visual and semantic) with varying dimensionalities, a common hurdle in multi-modal learning.
- Efficiency for Real-World Deployment: The significant reduction in parameters (0.08M) and computational cost (FLOPS) makes SalM² a practical solution for real-time applications in intelligent vehicles, where hardware constraints are a major concern.
Key Results
- State-of-the-Art Performance: SalM² achieves state-of-the-art (SOTA) or near-SOTA performance across multiple metrics (AUC, NSS, CC, SIM) on the evaluated datasets.
- Extreme Parameter Efficiency: The model uses only 0.08M trainable parameters, which is approximately 0.09% to 11.16% of other SOTA models.
- High Accuracy with Minimal Parameters: SalM² achieves over 98% of the performance of SOTA models while using a fraction of their parameters.
- Qualitative Improvements: Visualizations show that SalM² better captures the "top-down" information related to driving tasks (e.g., focusing on traffic lights, nearby vehicles, pedestrians) compared to other models.
Limitations and Future Work
- CLIP Input Resolution: The paper notes that CLIP, used for semantic feature extraction, is limited to 224x224 input images. While they use 256x256 for their model, increasing resolution further for CLIP might not yield proportional performance gains and significantly increases computational cost. They chose 256x256 as a balance.
- Dataset Specificity: While tested on multiple datasets, the performance might vary on datasets with even less rich semantic information.
- Future Work: The authors aim to explore more effective methods for using driving scene semantic information to guide driver attention prediction, with the goal of further improving prediction accuracy. They also implicitly suggest that further optimization of the Mamba architecture for vision tasks could lead to even more efficient models.
Summary updated