Abstract
The problem of predicting driver attention from the driving perspective is gaining increasing research focus due to its remarkable significance for autonomous driving and assisted driving systems. The driving experience is extremely important for safe driving, a skilled driver is able to effortlessly predict oncoming danger (before it becomes salient) based on the driving experience and quickly pay attention to the corresponding zones. However, the nonobjective driving experience is difficult to model, so a mechanism simulating the driver experience accumulation procedure is absent in existing methods, and the current methods usually follow the technique line of saliency prediction methods to predict driver attention. In this paper, we propose a FeedBack Loop Network (FBLNet), which attempts to model the driving experience accumulation procedure. By over-and-over iterations, FBLNet generates the incremental knowledge that carries rich historically-accumulative and long-term temporal information. The incremental knowledge in our model is like the driving experience of humans. Under the guidance of the incremental knowledge, our model fuses the CNN feature and Transformer feature that are extracted from the input image to predict driver attention. Our model exhibits a solid advantage over existing methods, achieving an outstanding performance improvement on two driver attention benchmark datasets.
Introduction
Predicting driver attention is a cornerstone of autonomous driving and Advanced Driver Assistance Systems (ADAS). However, most existing models treat this as a standard "saliency prediction" task—identifying what is visually prominent in an image, such as a bright red car or a flashing sign.
Human drivers do not just look at what is salient; they look at what is important based on experience. For example, an experienced driver might watch a car in the adjacent lane that might cut in, even if it isn't visually distracting yet. Existing architectures, typically relying on simple CNN or CNN-Transformer encoders, lack a mechanism to model this historically accumulated "driving experience," often leading to failures in complex or low-visibility traffic scenes. FBLNet addresses this by simulating the accumulation of driving experience through a feedback loop mechanism.
Methodology
The core innovation of FBLNet is the FeedBack Loop (FBL) and the resulting Incremental Knowledge ($K$). The architecture consists of several key components:
- Dual-Path Encoder: The model utilizes a ResNet-18 (CNN) to extract local detailed features and a Swin-Transformer to capture global, long-range dependencies in the traffic scene.
- The Feedback Loop: Inspired by Norbert Wiener’s cybernetics, the FBL takes features from the decoder and feeds them back to the front end during training.
- Incremental Knowledge ($K$): This feedback is used to iteratively update a persistent matrix $K$. Over thousands of training iterations, $K$ acts as a memory bank that stores "experience" about which areas of a scene are typically important.
- Guided Fusion Module: A specialized module uses the accumulated knowledge $K$ to guide the fusion of CNN and Transformer features. It uses an attention mechanism to ensure the model focuses on regions that are both visually relevant and "historically" significant.
Crucially, the feedback loop is active only during training to build $K$. During testing (inference), $K$ remains constant, acting as the "learned experience" the model applies to new images.
Experimental Setup
The authors evaluated FBLNet on two major driver attention benchmarks:
- BDDA (Berkeley DeepDrive Attention): Contains 1,232 videos focusing on critical maneuvers like emergency braking and occlusion.
- DADA (Driver Attention in Driving Accident scenarios): A massive dataset with over 658,000 frames across 2,000 videos, specifically categorized into 54 types of accident scenarios (e.g., highway, urban, rural).
Results and Discussion
FBLNet achieved state-of-the-art (SOTA) performance across both datasets. On the BDDA dataset, it achieved a 13.4% improvement in Correlation Coefficient (CC) and a 7.27% improvement in KL-divergence over previous best models. On the DADA dataset, it showed a 4.65% improvement in CC.
Qualitative results demonstrate that FBLNet correctly identifies pedestrians in deep shadows or small traffic lights that other models ignore because they aren't visually "loud" enough. This confirms that applying Wiener’s feedback theory helps stabilize the system and allows the predicted attention to inform and refine the input processing over time. Ablation studies further revealed that feeding back features from the middle of the decoder provided the best balance for updating the model's "experience."
Conclusion
FBLNet introduces a novel way to model "driving experience" as a form of long-term temporal information captured by a feedback mechanism. While the model significantly advances the state-of-the-art, challenges remain, such as predicting attention in empty scenes or handling extreme visual distractions like billboards. Future work could explore "on-the-fly" experience accumulation to allow the model to continue learning in real-time during deployment.
Summary updated