Introduction
Deep learning has significantly advanced robotic perception, yet autonomous agents often struggle with high-dimensional sensor noise. Unlike humans, who naturally filter out irrelevant visual stimuli—such as background posters or flickering lights—to focus on task-critical elements like the apex of a turn, robots frequently overfit to these distractors. This leads to poor robustness in out-of-distribution (OOD) scenarios and high data requirements for training. This paper explores a method to distill human biological attention into feature maps that guide robot learning pipelines toward task-relevant information.
Methodology
The proposed approach consists of three primary phases: data collection, attention modeling, and downstream integration.
Data Collection and Processing
Researchers utilized a miniature F1Tenth racecar operated manually by a human driver. The driver was equipped with a VPS 19 eye-tracking system to capture precise gaze points synchronized with first-person video and vehicle telemetry.
To transform discrete gaze points into a usable format for neural networks, the authors converted them into continuous heatmaps. These maps use a Gaussian distribution to represent the focus area and an exponential-decay function to account for temporal persistence, ensuring the model retains information about recently viewed areas.
Model Architecture
A U-Net++ architecture was trained to predict these human-like attention heatmaps directly from raw video frames. Once trained, this model can generate "attention priors" without requiring a human in the loop. These maps were then integrated into two distinct robotic tasks:
- Object Detection: Modifying YOLOv7 to include the attention map as a fourth input channel (RGB + Attention).
- Imitation Learning: Using the maps to "mark" input frames for a ResNet18-based agent predicting steering and velocity.
Key Results and Insights
The integration of human attention served as a powerful regularizer, filtering environmental perturbations and preventing the model from overfitting to irrelevant features.
Robustness to Illumination Changes
One of the most significant findings was the increased robustness to lighting variations. In tests where brightness was perturbed from 75% to 185% of the original levels, the standard YOLOv7 model experienced performance drops between 10% and 25%. In contrast, the attention-augmented version (Att-YOLOv7) maintained nearly constant performance across all tested conditions.
Learning Efficiency
In imitation learning tasks, the attention-aware agent demonstrated faster convergence and a lower Mean Squared Error (MSE) for driving commands. This advantage was most pronounced in data-scarce environments, suggesting that human-centric priors provide a significant "head start" by teaching the agent where to look before it begins learning how to act.
Conclusion and Future Work
By distilling human intuition into reusable feature maps, the researchers have demonstrated a path toward more robust and data-efficient robot learning. While currently focused on driving, future work will investigate the generalizability of these attention models to other domains, such as robotic manipulation and navigation in complex, multi-agent social environments.
Summary updated