TL;DR
This paper demonstrates that Vision Transformers (ViTs) pretrained with self-supervised learning can be effectively fine-tuned for monocular robot navigation using very few annotated images. They show that even coarse, low-resolution segmentation from these models is sufficient for robust lane following and obstacle avoidance on a real robot, highlighting the potential of efficient, self-supervised ViTs for embodied AI.
What Problem Does This Paper Address?
Training deep learning models for robotics, especially for tasks like navigation, typically requires a large number of meticulously annotated images. This annotation process is time-consuming and expensive, making it difficult to deploy robots in new, unseen environments. Furthermore, many state-of-the-art deep learning models are computationally intensive, posing a challenge for deployment on resource-constrained robotic hardware. This paper tackles the problem of learning effective perception models for monocular robot navigation with limited annotated data and a focus on efficient inference.
Methods
The core of this work lies in leveraging Vision Transformers (ViTs) pretrained using a label-free self-supervised method called DINO. Instead of training a full-resolution segmentation model, the authors propose a "coarse" semantic segmentation approach. This involves predicting labels at the 8x8 patch level of the input image. A ViT's patch encodings are fed into a simple fully-connected network to produce these coarse predictions.
A key aspect of their method is the flexibility in inference resolution. The same trained model can produce segmentation masks at different granularities by adjusting the input image resolution, allowing a trade-off between prediction detail and real-time performance. They also investigate adapting ViT architectures by varying the number of transformer blocks, finding that even shallow architectures (like a 1-block ViT) can yield good results.
The resulting segmentation model serves as the backbone for a visual servoing agent. This agent uses a potential-field-based controller that takes the segmentation output (specifically, masks for lane lines and obstacles) to generate steering commands for the robot, aiming to keep it centered in a lane or avoid obstacles.
Datasets
The authors collected their own dataset using the Duckietown platform, an inexpensive open-source platform for autonomous driving research. They created two object-rich driving scenes for data collection. A total of 100 images were sampled and manually labeled with 7 classes: Duckiebot, duckie, white lane marking, yellow lane marking, road sign, and human hand. This dataset was split into a 70-15-15 train-validation-test set.
Key Insights and Ideas
- Self-supervised ViTs are powerful for robotics: The DINO pretraining provides strong visual representations that transfer well to the downstream task of robot navigation, even with minimal fine-tuning data.
- Coarse segmentation is sufficient: Surprisingly, low-resolution (8x8 patch level) semantic segmentation is enough for robust lane following and obstacle avoidance. This significantly reduces computational overhead.
- Lightweight architectures shine: Contrary to the trend of ever-larger models, the authors found that shallower ViT architectures (specifically, a 1-block ViT) offered a better balance of performance and speed for real-time navigation.
- Resolution flexibility is key: The ability to adjust the input image resolution and thus the output segmentation granularity allows for dynamic adaptation to available computational resources and task requirements.
- Attention masks reveal task relevance: Finetuning the ViT model leads to attention masks that focus more specifically on objects relevant to the navigation task, compared to the more general attention of the pretrained model.
Key Results
- A ViT model pretrained with DINO was successfully trained for coarse image segmentation using only 70 annotated images.
- The 1-block ViT model, when run at 240p resolution, achieved excellent performance in lane following, with only a few minor infractions and zero major infractions over multiple test runs. This outperformed a baseline system and even a higher-capacity 3-block ViT in terms of navigation success.
- For obstacle avoidance, the 1-block ViT at 240p also performed best, with minimal infractions.
- The models demonstrated good performance across different input resolutions (240p, 480p, 960p), with the 240p resolution offering the best trade-off for real-time navigation.
- Inference speed was competitive, with models capable of running at a reasonable frame rate on a GPU, and even potentially on a CPU for lower resolutions.
Limitations and Future Work
- Coarse segmentation limitations: The authors acknowledge that their approach is not suited for tasks requiring high-resolution segmentation masks, where an encoder-decoder architecture would be preferred.
- Safety-critical applications: While the low-resolution predictions did not hinder navigation, the authors suggest further research into how these high-throughput, low-resolution predictions affect safety-critical applications by scaling the method to more challenging scenarios.
- Robot-specific pretraining: They propose that training perception models using SSL on sensory data collected directly from the robot itself, rather than generic datasets like ImageNet, could yield visual representations even better adapted to downstream visual servoing tasks.
Summary updated