Demo: Learning to Perceive Long-Range Obstacles Using Self-Supervision from Short-Range Sensors

Mirko Nava, Jérôme Guzzi, R. Omar Chavez-Garcia, Luca Maria Gambardella, Alessandro GiustiAAAI 2019 · DOI

Abstract

This paper presents a self-supervised learning approach that enables a mobile robot to perceive obstacles at long range using only a standard camera and short-range proximity sensors. By utilizing future sensor readings as training labels for current camera views, the robot learns to predict obstacles up to 30cm away—effectively tripling the range of its physical sensors without manual data labeling.

Introduction

Small, low-cost robots frequently rely on infrared (IR) or ultrasonic proximity sensors for obstacle avoidance. While reliable, these sensors typically have a very short range (often under 10cm), forcing the robot to react only immediately before a collision. This constraint leads to jerky movements and inefficient navigation.

To achieve smoother, proactive navigation, robots generally require expensive LIDAR or complex computer vision systems that depend on massive, manually labeled datasets. This work addresses the challenge of long-range perception on a budget: how can a robot autonomously learn to interpret visual data to predict distant obstacles using its existing, limited hardware?

Methodology

The researchers employ a temporal self-supervision strategy that allows the robot to learn from its own experiences. The process involves four key stages:

  1. Exploration: A "Mighty Thymio" robot autonomously explores various environments, recording synchronized video, odometry, and proximity sensor data.
  2. Automated Labeling: For any camera frame at time $t$, the system looks ahead in the recorded trajectory. If the robot later encounters an obstacle at a distance $d$ centimeters in front of its position at time $t$, that sensor "hit" becomes the ground-truth label for that specific distance in the earlier visual frame.
  3. Model Architecture: A Convolutional Neural Network (CNN) is trained to map single camera frames to a $31 \times 5$ grid of binary labels, representing predicted sensor outputs for 5 sensors across 31 distance increments (0 to 30 cm).
  4. Masked Loss: Because the robot's path is not always linear, future data is not available for every distance in every frame. A masked loss function is used to ensure the network learns only from distances where ground truth was successfully collected.
The Mighty Thymio robot uses its camera and short-range proximity sensors to learn long-range obstacle detection by projecting future sensor hits onto current visual frames.

Experimental Setup

The system was validated using a custom dataset collected by a Mighty Thymio robot equipped with five forward-facing IR sensors and a 720p webcam. The dataset includes 90 minutes of autonomous exploration across 10 diverse environments featuring various floor textures and obstacle types.

Results and Discussion

The core insight is that a robot's future experience serves as a perfect label for its current visual input, bypassing the need for human intervention. By associating physical sensor readings with visual features seen from a distance, the robot extends its effective perception range from ~8cm to 30cm.

Heatmap showing the model's predictive performance (AUC) across different distances and sensor positions. Accuracy typically decreases as distance increases.

Key results include:

  • Predictive Navigation: The multi-distance depth map allows for smoother steering, as the robot can anticipate and avoid obstacles much earlier than reactive IR-based systems.
  • Performance Decay: As shown in the performance heatmaps, prediction accuracy is highest at medium ranges and naturally decreases as the distance approaches the 30cm limit.
  • Blind Spots: Obstacles very close to the robot or at the extreme periphery proved harder to detect. This is attributed to the camera's field of view (FOV) being narrower than the proximity sensors' coverage in certain areas.

Conclusion

This self-supervised approach effectively triples a robot's perception range without requiring manual labeling. While successful, limitations remain regarding FOV mismatches and odometry drift on slippery surfaces. Future work will explore more complex geometric projections to incorporate data from turning maneuvers and improve the robustness of the predictive model.

Summary updated

BibTeX
@inproceedings{nava2019demo,
  author = {Mirko Nava and Jérôme Guzzi and R. Omar Chavez-Garcia and Luca Maria Gambardella and Alessandro Giusti},
  title = {{Demo: Learning to Perceive Long-Range Obstacles Using Self-Supervision from Short-Range Sensors}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33019867},
}
Figure