Off-Road Obstacle Avoidance through End-to-End Learning

Yann LeCun, Urs Muller, Jan Ben, Eric Cosatto, Beat FleppNeurIPS 2005 · OpenReview

Autonomous off-road navigation is notoriously difficult. Unlike paved roads with clear lane markings, off-road environments are highly unstructured, featuring diverse obstacles like tall grass, bushes, and rocks under varying lighting and weather conditions. Traditional approaches typically rely on active sensors (like LIDAR) or explicit stereo vision to build 3D range maps. However, active sensors can be bulky and power-hungry, while traditional stereo algorithms often fail in "messy" outdoor scenes where textures are repetitive or edges are poorly defined.

This paper presents a vision-based navigation system for off-road robots that bypasses traditional geometric mapping in favor of end-to-end supervised learning. By training a 6-layer convolutional neural network (CNN) to mimic the steering decisions of a human driver, the researchers demonstrated a robot capable of navigating complex, unstructured environments in real-time at speeds up to 2 m/s.

System Architecture and Hardware

The core of the system is a 6-layer Convolutional Neural Network (CNN). Instead of the network outputting a geometric map of the world, it maps raw pixels directly to steering angles—an approach known as end-to-end learning.

  • Architecture: The CNN processes a single pair of left/right images. It consists of layers of trainable, shift-invariant linear filters followed by non-linear saturation functions. This architecture allows the network to automatically learn relevant local features (like edges or textures) and potentially even estimate distances (disparity) implicitly.
  • Input: The network takes 6 planes of 149x58 pixels, representing the Y, U, and V color components for both the left and right cameras.
The modified 1/10th scale RC truck used as the experimental platform.

The hardware platform is a modified 1/10th scale RC truck (50cm long) equipped with two wireless color cameras. Processing is handled by a remote 1.4GHz Athlon PC, which communicates with the robot via radio.

Datasets and Environments

To train the system, the researchers collected a custom dataset utilizing supervised learning. A human driver remotely operated the robot through various terrains while the system recorded the video feed and the corresponding steering angles. The network was then trained to minimize the discrepancy between its predicted steering and the human's actual steering.

Example of the unstructured off-road environments the robot navigates.
  • Scale and Diversity: The dataset contains approximately 127,000 individual pairs of frames collected over 17 different days. Environments included parks, playgrounds, construction sites, and suburban yards.
  • Split: 95,000 frame pairs were used for training, and 32,000 were reserved for validation and testing.
  • Real-world Noise: The dataset intentionally included "bad" data, such as frames with wireless dropouts, motion blur, and extreme lighting differences between cameras, to ensure the final model was robust to real-world conditions.

Key Insights and Results

One of the most powerful insights from this work is that a CNN can learn to understand depth and distance implicitly. By providing stereo input, the network learns to combine features from both eyes to estimate the proximity of obstacles without being explicitly programmed with geometric rules. Furthermore, end-to-end learning eliminates the need for hand-tuned feature detectors (e.g., "look for green for grass"), as the network discovers which visual cues are most predictive of a safe path. Because the network is trained on raw data from two different cameras, it also learns to be robust to hardware imperfections, such as differences in white balance or brightness between the left and right sensors.

In practice, the system achieved impressive results:

  • Speed: The robot successfully navigated at 2 m/s, a high speed for a robot of that scale.
  • Accuracy: On the test set, the system achieved a classification error rate of 35.8% when steering was quantized into three bins (left, straight, right). The authors note that this numerical error rate is misleadingly high because there are often multiple "correct" ways to avoid an obstacle (e.g., turning left vs. right).
  • Efficiency: Despite having 3.15 million connections, the network was optimized to run at 10 cycles per second, including all radio and video processing overhead.

Limitations and Future Work

While highly effective for obstacle avoidance, the system has limitations. It occasionally made errors when driving directly toward a low sun, which would wash out the camera sensors. Additionally, the current model is purely reactive; it makes decisions based only on the current frame and has no memory. The authors suggest that using multiple successive frames could improve segmentation, though they noted that in supervised learning, this often leads the network to simply "copy" the previous steering angle rather than learning to see.

Finally, the system lacks high-level reasoning. The authors acknowledge that a fully autonomous system would need additional components for map building, global path planning, and spatial reasoning to complement this reactive obstacle avoidance layer.

Summary updated

Abstract

We describe a vision-based obstacle avoidance system for off-road mobile robots. The system is trained from end to end to map raw in put images to steering angles. It is trained in supervised mode to predict the steering angles provided by a human driver during training r uns collected in a wide variety of terrains, weather conditions, lighting conditions, and obstacle types. The robot is a 50cm off-road truck, with two f orwardpointing wireless color cameras. A remote computer process es the video and controls the robot via radio. The learning system is a lar ge 6-layer convolutional network whose input is a single left/right pair of unprocessed low-resolution images. The robot exhibits an excellent ability to detect obstacles and navigate around them in real time at speeds of 2 m/s.
BibTeX
@inproceedings{lecun2005off,
  author = {Yann LeCun and Urs Muller and Jan Ben and Eric Cosatto and Beat Flepp},
  title = {{Off-Road Obstacle Avoidance through End-to-End Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2005},
  url = {https://openreview.net/forum?id=ByZ7F8bO-r},
}
Figure