MergeNet: A Deep Net Architecture for Small Obstacle Discovery

Krishnam Gupta, Syed Ashar Javed, Vineet Gandhi, K. Madhava Krishna

TL;DR

MergeNet is a specialized deep learning architecture designed to detect small, hazardous road obstacles (like bricks or stones) using significantly less training data than traditional models. By fusing local features from vertical image "stripes" with global scene context, it achieves state-of-the-art performance with as few as 135 training images, outperforming previous benchmarks that used nearly ten times more data.

What Problem Does This Paper Address?

For autonomous vehicles, "small" obstacles (15–25 cm tall) are a massive safety headache. Standard sensors like 3D LIDAR often miss these objects at distances beyond 10 meters, and low-cost stereo cameras struggle to distinguish them from the road surface due to noise.

While deep learning is a natural solution, it faces two hurdles in this niche:

  1. Data Scarcity: Manually annotating thousands of tiny obstacles in diverse road conditions is incredibly labor-intensive and hard to scale.
  2. Visual Ambiguity: Standard networks often confuse road markings (like zebra crossings) with physical obstacles because they rely too heavily on appearance cues rather than geometric structure.

Methods

The authors propose MergeNet, a three-stage architecture that processes RGB-D (Color + Depth) data. The "secret sauce" is how it breaks down the scene:

  • Stripe-net: Instead of looking at the whole image, this network processes narrow vertical strips. This design choice exploits the fact that road depth typically follows a predictable gradient along the vertical axis. By sharing weights across these strips, the model remains "parsimonious"—it has fewer parameters, making it much harder to overfit on small datasets.
  • Context-net: This branch looks at the full RGB image to understand the global layout (e.g., where the sidewalk ends and the road begins). It provides the "big picture" that the narrow Stripe-net misses.
  • Refiner-net: This final stage acts as a fuser. It takes the detailed (but sometimes noisy) detections from the Stripe-net and the smooth (but coarse) labels from the Context-net to produce a final, high-fidelity segmentation map.

The training is staggered, allowing the model to learn simple geometric features first before moving on to complex global fusion.

Datasets

The model was trained and evaluated on the Lost and Found dataset, a specialized benchmark for small obstacle discovery.

  • Training Sets: The authors tested two versions: one trained on the full set (1,036 images) and a "reduced" version trained on just 135 images.
  • Testing Set: A challenging set of 1,200 images featuring varying illumination, non-uniform road textures, and "distractor" objects.

Key Insights and Ideas

  • Verticality is a Feature: In autonomous driving, the vertical axis of an image is intrinsically linked to depth. By forcing the network to look at vertical stripes, the authors effectively "baked" a geometric prior into the architecture.
  • Weight Sharing as Regularization: By using the same parameters to process every vertical slice of the road, the model learns a universal representation of "road-ness" and "obstacle-ness," which allows it to generalize from a tiny number of examples.
  • Complementary Strengths: The paper demonstrates that Stripe-net is excellent at finding the existence of small objects but bad at boundaries, while Context-net is great at boundaries but misses the objects. The Refiner-net's role in resolving this conflict is crucial for reducing false positives.

Key Results

  • Data Efficiency: With only 135 training images, MergeNet achieved an Instance-wise Detection Rate (IDR) of 73.4%. This is a 19% improvement over the previous state-of-the-art (FPHT Stixels) which used 1,000 images.
  • Peak Performance: When trained on the full 1,036 images, MergeNet reached an IDR of 82.05%, a nearly 30% jump over earlier benchmarks.
  • Speed: The system runs at 5 FPS on an NVIDIA GTX 1080 Ti, making it potentially viable for real-time driver assistance or autonomous navigation.
  • Robustness: The model significantly reduced false positives (lowering the Instance-wise False Positive rate from 2.03 to 0.69) by successfully ignoring road markings that trip up simpler models.

Limitations and Future Work

The authors identified three primary failure scenarios:

  1. Ultra-low obstacles: Objects like thin wooden planks are still occasionally missed.
  2. Proximity issues: Obstacles very close to the camera can be misclassified as "off-road" areas.
  3. Complex boundaries: Irregular artifacts at the very edge of the road (like patches of grass) can trigger false detections.

Future Work: The team plans to extend the architecture to detect a wider variety of classes, including pedestrians, potholes, and traffic signs, while maintaining the same data-efficient approach.

Summary updated

Abstract

We present here, a novel network architecture called MergeNet for discovering small obstacles for on-road scenes in the context of autonomous driving. The basis of the architecture rests on the central consideration of training with less amount of data since the physical setup and the annotation process for small obstacles is hard to scale. For making effective use of the limited data, we propose a multi-stage training procedure involving weight-sharing, separate learning of low and high level features from the RGBD input and a refining stage which learns to fuse the obtained complementary features. The model is trained and evaluated on the Lost and Found dataset and is able to achieve state-of-art results with just 135 images in comparison to the 1000 images used by the previous benchmark. Additionally, we also compare our results with recent methods trained on 6000 images and show that our method achieves comparable performance with only 1000 training samples.
BibTeX
@inproceedings{gupta2018mergenet,
  author = {Krishnam Gupta and Syed Ashar Javed and Vineet Gandhi and K. Madhava Krishna},
  title = {{MergeNet: A Deep Net Architecture for Small Obstacle Discovery}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2018},
  url = {https://openreview.net/forum?id=VNYdq1TBbbk},
}

Loading related content...

Figure