MergeNet: A Deep Net Architecture for Small Obstacle Discovery

Krishnam Gupta, Syed Ashar Javed, Vineet Gandhi, K. Madhava Krishna

Abstract

We present here, a novel network architecture called MergeNet for discovering small obstacles for on-road scenes in the context of autonomous driving. The basis of the architecture rests on the central consideration of training with less amount of data since the physical setup and the annotation process for small obstacles is hard to scale. For making effective use of the limited data, we propose a multi-stage training procedure involving weight-sharing, separate learning of low and high level features from the RGBD input and a refining stage which learns to fuse the obtained complementary features. The model is trained and evaluated on the Lost and Found dataset and is able to achieve state-of-art results with just 135 images in comparison to the 1000 images used by the previous benchmark. Additionally, we also compare our results with recent methods trained on 6000 images and show that our method achieves comparable performance with only 1000 training samples.

Introduction

For autonomous vehicles, detecting "small" obstacles (typically 15–25 cm tall) like bricks or stones is a critical safety challenge. Standard sensors, such as 3D LIDAR, often fail to detect these objects at distances exceeding 10 meters. While low-cost stereo cameras are an alternative, they frequently struggle to distinguish small objects from the road surface due to sensor noise and visual ambiguity.

Deep learning offers a potential solution, but it is hindered by two major obstacles: data scarcity and visual confusion. Manually annotating thousands of tiny obstacles across diverse conditions is labor-intensive, and standard networks often mistake road markings for physical hazards. MergeNet addresses these issues through a specialized architecture designed to achieve high performance with significantly less training data.

The MergeNet Architecture

MergeNet is a three-stage architecture that processes RGB-D (Color + Depth) data by decomposing the scene into local and global components.

The MergeNet architecture: Stripe-net and Context-net outputs are fused by Refiner-net for precise obstacle detection.

The architecture consists of three primary modules:

  • Stripe-net: This module processes narrow vertical strips of the image rather than the full frame. This design exploits the geometric reality that road depth typically follows a predictable vertical gradient. By sharing weights across these strips, the model maintains a low parameter count, which acts as a powerful regularizer against overfitting on small datasets.
  • Context-net: This branch analyzes the full RGB image to understand the global layout, identifying boundaries such as sidewalks and road edges. It provides the necessary "big picture" context that the localized Stripe-net lacks.
  • Refiner-net: The final stage fuses the detailed but noisy detections from Stripe-net with the smooth, coarse labels from Context-net. This results in a high-fidelity segmentation map that accurately identifies obstacles while maintaining clean boundaries.

Key Insights and Data Efficiency

The effectiveness of MergeNet stems from several core design principles. First, it treats verticality as a geometric feature, baking a depth prior directly into the architecture. Second, the use of weight sharing across vertical slices allows the model to learn a universal representation of "road-ness" and "obstacle-ness," enabling it to generalize from a very small number of examples.

The model was rigorously tested on the Lost and Found dataset, a benchmark specifically for small obstacle discovery. The authors evaluated the model using both a full set of 1,036 images and a drastically reduced set of only 135 images.

Results and Performance

MergeNet demonstrates remarkable data efficiency and robustness. Even when trained on just 135 images, it achieved an Instance-wise Detection Rate (IDR) of 73.4%, representing a 19% improvement over previous state-of-the-art methods that required nearly ten times more data.

Final segmentation output showing successful identification of small road obstacles with clean boundaries.

When trained on the full dataset (1,036 images), MergeNet reached an IDR of 82.05%. Beyond detection rates, the model significantly reduced false positives, lowering the Instance-wise False Positive rate from 2.03 to 0.69. This improvement is largely due to its ability to ignore road markings that often confuse simpler models. Furthermore, the system operates at 5 FPS on an NVIDIA GTX 1080 Ti, suggesting its potential for real-time application in driver assistance systems.

Conclusion and Future Work

While MergeNet sets a new benchmark for small obstacle detection, some challenges remain. The authors noted failure scenarios involving ultra-low obstacles (e.g., thin planks), objects very close to the camera being misclassified as off-road, and irregular artifacts at road edges. Future research will focus on expanding the architecture to detect a broader range of classes, such as pedestrians and potholes, while maintaining its signature data-efficient approach.

Summary updated

Abstract

We present here, a novel network architecture called MergeNet for discovering small obstacles for on-road scenes in the context of autonomous driving. The basis of the architecture rests on the central consideration of training with less amount of data since the physical setup and the annotation process for small obstacles is hard to scale. For making effective use of the limited data, we propose a multi-stage training procedure involving weight-sharing, separate learning of low and high level features from the RGBD input and a refining stage which learns to fuse the obtained complementary features. The model is trained and evaluated on the Lost and Found dataset and is able to achieve state-of-art results with just 135 images in comparison to the 1000 images used by the previous benchmark. Additionally, we also compare our results with recent methods trained on 6000 images and show that our method achieves comparable performance with only 1000 training samples.
BibTeX
@inproceedings{gupta2018mergenet,
  author = {Krishnam Gupta and Syed Ashar Javed and Vineet Gandhi and K. Madhava Krishna},
  title = {{MergeNet: A Deep Net Architecture for Small Obstacle Discovery}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2018},
  url = {https://openreview.net/forum?id=VNYdq1TBbbk},
}
Figure