Seeing Through the Grass: Semantic Pointcloud Filter for Support Surface Learning

Anqiao Li, Chenyu Yang, Jonas Frey, Joonho Lee, Cesar Cadena, Marco Hutter

Introduction

Autonomous ground robots operating in natural environments frequently encounter a fundamental limitation of standard exteroceptive sensing: the "rigid world" assumption. Most LiDAR-based mapping systems treat any laser return as a solid obstacle. In the wild, however, many surfaces are penetrable. Tall grass, ferns, and soft bushes can be stepped through, yet a standard LiDAR perceives them as impassable walls or high mounds.

This discrepancy leads to two critical failures in autonomous navigation. First, it creates "false obstacles," where a robot may refuse to enter a traversable meadow because it perceives the vegetation as a solid barrier. Second, it results in inaccurate mapping, where path planners attempt to "step over" grass, leading to unstable locomotion because the perceived support surface is significantly higher than the actual ground. To address this, we present the Semantic Pointcloud Filter (SPF), a multi-modal neural network designed to identify and filter penetrable vegetation to reveal the true underlying support surface.

The Semantic Pointcloud Filter (SPF)

The SPF is designed as a semantic-geometric filter that adjusts raw sensor measurements to align with the load-bearing ground. The architecture utilizes a U-Net-based Convolutional Neural Network (CNN) with an EfficientNet-B5 backbone.

Multi-modal Integration

The system processes a stacked tensor consisting of two primary inputs:

  1. RGB Images: A 3-channel image providing rich semantic context.
  2. Sparse Depth Maps: Generated by projecting raw LiDAR points into the camera frame to provide geometric grounding.

By combining these modalities, the network leverages the strengths of both: LiDAR provides precise geometry but struggles with thin structures, while RGB provides semantic clarity but is sensitive to lighting conditions. The model concurrently predicts a binary semantic mask—identifying regions as either "support surfaces" (penetrable ground) or "rigid obstacles" (trees, rocks, fences)—and a refined depth map that estimates the distance to the actual ground hidden beneath vegetation.

Learning from Interaction: Semi-Self-Supervised Training

A significant challenge in training such a system is the difficulty of obtaining ground truth data; humans cannot easily label the exact depth of the ground beneath a thick patch of grass from an image. The authors overcome this using a "semi-self-supervised" approach that turns the robot's physical interaction with the world into a teaching signal.

By recording the history of the robot's footholds—the points where its feet actually make contact with the solid ground—the system generates its own ground truth. These discrete contact points are interpolated using Gaussian Processes (GP) to create a continuous estimated support surface. This allows the robot to use its "sense of touch" (proprioception) to supervise its "sense of sight" (exteroception). Manual labeling was reserved only for a small subset of 300 images to assist with the semantic segmentation task.

Experimental Results

The SPF was evaluated using an ANYmal quadruped robot across diverse outdoor settings, including grasslands and forests in Perugia, Italy, and urban environments in Zurich, Switzerland.

Mapping Accuracy

In meadow terrain, the SPF achieved a 48% improvement in Root Mean Squared Error (RMSE) compared to raw LiDAR measurements. By shifting "penetrable" points downward to the estimated ground level while preserving "rigid" points, the filter produces a reconstructed point cloud that is far more accurate for navigation tasks.

Robustness and Navigation

Ablation studies confirmed that the multi-modal approach outperformed single-modality versions (RGB-only or LiDAR-only) by approximately 14%. Qualitatively, the filter successfully "lowered" the perceived height of tall grass in elevation maps. This allowed the robot to correctly identify and navigate traversable paths that were previously classified as blocked. In complex forest scenes, the system demonstrated the ability to correctly preserve rigid structures like fences while filtering out surrounding brush.

Conclusion and Future Work

The Semantic Pointcloud Filter represents a significant step toward robust off-road autonomy. By moving away from the rigid world assumption and utilizing self-supervised learning from physical interaction, robots can navigate complex natural terrains with greater confidence.

Future research will focus on improving the system's performance under extreme lighting conditions and exploring its generalization to different seasons, such as environments covered in snow or autumn leaves. Additionally, the authors aim to explore anomaly detection to further reduce the need for manual semantic labeling, moving toward a fully autonomous training loop.

Summary updated

BibTeX
@article{li2023seeing,
  author = {Anqiao Li and Chenyu Yang and Jonas Frey and Joonho Lee and Cesar Cadena and Marco Hutter},
  title = {{Seeing Through the Grass: Semantic Pointcloud Filter for Support Surface Learning}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2023},
  doi = {10.1109/lra.2023.3320016},
}
Figure