Improving Zero-shot 3D Semantic Mapping in Offroad with Geometric Priors

Eerik Alamikkotervo, Risto OjalaarXiv 2026 · arXiv

Navigating a robot through a structured city street is one thing; guiding it through a dense, unstructured forest is quite another. While modern Vision Foundation Models (VFMs) like Segment Anything (SAM) have revolutionized image segmentation, they often stumble in the wild, where visual boundaries between different types of vegetation are blurry at best.

A new paper titled "Improving Zero-shot 3D Semantic Mapping in Offroad with Geometric Priors" by Eerik Alamikkotervo and Risto Ojala proposes a clever way to fix this: by using the robot's 3D LiDAR data to "double-check" the vision model's homework.

TL;DR

The authors introduce a method to enhance zero-shot 3D semantic mapping in offroad environments by fusing Vision Foundation Model (VFM) predictions with geometric priors—specifically elevation and sphericity—derived from LiDAR point clouds. By using the VFM's most confident predictions to "anchor" the geometric properties of each class, they achieve significant performance boosts (up to +6.3 mIoU on RELLIS-3D) without any environment-specific training.

What Problem Does This Paper Address?

Autonomous offroad navigation requires a deep semantic understanding of the environment. A robot needs to know if it's looking at traversable grass, a crushable bush, or a solid tree trunk.

Current state-of-the-art VFMs are trained on massive datasets and can segment images "zero-shot" (without seeing the specific target domain before). However, these models are mostly trained on structured, human-centric scenes. In natural environments, they struggle because different types of vegetation often look visually similar.

While 3D LiDAR data provides rich geometric information, most offroad research simplifies this into a binary "traversable vs. non-traversable" map. This paper aims to bridge that gap, using 3D geometry to help VFMs distinguish between complex natural semantic classes.

Fig. 1: The proposed method improves semantic mapping by fusing the vision foundation model prediction with a geometric prior computed from the pointcloud.

Methods

The researchers developed a four-stage pipeline that operates without manual labeling:

  1. Zero-shot Segmentation: An open-vocabulary VFM (like SAM3 or DINOv2) generates initial semantic confidence maps from camera images.
  2. Point Cloud Accumulation: LiDAR scans are projected into these images to inherit semantic scores and then accumulated into a global 3D point cloud map.
  3. Geometric Feature Extraction: For every point in the 3D map, the system calculates two key features:
  4. Relative Elevation: The height of a point relative to the local ground.
  5. Sphericity: A measure of "fluffiness" derived from the local 3D covariance matrix. Flat surfaces (ground) have low sphericity, while voluminous objects (foliage) have high sphericity.
  6. Geometric Anchoring & Fusion: This is the "secret sauce." The system takes the top 5% most confident VFM predictions for each class and fits a 2D elevation-sphericity histogram. This histogram acts as a geometric signature for that class in that specific environment. Finally, these histograms are used as priors and fused with the original VFM scores using a Bayesian-inspired product.
Fig. 2: Geometric prior computation and utilization illus- trated. The most confident image segmentation predictions are used to fit the sphericity-elevation distributions for each class, which are then used to assign prior weights for each point in the fusion phase.

Datasets

The method was evaluated on two challenging real-world offroad datasets:

  • RELLIS-3D: Collected in rural Texas, featuring fields and trails dominated by grass, bushes, and small trees.
  • WildScenes: Captured on Australian forest trails, characterized by high, dense tree canopies.

Key Insights and Ideas

  • Self-Correction via Confidence: By only using the "Top 5%" most confident VFM predictions to build the geometric histograms, the model effectively ignores the VFM's noisier guesses, creating a high-quality "geometric anchor" for each class.
  • Sphericity as a Natural Discriminator: The authors found that sphericity is an excellent feature for natural scenes. Tree trunks and ground have low sphericity, while foliage is high. This geometric distinction is often much clearer than the visual distinction in a flat 2D image.
  • Natural vs. Structured Performance: The geometric prior significantly helps with natural classes (bush, grass, dirt) but has less impact on structured classes (poles, barriers). This is because natural classes have more distinct, consistent geometric signatures in unstructured environments compared to their visual appearance.
Fig. 3: Sphericity and elevation visualized. Tree trunks and even ground have low sphericity and tree foliage and vegetation high sphericity.

Key Results

The addition of geometric priors yielded substantial improvements across the board:

  • On RELLIS-3D: The SAM3 baseline improved by +6.3 in mIoU and +8.9 in accuracy. Some specific classes saw massive gains, such as Bush (+44.2 IoU) and Water (+28.7 IoU).
  • On WildScenes: The method provided a +1.7 mIoU and +2.1 accuracy boost.
  • Comparison to Baselines: The proposed method outperformed other zero-shot 3D segmentation approaches like OTAS, which often confuses visually similar classes like dirt and gravel.
Fig. 4: Elevation-sphericity histograms fitted on the top 5 % SAM3 detections on Rellis3D. Number of samples used for fitting each class in parentheses.

Limitations and Future Work

While powerful, the method has a few hurdles to clear:

  • Class Separability: Some classes share very similar geometric profiles (e.g., a fence vs. a barrier). Elevation and sphericity alone aren't enough to tell them apart.
  • VFM Dependency: If the vision model is consistently wrong—even when it's "confident"—the geometric prior will be built on bad data.
  • Histogram Sparsity: The current histogram approach treats an "empty" bin in a sparse area the same as one in a dense area. The authors suggest that future work could use spatial smoothing or parameterized distributions (like 2D Gaussians) to make the priors more robust.
  • Feature Expansion: Incorporating more geometric features (like planarity or linearity) could further improve the ability to distinguish between complex objects.

Summary updated

Abstract

Semantic 3D mapping is a powerful representation that enables platform agnostic mobile robot navigation in challenging offroad conditions. Current vision foundation models can accurately predict semantics zero-shot on structured scenes but performance is compromised in natural environments. On the other hand, 3D point clouds encode valuable information for discriminating semantics in scenarios where vision foundation models fail. In this work, we propose fitting a sphericity-elevation distribution for each semantic class to act as geometric anchor. Only the points with the most confident vision foundation model predictions are considered when fitting to produce high quality geometric prior distributions. The geometric prior is fused with the vision foundation model prediction to produce the final semantic mapping. The proposed prior improves SAM3 baseline by +6.3 in mIoU and +8.9 in accuracy on the RELLIS-3D dataset, and by +1.7 in mIoU and +2.1 in accuracy on the WildScenes dataset. The source code will be made public upon paper acceptance.
BibTeX
@misc{alamikkotervo2026improving,
  author = {Eerik Alamikkotervo and Risto Ojala},
  title = {{Improving Zero-shot 3D Semantic Mapping in Offroad with Geometric Priors}},
  year = {2026},
  eprint = {2611.24696},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24696},
}

Loading related content...

Figure