HiPFuse: Real-Time LiDAR Semantic Segmentation for Robotic Perception on Edge Devices

Haiyang Wu, George Vosselman, Ville LehtolaarXiv 2026 · arXiv

TL;DR

HiPFuse is a novel point–BEV (Bird’s-Eye-View) fusion framework designed for real-time LiDAR semantic segmentation on resource-constrained edge devices. By preserving vertical structure through height partitioning and fusing it with point-wise details, it achieves state-of-the-art efficiency, significantly improving downstream robotic tasks like odometry and mapping.

What Problem Does This Paper Address?

LiDAR semantic segmentation—the task of labeling every point in a 3D scan (e.g., "road," "tree," "car")—is a cornerstone of robotic perception. However, robots often operate on "edge" hardware with limited computational power.

To save time, many models project 3D points into a 2D Bird’s-Eye-View (BEV) grid. While efficient, this "flattens" the world, often mixing points from different heights (like a tree canopy overhanging a sidewalk) into the same grid cell. This loss of vertical detail leads to classification errors. Existing models that preserve 3D detail are often too slow for real-time use, while fast 2D models often lack the precision needed for robust navigation.

Methods

The authors introduce HiPFuse, which balances speed and detail through several key innovations:

  • Height-Partitioned Projection: Instead of collapsing all points into a single 2D layer, HiPFuse divides the space into $Z$ fixed height partitions. This preserves vertical separation throughout the spatial processing stages.
  • Point–BEV Fusion Backbone: The architecture (shown in Figure 2) features two branches that "talk" to each other. The point branch maintains individual point descriptors for fine detail, while the BEV branch uses 2D convolutions to learn broad spatial context. They exchange features across multiple scales.
  • Dual-Scale BEV Context (DSC) Module: To understand large structures without a massive computational hit, the DSC module (Figure 3) extracts context from a coarser, downsampled grid and fuses it back into the main feature map.
  • Auxiliary Supervision: During training, the model is given extra "hints" by supervising intermediate features with labels from 3D cells, forcing the network to learn height-aware distinctions early on.

![Fig.

  1. Overview of the HiPFuse architecture.](/figures/art_f02ee646099a4432869663cf293855bd)

Datasets

The model was evaluated on three distinct benchmarks:

  1. SemanticKITTI: The standard large-scale dataset for outdoor urban autonomous driving.
  2. MCD (Multi-Campus Dataset): A challenging outdoor dataset featuring non-repetitive scan patterns from Livox LiDARs.
  3. Semantic-ITC: A newly introduced benchmark for indoor semantic segmentation.

Key Insights and Ideas

  • Verticality is a First-Class Citizen: The authors show that simply adding height partitions isn't enough. You need specific components—like relative-z encoding and height-aligned auxiliary losses—to actually leverage that vertical information for better accuracy.
  • The "Sweet Spot" of Fusion: By combining 2D convolutions (which are highly optimized on modern hardware) with a sparse point branch, HiPFuse avoids the "heavy" math of 3D convolutions while outperforming pure projection-based methods.
  • Perception for Action: A major "aha" moment is the demonstration that better segmentation scores (mIoU) directly translate to better robot performance. High-quality semantic masks lead to more stable odometry and more reliable loop closure (recognizing a place you've been before).

Key Results

HiPFuse demonstrates a superior trade-off between accuracy and throughput, as seen in Figure 1.

  • Edge Performance: On an NVIDIA Jetson AGX Orin, HiPFuse achieves 63.7% mIoU at 14.2 FPS on SemanticKITTI and 62.5% mIoU at 22.4 FPS on MCD. Crucially, both exceed the standard 10 Hz LiDAR scan rate required for real-time operation.
  • Downstream Impact: When plugged into a robotic pipeline, HiPFuse reduced odometry error (ATE RMSE) by 10.6% and increased "accepted-loop" recall by 8.7 percentage points compared to baseline semantic front-ends.
  • Qualitative Improvements: As shown in Figure 4, HiPFuse produces more continuous road labels and better distinguishes boundaries (like sidewalks vs. roads) compared to previous real-time models.

![Fig.

  1. SemanticKITTI test accuracy versus measured Jetson AGX Orin throughput. Shading indicates throughput meeting the 10 Hz scan rate.](/figures/art_21b5886b4e3b47a182a0e0dc8bd3baff)

Limitations and Future Work

The primary limitation is that HiPFuse is sensor-specific. The spatial grid (height range, cell size, and number of partitions) must be manually configured and the model retrained for different LiDAR setups.

The authors suggest that future research should focus on adaptive height partitioning—allowing the model to automatically adjust its grid based on the environment—and developing "universal" models that can handle diverse sensors without extensive reconfiguration.

Summary updated

Abstract

Real-time robotic perception requires accurate LiDAR semantics under limited resources. Bird's-eye-view (BEV) aggregation is efficient but can mix differently labeled surfaces in columns. We present HiPFuse, a point-BEV fusion framework that exchanges point and height-partitioned BEV features across horizontal scales, combining point-wise detail with 2D spatial context from one LiDAR scan. Fixed height assignments preserve separate vertical aggregation groups, with auxiliary supervision of initial BEV features using targets from the same 3D cells. A dual-scale BEV context module adds broader context from a coarser grid. On SemanticKITTI and MCD, HiPFuse achieves 63.7% and 62.5% mIoU at 14.2 and 22.4 FPS on Jetson AGX Orin, respectively, exceeding the 10 Hz scan rate. Separate training and evaluation show indoor applicability. Replacing only the baseline semantic front end with HiPFuse in fixed robotic systems reduces odometry absolute trajectory error RMSE by 10.6% on average across two outdoor sequences and increases accepted-loop recall by 8.7 percentage points on SemanticKITTI. Qualitative semantic maps show more continuous road labeling. Our code is available at https://github.com/haiyang2022/HiPFuse.
BibTeX
@misc{wu2026hipfuse,
  author = {Haiyang Wu and George Vosselman and Ville Lehtola},
  title = {{HiPFuse: Real-Time LiDAR Semantic Segmentation for Robotic Perception on Edge Devices}},
  year = {2026},
  eprint = {2611.14025},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14025},
}
Figure