LSON-IP: Lightweight Sparse Occupancy Network for Instance Perception

Xinwang Zheng, Hanyu Cai, Yang Lu, Chengyu Lu, G YangWorld 2026 · DOI

TL;DR

This paper introduces LSON-IP, a novel framework that tackles the computational burden of 3D semantic occupancy prediction by moving from dense voxel grids to a sparse, instance-centric approach. By leveraging a Sparse Instance Aggregator (SIA) and a Differentiable Sparse Rendering (DSR) technique for 2D supervision, LSON-IP achieves competitive accuracy with over 60% reduction in computational cost, making it more suitable for real-time applications like autonomous driving.

What Problem Does This Paper Address?

Accurate 3D scene understanding is crucial for autonomous systems, and semantic occupancy prediction offers a detailed, voxel-level representation of geometry and semantics. However, existing methods that achieve high accuracy typically rely on dense 3D voxel grids. This dense representation leads to extremely high computational demands (memory and processing power), making real-time deployment challenging, especially for safety-critical applications where perceiving dynamic objects is paramount. Furthermore, obtaining dense 3D annotations for training is costly and time-consuming. The paper addresses this by questioning the necessity of dense prediction for all voxels, particularly when the focus is on dynamic instances, and proposes a more efficient, sparse approach that can be trained with readily available 2D supervision.

Methods

LSON-IP employs a "propose-then-refine" strategy:

  • Sparse Instance Query Proposal: Instead of a dense grid, the process begins by generating a sparse set of 3D "instance queries." These queries are initialized by first performing 2D instance segmentation on multi-view images. Pixels within detected instance masks are then "lifted" into 3D space using predicted depth and camera parameters. This creates an initial sparse cloud of points concentrated on potential objects.
  • Sparse Instance Aggregator (SIA): This novel attention-based module refines the initial sparse queries. It consists of stacked layers, each with:
  • Cross-Attention: Queries attend to multi-view 2D image features to gather photometric and geometric evidence. This step uses deformable attention to be robust to small errors in query positions.
  • Self-Attention: Queries interact with each other to model inter-object relationships and enforce geometric coherence, enabling the reconstruction of complete object shapes.
  • Differentiable Sparse Rendering (DSR): A key innovation to enable end-to-end training with 2D supervision. DSR creates a continuous density and semantic field from the discrete sparse particles by interpolating from the K-nearest neighbors. This allows for standard volume rendering techniques to be applied, generating 2D semantic and depth maps that can be compared to ground truth, thereby enabling gradient flow back to the sparse 3D representation.

Datasets

The paper evaluates LSON-IP on two large-scale autonomous driving benchmarks:

  • nuScenes: Provides 1000 scenes with 3D bounding box annotations for 23 object classes, used to create ground truth occupancy grids.
  • SemanticKITTI: Based on the KITTI Odometry Benchmark, offering point-wise semantic annotations for 22 sequences, used for semantic scene completion.

Key Insights and Ideas

  • Sparse is Sufficient (and Efficient): The core idea is that for instance-centric perception, focusing computational resources on sparse regions likely to contain objects is more efficient and can achieve comparable or better results than processing dense grids.
  • Instance-First Reasoning: The SIA module's design prioritizes understanding individual instances and their relationships, which is crucial for dynamic perception tasks.
  • Bridging Sparse 3D and 2D Supervision: DSR is a critical enabler, allowing the use of readily available 2D labels to train sparse 3D representations, circumventing the need for expensive 3D annotations.
  • Attention for Context: The use of both cross-attention (for 2D-3D fusion) and self-attention (for inter-query reasoning) within the SIA is vital for building coherent object representations from fragmented cues.

Key Results

  • Efficiency: LSON-IP achieves a computational reduction of over 60% compared to leading dense methods like TPVFormer and C-CONet. For instance, it uses 413 GFLOPs compared to TPVFormer's 1132 GFLOPs and C-CONet's 1810 GFLOPs on SemanticKITTI.
  • Accuracy:
    • On SemanticKITTI, LSON-IP achieves a competitive mIoU of 13.32%, close to OccFormer (13.46%), while being significantly more efficient.
  • On nuScenes-Occupancy, it sets a new state-of-the-art mIoU of 14.0%.
  • Per-Category Performance: LSON-IP shows strong performance on dynamic object classes and excels in large objects (Car, Bus) and small critical elements (Pedestrian, Traffic Cone).
  • Qualitative Results: Visualizations show crisp, well-defined point-cloud representations that accurately capture instance geometry and scene layout.

Limitations and Future Work

  • Small/Thin Objects: Performance on extremely small or thin objects (e.g., poles, traffic signs) is subpar due to potential failure in the sparse query proposal and over-smoothing by KNN interpolation in DSR.
  • Long-Tailed Categories: Infrequently occurring classes (e.g., 'motorcyclist') suffer from low IoU due to insufficient training data.
  • 2D Segmentation Dependency: The pipeline relies on a 2D instance segmentation proposal, meaning missed proposals in complex or occluded scenarios can lead to complete failure to perceive an instance.
  • Future Work: The authors suggest incorporating temporal cues for robustness, developing a more adaptive query proposal strategy, implementing a lightweight memory bank for temporal consistency across frames, and exploring self-supervised learning with unlabeled data to reduce reliance on 3D annotations.

Summary updated

Abstract

The high computational demand of dense voxel representations severely limits current vision-centric 3D semantic occupancy prediction methods, despite their capacity for granular scene understanding. This challenge is particularly acute in safety-critical applications like autonomous driving, where accurately perceiving dynamic instances often takes precedence over capturing the static background. This paper challenges the paradigm of dense prediction for such instance-focused tasks. We introduce the LSON-IP, a framework that strategically avoids the computational expense of dense 3D grids. LSON-IP operates on a sparse set of 3D instance queries, which are initialized directly from multi-view 2D images. These queries are then refined by our novel Sparse Instance Aggregator (SIA), an attention-based module. The SIA incorporates rich multi-view features while simultaneously modeling inter-query relationships to construct coherent object representations. Furthermore, to obviate the need for costly 3D annotations, we pioneer a Differentiable Sparse Rendering (DSR) technique. DSR innovatively defines a continuous field from the sparse voxel output, establishing a differentiable bridge between our sparse 3D representation and 2D supervision signals through volume rendering. Extensive experiments on major autonomous driving benchmarks, including SemanticKITTI and nuScenes, validate our approach. LSON-IP achieves strong performance on key dynamic instance categories and competitive overall semantic completion, all while reducing computational overhead by over 60% compared to dense baselines. Our work thus paves the way for efficient, high-fidelity instance-aware 3D perception.
BibTeX
@inproceedings{zheng2026lson,
  author = {Xinwang Zheng and Hanyu Cai and Yang Lu and Chengyu Lu and G Yang},
  title = {{LSON-IP: Lightweight Sparse Occupancy Network for Instance Perception}},
  booktitle = {World},
  year = {2026},
  doi = {10.3390/wevj17010031},
}
Figure