ALIGN: Advanced Query Initialization with LiDAR-Image Guidance for Occlusion-Robust 3D Object Detection

Janghyun Baek, Mincheol Chang, Seokha Moon, Seung Joon Lee, Jinkyu KimarXiv 2025 · DOI · arXiv

Abstract

Modern 3D object detectors increasingly rely on query-based architectures, yet their performance is heavily dictated by the quality of initial query placement. Current strategies, such as random or heatmap-based initialization, often overlook small, distant, or occluded objects. We introduce ALIGN (Advanced query initialization with LiDAR-Image GuidaNce), a novel framework that fuses 2D semantic segmentation with 3D LiDAR geometry to provide precise, object-aware query initialization. ALIGN addresses the "surface bias" of LiDAR sensors through class-specific depth offsets and ensures robust coverage via adaptive neighbor sampling. Evaluated on the nuScenes benchmark, ALIGN consistently improves state-of-the-art detectors, showing particularly strong gains for heavily occluded objects.

Introduction

The transition to query-based 3D object detection, inspired by the DETR architecture, has streamlined the detection pipeline. In these systems, a set of learnable queries probes 3D space to identify and localize objects. However, the sensitivity of these models to query initialization remains a significant bottleneck.

Comparison of query initialization strategies: (a) Random, (b) Heatmap-based, and (c) the proposed ALIGN method which uses object-centric and neighbor queries.

As shown in the figure above, existing methods typically fall into two categories:

  1. Random Initialization: Queries are distributed uniformly, leading to computational waste on background regions.
  2. Heatmap-based Initialization: Queries are placed based on confidence scores from Bird’s Eye View (BEV) heatmaps. While more efficient, this method struggles with occluded or small objects that produce weak visual signatures.

ALIGN seeks to bridge this gap by making query initialization both object-aware and occlusion-robust through the synergy of image semantics and LiDAR geometry.

The ALIGN Framework

ALIGN replaces heuristic initialization with a structured generator comprising three core modules:

Occlusion-Aware Center Estimation (OCE)

LiDAR sensors only capture the visible surface of an object, leading to an inherent "surface bias" where queries are initialized off-center. OCE projects LiDAR points onto 2D image segmentation masks to associate points with specific instances. It then applies a class-specific depth offset (e.g., shifting queries further for large trucks than for pedestrians) to move the query from the visible surface to the estimated geometric center.

Adaptive Neighbor Sampling (ANS)

To mitigate the impact of imperfect segmentation, ANS utilizes LiDAR clustering (DBSCAN) to identify potential object regions. It samples "neighbor" queries around these clusters and filters them using 2D segmentation masks to ensure they remain within semantically relevant areas.

Dynamic Query Balancing (DQB)

To prevent overfitting to detected objects and maintain global scene exploration, DQB adaptively balances the query budget between object-centric queries and background queries based on the detected scene density.

Experimental Results

The ALIGN framework was evaluated as a "plug-and-play" enhancement for several top-tier detectors (CMT, TransFusion, FUTR3D, and SparseFusion) on the nuScenes dataset.

Performance Gains

ALIGN delivered consistent improvements across all baselines. Notably, for the CMT-small model, it achieved a gain of +0.9 mAP and +1.2 NDS.

Robustness to Occlusion and Scale

The most significant impact was observed in challenging scenarios:

  • Occlusion: For objects with 0–40% visibility (Level 1), ALIGN improved mAP by +1.4%, nearly double the improvement seen for fully visible objects.
  • Small Objects: Detection of small or thin categories saw substantial boosts, such as +2.2% for traffic cones and +1.6% for bicycles in low-visibility conditions.

Conclusion

ALIGN demonstrates that intelligent query initialization, grounded in multi-modal fusion, is a powerful lever for improving 3D object detection. By correcting the surface bias of LiDAR and leveraging 2D semantic guidance, it achieves superior robustness to occlusion. While the additional segmentation and clustering modules introduce some computational overhead (approx. +51ms latency on an NVIDIA A6000), the significant accuracy gains—especially for safety-critical occluded objects—make it a compelling addition to modern autonomous driving stacks.

Summary updated

BibTeX
@misc{baek2025align,
  author = {Janghyun Baek and Mincheol Chang and Seokha Moon and Seung Joon Lee and Jinkyu Kim},
  title = {{ALIGN: Advanced Query Initialization with LiDAR-Image Guidance for Occlusion-Robust 3D Object Detection}},
  year = {2025},
  eprint = {2512.18187},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.18187},
}
Figure