AmodalBox: Automated 3D Bounding Box Annotation under Partial Visibility

In-Jae Lee, Namtae Kim, Robert Exley, Kwonyoung Ryu, Jaesik ParkarXiv 2026 · arXiv

TL;DR

AmodalBox is an automated 3D bounding box annotation pipeline that solves the "partial visibility" problem in autonomous driving datasets. By using Vision Foundation Models (VFMs) to generate complete 3D object priors from images and aligning them with sparse LiDAR data, it produces high-quality "amodal" annotations that significantly outperform existing auto-labeling baselines.

What Problem Does This Paper Address?

Training 3D object detectors for autonomous driving requires massive amounts of annotated data. While manual annotation is prohibitively expensive, current automated methods struggle with partial visibility.

In typical driving scenes, LiDAR point clouds are often sparse. An object might be occluded by another car, or its distance might mean LiDAR only hits a small fraction of its surface. Most auto-labelers try to fit boxes to these visible points (a "modal" box), which results in mislocalized or undersized annotations. To truly understand a scene, we need "amodal" boxes—annotations that represent the full spatial extent of an object, even the parts we can't see.

Methods

AmodalBox bridges the gap between sparse observations and full 3D geometry through a multi-stage pipeline:

  1. Instance & Motion Cues: The system uses 2D open-vocabulary detectors and trackers to identify objects and determine their motion state (static vs. dynamic).
  2. 3D Asset Generation: For every unique object instance, the pipeline selects an "anchor frame" with the best view. It then uses an image-to-3D generation model to create a complete 3D asset (represented as 3D Gaussians) of that specific object. This serves as the amodal geometric prior.
  3. Cross-Modal Spatial Alignment: This is the core innovation. To align the generated 3D asset with the real-world LiDAR points, AmodalBox uses dense feature matching. It renders views of the 3D asset and matches them to the actual camera images. These 2D matches are "lifted" into 3D-3D correspondences, providing a much more robust alignment cue than traditional geometric methods like ICP, which often get stuck in local minima when data is sparse.
  4. Motion-Aware Reasoning:
    • Static objects: LiDAR points from multiple frames are aggregated to create a denser cloud for better alignment.
  5. Dynamic objects: The object's metric scale is estimated once at the anchor frame and then propagated across the sequence, updating only position and orientation to prevent "size jitter."
Figure 2: Overview of AmodalBox. Given time-synchronized images and LiDAR scans, we first extract instance-level cues and estimate the motion state of each instance using VFMs [18, 20], scene flow [44], and PP scores [16]. We then align the amodal 3D prior [23] with the partial LiDAR point cloud across their coordinate spaces using correspondences obtained from image-level matching. Finally, we generate a 3D bounding box annotation from the aligned amodal asset.

Datasets

The authors evaluated AmodalBox on two major industry benchmarks:

  • nuScenes: A large-scale dataset with 10 classes and 360-degree sensor coverage.
  • Waymo Open Dataset (WOD): A dataset focused on high-quality LiDAR and camera data, evaluated here within the camera frustum for three main classes (Vehicle, Pedestrian, Cyclist).

Key Insights and Ideas

  • Amodal Priors are Essential: You can't accurately guess what you can't see without a good prior. By "hallucinating" a complete 3D shape from an image, the system gains the geometric constraints needed to fill in the gaps left by sparse LiDAR.
  • The 2D-to-3D Bridge: Purely geometric alignment (LiDAR-to-Mesh) is brittle. Using dense visual features from images to guide the 3D alignment makes the process significantly more robust to noise and sparsity.
  • Temporal Stability via Scale Propagation: For moving cars, the physical size doesn't change. Estimating scale once and locking it across frames prevents the "pulsing" box effect common in other automated systems.

Key Results

AmodalBox demonstrates a massive leap in auto-labeling quality:

  • nuScenes: 3D detectors trained on AmodalBox labels improved performance by 14.1 mAP over previous state-of-the-art automated baselines.
  • Waymo (WOD): For the vehicle class, it improved Average Precision (AP) by 12.45 at a 0.5 IoU threshold.
  • Handling Large Objects: The method is particularly effective for large vehicles like buses and trailers, where LiDAR often only captures one side or corner. AmodalBox successfully recovers the full length of these vehicles.
Figure 1: Annotation from AmodalBox. For diverse classes in the nuScenes dataset [1], Baseline often struggles under partial visibility, either missing objects or producing mislocalized boxes. In contrast, AmodalBox captures more complete object extents and produces spatially coherent annotations, while human annotations are shown as reference boxes. Best viewed in color and zoomed in.

Limitations and Future Work

  • Foundation Model Errors: Since the pipeline relies on VFMs, errors in the initial 2D detection (like reflections on glass being seen as "ghost" cars) can lead to false-positive 3D annotations.
  • Adverse Weather: Heavy rain or fog can degrade the quality of 2D feature matching, leading to alignment failures.
  • Small/Deformable Objects: Pedestrians remain a challenge. They are small, have very few LiDAR returns, and change shape (deform) as they walk. The authors found that for very sparse pedestrians (fewer than 15 points), a simple "tight-fit" box often works better than complex amodal alignment. Future work could involve using deformable 3D priors to better handle human motion.

Summary updated

Abstract

Recent progress in vision foundation models (VFMs) has made automated open-vocabulary 3D bounding box annotation increasingly feasible by jointly leveraging LiDAR point clouds and images. However, partial observations remain a major challenge in sparse driving scenes, where visible LiDAR points often cover only a small fraction of the object's surface and provide insufficient cues for estimating its full spatial extent. As a result, existing methods often produce mislocalized boxes or leave objects unlabeled. To address this limitation, we propose \textbf{\Ours{}}, an automated annotation framework that leverages image-generated 3D assets as amodal geometric priors for partially observed objects. By establishing correspondences between input images and rendered views of 3D assets, \Ours{} robustly aligns canonical amodal geometry with partially observed world-space LiDAR point clouds. To improve temporal consistency and reduce computational cost, \Ours{} generates a single asset per instance, estimates its metric scale once at a single frame and propagates it across frames for dynamic instances, and aggregates multi-frame observations for static instances. 3D object detectors trained on annotations from \Ours{} improve performance over strong baselines by 14.1 mAP on the nuScenes dataset and by 12.45 AP at IoU 0.5 for the vehicle class on the Waymo Open Dataset (WOD). Our project page is available at: \href{https://oliver0922.github.io/AmodalBox/}{https://oliver0922.github.io/AmodalBox/}
BibTeX
@misc{lee2026amodalbox,
  author = {In-Jae Lee and Namtae Kim and Robert Exley and Kwonyoung Ryu and Jaesik Park},
  title = {{AmodalBox: Automated 3D Bounding Box Annotation under Partial Visibility}},
  year = {2026},
  eprint = {2611.10821},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10821},
}
Figure