Far3D: Expanding the Horizon for Surround-view 3D Object Detection

Xiaohui Jiang, Shuailin Li, Yingfei Liu, Shihao Wang, Fan Jia, Tiancai Wang, Lijin Han, Xiangyu Zhang

TL;DR

Far3D is a novel sparse query-based framework that extends surround-view 3D object detection to a long-range horizon of 150 meters. By leveraging high-quality 2D object priors to generate 3D adaptive queries, combined with range-modulated denoising, it overcomes the severe convergence and error propagation challenges of long-distance detection to achieve state-of-the-art results—even surpassing several LiDAR-based methods.

What Problem Does This Paper Address?

While camera-based 3D object detection has progressed rapidly due to its low deployment cost, most existing models focus heavily on close-range perception (typically up to 50 meters, as benchmarked by datasets like nuScenes). However, real-world autonomous driving requires long-range perception to ensure safe braking distances at high speeds.

Directly scaling up existing architectures to cover longer distances (e.g., 150 meters) introduces two major roadblocks:

  1. Computational Explosion: Dense Bird's-Eye-View (BEV) methods suffer from a squared growth in computational cost as the perception range expands.
  2. Low Recall and Poor Convergence: Sparse query-based methods avoid this computational trap but struggle to detect distant objects. Their fixed global queries fail to adapt to dynamic scenarios, leading to poor recall. Furthermore, when projecting 2D image proposals into 3D space, even minor 2D errors are drastically amplified over long distances due to geometric frustum transformations, causing severe training instability.
Performance comparison showing that 2D recall significantly outperforms 3D recall at long ranges

Methods

To maintain efficiency, Far3D adopts a lightweight sparse query-based design and introduces three key innovations to tackle the long-range challenges:

  • Adaptive Query Generation: Instead of relying solely on static, learnable global queries, Far3D uses a 2D object detector (YOLOX) and a lightweight depth estimation network to generate reliable 2D box proposals and depths. These are projected into 3D space via spatial transformations to create 3D adaptive queries. This ensures excellent target coverage across all distances.
  • Perspective-Aware Aggregation: Distant objects occupy only a few pixels, whereas close objects span large regions. Far3D introduces a 3D spatial deformable attention mechanism that allows queries to dynamically sample and interact with multi-scale, multi-view features. Distant queries can focus on high-resolution features, while close queries leverage high-level contextual features.
  • Range-Modulated 3D Denoising: To combat the error propagation inherent in long-range geometric projections, the authors introduce a distance-aware denoising task during training. By adding position- and scale-modulated noise to ground-truth boxes, the model learns to actively reconstruct accurate 3D boxes from positive noisy queries and reject false positives.
Figure 2: Different cases of transformimg 2D points into 3D space. The blue dots indicate the centers of 3D objects in images. (a) shows the redundant prediction with the wrong depth, which is in yellow. (b) illustrates the error propaga- tion problem dominated by different ranges.

Datasets

The model was evaluated on two major autonomous driving benchmarks:

  • Argoverse 2: A challenging dataset featuring a wide 150-meter perception range, 26 object categories, and high-resolution multi-camera setups.
  • nuScenes: Used primarily to validate the model's generalization capability across standard close-to-mid-range perception tasks.

Key Insights and Ideas

  • The 2D-to-3D Recall Bridge: The core realization is that while 3D object recall drops precipitously at long distances, 2D object detectors remain highly robust. Using 2D detections as a prior "bridge" drastically improves the initialization of 3D queries.
  • Distance-Dependent Difficulty: Denoising cannot be treated uniformly across an entire scene. Because geometric frustum transformations amplify errors non-linearly over distance, training signals must be modulated based on object range to prevent the model from overfitting to dense, easy-to-detect close objects.

Key Results

Far3D achieves outstanding performance on the long-range Argoverse 2 validation set:

  • Using a standard VoV-99 backbone, Far3D achieved a 0.244 mAP and 0.181 CDS (Composite Detection Score), outperforming the baseline StreamPETR by 4.1% mAP and 3.5% CDS.
  • When scaled up with a ViT-Large backbone, Far3D achieved 0.316 mAP, remarkably outperforming several state-of-the-art LiDAR-based methods (such as CenterPoint at 0.274 mAP and FSD at 0.291 mAP).

On the nuScenes dataset, Far3D maintained its competitive edge, achieving state-of-the-art results with 51.0% mAP / 59.4% NDS on the validation split and 63.5% mAP / 68.7% NDS on the test split.

Limitations and Future Work

Despite its strengths, the authors note a couple of limitations:

  1. Long-Tail Performance: Like many deep learning models, Far3D still exhibits lower performance on rare, long-tail object classes, which drags down the overall average precision.
  2. Evaluation Metrics: Current unified metrics evaluate close-range and long-range objects under the same criteria. The authors suggest that developing dynamic, practical evaluation metrics tailored specifically to different distance zones is an important direction for future research.

Summary updated

Abstract

Recently 3D object detection from surround-view images has made notable advancements with its low deployment cost. However, most works have primarily focused on close perception range while leaving long-range detection less explored. Expanding existing methods directly to cover long distances poses challenges such as heavy computation costs and unstable convergence. To address these limitations, this paper proposes a novel sparse query-based framework, dubbed Far3D. By utilizing high-quality 2D object priors, we generate 3D adaptive queries that complement the 3D global queries. To efficiently capture discriminative features across different views and scales for long-range objects, we introduce a perspective-aware aggregation module. Additionally, we propose a range-modulated 3D denoising approach to address query error propagation and mitigate convergence issues in long-range tasks. Significantly, Far3D demonstrates SoTA performance on the challenging Argoverse 2 dataset, covering a wide range of 150 meters, surpassing several LiDAR-based approaches. Meanwhile, Far3D exhibits superior performance compared to previous methods on the nuScenes dataset. The code is available at https://github.com/megvii-research/Far3D.
BibTeX
@inproceedings{jiang2024far,
  author = {Xiaohui Jiang and Shuailin Li and Yingfei Liu and Shihao Wang and Fan Jia and Tiancai Wang and Lijin Han and Xiangyu Zhang},
  title = {{Far3D: Expanding the Horizon for Surround-view 3D Object Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i3.28033},
}
Figure