Test-Time Correction: An Online 3D Detection System via Visual Prompting

Hanxue Zhang, Zhengeng Yang, Yanan Sun, Li‐Qun Chen, Fei Xia, Fatma Güney, Hongyang Li

TL;DR

This paper introduces Test-time Correction (TTC), a novel online system that allows deployed 3D object detectors in autonomous vehicles to correct their own errors in real-time without retraining. By using "visual prompts" derived from auxiliary information like 2D detections or user clicks, TTC significantly improves safety and adaptability in challenging driving scenarios.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on accurate 3D object detection to navigate safely. Traditional 3D detectors are trained offline and then deployed. If they encounter novel scenarios, domain shifts, or simply make mistakes during inference (e.g., missing an object), these systems cannot adapt or correct themselves without a costly and time-consuming retraining process. This limitation poses a significant safety risk, as errors in perception can lead to dangerous driving decisions. The authors aim to bridge this gap by enabling online, test-time error correction for deployed 3D detectors.

Methods

The core innovation is the Test-time Correction (TTC) system, which acts as an add-on to existing 3D detectors. It comprises two main components:

  • Online Adapter (OA) Module: This plug-and-play module takes various forms of prompts and transforms them into queries that traditional 3D detectors can understand. Crucially, it's designed for parameter-free, prediction-level rectification, meaning it doesn't alter the original detector's weights.
  • Visual Prompt Buffer: This component stores "visual prompts" – image-based descriptions of objects that the system needs to pay extra attention to. These prompts can originate from:
  • Mismatches with 2D detections: When a 2D detector sees something the 3D detector misses.
  • Road descriptions: Textual information that can be used to retrieve relevant visual prompts.
  • User clicks: Direct human input indicating a specific object.
  • Image patches of risky objects: Captured from previous frames.

The key novelty lies in visual prompts. Unlike text, box, or point prompts, visual prompts are image snippets of the target object itself. This allows for more direct and flexible guidance. The OA module uses these visual prompts to generate queries, locate corresponding objects in the current frame, and guide the 3D detector to output corrected bounding boxes. The system also includes mechanisms to manage the prompt buffer, preventing it from growing indefinitely and ensuring efficient operation.

Datasets

The primary dataset used for training and evaluation is nuScenes [19], a popular multimodal dataset for autonomous driving research. The authors also mention using external images for testing robustness to out-of-domain visual prompts.

Key Insights and Ideas

  • Visual Prompts are Powerful: The paper strongly advocates for using image patches of objects as prompts, arguing they are more natural and informative than text, boxes, or points for guiding 3D detection.
  • Online Correction is Crucial for Safety: The core idea is that safety-critical systems like autonomous vehicles must be able to correct errors on the fly, rather than waiting for offline updates.
  • No Retraining Needed: A major advantage is that TTC works by adding a module to frozen pre-trained 3D detectors, meaning no expensive retraining is required for adaptation.
  • Robustness to Diverse Inputs: TTC demonstrates impressive robustness to variations in prompt sources, styles, viewpoints, and even noisy or rapidly changing prompts.
  • Addressing Instance Ambiguity: The "one-to-N mapping" strategy in visual prompt alignment is key to handling situations where multiple similar objects are present.

Key Results

  • Significant Performance Boost: TTC consistently improves the performance of various offline 3D detectors. For example, it boosts mAP by 2.4% to 9.7% across different detector types (monocular, multi-view, BEV).
  • Out-of-Distribution Performance: TTC shows substantial gains in challenging scenarios:
  • Long-range rectification: Improves mAP by 10.1% when 80% of distant object labels are removed during training.
  • Vehicle-focused rectification: Achieves 20.0% mAP improvement when 80% of vehicle labels are discarded.
  • Novel object rectification: Detects unseen classes with 7.2% mAP improvement.
  • Domain shift: Shows 3.8% mAP improvement in scenarios like nighttime or rainy conditions.
  • Reduced False Negatives: TTC significantly reduces false negatives (missed detections) with only marginal increases in false positives.
  • Efficiency: TTC introduces minimal overhead in terms of model size and inference latency compared to LLM-based methods.

Limitations and Future Work

  • Generalization Constraints: The current generalization ability is limited by model size and data volume. Future work could involve scaling up data and parameters.
  • Human Feedback Dependency: While TTC can use automated feedback, relying on human clicks can be challenging in L3 autonomous driving. Future work could explore simplifying feedback collection (e.g., eye-tracking).
  • Focus on Missed Detections: The current work primarily addresses missed detections. Future research could explore addressing false positives.
  • Sensor Fusion: The current TTC system is visual-domain focused. Extending it to other sensors like LiDAR is a promising direction.
  • Cross-Dataset Transfer: While TTC shows some cross-dataset transferability, camera intrinsic/extrinsic mismatches remain a challenge. Techniques like "K-sync" help, but further research is needed.

Summary updated

Abstract

This paper introduces Test-time Correction (TTC), an online 3D detection system designed to rectify test-time errors using various auxiliary feedback, aiming to enhance the safety of deployed autonomous driving systems. Unlike conventional offline 3D detectors that remain fixed during inference, TTC enables immediate online error correction without retraining, allowing autonomous vehicles to adapt to new scenarios and reduce deployment risks. To achieve this, we equip existing 3D detectors with an Online Adapter (OA) module-a prompt-driven query generator for real-time correction. At the core of OA module are visual prompts: image-based descriptions of objects of interest derived from auxiliary feedback such as mismatches with 2D detections, road descriptions, or user clicks. These visual prompts, collected from risky objects during inference, are maintained in a visual prompt buffer to enable continuous correction in future frames. By leveraging this mechanism, TTC consistently detects risky objects, achieving reliable, adaptive, and versatile driving autonomy. Extensive experiments show that TTC significantly improves instant error rectification over frozen 3D detectors, even under limited labels, zero-shot settings, and adverse conditions. We hope this work inspires future research on post-deployment online rectification systems for autonomous driving.
BibTeX
@article{zhang2025test,
  author = {Hanxue Zhang and Zhengeng Yang and Yanan Sun and Li‐Qun Chen and Fei Xia and Fatma Güney and Hongyang Li},
  title = {{Test-Time Correction: An Online 3D Detection System via Visual Prompting}},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  doi = {10.1109/tpami.2025.3642076},
}
Figure