Test-time Correction with Human Feedback: An Online 3D Detection System via Visual Prompting

Zetong Yang, Hanxue Zhang, Yanan Sun, Li Chen, Fei Xia, Fatma Guney, Hongyang LiSubmission to ICLR 2025 · Withdrawn

TL;DR

Existing 3D object detectors in autonomous vehicles are frozen post-deployment, meaning missed objects cannot be corrected without days or weeks of offline re-annotation and retraining. The Test-Time Correction (TTC) system enables immediate, test-time error correction via human feedback by converting user clicks into persistent "visual prompts." Without updating model weights, TTC detects and tracks previously missed or out-of-distribution objects across subsequent streaming video frames in real time.

What Problem Does This Paper Address?

Vision-based 3D object detection is foundational for autonomous driving perception, planning, and safety. However, standard 3D detectors follow an offline train-and-deploy paradigm. When a deployed model encounters domain shifts (e.g., adverse weather) or rare edge cases (e.g., wild animals on roads) and misses an obstacle, current workflows require manual data collection, annotation, and model retraining—a loop that takes days or weeks.

In safety-critical driving scenarios, immediate remediation is required. While interactive vision models (like SAM) use points or bounding boxes, these single-frame prompts do not naturally persist over streaming temporal video. Text-based prompting can be ambiguous and multimodal LLM-based 3D models suffer from excessive latency. TTC bridges this gap by enabling instant, continuous error rectification during online inference.

Methods

The TTC framework augments traditional 3D detectors (monocular, multi-view, or BEV-based) with two key components:

  1. Online Adapter (OA) Module:
  2. Acts as a bridge between human prompts and the detector’s transformer decoder by converting prompts into queries.
  3. Converts user interaction (e.g., a click or bounding box on an overlooked object) into visual prompts—cropped image patches describing the target object.
  4. Visual Prompt Alignment: Aligns image features with visual prompt features using MLPs and cosine similarity.
  5. Instance Ambiguity Handling: Instead of a strict 1-to-1 match, OA regresses $N$ spatial candidate peaks ($N=4$) from the similarity map, enabling it to detect multiple similar objects (e.g., a row of traffic cones) from a single prompt.
  6. Dynamic Visual Prompt Buffer:
  7. A queue that stores active visual prompts to track and detect previously missed objects in future frames.
  8. Employs an automatic dequeue mechanism to purge visual prompts when detection confidence drops below a threshold or when objects exit the camera's field of view, keeping memory and computation bounded.
[Missed Object] ──> [User Click] ──> [Visual Prompt Buffer] ──> [OA Module / Alignment] ──> [Detector Output]
                                              ▲                              │
                                              └── Continuous Temporal Loop ──┘

Datasets

  • nuScenes: Used for primary training and evaluation (1,000 driving scenes). To evaluate out-of-distribution robustness without classification labels, the authors introduce the Entity Detection Score (EDS), a class-agnostic version of nuScenes Detection Score (NDS) weighted by recall.
  • Distant Object Detection: 80% of labels beyond 30m were withheld during training.
  • Vehicle-focused Detection: 80% of vehicle labels withheld during training.
  • Novel Object Detection: All annotations for "bus" and "truck" withheld during training.
  • Domain Shift: Training excluded "Nighttime" and "Rainy" scenes.
  • In-the-Wild Real-World Driving Footage: Unlabeled driving clips sourced from YouTube (e.g., deer/animals crossing highways) to test open-world zero-shot capabilities.

Key Insights and Ideas

  • Visual Prompts as In-Context Guidance: Much like in-context learning in Large Language Models (LLMs), visual patches serve as visual context tokens that guide the detector queries without requiring backpropagation or parameter updates at test time.
  • View- and Style-Invariant Matching: Visual prompts do not require an exact pose match. A single web-sourced stock image or an object patch captured from an earlier frame can identify and track instances across varying viewing angles, lighting conditions, and even toy/Lego-stylized representations.
  • Real-Time Efficiency vs. LLMs: Unlike heavy 3D LLM approaches (e.g., LL3DA running at ~0.42 Hz), TTC adds minimal latency over base detectors (e.g., TTC-MonoDETR runs at 9.1 FPS compared to MonoDETR's 11.1 FPS on an A100 GPU), making it practical for streaming autonomous platforms.

Key Results

  • Consistent Gains Across Detectors: Adding TTC to pre-trained architectures improved Entity Detection Score (EDS) across the board on nuScenes without retraining:
  • MonoDETR: +5.0% EDS (38.2% $\rightarrow$ 43.2%)
  • MV2D: +12.7% EDS (38.3% $\rightarrow$ 51.0%)
  • BEVFormer: +11.4% EDS (35.8% $\rightarrow$ 47.2%)
  • Sparse4Dv2: +14.5% EDS (37.6% $\rightarrow$ 52.1%)
  • Handling Out-of-Distribution Scenarios (using TTC-MonoDETR):
  • Distant objects (>30m): Improved distant EDS by +14.4% (from 0.0% to 14.4%).
  • Underrepresented vehicles: Improved vehicle EDS by +21.6% (from 7.3% to 28.9%).
  • Unseen categories (Zero-shot bus/truck): Improved EDS from 0.0% to 13.6%.
  • Adverse weather/night: Improved EDS by +4.7%.
  • Web-derived Prompts: Using stock Internet pictures of cars and buses boosted detection of under-annotated categories by +18.4% EDS (cars) and +16.2% EDS (buses).

Limitations and Future Work

  • Human Delay: The system relies on initial human feedback; while the model runs in real time, human reaction time introduces an unavoidable latency before an error is flagged.
  • Scale of Model and Training Data: The current framework was developed and evaluated at a moderate scale primarily within nuScenes.
  • Future Directions: The authors highlight scaling visual promptable 3D detection by pretraining on large-scale 2D datasets combined with 3D driving data, expanding to multi-modal prompts (combining language descriptions with visual prompts), and applying visual prompting to V2X communication and automated data labeling.

Summary updated

Abstract

This paper introduces Test-time Correction (TTC) system, a novel online 3D detection system designated for online correction of test-time errors via human feedback, to guarantee the safety of deployed autonomous driving systems. Unlike well studied offline 3D detectors frozen at inference, TTC explores the capability of instant online error rectification. By leveraging user feedback with interactive prompts at a frame, e.g., a simple click or draw of boxes, TTC could immediately update the corresponding detection results for future streaming inputs, even though the model is deployed with fixed parameters. This enables autonomous driving systems to adapt to new scenarios flexibly and decrease deployment risks reliably without additional expensive training. To achieve such TTC system, we equip existing 3D detectors with OA module, an online adapter with prompt-driven design for online correction. At the core of OA module are visual prompts, images of missed object-of-interest for guiding the corresponding detection and subsequent tracking. Those visual prompts, belonging to missed objects through online inference, are maintained by the visual prompt buffer for continuous error correction in subsequent frames. By doing so, TTC consistently detects online missed objects and immediately lowers down driving risks. It achieves reliable, versatile, and adaptive driving autonomy. Extensive experiments demonstrate significant gain on instant error rectification over pre-trained 3D detectors, even in challenging scenarios with limited labels, zero-shot detection, and adverse conditions. We hope this work would inspire the community to investigate online rectification systems for autonomous driving post-deployment. Code would be publicly shared.
BibTeX
@article{yang2025test,
  author = {Zetong Yang and Hanxue Zhang and Yanan SUN and Li Chen and Fei Xia and Fatma Guney and Hongyang Li},
  title = {{Test-time Correction with Human Feedback: An Online 3D Detection System via Visual Prompting}},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  doi = {10.1109/tpami.2025.3642076},
}
Figure