Test-time Correction with Human Feedback: An Online 3D Detection System via Visual Prompting

Zetong Yang, Hanxue Zhang, Yanan Sun, Li Chen, Fei Xia, Fatma Güney, Hongyang Li

Abstract

Test-time Correction (TTC) is a novel online system designed to address the limitations of static 3D object detectors in autonomous driving. Unlike traditional models that require offline retraining to fix perception errors, TTC enables real-time rectification during the inference phase. By utilizing "visual prompts"—image patches derived from auxiliary sources like 2D detections or user feedback—the system can instantly identify and track missed or risky objects. This approach significantly enhances the safety and adaptability of autonomous systems in novel or adverse conditions without the need for parameter updates.

The Challenge: The "Frozen Model" Limitation

Most 3D object detectors currently deployed in autonomous vehicles follow a rigid "offline" lifecycle. These models are trained on massive datasets, frozen, and then deployed. When a model encounters a scenario outside its training distribution (a "domain shift") or fails to detect a critical object, the traditional fix involves a lengthy cycle: collecting new data, manual annotation, and model retraining. This process typically takes days or weeks.

The TTC system shifts from a slow offline correction loop to a swift online response, reducing the time to fix perception errors from weeks to seconds.

In high-stakes driving environments, such a delay represents a significant safety risk. TTC addresses this by proposing a system that enables instant, online error correction, allowing the vehicle to adapt to immediate risks as they occur.

Methodology: Visual Prompting and Online Adaptation

The core innovation of TTC is the transition from static category-based detection to dynamic Visual Prompting. Instead of relying solely on pre-learned weights, the system uses auxiliary information to generate specific queries for the detector.

Key Components of the TTC System

  • Visual Prompts: Rather than using ambiguous text or frame-specific bounding boxes, TTC uses image patches of objects. These prompts can be sourced from mismatched 2D detections, web-searched images, or direct human input (e.g., a user click on a missed object).
  • Online Adapter (OA): This is a plug-and-play module compatible with existing 3D detectors like BEVFormer or MonoDETR. It extracts features from visual prompts and aligns them with the live camera feed using a contrastive learning mechanism.
  • Visual Prompt Buffer: To maintain temporal consistency, a dynamic queue stores prompts of "risky" objects. This allows the system to "remember" and track specific objects across multiple frames. A dequeue policy ensures the system remains efficient by removing stale or redundant prompts.
  • One-to-N Mapping: To resolve "instance ambiguity"—such as identifying which of several traffic cones a prompt refers to—the system predicts multiple candidate positions for a single prompt, ensuring comprehensive coverage.
The TTC System architecture (left) manages a prompt buffer to correct misses across frames, while the TTC-3D Detector (right) integrates the Online Adapter (OA) module to align visual prompts with image features.

Experimental Setup and Datasets

The researchers evaluated TTC primarily using the nuScenes dataset, a benchmark featuring complex urban traffic sequences. To test the system's robustness, they conducted several specialized evaluations:

  • Cross-Dataset Generalization: Models trained on nuScenes were tested on the Waymo Open Dataset to simulate domain shifts.
  • Zero-Shot Detection: Using real-world footage from YouTube, the team tested whether the system could detect classes (like deer) that were entirely absent from the original training data.

Key Insights

  • Superiority of Visual Cues: The authors demonstrate that image-based prompts are more effective than text for real-time driving. While a text prompt like "blue truck" is vague, an image patch provides the high-fidelity spatial and appearance cues necessary to distinguish specific instances in a crowded scene.
  • Stability via Parameter-Free Rectification: TTC does not update the weights of the underlying 3D detector. This prevents "catastrophic forgetting" and ensures that the system's core knowledge remains stable while it adapts to new prompts.
  • Turning Failures into Feedback: The system effectively turns perception failures into actionable prompts. For instance, if a high-confidence 2D detector identifies an object the 3D detector missed, that 2D crop becomes a visual prompt that "wakes up" the 3D system.

Results and Performance

TTC demonstrated substantial performance improvements across various architectures:

  • General Accuracy: Mean Average Precision (mAP) increased by 2.4% for monocular detectors, 9.7% for multi-view detectors, and 9.3% for Bird’s-Eye-View (BEV) detectors.
  • Long-Range and Zero-Shot: In long-range scenarios (>30m), mAP improved by 10.1%. In zero-shot settings, it achieved a 7.2% mAP improvement over a baseline of 0%.
  • Real-Time Efficiency: The system maintains high frame rates; for example, TTC-MonoDETR runs at 9.1 FPS (compared to the base 11.1 FPS), making it a practical solution for live deployment.

Conclusion and Future Directions

TTC represents a significant shift toward adaptive, human-in-the-loop perception systems. While currently dependent on the quality of auxiliary feedback, future work aims to scale the Online Adapter's training data and extend the prompting mechanism to other sensor modalities, such as LiDAR, to create a more robust, multi-modal online correction system.

Summary updated

BibTeX
@article{yang2025test,
  author = {Zetong Yang and Hanxue Zhang and Yanan Sun and Li Chen and Fei Xia and Fatma Güney and Hongyang Li},
  title = {{Test-time Correction with Human Feedback: An Online 3D Detection System via Visual Prompting}},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  doi = {10.1109/tpami.2025.3642076},
}
Figure