Check Your Blind Spots: Decision-Guided Perception by Probing the Planner with Hypothetical Objects

Hoang Pham Cong, Long D. Nguyen, Ho Viet Duc Luong, Nguyen Tran Minh Nhat, Binh P. Nguyen

TL;DR

Decision-Guided Speculative Perception (DGSP) is a novel framework that uses an autonomous vehicle's planner to decide which image regions require high-resolution processing. By "probing" the planner with hypothetical objects to see if they would change the vehicle's action, DGSP achieves the decision-making fidelity of full-resolution perception while processing only 25% of the pixels.

What Problem Does This Paper Address?

Autonomous vehicles (AVs) face a constant trade-off between perception accuracy and computational cost. Processing megapixel camera feeds at native resolution is computationally expensive, yet downscaling images can lead to missing small or distant objects—like a pedestrian stepping into the road far ahead.

Existing "adaptive-resolution" methods try to save computation by focusing on "interesting" regions based on image cues (like saliency) or initial low-resolution detections. However, these methods have two major blind spots:

  1. They can't find what they've already missed: If a low-resolution pass misses an object entirely, there's no "cue" to tell the system to look closer at that spot.
  2. They don't know what matters: A car parked safely on a side street might get as much high-resolution "attention" as a car about to cut into the AV's lane, simply because both look like cars.

The authors argue that perception should be driven by the intended action. Just as a human driver checks their blind spot specifically because a car there would change their plan to change lanes, an AV should focus its high-resolution "eyes" on areas where an unseen object would actually force a change in behavior.

Methods

The authors introduce Decision-Guided Speculative Perception (DGSP). Instead of relying on image features to find objects, DGSP "asks" the vehicle's planner where objects would matter.

Figure 1. Probing the planner shows where high resolution is needed. Schematic top view: a low-resolution draft detects a lead vehicle (DECELERATE) but misses a cut-in vehicle requiring

The process works in an iterative loop:

  1. Drafting: A "cheap" detector runs on a low-resolution version of the full frame to create an initial scene "draft."
  2. Probing: DGSP probes the planner by inserting hypothetical objects into a grid of locations. If an object at a certain location changes the planned action (e.g., from "Keep Speed" to "Brake"), that location is added to a decision-sensitivity field.
  3. Support Identification: It also identifies the "support"—the specific existing detections that are currently justifying the vehicle's action.
  4. Verification: The system then runs a high-resolution "reference" detector only on small crops covering the sensitivity field and the support.
  5. Iteration: The scene is updated with these high-res results, and the process repeats until the planned action is stable.
Figure 2. Overview of DGSP on a nuScenes frame. (1) The cheap detector finds a truck and a pedestrian; the truck demands DECELERATE and forms the support. (2) Probing the planner with hypothetical objects (“?”) gives the field (orange): at this ego speed, empty at

Crucially, the authors provide theoretical proofs that for certain classes of planners (like those used for emergency braking or trajectory selection), this iterative approach is guaranteed to return the exact same action as a full-frame, high-resolution perception system.

Datasets

The researchers evaluated DGSP on three major autonomous driving datasets:

  • nuScenes: A large-scale dataset for autonomous driving with diverse urban scenes.
  • Argoverse 2: A dataset focused on long-range perception and complex scenarios.
  • Argoverse-HD: A version of Argoverse with high-frame-rate annotations.
  • CARLA: Used for closed-loop simulation experiments to test how DGSP affects actual driving behavior.

Key Insights and Ideas

  • Planner as a Specification: The planner can act as a "specification" for perception. It defines exactly what information is needed to make a safe decision, allowing the system to ignore irrelevant details.
  • Probing the "Unseen": By using hypothetical objects, the system can "see" its own blind spots. It doesn't need to detect an object to know that a specific area is dangerous if occupied.
  • Decision Fidelity over Detection Accuracy: The goal isn't to detect every single object perfectly; it's to ensure the vehicle takes the correct action. This shift in objective allows for massive computational savings without sacrificing safety.

Key Results

DGSP demonstrates that "looking where it matters" is highly effective:

  • High Fidelity: It reproduces the action of a full-resolution system on ~98% of frames across all datasets.
  • Efficiency: It achieves this while processing only 22–27% of the pixels compared to the full-resolution reference.
  • Superior to Baselines: At the same pixel cost as uniform downscaling, DGSP is significantly more precise, has fewer false brakes, and recovers many more "decision-relevant" objects that low-resolution passes miss.
Figure 3. Fidelity and decision quality versus pixel cost. No baseline has both fewer pixels and a higher agreement (top, Table 1), and none has a significantly higher F1 (bottom, Table 2). F: the reference (open marker: 100% by definition).
  • Missed Object Recovery: In cases where the initial low-res draft missed a critical object (like a pedestrian), DGSP's sensitivity field successfully "probed" that area and recovered the object in 85–97% of cases where the high-res reference could see it.
Figure 4. Complementary roles of the field and the support. Test frames of Argoverse 2 (a) and nuScenes (b) (cropped views; selection: Appendix F). Dashed blue: replaced draft boxes. (a) The footprint covers a pedestrian 37 m ahead that the draft misses; once found, it joins the support (blue crop), and KEEP becomes the required DECELERATE. (b) Verifying the support restores the required KEEP.

Limitations and Future Work

  • State Estimation (Lifting): The system relies on "lifting" 2D boxes to 3D world coordinates. Errors in monocular depth estimation can lead to incorrect sensitivity fields.
  • Temporal Consistency: Because it operates on crops, maintaining a consistent history of objects over time is more challenging than with full-frame inference.
  • Planner Complexity: While the paper covers common planner types, extremely complex or "black-box" neural planners might require more sophisticated probing techniques.
  • Single Camera Focus: The current work focuses on single-camera setups; extending this to multi-camera, 360-degree views is a natural next step.

Summary updated

Abstract

Not every perception error changes the planner's action. Yet adaptive-resolution perception chooses where to spend high-resolution computation from image cues, which can neither reliably point to an object that low resolution missed nor tell which objects matter to the planner. We observe that the planner can say where an object would matter before anything is seen: inserting a hypothetical object at a location tests whether its action would change. Decision-Guided Speculative Perception (DGSP) treats the low-resolution scene and its action as a draft and probes the planner in this way to obtain a decision-sensitivity field, the locations where an unseen object would change the action. Only this field and the support, the detections on which the current action rests, are verified at high resolution, repeatedly until the action is stable. For two planner classes, we prove that DGSP returns exactly the action of whole-frame high-resolution perception if crop detections match whole-frame ones and the crops cover the field, conditions that real images meet approximately. On three driving datasets, it reproduces this action on about 98% of frames with about a quarter of the pixels. At the pixel cost of uniform downscaling, it repairs more of the draft's disagreements with this action, brakes falsely less often, and recovers many more of the decision-relevant objects that the draft misses.
BibTeX
@misc{cong2026check,
  author = {Hoang Pham Cong and Long D. Nguyen and Ho Viet Duc Luong and Nguyen Tran Minh Nhat and Binh P. Nguyen},
  title = {{Check Your Blind Spots: Decision-Guided Perception by Probing the Planner with Hypothetical Objects}},
  year = {2026},
  eprint = {2611.23030},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23030},
}
Figure