TL;DR
Decision-Guided Speculative Perception (DGSP) is a novel framework that uses an autonomous vehicle's planner to decide which image regions require high-resolution processing. By "probing" the planner with hypothetical objects to see if they would change the vehicle's action, DGSP achieves the decision-making fidelity of full-resolution perception while processing only 25% of the pixels.
What Problem Does This Paper Address?
Autonomous vehicles (AVs) face a constant trade-off between perception accuracy and computational cost. Processing megapixel camera feeds at native resolution is computationally expensive, yet downscaling images can lead to missing small or distant objects—like a pedestrian stepping into the road far ahead.
Existing "adaptive-resolution" methods try to save computation by focusing on "interesting" regions based on image cues (like saliency) or initial low-resolution detections. However, these methods have two major blind spots:
- They can't find what they've already missed: If a low-resolution pass misses an object entirely, there's no "cue" to tell the system to look closer at that spot.
- They don't know what matters: A car parked safely on a side street might get as much high-resolution "attention" as a car about to cut into the AV's lane, simply because both look like cars.
The authors argue that perception should be driven by the intended action. Just as a human driver checks their blind spot specifically because a car there would change their plan to change lanes, an AV should focus its high-resolution "eyes" on areas where an unseen object would actually force a change in behavior.
Methods
The authors introduce Decision-Guided Speculative Perception (DGSP). Instead of relying on image features to find objects, DGSP "asks" the vehicle's planner where objects would matter.
The process works in an iterative loop:
- Drafting: A "cheap" detector runs on a low-resolution version of the full frame to create an initial scene "draft."
- Probing: DGSP probes the planner by inserting hypothetical objects into a grid of locations. If an object at a certain location changes the planned action (e.g., from "Keep Speed" to "Brake"), that location is added to a decision-sensitivity field.
- Support Identification: It also identifies the "support"—the specific existing detections that are currently justifying the vehicle's action.
- Verification: The system then runs a high-resolution "reference" detector only on small crops covering the sensitivity field and the support.
- Iteration: The scene is updated with these high-res results, and the process repeats until the planned action is stable.
Crucially, the authors provide theoretical proofs that for certain classes of planners (like those used for emergency braking or trajectory selection), this iterative approach is guaranteed to return the exact same action as a full-frame, high-resolution perception system.
Datasets
The researchers evaluated DGSP on three major autonomous driving datasets:
- nuScenes: A large-scale dataset for autonomous driving with diverse urban scenes.
- Argoverse 2: A dataset focused on long-range perception and complex scenarios.
- Argoverse-HD: A version of Argoverse with high-frame-rate annotations.
- CARLA: Used for closed-loop simulation experiments to test how DGSP affects actual driving behavior.
Key Insights and Ideas
- Planner as a Specification: The planner can act as a "specification" for perception. It defines exactly what information is needed to make a safe decision, allowing the system to ignore irrelevant details.
- Probing the "Unseen": By using hypothetical objects, the system can "see" its own blind spots. It doesn't need to detect an object to know that a specific area is dangerous if occupied.
- Decision Fidelity over Detection Accuracy: The goal isn't to detect every single object perfectly; it's to ensure the vehicle takes the correct action. This shift in objective allows for massive computational savings without sacrificing safety.
Key Results
DGSP demonstrates that "looking where it matters" is highly effective:
- High Fidelity: It reproduces the action of a full-resolution system on ~98% of frames across all datasets.
- Efficiency: It achieves this while processing only 22–27% of the pixels compared to the full-resolution reference.
- Superior to Baselines: At the same pixel cost as uniform downscaling, DGSP is significantly more precise, has fewer false brakes, and recovers many more "decision-relevant" objects that low-resolution passes miss.
- Missed Object Recovery: In cases where the initial low-res draft missed a critical object (like a pedestrian), DGSP's sensitivity field successfully "probed" that area and recovered the object in 85–97% of cases where the high-res reference could see it.
Limitations and Future Work
- State Estimation (Lifting): The system relies on "lifting" 2D boxes to 3D world coordinates. Errors in monocular depth estimation can lead to incorrect sensitivity fields.
- Temporal Consistency: Because it operates on crops, maintaining a consistent history of objects over time is more challenging than with full-frame inference.
- Planner Complexity: While the paper covers common planner types, extremely complex or "black-box" neural planners might require more sophisticated probing techniques.
- Single Camera Focus: The current work focuses on single-camera setups; extending this to multi-camera, 360-degree views is a natural next step.
Summary updated