Transcendental Idealism of Planner: Evaluating Perception from Planning Perspective for Autonomous Driving

Weixin Li, Xiaodong Yang

TL;DR

The Transcendental Idealism of Planner (TIP) framework shifts autonomous vehicle (AV) perception evaluation from "how accurate is the detection?" to "how much does this error actually matter for driving?" By embedding perception and planning into a shared geometric space, TIP provides a principled, parameter-free metric that identifies critical safety risks that traditional computer vision metrics often miss.

What Problem Does This Paper Address?

In the world of autonomous driving, not all mistakes are created equal. Missing a parked car 50 meters behind you is a trivial error; missing a pedestrian 5 meters in front of you is a catastrophe.

Traditional perception metrics like mean Average Precision (mAP) or the nuScenes Detection Score (NDS) are "deployment-agnostic"—they treat every detection error with similar weight regardless of its context. Newer "ego-centric" metrics try to fix this by weighting errors based on distance, but they still rely on handcrafted rules and manual tuning.

The authors argue that the only way to truly measure a perception module's performance is through the lens of the planner—the "brain" that actually decides how the car moves. As illustrated below, a small change in behavior (like a slight swerve) can sometimes have a much higher "cost" (like hitting an object) than a large, safe detour.

Illustration of how behavior change doesn't always correlate with driving cost.

Methods

The authors propose a framework inspired by Immanuel Kant’s philosophy: we don't perceive objects "as they are," but as our cognitive faculties interpret them. In an AV, the "cognitive faculty" is the planner.

  1. Expected Utility Maximization (EUM): The planning process is modeled as a search for the action that maximizes "utility" (a score representing safety, comfort, and progress).
  2. Hilbert Space Embedding: The authors prove that under practical conditions, both the perception output (the world state) and the planner’s utility function can be represented as vectors in a high-dimensional Hilbert space.
  3. Geometric Interpretation: In this space, the planner’s objective becomes a simple inner product between the perception vector and the utility vector.
The geometric interpretation of planning in a Hilbert space.

This geometric approach allows the authors to decompose any perception error into two orthogonal components:

  • Planning-Invariant Error (PIE): Noise that doesn't change the planner's final decision.
  • Planning-Critical Error (PCE): Noise that directly impacts the decision-making process.

Key Insights and Results

The researchers validated TIP using a mix of synthetic data (1,000 curated scenarios with controlled noise) and real-world data (780,000 annotated LiDAR sweeps, plus the nuScenes dataset).

  • The "Near-Miss" Detection: TIP can identify "critical errors" where the AV's final behavior might look identical to the ground truth case, but the internal "preference" for that action has plummeted, indicating a dangerously thin safety margin.
  • Parameter-Free Evaluation: Unlike metrics requiring manual calibration, TIP naturally derives weights from the planner's own utility function.
  • Human Alignment: In subjective tests with human drivers, TIP's severity rankings agreed with human judgment significantly more often than NDS or previous state-of-the-art behavior metrics.
  • Training Guidance: The authors found that perception models don't always get "better" for driving as training progresses. TIP scores often peaked early in training, even as traditional NDS scores continued to climb, suggesting that over-optimizing for detection accuracy can sometimes hurt overall system performance.

Limitations and Future Work

The primary limitation of TIP is its dependence on the planner. Because it evaluates perception through the planner's reactions, a flawed or unverified planner will lead to biased perception metrics. The authors emphasize that the planner used for evaluation must be reasonable and validated. Future work could involve extending this framework to multi-agent scenarios or applying TIP to "end-to-end" driving models where the boundary between perception and planning is blurred.

Summary updated

Abstract

Evaluating the performance of perception modules in autonomous driving is one of the most critical tasks in developing the complex intelligent system. While module-level unit test metrics adopted from traditional computer vision tasks are feasible to some extent, it remains far less explored to measure the impact of perceptual noise on the driving quality of autonomous vehicles in a consistent and holistic manner. In this work, we propose a principled framework that provides a coherent and systematic understanding of the impact an error in the perception module imposes on an autonomous agent's planning that actually controls the vehicle. Specifically, the planning process is formulated as expected utility maximisation, where all input signals from upstream modules jointly provide a world state description, and the planner strives for the optimal action by maximising the expected utility determined by both world states and actions. We show that, under practical conditions, the objective function can be represented as an inner product between the world state description and the utility function in a Hilbert space. This geometric interpretation enables a novel way to analyse the impact of noise in world state estimation on planning and leads to a universal metric for evaluating perception. The whole framework resembles the idea of transcendental idealism in the classical philosophical literature, which gives the name to our approach.
BibTeX
@inproceedings{li2023transcendental,
  author = {Weixin Li and Xiaodong Yang},
  title = {{Transcendental Idealism of Planner: Evaluating Perception from Planning Perspective for Autonomous Driving}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=O0G4GsQ3Jy},
}
Figure