Beyond Infraction Scores: Failure-Boundary Profiles for Closed-Loop Evaluation of Driving and Robot Navigation Agents

Milan Bulat, Gino Ardelean, Ellington KirbyarXiv 2026 · arXiv

TL;DR

Standard driving benchmarks often fail to distinguish between an agent that stops safely and one that recklessly "crashes through" an obstacle. This paper introduces failure-boundary profiles, a new evaluation framework that separates an agent's decision to attempt a maneuver (commitment) from its ability to execute it safely (competence) across controlled scenario variations.

What Problem Does This Paper Address?

In the world of autonomous driving (AD) and robot navigation, "closed-loop" evaluation—where the agent's actions affect the environment—is the gold standard. Currently, benchmarks like CARLA use a Driving Score (DS), which is essentially route completion multiplied by an infraction penalty.

However, the authors argue that the DS is a blunt instrument. It can reward "aggressive" agents that ram through narrow gaps (gaining route completion despite a collision penalty) while penalizing "safe" agents that stop correctly at a blockage (losing route completion). This makes it nearly impossible to tell if an agent is failing because it’s a bad driver or because it’s making bad decisions about when to proceed.

Methods

The authors propose a more granular approach using Failure-Boundary Profiles. Instead of fixed routes, they evaluate agents along controlled axes within "scenario families." For example, they might vary the width of a corridor or the time gap in oncoming traffic.

Key components include:

  • Commitment Curve: The probability that an agent attempts a passage as a factor (like width) changes.
  • Competence Curve: The probability that an agent passes through without any contact.
  • Decision Accuracy (DA): A summary metric that credits an agent for "clean passes" when a route is passable and "correct stops" when it is blocked.
  • The SCL Pipeline: A novel "Social Chaos Lab" pipeline that converts natural-language requests into deterministic, certified scenario families.
The SCL scenario-generation pipeline turns natural language intents into validated, certified scenario families.

Datasets

The researchers evaluated seven driving agents and one warehouse robot across two major simulation environments:

  • CARLA: Seven scenario families including the "Charger Gauntlet" (parked cars narrowing a lane), "Machine Intrusion" (vending machines in the road), and "Oncoming Flow" (left turns across traffic).
  • NVIDIA Isaac Sim: Three warehouse families (Slab Pincer, Floor Stock, and Composed Stacks) where a robot must navigate narrow aisles or choose to detour.
Figure 3. The ten scenario families. One picture per family, cut from the recordings of the campaigns the paper reports: (a) to (g) the front cameras of TFv6 in the CARLA families, (h) to (j) the top-down view of the warehouse families. (a) The Charger gauntlet at its closed end, the two parked cars meeting across the lane. (b) The vending machine at the lane edge, the shallowest intrusion of the sweep. (c) The narrowest corridor: two street barriers in a row on the right, opposite the vending machine on the left curb. (d) The identity ladder with the slide in place. (e) The cluttered intrusion, the machine and two bins at the lane edge. (f) The slowest oncoming flow and (g) a mid-range spacing, the car waiting at the junction to turn left across the oncoming lane. (h) The slab pincer at a gap certified blocked, (i) the floor stock and (j) the composed stacks at their narrowest passable gaps, the robot approaching the row.

Key Insights and Ideas

  • The Commitment-Competence Gap: Many agents suffer from a "gap" where they commit to a maneuver they are no longer competent enough to finish cleanly. Visualizing these as separate curves allows researchers to see exactly where an agent's "ego" exceeds its "skill."
  • The "Stop Side" Failure: Learned agents (neural networks) are surprisingly bad at waiting. While some agents stop at static blockages, almost all of them failed to wait for safe gaps in oncoming traffic, a behavior the rule-based "Expert" handled easily.
  • Fine-tuning Trade-offs: In the warehouse robot experiments, fine-tuning the model improved its ability to navigate narrow gaps (competence) but completely destroyed its ability to stop at blocked ones (commitment). Without these profiles, a developer might only see the improved navigation and miss the dangerous loss of stopping logic.

Key Results

The results highlight the deceptive nature of aggregate scores. For instance, the agent BridgeDrive had the fourth-highest Driving Score (71.8) but the lowest Decision Accuracy (0.26) because it literally never stopped for a blockage, preferring to displace parked cars by up to 55 meters to make "progress."

Comparison of Driving Score and Commitment. Note how BridgeDrive (red) maintains 100% commitment even when the corridor is too narrow to pass, leading to a higher DS than the safer TFv6 (blue) at narrow widths.
  • Top Performer: The rule-based Expert reached a Decision Accuracy of 0.74.
  • Best Learned Agent: TFv6 led the learned agents with a DA of 0.60.
  • The "Wait" Problem: On wait-labeled windows in traffic, the Expert waited 13 of 29 times, while most learned agents waited 0 times.

Limitations and Future Work

  • Simulation Gap: All results are from CARLA and Isaac Sim; it remains to be seen if these specific failure boundaries translate to real-world hardware.
  • Computational Cost: Running these profiles is expensive. Evaluating seven agents took roughly 220 GPU hours, which might be prohibitive for smaller research teams.
  • Sensitivity: The "Decision Accuracy" metric depends on human-defined constants (like how much clearance a car "should" have). Changing these constants can reorder the middle-ranking agents.

The authors suggest that future work should focus on "densifying" runs near the failure boundaries to increase precision and exploring how these profiles can be used to guide safer model training.

Summary updated

Abstract

Closed-loop driving benchmarks in CARLA rank agents by a driving score that multiplies route completion by an infraction penalty. This score does not distinguish an agent that stops in front of an obstacle from one that pushes past it. In a corridor closed by two parked cars, for example, one agent that drives through with contact averages 52.2 on the blocked widths, while the agents that stop average between 37.8 and 40.6. We propose failure-boundary profiles, an evaluation in which each scenario family varies one measured factor, such as corridor width or the time gap in oncoming traffic, and each variant is certified from measurement as one where the agent should proceed or one where it should stop. Along this axis we estimate two curves per agent, commitment (the probability of attempting the passage) and competence (the probability of passing without contact), and locate where each crosses one half, with intervals that account for the simulator's run-to-run variance. The certified labels also give a summary metric, decision accuracy, which credits clean passes on passable variants and stops on blocked ones. We evaluate seven driving agents on seven CARLA families. The rule-based expert reaches a decision accuracy of 0.74 and TFv6, the highest-scoring learned agent, 0.60. Agents with similar driving scores fail in different ways, and agents that reliably stop at a blocked corridor still rarely wait for a gap in oncoming traffic. We also apply the method, unchanged, to a warehouse robot in Isaac Sim. The navigation policy we test stops at blocked gaps in only one of three families, and two fine-tuned versions of it pass more gaps cleanly but no longer stop at all.
BibTeX
@misc{bulat2026beyond,
  author = {Milan Bulat and Gino Ardelean and Ellington Kirby},
  title = {{Beyond Infraction Scores: Failure-Boundary Profiles for Closed-Loop Evaluation of Driving and Robot Navigation Agents}},
  year = {2026},
  eprint = {2611.06453},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06453},
}
Figure