Introduction
The automotive industry is undergoing a paradigm shift from modular autopilot architectures—where perception, planning, and control are distinct components—to end-to-end (E2E) AI solutions. These E2E models map raw sensor data directly to control signals, offering lower development costs and eliminating "compositional" errors between modules. However, E2E models are essentially "black boxes": they are non-deterministic, difficult to interpret, and often exhibit unpredictable behaviors.
Standard evaluation benchmarks, such as the Carla Leaderboard, rely on randomized scenarios and quantitative metrics like "Route Completion." While useful, these scores often fail to distinguish between unavoidable accidents caused by adversarial agents and fundamental failures of the autopilot's logic. This research addresses the need for qualitative, safety-focused testing by introducing Critical Configuration Testing (CCTest) to probe the "Achilles' heel" of modern AI driving systems.
Methodology: Critical Configuration Testing (CCTest)
Unlike standard benchmarks that rely on random traffic generation, CCTest is "safe by construction." It utilizes predictive analysis of vehicle dynamics—specifically braking and acceleration capabilities—to ensure that a safe driving policy theoretically exists for every test case. Consequently, any collision or traffic violation observed during testing is indisputably the fault of the autopilot.
The evaluation process involves several key steps:
- Empirical Dynamics Estimation: Since E2E models lack explicit physics parameters, their acceleration and deceleration (A/D) functions were experimentally estimated.
- Critical Context Selection: Testing focused on four high-stakes scenarios: Merging, Lane Changes, Crossing with Yield Signs, and Crossing with Traffic Lights.
- Non-determinism Assessment: Each test case was executed five times to determine if the AI's decision-making remained consistent in identical conditions.
The study evaluated four prominent E2E models—InterFuser, MILE, Transfuser, and LMDrive—across established benchmarks including Longest6, Town05 Long, NEAT, and Langauto. These models represent diverse architectures, from Transformer-based fusion to Large Language Model (LLM) integration.
Analysis of E2E Autopilot Performance
The results reveal a "safety paradox": AI autopilots frequently fail in simple scenarios while succeeding in more critical ones. This suggests that their safety policies are not robustly learned but are instead artifacts of specific training data distributions.
Key Findings:
- Non-Determinism: In "transition zones"—where a vehicle must decide whether to stop or proceed—models like InterFuser produced conflicting outcomes (e.g., safe passage vs. accident) across identical runs.
- Low Speed as a Mask: Many models maintain very low average speeds (4.0–6.5 m/s). This cautiousness often masks underlying design flaws that would likely result in catastrophic failures at highway speeds.
- Unique Failure Modes: E2E models exhibited "hallucinatory" behaviors not seen in modular systems, such as arbitrary route changes (e.g., turning when instructed to go straight) and driving onto sidewalks.
- Leaderboard Bias: Analysis showed that approximately 24% of accidents in standard leaderboard scenarios were caused by background NPCs, yet the ego-vehicle was still penalized. CCTest eliminates this bias by ensuring the ego-vehicle is always the responsible party.
Model-Specific Observations:
- InterFuser: Despite its high leaderboard ranking, it occasionally ignored red lights entirely in non-critical settings.
- MILE: Showed extreme over-cautiousness, often failing to overtake stationary obstacles even when safe to do so.
- Transfuser: Its low top speed (4 m/s) sometimes made it physically impossible to clear intersections before a light change, forcing a "cautious" profile.
- LMDrive: Demonstrated the poorest performance, with frequent route timeouts and road deviations.
Conclusion and Future Directions
While E2E models show promise, they currently lack the "safe caution" inherent in well-designed modular systems. Their failures are often not software bugs in the traditional sense, but a fundamental failure to internalize strict traffic rules, such as yielding.
Future research should focus on:
- Hybrid Assessment: Combining qualitative safety-critical testing with general proficiency metrics.
- High-Speed Testing: Evaluating models at realistic speeds to uncover latent control flaws.
- Locality of Knowledge: Improving how E2E models integrate navigation instructions with local obstacle avoidance to prevent nonsensical route deviations.
Summary updated