In traditional robotics and autonomous systems, the standard design paradigm follows a sequential "sense-plan-act" pipeline. A self-driving car, for instance, might first detect objects in its environment, link those detections over time to track them, predict their future trajectories, and finally plan a safe path.
[ Raw Sensors ] ──> [ Perception ] ──> [ Tracking ] ──> [ Prediction ] ──> [ Planning ] ──> [ Control ]
While this modularity makes systems easier to engineer and debug in isolation, it introduces a critical vulnerability: cascading error propagation. If an upstream module makes a minor mistake—such as a detector missing a vehicle obscured by foliage—that error is passed downstream as an absolute fact. Because downstream modules lack access to the original sensor data, they cannot correct or recover from the mistake, often leading to catastrophic system failures.
The Fragility of the Rigid Hand-Off
The core reason modular pipelines fail in challenging environments is that each interface acts as an information bottleneck. When a perception module converts raw sensor data into discrete geometric representations (like bounding boxes or occupancy grids), it discards the rich, probabilistic context of the original signal.
As noted in LLM4Drive: A Survey of Large Language Models for Autonomous Driving, this fragmentation causes severe information degradation. Downstream components are forced to make safety-critical decisions based on incomplete and potentially flawed summaries of the world.
Consider a robotic manipulation task. If a system separates object detection, pose estimation, and grasp planning into isolated steps, a millimeter-level error in the initial geometric estimate will propagate directly to the execution phase. As Agrawal et al. observe, these sequential dependencies frequently cause total grasp failures, even when each individual module performs reasonably well according to isolated computer vision metrics.
This compounding effect means that a pipeline's overall reliability is bounded by its most error-prone component. In open-world deployments, like the home environments evaluated in Demonstrating OK-Robot, minor inaccuracies across perception, navigation, and manipulation accumulate rapidly, causing the entire system to fail despite the high quality of the underlying off-the-shelf models.
Furthermore, this architecture complicates post-mortem analysis. When an accident occurs, isolating the root cause is exceptionally difficult because an error in one module manifests as a failure in another. As Sun et al. highlight in their work on the ACAV framework, the constant exchange of data between interdependent modules creates a complex web of causality that obscures whether a crash was caused by a tracking failure, a prediction error, or a planning anomaly.
[ Perception Error ] (Missed obstacle or swapped ID)
│
▼
[ Prediction Error ] (Miscalculated future trajectory)
│
▼
[ Planning Failure ] (Kinematically impossible path generated)
Unifying the Pipeline via End-to-End Learning
To eliminate these bottlenecks, recent research has shifted toward end-to-end (E2E) differentiable architectures. By treating the entire perception-to-control stack as a single network, the system can optimize all components simultaneously under a shared objective.
Instead of passing rigid symbolic data between modules, E2E systems pass continuous, high-dimensional feature representations—such as Bird's-Eye-View (BEV) feature maps. Luo et al. demonstrated the power of this approach with an architecture that performs 3D detection, tracking, and motion forecasting within a single convolutional network. Because the system performs joint spatial-temporal reasoning on a shared feature space, it can recover from early-stage detection misses; a vehicle missed in a single frame can still be tracked and accounted for based on temporal context.
┌──> Detection ──┐
[ Raw Sensors ] ──> [ Shared BEV Feature Space ] ──┼──> Tracking ──┼──> [ Joint Optimization ]
└──> Forecasting ──┘
This holistic integration alters how the perception system behaves. In a traditional stack, the detector is trained on generic computer vision metrics (like mean Average Precision). In an E2E stack, the perception layers learn features specifically tailored to the downstream task.
For instance, Roddick developed a dense transformer layer that maps monocular images directly to BEV semantic maps, bypassing multi-stage vision pipelines that separate ground plane estimation from object detection. Similarly, Hu integrated perception, prediction, and planning into a unified network using spatial-temporal feature learning, ensuring that the system corrects perceptual ambiguities before they influence vehicle control.
By removing intermediate hand-offs, E2E systems also achieve significant latency reductions, which is vital for high-speed maneuvers. Müller demonstrated that mapping noisy depth images directly to trajectories allows quadrotors to navigate complex environments at speeds up to 10 m/s, outperforming modular stacks that fail when downstream mapping errors generate kinematically impossible paths.
Alternative Paradigms: Multi-Hypothesis and Imperative Planning
While pure end-to-end learning offers an elegant solution to error propagation, it can suffer from a lack of interpretability and a susceptibility to edge cases. Researchers have therefore developed alternative frameworks that preserve some modular boundaries while breaking the sequential dependency chain.
Multi-Hypothesis Tracking
One approach to preventing upstream errors from ruining downstream plans is to avoid committing to a single interpretation of the world too early. In traditional tracking, an identity swap (e.g., confusing two nearby pedestrians) can cause a prediction module to miscalculate their future paths.
To mitigate this, Weng et al. introduced MTP (Multi-Hypothesis Tracking and Prediction). Instead of passing a single "best" tracking history to the prediction engine, MTP maintains multiple tracking hypotheses simultaneously. This ensures that even if the primary tracker makes a mistake, the downstream prediction module receives at least one accurate historical trajectory, reducing final prediction errors significantly.
Imperative and Bilevel Planning
Another strategy is to embed physics-based control constraints directly into the learning loop, creating a feedback mechanism between planning and perception.
Wang et al. and Xiao et al. developed frameworks utilizing Imperative Learning (IL) and bilevel optimization. By embedding Model Predictive Control (MPC) directly into the training process, the planning requirements explicitly guide the perception system. If a perceptual feature doesn't impact the feasibility of the physical path, the network learns to ignore its noise; conversely, it prioritizes precision for features that are critical to maintaining balance or avoiding obstacles. This tight mathematical coupling prevents the generation of paths that are feasible in a flawed internal map but impossible in reality.
The Shift to Holistic Evaluation
The realization that modular errors compound has also forced a re-evaluation of how autonomous systems are tested. Historically, researchers evaluated components in isolation—testing a detector on an image dataset or a trajectory predictor on pre-recorded ground-truth tracks.
However, as Cai et al. demonstrate, high accuracy in isolated trajectory prediction does not necessarily translate to safer driving performance because standard benchmarks ignore how sensor noise and occlusions compound with prediction errors. This insight is echoed by Dreossi et al., who argue that evaluating components in isolation fails to capture how localized perception errors propagate through multi-stage control loops.
Ultimately, overcoming cascading error propagation requires moving away from isolated optimization. Whether through unified end-to-end neural networks, parallelized co-training, or multi-hypothesis tracking, modern autonomous architectures succeed by ensuring that information flows fluidly across the entire system, allowing downstream modules to reason through the uncertainty of the physical world.
Key Foundational Work
Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting With a Single Convolutional Net (2018) — contributed 2 concept elements to this direction.