End-to-End Sensorimotor Control for Autonomous Systems

31 concepts28 papers1 proposal

When designing an autonomous vehicle, the traditional engineering instinct is to divide and conquer. We break the problem into a linear pipeline: a perception module detects obstacles, a localization module places the vehicle on a map, a planning module charts a safe trajectory, and a control module actuates the motors to follow that path.

While this modular approach is highly interpretable, it suffers from a fundamental flaw: it behaves like a high-stakes game of "telephone." If the perception module miscalculates an object's distance by a few inches, that error propagates to the planner, which generates a flawed trajectory, forcing the controller to make an unstable correction.

End-to-end sensorimotor control eliminates these intermediate boundaries. Instead of passing symbolic representations (like bounding boxes or coordinates) between isolated software blocks, an end-to-end architecture uses a unified, differentiable model to map raw sensory data—such as camera pixels, event streams, or onboard telemetry—directly to low-level actuator commands.

Traditional Modular Pipeline:
[Raw Sensors] ──> (Perception) ──> (Localization) ──> (Planning) ──> (Control) ──> [Actuators]
                                                                        ▲
                                                                 Error Cascades

End-to-End Sensorimotor Control:
[Raw Sensors] ───────────────────────> (Deep Neural Network) ───────────────────────> [Actuators]
                                                                        ▲
                                                               Joint Optimization

Why Direct Mapping Works: Latency and Error Mitigation

To understand why this approach works, consider the physical constraints of agile robotics. In high-speed scenarios like autonomous drone racing, milliseconds matter. As surveyed by Drew Hanover et al. in Autonomous Drone Racing: A Survey, traditional architectures introduce significant computational latency because each module must finish processing before the next can begin.

By compressing the entire pipeline into a single forward pass of a deep neural network, end-to-end models drastically reduce this latency. As demonstrated by Davide Scaramuzza 0001 et al. in Learning Agile, Vision-based Drone Flight: from Simulation to Reality, directly mapping sensory observations to control actions minimizes both latency and error propagation. Because the network is optimized holistically, it learns to inherently compensate for sensor noise and dynamic uncertainties rather than letting them accumulate across software boundaries.

Furthermore, traditional controllers often struggle when a vehicle reaches its physical limits, such as aerodynamic turbulence or actuator saturation. In Aggressive Online Control of a Quadrotor via Deep Network Representations of Optimality Principles, Shuo Li et al. show that predicting control actions directly from state inputs allows a vehicle to operate safely at its physical limits. Because the model does not rely on a secondary tracking layer (like a PID loop) to follow a pre-planned path, it avoids the tracking errors that typically occur during aggressive maneuvers. This precise execution at high speeds is further validated by Matthias Müller et al. in Deep Drone Acrobatics, where direct sensorimotor mapping enabled quadrotors to successfully execute extreme, high-G maneuvers using only onboard sensing.

Joint Optimization of Perception and Action

In a modular system, the perception team optimizes for object detection accuracy (e.g., mean Average Precision), while the control team optimizes for trajectory smoothness. However, maximizing detection accuracy does not guarantee better driving. A model might spend significant computational effort identifying a distant pedestrian on a sidewalk who poses zero threat to the vehicle's immediate path.

End-to-end learning forces the system to build a unified representation where perception and action are optimized jointly. As highlighted by Sampo Kuutti et al. in A Survey of Deep Learning Applications to Autonomous Vehicle Control, this single differentiable model learns to extract only the visual features relevant to the control task.

This concept dates back to early foundational work in imitation learning. For instance, Yann LeCun et al. demonstrated in Off-Road Obstacle Avoidance through End-to-End Learning that a convolutional neural network could learn to navigate complex, unstructured off-road environments simply by mimicking human steering decisions. The network bypassed hand-coded geometric mapping entirely, learning internal representations of "steerable space" directly from raw pixels.

By stripping away human-defined heuristics, the network can discover non-linear control strategies that a human engineer might never explicitly program. This is particularly powerful when pushing the envelope of control down to the lowest level of hardware actuation. Jonas Eschmann et al. pushed this boundary in Learning to Fly in Seconds by training a reinforcement learning policy that maps raw sensor inputs directly to motor RPMs. Bypassing intermediate high-level commands allowed the neural network to directly absorb and counter high-frequency sensor noise and complex aerodynamic non-linearities in real time.

The Hidden Vulnerabilities of Direct Mapping

While the simplicity of direct sensor-to-actuator mapping is elegant, it introduces unique failure modes that highlight why the paradigm remains an active area of research.

Because end-to-end models operate as a "black box," they are prone to exploiting statistical shortcuts in training data rather than learning true causal relationships. In Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?, Zhiqi Li et al. uncovered a startling reality: many state-of-the-art end-to-end driving models were largely ignoring the visual environment entirely. Instead, the models were over-relying on the vehicle's own historical velocity and acceleration (ego status) to predict future trajectories, failing catastrophically when presented with novel environmental layouts.

Furthermore, bypassing intermediate perception steps means the model lacks an explicit semantic understanding of the world, making it highly vulnerable to distribution shifts and adversarial attacks. As demonstrated by Adith Boloor et al. in Simple Physical Adversarial Examples against End-to-End Autonomous Driving Models, end-to-end controllers can be easily fooled into crashing by minor, physically realizable alterations to the environment—such as painting simple black lines on a road surface. Because the entire control loop depends on the model's internal, unconstrained representation of the input imagery, these harmless-looking patterns distort the network's interpretation of road geometry, leading to catastrophic steering commands.

Expanding the Paradigm

Despite these challenges, the efficiency gains of bypassing error-prone intermediate steps continue to drive adoption across diverse robotic domains. In social robotics, Rashid Alyassi et al. note in Social robot navigation: a review and benchmarking of learning-based methods that end-to-end navigation models significantly reduce system failures by avoiding the compounding tracking errors common in traditional pedestrian-tracking pipelines. Similar transitions toward direct mapping are being explored in maritime environments (Yuanyuan Qiao 0002 et al. in Survey of Deep Learning for Autonomous Surface Vehicles in Marine Environments) and minimalist multi-agent swarms (Connor Mattson et al. in Exploring Behavior Discovery Methods for Heterogeneous Swarms of Limited-Capability Robots).

The core insight of end-to-end control is not that perception and planning are unnecessary, but that forcing them through rigid, human-designed interfaces limits a system's ultimate performance. By treating the mapping from sensor to actuator as a single, continuous optimization problem, autonomous systems can trade human interpretability for fluid, low-latency reflexes capable of operating at the absolute limits of physical hardware.

Key Foundational Work

Off-Road Obstacle Avoidance through End-to-End Learning (2005) — contributed 1 concept elements to this direction.
Comparing Different Cognitive Paradigms with a Virtual Laboratory (2003) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_end_to_end_control_policy_443a79c8,
  title = {end-to-end control policy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/end-to-end-control-policy/rdr_443a79c89ece4f3094bcb7aa6eef0733},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_end_to_end_control_policy_443a79c8_v1,
  title = {end-to-end control policy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/end-to-end-control-policy/rdr_443a79c89ece4f3094bcb7aa6eef0733/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake