Direct Perception to Action Mapping

56 concepts52 papers1 proposal

Traditional robotics relies on a modular "sense-plan-act" pipeline. A robot captures an image, estimates its 3D state or reconstructs the environment's geometry, computes a trajectory, and finally executes motor commands. While conceptually clean, this architecture is notoriously brittle: errors accumulate at every boundary, and precise camera calibration is mandatory.

Direct perception-to-action mapping—often termed end-to-end visuomotor control—collapses this pipeline. Instead of building intermediate geometric representations, a single neural network maps raw visual inputs (like RGB pixels or optical flow) directly to low-level motor commands (such as joint velocities, body rates, or steering angles).

Early foundations by Nakamura et al. demonstrated that tightly coupling raw optical flow to motor commands allows mobile robots to navigate and avoid obstacles in real-time without any 3D scene reconstruction. By bypassing manual feature engineering, systems can discover the most predictive visual cues directly from data, as shown in early off-road autonomous driving work by LeCun et al.. In modern autonomous driving, Xu et al. showed that end-to-end models trained on raw pixels can outperform modular systems that rely on pre-processed object detections and high-definition maps, capturing environmental nuances that traditional pipelines discard.

The Intuition of Visual Reactivity

Why does mapping pixels directly to actions work so well in practice? The core intuition lies in relative visual reactivity.

Consider an agile drone racing through a gate. A traditional system calculates the drone's absolute 3D position and the gate's absolute coordinates in a global map. If a landmark shifts or a gust of wind occurs, the state estimation loop can fail. In contrast, as demonstrated by Geles et al., an end-to-end policy maps visual features directly to collective thrust and body rates. Because the agent reacts to the relative visual position of targets rather than absolute coordinates, it naturally compensates for physical perturbations.

Modular Pipeline:  [Pixels] -> [State Estimation] -> [Motion Planning] -> [Motor Torques]  (Brittle)
Direct Mapping:    [Pixels] --------------------------------------------> [Motor Torques]  (Robust)

Furthermore, direct mapping eliminates the need for explicit physical modeling. In highly non-linear or unstructured environments—such as autonomous underwater manipulation (Liu et al.), robotic colonoscopy (Nguyen et al.), or surgical needle picking (Li et al.)—modeling fluid dynamics, friction, or tissue deformation is mathematically intractable. Similarly, for tabletop object rearrangement using nonprehensile pushing, Yuan et al. utilized a frame-by-frame reactive policy to eliminate the need for explicit physical modeling of friction or object dynamics.

By bypassing explicit 3D reconstructions in favor of direct visual feedback, systems can navigate complex, unstructured environments fluidly, whether maneuvering through modern orchards (Jain et al.) or handling out-of-distribution driving errors via bio-inspired recurrent networks (Hu).

Learning the Sensorimotor Loop

How do we teach a network to map high-dimensional pixels to low-level actions? Two primary paradigms dominate: Imitation Learning and Reinforcement Learning.

Imitation Learning (Behavior Cloning)

The network is trained via supervised learning to mimic human experts. For example, Zhao et al. scaled this approach to autonomous carotid ultrasonography, mapping real-time visual inputs directly to 6-DoF robotic adjustments. To scale data collection for behavior cloning, Young utilized inexpensive assistive tools to collect "in-the-wild" visual demonstrations, allowing networks to learn multi-stage manipulation sequences directly from raw video. Similarly, Wei demonstrated that visual behavior cloning can map inputs directly to high-degree-of-freedom joint commands to chain complex behaviors together.

Reinforcement Learning (RL)

When expert demonstrations are unavailable, agents learn via environmental interaction. Yarats et al. showed that model-free RL can master high-dimensional continuous control tasks, like humanoid locomotion, directly from raw pixels without needing an explicit latent world model. Alternatively, model-based approaches train policies within a learned latent world model to adapt to diverse object morphologies (Li et al.).

In multi-stage tasks, offline RL can automatically "stitch together" separate visual datasets to execute chained behaviors without manual task decomposition (Singh et al.). Evidence from Yu et al. further suggests that visual perception tasks benefit from direct output policies rather than explicit chain-of-thought reasoning, as visual logic is more effectively captured through direct mapping from input to grounded coordinates.

Overcoming the Sample Efficiency Bottleneck

Training end-to-end policies directly in the real world is dangerously slow and computationally expensive. To make direct perception-to-action mapping practical, researchers employ three key strategies:

  • Differentiable Simulation: Traditional RL treats the environment as a black box. Heeg et al. leveraged differentiable simulators to backpropagate gradients directly through both physics and camera models, accelerating the training of vision-based quadrotor controllers by up to 70 times.
  • Extreme Domain Randomization: To bypass the need for real-world data entirely, Sadeghi trained a deep RL navigation policy entirely within a 3D CAD environment using highly randomized visual textures, achieving zero-shot transfer to real-world indoor flight.
  • Visual Subgoals and Action Space Alignment: Instead of mapping directly from the current image to the final action in one massive leap, recent architectures decouple the problem visually or spatially. Black et al. introduced a method where a diffusion model synthesizes a "hypothetical future" image as a subgoal, allowing a low-level policy to focus purely on the geometric delta between the current and desired visual states.

To smooth out action generation, Ding et al. conditioned Riemannian flow matching fields on visual observations for robust skill learning. Spatially, Pajarinen et al. demonstrated that mapping the reinforcement learning action space directly to image pixel coordinates facilitates a tighter coupling between visual perception and end-effector control, reducing the complexity of coordinating mobile bases with robotic arms.

By removing the intermediate abstractions of traditional robotics, direct perception-to-action mapping allows agents to operate with the fluid, reactive immediacy characteristic of biological systems.

Key Foundational Work

Thoughts About a Visually Guided Grasp Reflex (1973) — contributed 1 concept elements to this direction.
Motion Sketch: Acquisition of Visual Motion Guided Behaviors (1995) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_end_to_end_visuomotor_control_943af19f,
  title = {end-to-end visuomotor control},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/end-to-end-visuomotor-control/rdr_943af19fff14439395d7c6c4ee59662a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_end_to_end_visuomotor_control_943af19f_v1,
  title = {end-to-end visuomotor control},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/end-to-end-visuomotor-control/rdr_943af19fff14439395d7c6c4ee59662a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake