Decoupled Perception and Control for Robot Learning

125 concepts93 papers

When we train an embodied agent to navigate a room or grasp an object using an end-to-end neural network, we are asking it to solve two fundamentally distinct problems simultaneously: representation (interpreting high-dimensional visual pixels to understand what and where things are) and dynamics (translating that understanding into precise physical forces or motor commands).

This joint optimization creates a severe vulnerability known as the visual domain gap. If a robot trained to manipulate an object in a pristine simulator is deployed in a real-world room with slightly different lighting, a flickering background, or a novel floor texture, the end-to-end policy often fails completely. The network cannot distinguish between a change in visual appearance and a change in underlying physical structure.

To solve this, a prominent research direction focuses on perception-control decoupling. By breaking the end-to-end sensorimotor loop into modular components separated by low-dimensional, human-interpretable intermediate abstractions, researchers can isolate visual complexity from physical execution. This architecture ensures that changes in the visual environment do not require retraining low-level motor skills, and vice versa.


The Core Mechanism: Intermediate Abstractions

The defining feature of a decoupled architecture is the introduction of a structural bottleneck between the perception system and the control policy. Instead of passing raw images directly to an actuator model, a specialized perception module processes the visual input to extract an intermediate representation. This representation captures the essential geometry or semantics of the scene while discarding irrelevant visual noise.

[Raw Visual Input] ──> [Perception Module] ──> [Intermediate Abstraction] ──> [Control Policy] ──> [Motor Actions]

Depending on the task, these intermediate abstractions generally fall into three categories:

1. Affordances and Indicators

Instead of mapping out every pixel in a scene, the perception module can extract a compact set of task-relevant physical parameters. In autonomous driving, Chen et al. pioneered this via "direct perception," training a convolutional neural network to estimate discrete affordance indicators—such as the distance to lane markings and preceding cars—which are then fed into a rule-based controller. Similarly, Sauer et al. leverage intermediate physical affordances to combine the representational power of deep learning with stable, classical control laws.

2. Geometric and Spatial Maps

For complex navigation, the intermediate layer often takes the form of a spatial cost map or voxel grid. Drews demonstrates this by training a neural network to transform monocular camera images directly into a bird's-eye view cost map, allowing a downstream Model Predictive Control (MPC) algorithm to optimize trajectories without ever processing raw pixels. To ensure navigation logic relies entirely on spatial structure rather than domain-specific visual semantics, Li et al. feed geometric features from an occupancy voxel grid into a lightweight policy. For room rearrangement, Trabucco et al. construct explicit 3D semantic maps to isolate object search from deterministic movement planners.

3. Waypoints and Trajectories

Another common abstraction is a sequence of future spatial coordinates. Müller et al. design a modular navigation architecture that "sandwiches" a high-level path-planning policy between a semantic mapping perception module and a low-level waypoint controller. This approach is mirrored in work by Li et al. and Wildi et al., who decouple intent from mechanical execution by predicting future spatial waypoints that a classical feedback controller then executes. For high-speed drone racing, Li utilizes local 3D trajectories as the intermediate representation, ensuring that updates to visual processing do not disrupt or require retraining the underlying vehicle dynamics.


Why Decoupling Works: Bridging the Reality Gap

Decoupling perception from control provides two massive advantages over end-to-end alternatives: seamless sim-to-real transfer and sample efficiency.

Visual Robustness and Domain Adaptation

Because the control policy operates on a clean, abstract geometric or semantic space, it remains entirely agnostic to the visual domain of the input. To transfer a policy from simulation to reality, developers only need to align the perception module's output.

For example, Yuan isolates the visual domain shift using CycleGAN to translate real-world images into simulated styles, allowing a control policy trained entirely in simulation to execute contact-rich assembly tasks in the real world without modification. To handle environmental noise, Wang et al. introduce a dedicated filtering module that processes raw observations before passing them to a frozen action policy, ensuring that visual perturbations do not destabilize low-level motor skills.

Data Efficiency via Geometric Logic

End-to-end reinforcement learning requires millions of samples because it must discover geometric relationships through trial and error. Decoupling allows systems to use simple, interpretable geometric logic instead.

Sundaresan et al. demonstrate this in deformable object manipulation by using Dense Depth Object Descriptors (DDODs) to identify corresponding points on a rope, allowing a robot to tie knots without requiring end-to-end reinforcement learning or massive real-world datasets. Similarly, Hadjivelichkov et al. decouple the perception of functional object regions (like handles) from skill execution, enabling standard geometric controllers to successfully execute tasks like grasping or pouring zero-shot once the functional part is visually matched.

[Raw Pixels] ──> [Foundation Model / Segment Anything] ──> [Geometric Mask] ──> [Frozen Motor Policy]

This modularity also allows developers to leverage powerful, pre-trained foundation models for the perception step without needing to fine-tune them on robot dynamics. Mosbach utilizes the Segment Anything Model (SAM) to isolate object identification from motor execution, converting segmentation masks into geometric representations that a control policy can immediately act upon to grasp arbitrary objects.


Modularity Across Diverse Robotic Domains

The principle of decoupling manifests across a wide variety of physical platforms, from legged robots to aerial vehicles.

Loco-Manipulation and Articulated Objects

For robots performing complex physical interactions, modularity prevents the high degrees of freedom from overwhelming the learning process. Wang et al. decouple perception from locomotion and manipulation in quadrupedal whole-body grasping, allowing reinforcement learning policies to focus exclusively on physical control while specialized vision models handle tricky transparent or specular objects. For articulated objects like drawers and containers, Liu et al. decompose the task into distinct stages: 6D pose estimation, 3D shape reconstruction, and reinforcement learning-based control, optimizing each independently.

Active Vision and Sensory Control

When perception and control are decoupled, an interesting question arises: how should the robot move its sensors to see better? Rather than relying on fixed camera angles, modern decoupled architectures often feature a dual-policy setup that separates motor control from sensory control.

  • Shang introduces SUGARL, a dual-head policy architecture that separates motor control from sensory control while sharing a common visual representation, optimizing task execution and active observation strategies simultaneously.
  • Wan et al. propose MOSER, which uses one policy to execute physical tasks while a separate sensory policy independently optimizes the camera viewpoint to find the most informative perspective.
  • Cho et al. coordinate physical actions and visual perception via a modular diffusion architecture, employing separate models for arm control, object motion prediction, and viewpoint priors to ensure planned physical movements are always supported by a clear line of sight.

A Nuanced Caveat: The Risk of Blind Separation

While strict decoupling simplifies system design and improves generalization, it introduces a subtle risk: if the perception module is trained completely independently of the downstream task, it may discard subtle visual information that is critical for control.

To address this limitation, Genewein et al. formulate an information-theoretic update principle for perception-action coupling. They argue that while perception and action should be treated as separate communication channels to maintain computational efficiency, they must be optimized online relative to one another. This ensures that the abstractions created by the perception module are always functionally driven by the specific requirements of the downstream task.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_modular_navigation_architecture_30f67638,
  title = {modular navigation architecture},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modular-navigation-architecture/rdr_30f676389db74209802136cfb7e8dfd0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_modular_navigation_architecture_30f67638_v1,
  title = {modular navigation architecture},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modular-navigation-architecture/rdr_30f676389db74209802136cfb7e8dfd0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake