The Bidirectional Coupling of Perception and Action

24 concepts14 papers1 proposal

For decades, classical cognitive science treated the mind like a sandwich: perception was the input, action was the output, and cognition was the filling in between. In this view, the visual system passively reconstructs a 3D model of the world before handing it off to the motor system to plan a movement.

However, a growing body of research across neuroscience, developmental psychology, and embodied AI suggests a far more integrated reality. Perception and action are not separate sequential stages; they are bidirectionally coupled, sharing a unified representational domain and co-developing through closed-loop feedback. To truly understand how an agent interacts with the world, we cannot look at sensory processing or motor control in isolation. We must look at how they shape one another.

The Shared Space of Common Coding

Why does executing an action change what we see? If perception and action were separate modules, moving your legs should have no bearing on how your visual cortex interprets a shape. Yet, evidence shows a direct, top-down influence of motor activity on sensory interpretation.

In a 2012 study on action orientation, researchers found that when human subjects were walking, they were significantly more likely to perceive an ambiguous silhouette as walking toward them. This phenomenon is explained by the principle of common coding: action execution and perception share a unified representational domain. When you engage a specific motor program, it generates a top-down signal that primes the visual system to prioritize congruent, socially relevant interpretations.

[ Motor System: Walking ] ──(Shared Representation)──> [ Visual System: Primed for Approach ]

This cross-talk is made possible because neural assemblies are not confined to isolated sensory regions. As Roelfsema reviews in Solutions for the Binding Problem, the brain solves the challenge of linking distributed features by forming dynamic "interaction skeletons" of neural connections. These assemblies extend across visual, parietal, and motor cortices, globally synchronizing sensory input with motor planning to ensure behavioral consistency.

Active Vision: Eye Movements as Motor Commands

The coupling becomes even more explicit when we examine how agents acquire information. We often think of eyes as passive cameras, but in reality, vision is an active, behavior-driven process.

In his foundational paper on animate vision, Ballard argued that shifting from ego-centric coordinates to behavior-driven vision requires tight visuo-motor control loops. Instead of building a complete 3D map of a scene, an active agent moves its eyes to extract just the information needed for the task at hand.

This means eye movements are a fundamental component of the motor command chain rather than a passive input stage. As Arai et al. demonstrate in an analysis of skilled physical tasks, visual tracking and limb movements are neurologically linked. When an expert athlete intercepts a ball, their gaze behavior and motor execution interact critically; isolating visual prediction from the physical movement overlooks how the brain coordinates precise physical actions. Similarly, Fix et al. developed computational models showing a tight, inseparable coupling between low-level feature extraction and the spatial processing required to guide attention and eye movements.

Co-Development Through Active Efficient Coding

If perception and action are so deeply intertwined, how do they develop in a growing organism or an autonomous agent? They cannot be trained sequentially. Because an agent's actions change the statistics of its sensory input, the sensory system must learn from data that is constantly being altered by the motor system.

To solve this reciprocal dependency, Zhang et al. proposed a unified optimization principle known as Active Efficient Coding (AEC) in Intrinsically Motivated Learning of Visual Motion Perception and Smooth Pursuit.

Consider the problem of learning to track a moving object (smooth pursuit). A classical reinforcement learning agent might need an external reward function to learn to keep the object centered. In contrast, an AEC agent is driven by a single, internal motivation: to represent sensory input as accurately and efficiently as possible (sparse coding).

┌────────────────────────────────────────────────────────┐
│                                                        │
▼                                                        │
[ Motor Action: Eye Movement ]                           │ (Alters statistics)
│                                                        │
▼                                                        │
[ Sensory Input: Retinal Image ]                         │
│                                                        │
▼                                                        │
[ Sparse Coding Neural Network ] ──(Intrinsic Reward)────┘ (Maximize Coding Efficiency)

When the agent's eyes successfully track an object, the image stabilizes on the retina. A stabilized image is redundant and far easier for the neural network to compress efficiently. Therefore, the agent learns to execute smooth pursuit movements not because of an external goal, but because its behavior shapes the sensory statistics to maximize coding efficiency.

This bidirectional co-development explains how complex biological behaviors mature. For example, Zhang et al. applied this to model the development of optokinetic nystagmus, while Klimmasch et al. demonstrated that the active control of eye vergence directly shapes the development of neural disparity tuning. Motor behavior is not just a consumer of sensory representations; it actively carves them.

Implications for Embodied AI and Robotics

Understanding this bidirectional coupling is transforming how we build artificial agents. Traditional robotic architectures that separate perception pipelines from control loops struggle in dynamic, unpredictable environments.

When models focus exclusively on passive observation, they lack a functional link to their own motor capabilities. As Zhong et al. argue, integrating observational learning with active motor execution is necessary to develop comprehensive sensorimotor primitives.

Modern embodied AI systems are bridging this gap by building high-fidelity, closed-loop architectures:

  • Biomimetic Control: Nakada et al. developed virtual human models that combine deep neural networks with biomechanically accurate musculoskeletal systems. By learning to foveate and track targets simultaneously, these agents achieve natural, adaptive interception behaviors.
  • Direct Sensory-to-Motor Mapping: Rather than translating visual inputs into idealized spatial coordinates, Lin et al. demonstrated with SMS-Human that feeding raw binocular visual input directly into motor control systems facilitates human-like eye-head-hand tracking.
  • Physical Action-Perception Dependencies: In hardware setups like eye-in-hand configurations, moving a robotic limb inherently changes the camera's perspective. To exploit this, Shang introduced SUGARL, a reinforcement learning framework where an agent optimizes a sensorimotor reward based on how well it can predict its own actions from its observations, outperforming traditional decoupled models.

Ultimately, whether analyzing the human brain during a sensorimotor conflict like the rubber-hand illusion—where Rood showed that perceptual drift generates involuntary physical forces—or training a humanoid robot, the lesson is the same. Perception and action are two sides of the same coin. By treating them as a unified, bidirectional system, we gain a clearer picture of both biological intelligence and the path toward truly adaptive machines.

Key Foundational Work

Reference Frames for Animate Vision (1989) — contributed 1 concept elements to this direction.
Solutions for the Binding Problem (1998) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bidirectional_common_coding_2f7ea0d5,
  title = {bidirectional common coding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-common-coding/rdr_2f7ea0d5e5674d76a6849385c156d13f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_bidirectional_common_coding_2f7ea0d5_v1,
  title = {bidirectional common coding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-common-coding/rdr_2f7ea0d5e5674d76a6849385c156d13f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake