In the history of legged robotics, the most robust controllers have often been "blind." By relying exclusively on proprioception—the internal sense of joint angles, velocities, and inertial forces—robots like the Unitree Go1 or Boston Dynamics’ platforms can maintain balance even when pushed or when traversing slippery surfaces. However, a proprioception-only robot is fundamentally reactive: it only knows it has encountered an obstacle once it physically makes contact with it.
A growing body of research is now focused on bridging the gap between this internal "feel" and exteroception (external sensing like vision or LiDAR). The goal is to move from reactive balancing to proactive navigation, allowing robots to adjust their gait before they step into a hole or trip over a curb.
The Reactive Bottleneck
Traditional reinforcement learning (RL) for locomotion often prioritizes proprioceptive history. As Ilija Radosavovic, Jitendra Malik, and colleagues demonstrated in Learning Humanoid Locomotion with Transformers, a causal transformer can learn to adapt to slopes and obstacles "in-context" by analyzing the sequence of its own past movements. While impressive, this approach is inherently limited. Without vision, the robot must "feel" the slope to know it is there.
This lack of foresight is a recurring theme across the field. Xiangyu Miao and Hang Lai argue in PALo: Learning Posture-Aware Locomotion for Quadruped Robots that proprioceptive-only systems cannot perform high-speed navigation in cluttered spaces because they cannot anticipate environmental changes. Similarly, Haojie Shi and Lei Han note in Terrain-Aware Quadrupedal Locomotion via Reinforcement Learning that navigating discrete terrains—like stepping stones—is impossible without integrating visual data directly into the trajectory generation.
Fusing Internal and External States
The core challenge is not just adding a camera, but fusing high-bandwidth visual data with high-frequency physical feedback. Researchers are using several architectural patterns to achieve this:
- Teacher-Student Distillation: A common strategy is to train an "expert" policy in simulation that has access to "privileged" information (like exact terrain height or friction). This expert is then distilled into a "student" policy that only uses the sensors available on the real robot. Hang Lai and Weinan Zhang used this in Terrain Transformer (TERT) to allow quadrupeds to climb stairs by learning to map visual features to stable gait parameters.
- Information Bottlenecks: To ensure the robot doesn't over-rely on noisy sensors, Haoran He and Peilin Wu proposed the Historical Information Bottleneck (HIB). This framework compresses the robot's history into a compact representation, forcing the model to ignore sensor noise while retaining the critical physical "essence" needed for sim-to-real transfer.
- Generative Planners: New methods are moving beyond simple RL. GVS Mothish and Shishir Kolathaya introduced BiRoDiff, which uses diffusion policies to generate bipedal locomotion. By integrating exteroceptive data into the latent space of a diffusion model, the robot can "imagine" stable paths through unseen terrains.
Resilience and Safety
Integration also goes the other way: exteroceptive systems need proprioceptive grounding to be safe. Jin Jin and Marco Hutter demonstrated in Resilient Legged Local Navigation that robots relying solely on vision are vulnerable to "invisible" hazards like glass or deep shadows. Their framework allows a robot to "feel" its way through an environment when vision fails, maintaining an 80% success rate even when blinded.
Furthermore, Wenhao Yu introduced Gameplay Filters, a safety mechanism where the robot "imagines" worst-case environmental interactions. This requires a tight coupling of the robot's internal dynamics and its external surroundings to preemptively block dangerous actions.
The Role of Foundation Models
The most recent frontier involves Large Language Models (LLMs) and Vision-Language Models (VLMs). These models provide high-level "common sense" that traditional controllers lack. Yecheng Jason Ma and Linxi Fan developed DrEureka, which uses LLMs to autonomously configure the simulation parameters needed for a robot to learn to walk on a yoga ball—a task requiring precise coordination of visual balance and physical torque.
However, as Boyu Li points with Proprio-MLLM, these high-level planners often fail because they lack an "inner sense" of the robot's own body limits. By explicitly teaching models about their own proprioception, researchers are achieving significantly higher success rates in complex, long-horizon tasks.
Ultimately, the research direction is clear: the next generation of robots will not just "see" the world or "feel" the ground, but will possess a unified representation that treats its own body and the external environment as a single, continuous system for action.