PAIR: Proprioception-Aligned Image-Ray Representations for Viewpoint-Robust Manipulation

Sung-Wook Lee, Xuhui Kang, Dylan Losey, Yen-Ling KuoarXiv 2026 · arXiv

TL;DR

PAIR (Proprioception-Aligned Image-Ray) is a new framework that makes robot manipulation policies significantly more robust to changes in camera angles. By fusing camera-ray geometry with visual features and aligning them using the robot's own internal sense of position (proprioception) via contrastive learning, PAIR enables robots to generalize to unseen viewpoints with much higher success rates than standard methods.

What Problem Does This Paper Address?

In robot imitation learning, we want a robot to watch a human perform a task and then do it itself. Most modern policies rely on vision, but images are tricky: they are 2D projections of a 3D world. If you move the camera, the same physical state (e.g., the gripper holding a mug) looks completely different. Conversely, two very different robot poses might look identical from specific angles.

This "visual ambiguity" makes it hard for robots to learn a consistent mapping from pixels to actions. If a robot is trained on demonstrations from only one or two viewpoints, it often fails when the camera is moved even slightly. While some researchers have tried adding camera information as an input, simply "dumping" camera data into a model doesn't necessarily teach the model how that geometry relates to the task at hand.

Fig. 1: Resolving visual ambiguity with camera geometry. (A) Robot datasets often contain visually confounding observations, where similar robot appearances correspond to substantially different robot states. (B) Camera-ray information provides explicit camera-to-robot geometry to help disambiguate such observations.

Methods

The authors introduce PAIR, which tackles this problem through two main innovations:

  1. Ray Positional Encoding: Instead of just using standard 2D pixel coordinates, PAIR injects 3D geometric information directly into the vision encoder. They use Plücker coordinates to represent camera rays. For every image patch, they calculate two rays: one relative to the robot's base and one relative to its end-effector. These rays act as "geometry-aware positional embeddings," telling the model exactly where in 3D space each part of the image is located.
  2. Proprioceptive Alignment: To ensure the model actually understands this geometry, the authors use Supervised Contrastive Learning. They force the model to produce similar internal representations for observations that have similar robot states (proprioception), even if they look different because of the camera angle.

The framework is highly flexible; the authors demonstrate it working with both Diffusion Policies (a popular class of generative models for robotics) and Vision-Language-Action (VLA) models like SmolVLA.

Overview of the PAIR architecture, showing how image and ray features are fused and then aligned using proprioception similarity.

Datasets

The researchers evaluated PAIR across a diverse set of benchmarks:

  • MimicGen: A simulated benchmark for single-arm tasks like stacking blocks and assembling pieces.
  • AlohaGym: A bimanual (two-armed) simulation focused on cube transfer.
  • LIBERO: A large-scale multi-task benchmark involving 130 different manipulation tasks.
  • Real-World Franka: A physical setup using a Franka Research 3 arm and two RealSense cameras, performing tasks like stacking cubes, loading a toaster, and hanging a mug.

For all benchmarks, they re-collected data with diverse, randomized camera viewpoints to specifically test for viewpoint robustness.

Key Insights and Ideas

  • Geometry is the "Bridge": Proprioception alone is ambiguous because it's in a fixed physical frame, while vision is in a camera frame. Camera rays provide the mathematical bridge needed to align these two worlds.
  • Dual-Reference Rays Matter: Using rays relative to both the base (static anchor) and the end-effector (dynamic target) provides better geometric cues than using just one. Ablation studies showed that removing either one leads to a performance drop.
  • Attention Grounding: Visualizations show that PAIR-trained models focus their "attention" much more accurately on the robot's gripper and the task-relevant objects compared to standard models, which often get distracted by background elements when the viewpoint changes.
Attention maps showing that PAIR concentrates the model's focus on the end-effector and workspace, whereas baseline models focus on irrelevant regions.

Key Results

PAIR consistently outperformed standard Behavior Cloning (BC) baselines:

  • Simulation: In single-task settings, PAIR improved success rates by an average of 12.7% over standard BC.
  • Real-World: The gains were even more dramatic in physical experiments, where PAIR outperformed the baseline by 26.7% in task completion.
  • Efficiency: Compared to other alignment methods like Time-Contrastive Networks (TCN), PAIR is significantly more computationally efficient, requiring 58.8% fewer FLOPs and 60% fewer decoded frames during training.
Success rates in real-world tasks, showing PAIR (blue) significantly outperforming the BC baseline (red).

Limitations and Future Work

While PAIR is a significant step forward, the authors note a few hurdles:

  • Viewpoint Diversity: The effectiveness of the method still depends on having at least some diversity of viewpoints in the training data.
  • Calibration Overhead: The method requires accurate camera extrinsics (knowing exactly where the camera is relative to the robot), which can be time-consuming to set up in the real world.
  • Computational Cost: While more efficient than some alternatives, the proprioceptive alignment objective still adds extra computational overhead during the training phase compared to simple behavior cloning.

Future work includes scaling PAIR to even larger real-world datasets and exploring how to use camera pose estimation to remove the need for manual calibration.

Summary updated

Abstract

While imitation learning has made substantial progress toward generalizable robot manipulation through scaling, viewpoint robustness remains a major bottleneck. Visuomotor policies rely heavily on visual inputs, yet images provide an inherently ambiguous signal under viewpoint change: similar scenes may appear markedly different across viewpoints, while distinct states may share certain visual overlaps. A policy must therefore infer not only the task-relevant state, but also how it is visually projected under a particular camera configuration. This entanglement of viewpoint-dependent appearance and task-relevant state can make learning inefficient and limit generalization to unseen viewpoints. Recent work has begun to address this challenge through camera-ray conditioning and representation learning using geometric cues to induce invariance. However, providing camera information as additional context does not by itself ground visual representations in how the task scene is visually projected, while geometric alignment without camera context can introduce spurious correspondences by ignoring viewpoint-dependent projection. We propose PAIR, a contrastive learning framework that aligns joint image-ray representations using robot proprioception. PAIR encourages observations with similar robot states to remain close in representation space across viewpoints, providing a structured grounding signal for learning viewpoint-robust features. PAIR is a highly adaptable framework: it can be used to train Diffusion Policy (DP) or post-train vision-language-action (VLA) models. In simulated and real-world manipulation benchmarks with camera variation, PAIR consistently outperforms the behavior cloning baseline without ray conditioning by 12.7% and 26.7% points, respectively, demonstrating stronger generalization to unseen viewpoints. Project website: PAIRrobot.github.io.
BibTeX
@misc{lee2026pair,
  author = {Sung-Wook Lee and Xuhui Kang and Dylan Losey and Yen-Ling Kuo},
  title = {{PAIR: Proprioception-Aligned Image-Ray Representations for Viewpoint-Robust Manipulation}},
  year = {2026},
  eprint = {2611.11798},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11798},
}
Figure