Weakly Supervised Correspondence Learning

Zihan Wang, Zhangjie Cao, Yilun Hao, Dorsa Sadigh

Abstract

Training robots to mimic behaviors across different physical embodiments—such as a four-legged agent teaching a five-legged one—remains a significant challenge in robotics. Traditional methods rely either on expensive, perfectly paired expert data or unreliable unsupervised techniques prone to compounding errors. This paper introduces Weakly Supervised Correspondence Learning (WeaSCL), a framework that bridges this gap by utilizing two forms of "weak" supervision: the temporal order of actions and simple, human-labeled spatial abstractions. Our results demonstrate that WeaSCL significantly outperforms unsupervised baselines across cross-morphology, cross-physics, and cross-modality tasks, approaching the performance of fully supervised oracles.

Introduction

In robotics, correspondence learning is the process of finding a functional mapping between the states and actions of two distinct agents. This is essential for imitation learning, where a "student" robot must learn from a "teacher" that may have a different number of joints, different limb lengths, or different physical properties. For example, mapping the movements of a human arm to a robotic industrial manipulator requires the model to understand how a human shoulder and elbow correspond to the robot's specific joint configuration.

Existing approaches generally fall into two categories, both with significant drawbacks:

  1. Strictly Supervised Learning: This requires experts to manually pair thousands of identical states between two robots. This process is labor-intensive, scales poorly, and is often physically impossible for complex, high-dimensional systems.
  2. Unsupervised Learning (Cycle-Consistency): Methods like Domain-Conditioned Cycle-consistency (DCC) attempt to learn mappings by ensuring that a state translated from Agent A to Agent B and back again returns to the starting point. However, these models frequently suffer from compounding errors, where small inaccuracies accumulate over time, and misalignment, where the model learns a mathematically consistent but physically nonsensical mapping.

WeaSCL addresses these issues by asking: Can we use "weak" supervision that is easy for humans to provide but strong enough to stabilize the learning process?

Methodology

WeaSCL introduces two primary innovations to stabilize correspondence learning and prevent the drift associated with unsupervised methods.

Multi-Step Dynamics Cycle-Consistency

Standard cycle-consistency models typically only verify if the immediate next step is consistent. WeaSCL extends this by looking ahead across a temporal horizon (e.g., 5 to 10 steps). By enforcing consistency over a sequence of actions rather than a single transition, the model is forced to account for long-term dynamics. This multi-step approach acts as a temporal guardrail, significantly reducing the "drift" that causes single-step models to diverge from the intended trajectory.

Paired Abstractions

The core of the "weak" supervision in WeaSCL lies in paired abstractions. Instead of requiring humans to align complex 100+ dimensional joint vectors, the framework asks for alignment on simple, intuitive features. For instance, a human can easily identify if two different robots are at the same (x, y) coordinate or if their grippers are at the same height.

  • The system trains a similarity function based on these easily labeled pairs.
  • This function serves as a constraint during the main training phase, ensuring that the complex state-mapping does not violate basic physical truths (e.g., if the teacher is jumping, the student's mapped state should not show it crouching).

Experimental Evaluation

The authors evaluated WeaSCL across three diverse and challenging scenarios:

  1. Cross-Morphology (MuJoCo): Mapping between agents with different skeletal structures, including 2-legged vs. 3-legged HalfCheetah, 3-link vs. 4-link Swimmer, and 4-legged vs. 5-legged Ant.
  2. Cross-Physics: Mapping between agents operating under different environmental constants, such as varying gravity or friction levels (using Hopper and Walker2d).
  3. Cross-Modality (Franka Panda Robot): A sim-to-real task requiring the model to map raw RGB camera images (visual state) to the actual joint configurations (physical state) of a real-world robotic arm.

Key Results and Insights

WeaSCL consistently outperformed the state-of-the-art unsupervised baseline, DCC, across all metrics.

  • Performance Gains: In the complex Ant morphology task (4-legged to 5-legged), WeaSCL achieved a performance score of 971.08. This nearly doubles the score of the single-step DCC-1 (447.50) and closely rivals the "Oracle" score of 991, which represents the theoretical limit of perfect supervision.
  • Error Suppression: Qualitative analysis shows that while unsupervised errors grow exponentially over time, WeaSCL’s error remains relatively flat. This confirms that multi-step loss effectively mitigates compounding errors.
  • Real-World Viability: On the Franka Panda robot, WeaSCL achieved the lowest joint-state estimation error from visual inputs, proving more robust than standard cycle-consistency or domain randomization techniques.
  • The Power of "Loose" Alignment: A critical insight from the study is that abstractions do not need to be perfect. Even "loose" similarity signals, such as end-effector positions, are sufficient to prevent "mode collapse," where a model erroneously maps multiple distinct source states to a single target state.

Discussion and Future Work

While WeaSCL significantly reduces the burden of data labeling, it does not eliminate it. The method still requires a human-in-the-loop to select relevant abstractions (e.g., deciding that "height" is a useful signal for a jumping task). Furthermore, while labeling 1,000 similarity pairs is easier than full-state pairing, it still represents a bottleneck for massive scaling.

Future research directions include automated abstraction discovery, where the model autonomously determines which features (like velocity or center of mass) serve as the most effective weak signals. This would further move correspondence learning toward a fully autonomous pipeline, capable of adapting to new robotic bodies with minimal human intervention.

Summary updated

Abstract

Correspondence learning is a fundamental problem in robotics, which aims to learn a mapping between state, action pairs of agents of different dynamics or embodiments. However, current correspondence learning methods either leverage strictly paired data-which are often difficult to collect-or learn in an unsupervised fashion from unpaired data using regularization techniques such as cycle-consistency-which suffer from severe misalignment issues. We propose a weakly supervised correspondence learning approach that trades off between strong supervision over strictly paired data and unsupervised learning with a regularizer over unpaired data. Our idea is to leverage two types of weak supervision: i) temporal ordering of states and actions to reduce the compounding error, and ii) paired abstractions, instead of paired data, to alleviate the misalignment problem and learn a more accurate correspondence. The two types of weak supervision are easy to access in real-world applications, which simultaneously reduces the high cost of annotating strictly paired data and improves the quality of the learned correspondence. We show the videos of the experiments on our website.
BibTeX
@inproceedings{wang2022weakly,
  author = {Zihan Wang and Zhangjie Cao and Yilun Hao and Dorsa Sadigh},
  title = {{Weakly Supervised Correspondence Learning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  url = {https://openreview.net/forum?id=05SNARBhl5},
}
Figure