Ag2x2: Robust Agent-Agnostic Visual Representations for Zero-Shot Bimanual Manipulation

Ziyin Xiong, Yinghan Chen, Puhao Li, Yixin Zhu, Tengyu Liu, Siyuan Huang

Abstract

Bimanual manipulation remains a significant challenge in robotics due to the complexities of coordination and the high cost of expert demonstrations. We present Ag2x2, a framework that enables robots to acquire complex bimanual skills by watching human videos. By selectively preserving hand position data while masking human appearance, Ag2x2 creates "coordination-aware" visual representations. Our method achieves a 73.5% average success rate across 13 diverse tasks, outperforming both previous agent-agnostic methods and manually engineered reward functions in zero-shot transfer scenarios.

Introduction

Bimanual manipulation—the coordinated use of two hands—is a fundamental human capability but notoriously difficult for robots to master. Traditional approaches rely on expensive expert demonstrations or brittle, manually engineered reward functions that often fail to generalize. Recent "agent-agnostic" methods attempt to learn from human videos by masking the human entirely to focus on object movement. However, for bimanual tasks, total masking removes critical information about how the hands must coordinate relative to each other. Ag2x2 bridges this gap by maintaining an agent-agnostic representation that remains coordination-aware.

Methodology

Overview of the Ag2x2 pipeline: from human video processing with selective inpainting to robot skill execution.

The Ag2x2 framework operates in two distinct stages:

1. Representation Learning

The system learns from the EpicKitchen dataset of ego-centric human videos. The core innovation is selective inpainting. Using off-the-shelf models (HaMeR and ODISE), the system detects hands and segments the human body. It then erases the human body while preserving the 2D coordinates of the hands. These coordinates are treated as specialized position tokens for a ViT-Large visual encoder. This allows the model to learn the relationship between object states and hand trajectories without being biased by human-specific kinematics or appearance.

2. Skill Learning

The learned representation guides Reinforcement Learning (PPO) in the IsaacGym simulator. Ag2x2 introduces a "Tilted Reward" function, which calculates the embedding similarity between the robot's current state and a goal image. This function encourages the agent to exceed the initial state's similarity to the goal, facilitating more effective exploration during the early phases of training.

Experimental Evaluation

Bimanual task execution across diverse environments including drawer manipulation, door closing, and rope straightening.

Ag2x2 was evaluated across 13 bimanual tasks from the Bi-DexHands and PerAct2 benchmarks, including closing doors, lifting pots, and straightening deformable rope.

Key Results:

  • Superior Success Rates: Ag2x2 achieved a 73.5% average success rate, significantly higher than the previous state-of-the-art agent-agnostic method, Ag2Manip (56.4%).
  • Outperforming Expert Rewards: The system even surpassed policies trained with manually designed expert rewards (63.2%), suggesting that visual embeddings capture semantic task completion better than geometric heuristics.
  • Deformable Object Mastery: The system successfully learned to straighten a rope, a task involving complex tension and positioning that is typically difficult to model.
  • Smoothness: Trajectory analysis indicated that Ag2x2 produces movements with smoothness comparable to expert-designed rewards, avoiding the jittery motions common in RL.

Discussion and Limitations

While Ag2x2 demonstrates robust zero-shot capabilities, it currently relies on static goal images, which can limit its ability to capture intermediate dynamics (e.g., the intent behind a throwing motion). Furthermore, the current implementation focuses on end-effector positions rather than complex, articulated finger movements. Future work will explore high-level planners to generate intermediate visual waypoints for multi-stage tasks and extending the representation to more dexterous, finger-level manipulation.

Summary updated

BibTeX
@inproceedings{xiong2025ag,
  author = {Ziyin Xiong and Yinghan Chen and Puhao Li and Yixin Zhu and Tengyu Liu and Siyuan Huang},
  title = {{Ag2x2: Robust Agent-Agnostic Visual Representations for Zero-Shot Bimanual Manipulation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11246626},
}
Figure