One Hand Watches The Other: Dynamic Multi-Agent Cooperation for Sample-Efficient Bimanual Manipulation in Dynamic Environments

Jan Ole von Hartz, Abhinav Valada, Joschka BoedeckerarXiv 2026 · arXiv

TL;DR

DynaMAC is a lightweight, policy-agnostic framework that allows robots to coordinate bimanual tasks and manipulate moving objects with extreme sample efficiency. By automatically detecting when an object is grasped, it resolves a fundamental causal limitation in multi-stream learning, enabling zero-shot generalization from static training demonstrations to highly dynamic real-world environments and human-robot handovers.

What Problem Does This Paper Address?

Imitation learning has made massive strides in robotics, but it typically suffers from a severe data bottleneck: teaching a robot a complex task requires a massive number of demonstrations. To combat this, researchers use multi-stream learning, which breaks down a robot's actions relative to multiple object-centric coordinate frames (e.g., modeling a trajectory relative to a mug or a coffee machine).

While highly sample-efficient, standard multi-stream learning relies on a flawed assumption: that these reference frames are strictly exogenous (independent of the robot's actions). The moment a robot arm grasps an object, this causal assumption collapses. The object is no longer moving independently; its position is rigidly dictated by the robot. In standard multi-stream architectures, this causes the mathematical fusion mechanism (Product-of-Experts) to experience "model collapse"—the variance drops to zero, and the policy freezes. This limitation has historically restricted multi-stream learning to unimanual, static environments.

Fig. 1: DynaMAC brings multi-stream learning to dynamic environments and bimanual manipulation. By estimating kinematic links from demonstrations, it distinguishes independently moving objects (blue) from those manipulated by the robot (orange). Disentangling these causal relationships enables multi- stream learning using the dynamic object frames as task parameters. These dynamic task parameters further enable effective multi-agent coordination without an explicit leader-follower relationship or coordination module. When the cube is lifted (green), the right arm’s pose informs the handover (pink).

Methods

The authors introduce DynaMAC (Dynamic Multi-Agent Cooperation) to resolve this causal breakdown through three key mechanisms:

  1. Kinematic Link Analysis: DynaMAC monitors the precision (inverse variance) of each object-centric stream. When the robot grasps an object, the stream's variance plummets. DynaMAC uses an information-theoretic metric called the Geometric Mean Standard Deviation to detect this spike in mutual information and dynamically filters out the object frame to prevent model collapse.
  2. Virtual Frame Supplementation: Once an object frame is filtered out, the robot loses its spatial reference for post-grasp behavior. DynaMAC compensates by taking a "snapshot" of the end-effector's pose at the start of the skill and injecting it as a static, virtual frame. This acts as a primitive spatial memory.
  3. Decentralized Bimanual Coordination: Instead of using a complex centralized controller or a rigid leader-follower architecture, DynaMAC treats bimanual manipulation as a multi-agent problem. Each arm runs its own independent policy but treats the opposite arm's end-effector as a dynamic task parameter. Leader-follower roles emerge naturally based on which arm has higher spatial precision at any given moment.
Fig. 3: Kinematic link detection. The left y-axes show the local end-effector poses (green), averaged over the pose dimensions; the shaded region indicates the total standard deviation. The right y-axes show the precision determinants, where blue denotes no kinematic link between object and robot, and orange denotes a link. Dotted lines indicate skill boundaries estimated with TAPAS [2]. Panel (A): initially, neither mug nor tree is linked to the robot. Panel (B): external mug motion (À) triggers an updated prediction. Panel (C): after grasping, the mug becomes kinematically linked and follows the end-effector (À). Panel (D): the link ends after placement. Short-lived precision peaks at pre-grasp and during mug–tree alignment do not constitute kinematic links.

Datasets

To rigorously test their method, the authors introduced DynaBench, a novel benchmark built on top of RLBench.

Unlike standard benchmarks that restrict objects to safe spawn zones, DynaBench uses a sample-then-verify scheme across two separate valid static configurations and interpolates between them. This creates highly dynamic scenes where the direction, distance, and velocity of object movements vary automatically across episodes, providing a rigorous testbed for dynamic reference frames, bimanual coordination, and external perturbations.

Key Insights and Ideas

  • Zero-Shot Static-to-Dynamic Generalization: Crucially, DynaMAC can be trained entirely on static demonstrations and deployed directly into dynamic environments where objects are moving. This drastically simplifies data collection, as gathering high-quality demonstrations for moving targets is notoriously difficult.
  • Information-Theoretic Foundation: The paper provides a missing theoretical link in multi-stream literature by proving that frame selection and kinematic link detection are fundamentally grounded in maximizing mutual information and minimizing conditional entropy.
  • Fluid Role Reversal: Because coordination is decentralized, the robot arms don't need pre-defined roles. If one arm is perturbed by an external force (or a human collaborator), the other arm automatically senses the change through its dynamic frame tracking and re-coordinates in real-time.
Fig. 5: Frame selection result for PlaceCups. The Mug frame is used initially, but is ignored once kinematically linked. The virtual end-effector frames complement the dynamic frames. For example, eeGrasp guides grasping and lifting, while eePlace guides placing. Distractor frames are rejected entirely.

Key Results

The authors evaluated DynaMAC against leading deep learning baselines, including Diffusion Policy and MiDiGaP, across both simulation and real-world bimanual Franka Emika robots.

  • Performance Leap: In unimanual dynamic environments, DynaMAC outperformed Diffusion Policy by 35 percentage points on average while requiring 20 times fewer samples (5 vs 100 demonstrations). At data parity (5 demonstrations), DynaMAC outperformed Diffusion Policy by 89 percentage points.
  • Bimanual Resilience: In dynamic bimanual tasks involving trajectory perturbations, standard multi-stream methods failed completely (0% success), while DynaMAC maintained a 96% success rate.
  • Human-Robot Collaboration: When evaluated on a real-world hardware setup, DynaMAC successfully generalized zero-shot to human-robot handovers, dynamically adjusting its trajectory to meet a human hand in free space.

Limitations and Future Work

DynaMAC inherits a few structural constraints from its underlying multi-stream architecture. It relies on an automated skill segmentation module (like TAPAS) to divide long-horizon tasks into distinct phases, and it requires an external vision module to track object poses. Furthermore, under severe visual occlusion, its performance can degrade.

For future work, the authors suggest extending DynaMAC's decentralized framework to whole-body coordination in mobile manipulation and exploring deeper integrations into human-robot collaborative workflows.

Summary updated

Abstract

Multi-stream robot manipulation policies achieve unparalleled sample efficiency and generalization by modeling actions relative to environmental reference frames. However, existing approaches typically assume these frames to be strictly exogenous. This causal assumption collapses in dynamic settings, such as when a single robot arm manipulates a moving object or when two arms coordinate, where each arm effectively becomes part of the dynamic environment of the other. We propose DynaMAC, a lightweight, policy-agnostic framework that resolves this causal limitation while preserving the sample efficiency, computational speed, and flexibility of multi-stream policies, DynaMAC treats the opposite arm as a dynamic task parameter, thereby providing a unified formulation for dynamic manipulation and bimanual coordination without requiring an explicit leader-follower relationship. To rigorously evaluate these capabilities, we introduce DynaBench, a novel benchmark for robot manipulation in dynamic environments. Across both dynamic environments and bimanual manipulation tasks, DynaMAC outperforms leading probabilistic and generative baselines by over 35 percentage points while requiring 20 times fewer samples. Crucially, DynaMAC generalizes zero-shot from static demonstrations to dynamic environments, substantially simplifying data collection and establishing an elegant bridge toward human-robot collaboration.
BibTeX
@misc{hartz2026one,
  author = {Jan Ole von Hartz and Abhinav Valada and Joschka Boedecker},
  title = {{One Hand Watches The Other: Dynamic Multi-Agent Cooperation for Sample-Efficient Bimanual Manipulation in Dynamic Environments}},
  year = {2026},
  eprint = {2607.22119},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.22119},
}
Figure