Learning from My Partner's Actions: Roles in Decentralized Robot Teams

Dylan P. Losey, Mengxi Li, Jeannette Bohg, Dorsa Sadigh

TL;DR

Decentralized robot teams often lack explicit communication channels and must coordinate implicitly through physical actions, but directly inferring a teammate's intent triggers intractable recursive reasoning ("what do you think I think you think"). This paper resolves this deadlock by structuring interactions into alternating speaker and listener roles, showing theoretically and empirically that dynamic role-switching enables implicit action-based communication to match the performance of explicit message passing.

What Problem Does This Paper Address?

When multiple robots collaborate on physical manipulation tasks (such as jointly carrying a heavy table across an obstacle-filled room), occlusions often prevent any single agent from observing the entire environment. While robots could theoretically coordinate via explicit radio broadcasts, communication can be bandwidth-constrained, jammed, or unavailable.

Humans overcome this by communicating implicitly through their actions—for instance, by guiding an object away from an unseen obstacle. For robots, however, inferring state information from a teammate's actions is mathematically challenging: a partner’s force could stem from exploiting their own view, reacting to your past actions, or exploratory behavior. In a decentralized Markov decision process (Dec-MDP), attempting to simultaneously learn from and react to a partner causes cyclic dependencies and infinite recursion.

Methods

The authors break this mutual dependency by introducing complementary, asymmetric behavioral roles:

  • Speaker Role: The speaker acts purely greedily on its local observation history $s_i^{0:t}$, ignoring partner feedback to focus exclusively on task exploitation. Because its behavior does not depend on the partner's actions, its actions become a reliable, interpretable signal of its local observations.
  • Listener Role: The listener observes the speaker’s actions, models the partner explicitly as a speaker policy, and infers the unobserved system state using Bayesian inference or gradient inversion before executing an action.
  • Dynamic Role Alternation: To prevent the speaker from remaining blind to the listener’s observations, agents dynamically alternate roles at regular time intervals $T$.

Theoretical Analysis

  • Matching Centralized Systems (Theorem 1): If the action-to-state mapping is surjective, rapidly switching roles ($dt \to 0$) converges to the optimal centralized policy that has full state information (extended to $N$-agent teams in the appendix).
  • System Stability (Proposition 1): In linear feedback systems ($a = -Ks$), static role assignment can render a controllable system inherently unstable. Alternating roles is necessary to preserve closed-loop stability.
  • Information vs. Stochasticity (Proposition 3): If the target centralized policy is stochastic ($a^* = -K^*s + n$), speakers must reduce their action variance ($\sigma_1^2 \le w_1^2$). Choosing more deterministic actions reduces ambiguity, helping the listener reconstruct the speaker’s hidden state.

Datasets

The authors evaluate their approach on custom synthetic and robotic testbeds rather than standard static datasets:

  • Simulated Collaborative Transport: A 2D simulation where two point-mass robots rigidly carry a rod to a target across environments containing $n \in \{2, 4, 8\}$ circular obstacles (evaluated over 1,000 randomized configurations). Tested under both known obstacle radii and unknown obstacle geometry (where observation dimension exceeds action dimension).
  • Physical Robot Manipulation: An experimental manipulation platform featuring two 7-DoF Franka Emika Panda arms holding a rigid rod. Each robot's camera view is restricted so that each arm can see only one of two obstacles in the shared workspace.

Key Insights and Ideas

  • Asymmetry Eliminates Recursion: Imposing an asymmetric "speaker-listener" structure removes policy interdependence ($π_1$ inside $π_2$ inside $π_1$), turning an intractable recursive belief-updating problem into a standard state estimation problem.
  • Static Roles Cause Instability: Having one permanent leader and one permanent follower fails; without feedback from the follower's private state, the coupled dynamics can diverge even in simple linear systems.
  • Determinism as a Communication Channel: When policies require exploration or noise, the speaker must deliberately suppress variance to make its actions informative, highlighting an explicit trade-off between randomness and legibility in multi-agent control.

Key Results

  • Known Obstacle Geometry: In simulations where actions can fully convey obstacle coordinates, dynamic role switching at every timestep ($T=1$) reached near-optimal success rates: 99.5% ($n=2$), 98.2% ($n=4$), and 92.4% ($n=8$). This significantly outperformed static speaker-listener pairs (87.2%, 73.5%, 53.9%) and symmetric speaker-speaker teams (75.7%, 54.5%, 30.5%).
  • High-Dimensional / Partial Information: When obstacle radii were variable and could not be conveyed in a single 2-DoF action, dynamic role switching closely matched the performance curve of explicit message-passing systems at equivalent frequencies.
  • Robustness to Noise: When zero-mean Gaussian noise was added to action observations (coefficient of variation $= 0.1$), dynamic roles ($T=1$) maintained an 88.4% success rate for $n=2$, tracking explicit noisy communication (92.5%).
  • Hardware Deployment: Two Panda arms successfully negotiated opposing plans (one arm wanting to move forward, the other backward) solely via physical interaction forces. The dynamic role team successfully coordinated around both obstacles, whereas static role teams collided with the obstacle invisible to the speaker.

Limitations and Future Work

  • Dec-MDP Assumption: The formulation assumes joint full observability (together, the agents observe the full state). It does not yet account for Dec-POMDP settings where aspects of the environment are hidden from both agents.
  • Switching Frequency Constraints: Theoretical convergence assumes arbitrarily fast role transitions ($dt \to 0$); on real physical hardware, rapid switching is bounded by actuator dynamics, control loop latency, and mechanical compliance.
  • Scalability and Role Scheduling: While extended theoretically to $N$ agents, the round-robin listener schedule requires high alternation rates that may become difficult to coordinate as team sizes grow.

Summary updated

Abstract

When teams of robots collaborate to complete a task, communication is often necessary. Like humans, robot teammates should implicitly communicate through their actions: but interpreting our partner...
BibTeX
@inproceedings{losey2019learning,
  author = {Dylan P. Losey and Mengxi Li and Jeannette Bohg and Dorsa Sadigh},
  title = {{Learning from My Partner's Actions: Roles in Decentralized Robot Teams}},
  booktitle = {Conference on Robot Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=OcLtr7MCjvC},
}
Figure