Robustifying a Policy in Multi-Agent RL with Diverse Cooperative Behaviors and Adversarial Style Sampling for Assistive Tasks

Takayuki Osa, Tatsuya Harada

TL;DR

Researchers have developed a framework to make caregiver robots more robust by training them against a wide variety of simulated human behaviors. By using mutual information to autonomously generate diverse "human styles" and adversarial sampling to focus on the most challenging ones, the resulting robotic policies perform significantly better when paired with new, unseen partners.

What Problem Does This Paper Address?

In assistive robotics—such as a robot arm feeding a person with motor impairments—the task is often modeled as a "cooperative multi-agent" problem. Usually, researchers use co-optimization, where the robot (caregiver) and the simulated human (care-receiver) learn to work together simultaneously.

The "aha" problem is that these policies become brittle. Because the robot only learns to assist one specific version of a human, it often fails if a real person moves their head differently or has a different "style" of interaction. In the real world, human behavior is diverse and unknown. This paper tackles the challenge of training a caregiver policy that remains effective even when the care-receiver's behavior deviates from what was seen during training.

Methods

The authors propose a framework consisting of two main components:

  1. Latent-conditioned Proximal Policy Optimization (LPPO): To avoid the tedious task of manually "hard-coding" different human behaviors (e.g., "this human is jerky," "this human is still"), the authors use a latent variable $z_r$ to represent behavior styles. They maximize the mutual information between this latent variable and the human's actions. This forces the simulated human to learn a wide "menu" of distinct, recognizable behaviors autonomously.
  2. Adversarial Style Sampling: During training, the robot doesn't just practice with a random human style. Instead, it uses an $\epsilon$-greedy strategy to frequently sample the "worst-case" human style—the one that currently results in the lowest success rate. This is a form of adversarial training applied to a cooperative task; by learning to succeed with the most difficult partners, the robot becomes more robust overall.

Datasets

The study utilizes Assistive Gym, a physics-based simulation framework built on the PyBullet engine. They evaluated their method on four specific feeding tasks involving different robotic platforms:

  • FeedingPR2Human-v1 (PR2 mobile manipulator)
  • FeedingJacoHuman-v1 (Jaco arm)
  • FeedingBaxterHuman-v1 (Baxter research robot)
  • FeedingSawyerHuman-v1 (Sawyer arm)

Key Insights and Ideas

  • Cooperation requires Robustness: In multi-agent RL, "cooperation" is often treated as a solved problem if agents share a reward. This paper highlights that shared rewards lead to over-specialization, which is a liability when one agent is a variable human.
  • Mutual Information as a Proxy for Diversity: Instead of engineering rewards for "diversity," the authors show that maximizing mutual information effectively "spreads out" the latent space of behaviors, creating a diverse set of test subjects for the robot.
  • Adversarial Training for Cooperation: Adversarial techniques are usually associated with zero-sum games (like GANs or Chess). Here, the authors show that "adversarial sampling" of a partner's style is a powerful tool for improving the lower bound of performance in cooperative settings.

Key Results

The authors compared their proposed method (PPO-LPPO-adv) against standard co-optimization (PPO-PPO) and other baselines.

  • Generalization to New Partners: When the trained caregiver robots were tested against a completely new care-receiver (trained separately with a different algorithm, TD3), the proposed method maintained much higher performance.
  • Quantitative Gains: In the FeedingPR2Human task, the standard PPO-PPO baseline's performance dropped from a "training" score of 114.9 to a "test" score of 47.0 when the partner changed. In contrast, the proposed PPO-LPPO-adv maintained a score of 77.1 on the same test, demonstrating significantly higher robustness.
  • Stability: The LPPO-based methods showed more stable learning curves compared to other diversity-seeking algorithms like LTD3, which often saw performance drops late in training.

Limitations and Future Work

  • Sim-to-Real Gap: The entire study was conducted in simulation. While Assistive Gym is a sophisticated physics environment, real-world human muscle movements and sensor noise present additional challenges.
  • Scope of Robustness: The paper focuses specifically on robustness to behavioral styles. It does not explicitly address robustness to environmental changes (like lighting or obstacles) or hardware failures.
  • Future Directions: The authors suggest exploring other types of robustness and eventually deploying these robust policies on physical robotic platforms to validate the simulation findings.

Summary updated

Abstract

Autonomous assistance of people with motor impairments is one of the most promising applications of autonomous robotic systems. Recent studies have reported encouraging results using deep reinforcement learning (RL) in the healthcare domain. Previous studies showed that assistive tasks can be formulated as multi-agent RL, wherein there are two agents: a caregiver and a care-receiver. However, policies trained in multi-agent RL are often sensitive to the policies of other agents. In such a case, a trained caregiver’s policy may not work for different care-receivers. To alleviate this issue, we propose a framework that learns a robust caregiver’s policy by training it for diverse care-receiver responses. In our framework, diverse care-receiver responses are autonomously learned through trials and errors. In addition, to robustify the care-giver’s policy, we propose a strategy for sampling a care-receiver’s response in an adversarial manner during the training. We evaluated the proposed method using tasks in an Assistive Gym. We demonstrate that policies trained with a popular deep RL method are vulnerable to changes in policies of other agents and that the proposed framework improves the robustness against such changes.
BibTeX
@inproceedings{osa2024robustifying,
  author = {Takayuki Osa and Tatsuya Harada},
  title = {{Robustifying a Policy in Multi-Agent RL with Diverse Cooperative Behaviors and Adversarial Style Sampling for Assistive Tasks}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  url = {https://openreview.net/forum?id=Pkey6AEihT},
}

Loading related content...

Figure