Robustifying a Policy in Multi-Agent RL with Diverse Cooperative Behaviors and Adversarial Style Sampling for Assistive Tasks

Takayuki Osa, Tatsuya Harada

Abstract

Assistive robotics, particularly tasks involving close human-robot interaction like feeding, are often modeled as cooperative multi-agent reinforcement learning (MARL) problems. However, standard co-optimization techniques frequently result in brittle policies that fail when paired with human partners whose behaviors deviate from those seen during training. This paper presents a framework to enhance the robustness of caregiver robot policies by training them against a diverse set of simulated human behaviors. By leveraging mutual information to autonomously generate distinct human "styles" and employing adversarial sampling to prioritize training on the most challenging partners, the proposed method significantly improves generalization to unseen human behaviors in high-fidelity simulations.

Introduction

In the field of assistive robotics, the primary goal is to develop agents capable of providing physical support to individuals with motor impairments. Tasks such as robotic feeding require a high degree of coordination between the robot (the caregiver) and the human (the care-receiver). While multi-agent reinforcement learning (MARL) provides a robust framework for learning these cooperative behaviors, a significant challenge remains: the "co-optimization trap." When a robot and a simulated human learn together, they often over-specialize to each other's specific quirks, leading to a lack of robustness when the robot is deployed alongside a real human with a different interaction style.

Methodology

To address this brittleness, we propose a two-pronged framework designed to expose the robot to a wide spectrum of cooperative behaviors during the training phase.

1. Latent-conditioned Proximal Policy Optimization (LPPO)

Manually defining the vast array of possible human behaviors—ranging from jerky movements to static postures—is impractical. Instead, we utilize Latent-conditioned Proximal Policy Optimization (LPPO). This method introduces a latent variable, $z_r$, which represents different behavioral styles. By maximizing the mutual information between this latent variable and the human agent's resulting actions, the system autonomously learns a diverse "menu" of recognizable and distinct behaviors. This ensures that the robot is trained against a representative variety of human partners without requiring manual heuristic engineering.

2. Adversarial Style Sampling

Simply having a diverse set of partners is insufficient if the robot only masters the easiest interactions. We implement an adversarial sampling strategy using an $\epsilon$-greedy approach. During training, the framework identifies which human styles currently result in the lowest success rates for the robot. The system then prioritizes these "worst-case" styles for training sessions. By focusing on these difficult edge cases, the robot's lower bound of performance is raised, leading to a more robust policy overall.

Experimental Setup

The framework was evaluated using Assistive Gym, a physics-based simulation environment built on the PyBullet engine. We focused on four distinct feeding tasks involving various robotic platforms to ensure the generalizability of our findings:

  • FeedingPR2Human-v1 (PR2 mobile manipulator)
  • FeedingJacoHuman-v1 (Jaco arm)
  • FeedingBaxterHuman-v1 (Baxter research robot)
  • FeedingSawyerHuman-v1 (Sawyer arm)

Results and Discussion

The proposed method, PPO-LPPO-adv, was benchmarked against standard co-optimization (PPO-PPO) and other diversity-seeking baselines.

Generalization to Unseen Partners

The most critical test of robustness is performance when paired with a completely new partner. We tested our trained caregiver policies against care-receivers trained independently using a different algorithm (TD3). While the standard PPO-PPO baseline experienced a catastrophic performance drop—falling from a training score of 114.9 to a test score of 47.0 in the PR2 task—our PPO-LPPO-adv method maintained a score of 77.1. This demonstrates that the robot learned generalized assistance strategies rather than just memorizing a specific partner's movements.

Stability and Diversity

Our findings indicate that mutual information is a highly effective proxy for behavioral diversity. Unlike other diversity-seeking algorithms like LTD3, which can exhibit instability or performance degradation late in the training cycle, the LPPO-based approach remained stable. The adversarial sampling further ensured that the robot did not "forget" how to handle difficult partners as it optimized for the average case.

Conclusion

This research highlights that in cooperative MARL, shared rewards can inadvertently lead to fragile over-specialization. By treating the diversity of a partner's style as a feature to be explored through mutual information and challenged through adversarial sampling, we can develop robotic policies that are far better prepared for the unpredictability of human interaction. Future work will focus on bridging the sim-to-real gap, testing these robust policies on physical hardware with real-world sensor noise and human variability.

Summary updated

Abstract

Autonomous assistance of people with motor impairments is one of the most promising applications of autonomous robotic systems. Recent studies have reported encouraging results using deep reinforcement learning (RL) in the healthcare domain. Previous studies showed that assistive tasks can be formulated as multi-agent RL, wherein there are two agents: a caregiver and a care-receiver. However, policies trained in multi-agent RL are often sensitive to the policies of other agents. In such a case, a trained caregiver's policy may not work for different care-receivers. To alleviate this issue, we propose a framework that learns a robust caregiver's policy by training it for diverse care-receiver responses. In our framework, diverse care-receiver responses are autonomously learned through trials and errors. In addition, to robustify the care-giver's policy, we propose a strategy for sampling a care-receiver's response in an adversarial manner during the training. We evaluated the proposed method using tasks in an Assistive Gym. We demonstrate that policies trained with a popular deep RL method are vulnerable to changes in policies of other agents and that the proposed framework improves the robustness against such changes.
BibTeX
@inproceedings{osa2024robustifying,
  author = {Takayuki Osa and Tatsuya Harada},
  title = {{Robustifying a Policy in Multi-Agent RL with Diverse Cooperative Behaviors and Adversarial Style Sampling for Assistive Tasks}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  url = {https://openreview.net/forum?id=Pkey6AEihT},
}
Figure