Abstract
Assistive robotics, particularly tasks involving close human-robot interaction like feeding, are often modeled as cooperative multi-agent reinforcement learning (MARL) problems. However, standard co-optimization techniques frequently result in brittle policies that fail when paired with human partners whose behaviors deviate from those seen during training. This paper presents a framework to enhance the robustness of caregiver robot policies by training them against a diverse set of simulated human behaviors. By leveraging mutual information to autonomously generate distinct human "styles" and employing adversarial sampling to prioritize training on the most challenging partners, the proposed method significantly improves generalization to unseen human behaviors in high-fidelity simulations.
Introduction
In the field of assistive robotics, the primary goal is to develop agents capable of providing physical support to individuals with motor impairments. Tasks such as robotic feeding require a high degree of coordination between the robot (the caregiver) and the human (the care-receiver). While multi-agent reinforcement learning (MARL) provides a robust framework for learning these cooperative behaviors, a significant challenge remains: the "co-optimization trap." When a robot and a simulated human learn together, they often over-specialize to each other's specific quirks, leading to a lack of robustness when the robot is deployed alongside a real human with a different interaction style.
Methodology
To address this brittleness, we propose a two-pronged framework designed to expose the robot to a wide spectrum of cooperative behaviors during the training phase.
1. Latent-conditioned Proximal Policy Optimization (LPPO)
Manually defining the vast array of possible human behaviors—ranging from jerky movements to static postures—is impractical. Instead, we utilize Latent-conditioned Proximal Policy Optimization (LPPO). This method introduces a latent variable, $z_r$, which represents different behavioral styles. By maximizing the mutual information between this latent variable and the human agent's resulting actions, the system autonomously learns a diverse "menu" of recognizable and distinct behaviors. This ensures that the robot is trained against a representative variety of human partners without requiring manual heuristic engineering.
2. Adversarial Style Sampling
Simply having a diverse set of partners is insufficient if the robot only masters the easiest interactions. We implement an adversarial sampling strategy using an $\epsilon$-greedy approach. During training, the framework identifies which human styles currently result in the lowest success rates for the robot. The system then prioritizes these "worst-case" styles for training sessions. By focusing on these difficult edge cases, the robot's lower bound of performance is raised, leading to a more robust policy overall.
Experimental Setup
The framework was evaluated using Assistive Gym, a physics-based simulation environment built on the PyBullet engine. We focused on four distinct feeding tasks involving various robotic platforms to ensure the generalizability of our findings:
- FeedingPR2Human-v1 (PR2 mobile manipulator)
- FeedingJacoHuman-v1 (Jaco arm)
- FeedingBaxterHuman-v1 (Baxter research robot)
- FeedingSawyerHuman-v1 (Sawyer arm)
Results and Discussion
The proposed method, PPO-LPPO-adv, was benchmarked against standard co-optimization (PPO-PPO) and other diversity-seeking baselines.
Generalization to Unseen Partners
The most critical test of robustness is performance when paired with a completely new partner. We tested our trained caregiver policies against care-receivers trained independently using a different algorithm (TD3). While the standard PPO-PPO baseline experienced a catastrophic performance drop—falling from a training score of 114.9 to a test score of 47.0 in the PR2 task—our PPO-LPPO-adv method maintained a score of 77.1. This demonstrates that the robot learned generalized assistance strategies rather than just memorizing a specific partner's movements.
Stability and Diversity
Our findings indicate that mutual information is a highly effective proxy for behavioral diversity. Unlike other diversity-seeking algorithms like LTD3, which can exhibit instability or performance degradation late in the training cycle, the LPPO-based approach remained stable. The adversarial sampling further ensured that the robot did not "forget" how to handle difficult partners as it optimized for the average case.
Conclusion
This research highlights that in cooperative MARL, shared rewards can inadvertently lead to fragile over-specialization. By treating the diversity of a partner's style as a feature to be explored through mutual information and challenged through adversarial sampling, we can develop robotic policies that are far better prepared for the unpredictability of human interaction. Future work will focus on bridging the sim-to-real gap, testing these robust policies on physical hardware with real-world sensor noise and human variability.
Summary updated