Learning Human-Aware Robot Policies for Adaptive Assistance

Jason Qin, Shikun Ban, Wentao Zhu, Yizhou Wang, Dimitris Samaras

Abstract

Developing robots that can assist humans efficiently, safely, and adaptively is crucial for real-world applications such as healthcare. While previous work often assumes a centralized system for co-optimizing human-robot interactions, we argue that real-world scenarios are much more complicated, as humans have individual preferences regarding how tasks are performed. Robots typically lack direct access to these implicit preferences. However, to provide effective assistance, robots must still be able to recognize and adapt to the individual needs and preferences of different users. To address these challenges, we propose a novel framework in which robots infer human intentions and reason about human utilities through interaction. Our approach features two critical modules: the anticipation module is a motion predictor that captures the spatial-temporal relationship between the robot agent and user agent, which contributes to predicting human behavior; the utility module infers the underlying human utility functions through progressive task demonstration sampling. Extensive experiments across various robot types and assistive tasks demonstrate that the proposed framework not only enhances task success and efficiency but also significantly improves user satisfaction, paving the way for more personalized and adaptive assistive robotic systems.

Introduction

Assistive robots often fail in real-world settings because they focus exclusively on task completion (e.g., feeding a user) while ignoring the user's personal preferences (e.g., "be gentler"). This creates a fundamental "reward misalignment." A robot’s primary objective is usually a Task Reward (e.g., "Did the food reach the mouth?"), but humans prioritize a secondary layer of Preference Rewards (e.g., "Don't use too much force," "Don't spill").

These preferences are often "opaque"—the robot cannot directly access the human's internal state, and preferences vary significantly between individuals. Traditional solutions rely on centralized co-optimization or frequent, intrusive user feedback. This work introduces a decentralized framework that allows robots to autonomously infer hidden human preferences and predict future human movements through interaction alone, improving both task success and user satisfaction without explicit feedback.

Methodology

The proposed framework integrates two primary components with a Reinforcement Learning (PPO) backbone to create a human-aware policy:

  1. Anticipation Module: This motion predictor captures the spatial-temporal relationship between the robot and the user. By analyzing the last $k$ frames of joint positions, it predicts the human's future motion. The robot’s policy uses this foresight to generate actions that align with the user's likely trajectory.
  2. Utility Module: This module estimates the weights of the human’s underlying utility function. Instead of querying the user, it employs Markov Chain Monte Carlo (MCMC) sampling on the robot's interaction history. As the robot’s policy improves, the demonstrations in its buffer become more consistent, allowing the module to progressively refine its estimate of what the human values (e.g., prioritizing "low force" over "speed").
The proposed framework architecture featuring the Utility and Anticipation modules.

The final robot reward is a combination of the standard task reward and an estimated preference reward derived from the utility module: $r_R = \hat{r}_{pref} + r_{task}$.

Experimental Setup

The framework was evaluated using Assistive Gym, a high-fidelity physics simulation for physical human-robot interaction.

  • Tasks: Three scenarios were tested: Feeding, Drinking (requiring complex rotations), and Bed Bathing (requiring constant skin contact).
  • Robots: To ensure generalization, the policy was tested on four robotic platforms: Sawyer, Baxter, Jaco, and PR2.
  • User Profiles: Various "human preference settings" were simulated by varying the weights of penalties for "Hits" (spillage), "Force" (accidental contact), and "High Force" (excessive pressure).

Results and Key Insights

The framework effectively turns a misaligned multi-agent problem into a collaborative one. By estimating the human's reward, the robot can optimize for a "Human Reward" it doesn't actually have direct access to.

Training curves showing the proposed method outperforming PPO and TD3 baselines in mean human reward.
  • Superior Satisfaction: In the Feeding task, the proposed method achieved a Human Reward of 118.656, significantly outperforming standard PPO (109.063) and TD3 (87.069).
  • Safety and Efficiency: In the Drinking task, the method reduced the "Hit" penalty (spillage) to less than a third of the baseline PPO level while maintaining a 100% success rate.
  • Robust Generalization: While baselines like TD3 failed completely (0% success) on complex tasks like Bed Bathing, the proposed policy maintained a 60% success rate. Furthermore, performance gains were consistent across all four robot types, proving the framework is robot-agnostic.
  • Dynamic Foresight: The anticipation module adapts its horizon based on task stage. When far from the user, it utilizes longer foresight (10 steps) for planning; when close to the user's face, it shifts to short-term foresight (5 steps) for precision.

Conclusion

This research demonstrates that implicit inference is key to adaptive assistance. By treating human behavior as a signal of underlying utility, robots can self-correct to meet user needs. While currently limited to simulation, future work will focus on deploying this framework on physical robotic platforms with real human subjects and incorporating explicit communication (speech/gestures) to complement motion-based inference.

Summary updated

Abstract

Developing robots that can assist humans efficiently, safely, and adaptively is crucial for real-world applications such as healthcare. While previous work often assumes a centralized system for co-optimizing human-robot interactions, we argue that real-world scenarios are much more complicated, as humans have individual preferences regarding how tasks are performed. Robots typically lack direct access to these implicit preferences. However, to provide effective assistance, robots must still be able to recognize and adapt to the individual needs and preferences of different users. To address these challenges, we propose a novel framework in which robots infer human intentions and reason about human utilities through interaction. Our approach features two critical modules: the anticipation module is a motion predictor that captures the spatial-temporal relationship between the robot agent and user agent, which contributes to predicting human behavior; the utility module infers the underlying human utility functions through progressive task demonstration sampling. Extensive experiments across various robot types and assistive tasks demonstrate that the proposed framework not only enhances task success and efficiency but also significantly improves user satisfaction, paving the way for more personalized and adaptive assistive robotic systems. Code and demos are available at https://asonin.github.io/Human-Aware-Assistance/.
BibTeX
@article{qin2025learning,
  author = {Jason Qin and Shikun Ban and Wentao Zhu and Yizhou Wang and Dimitris Samaras},
  title = {{Learning Human-Aware Robot Policies for Adaptive Assistance}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2025},
  doi = {10.1109/lra.2025.3583161},
}
Figure