Learning Social Affordance for Human-Robot Interaction

Tianmin Shu, Michael S. Ryoo, Song-Chun Zhu

Abstract

In this paper, we present an approach for robot learning of social affordance from human activity videos. We consider the problem in the context of human-robot interaction: Our approach learns structural representations of human-human (and human-object-human) interactions, describing how body-parts of each agent move with respect to each other and what spatial relations they should maintain to complete each sub-event (i.e., sub-goal). This enables the robot to infer its own movement in reaction to the human body motion, allowing it to naturally replicate such interactions. We introduce the representation of social affordance and propose a generative model for its weakly supervised learning from human demonstration videos. Our approach discovers critical steps (i.e., latent sub-events) in an interaction and the typical motion associated with them, learning what body-parts should be involved and how. The experimental results demonstrate that our Markov Chain Monte Carlo (MCMC) based learning algorithm automatically discovers semantically meaningful interactive affordance from RGB-D videos, which allows us to generate appropriate full body motion for an agent.

Introduction

In robotics, "affordance" traditionally refers to the action possibilities offered by an object, such as a handle affording pulling. However, Human-Robot Interaction (HRI) requires a more complex "social affordance," where a robot must perceive action possibilities provided by another living agent. Social interactions are dynamic and context-dependent; for instance, a human hand only affords grabbing if the person intends to shake hands. Capturing these nuances from raw video is challenging due to noisy skeleton data and the difficulty of manually defining the temporal "grammar" of social gestures like high-fives.

Overview of social affordance learning. The model decomposes a 'Shake Hands' interaction into latent sub-events and identifies affordable joints.

Methodology

The proposed framework utilizes a hierarchical graphical model to represent social affordance, learned via a weakly supervised approach from RGB-D videos.

  1. Hierarchical Representation: Interactions are decomposed into latent sub-events (sub-goals). For each sub-event, the model identifies relevant "affordable" body joints and their required trajectories.
  2. Joint Selection and Grouping: To handle noise, the model uses a Chinese Restaurant Process (CRP) to automatically cluster relevant joints into functional groups while relegating irrelevant joints to a "Null" group.
  3. Learning via MCMC: A Markov Chain Monte Carlo (MCMC) algorithm is employed with two loops: an outer loop (Metropolis-Hastings) to discover temporal boundaries of sub-events, and an inner loop (Gibbs sampling) to optimize joint groupings.
  4. Motion Synthesis: For real-time interaction, the robot uses Dynamic Programming to infer the current sub-event based on human movement and synthesizes its response by maximizing learned spatial and motion potentials.

UCLA Human-Human-Object Interaction (HHOI) Dataset

To evaluate the model, the researchers introduced the UCLA HHOI dataset. It includes three human-human interactions (shake hands, high-five, pull up) and two human-object-human interactions (throw and catch, hand over a cup). The dataset consists of 23.6 instances per interaction performed by 8 actors, captured using Microsoft Kinect v2, providing a benchmark for multi-agent interactions.

Experimental Results

The proposed model achieved an average joint distance error of 0.073 meters, significantly outperforming a standard Hidden Markov Model (HMM) baseline (0.281m). Qualitative results from a user study showed that synthesized motions for "shake hands" and "throw and catch" were rated as natural as human-to-human ground truth.

Qualitative comparison of motion synthesis. Our model (middle row) closely matches ground truth (top row) and outperforms the HMM baseline (bottom row) across various social interactions.

The generative model effectively filters sensor jitter, sometimes producing motions rated more natural than the raw ground truth. Despite the complexity of MCMC, synthesis runs at approximately 5 fps in unoptimized code, indicating potential for real-time deployment.

Conclusion and Future Work

While successful, the current model assumes a humanoid morphology. Future work will focus on motion transfer—mapping learned social affordances onto robots with different physical structures and degrees of freedom—and testing the framework on physical robotic systems.

Summary updated

Abstract

In this paper, we present an approach for robot learning of social affordance from human activity videos. We consider the problem in the context of human-robot interaction: Our approach learns structural representations of human-human (and human-object-human) interactions, describing how body-parts of each agent move with respect to each other and what spatial relations they should maintain to complete each sub-event (i.e., sub-goal). This enables the robot to infer its own movement in reaction to the human body motion, allowing it to naturally replicate such interactions. We introduce the representation of social affordance and propose a generative model for its weakly supervised learning from human demonstration videos. Our approach discovers critical steps (i.e., latent sub-events) in an interaction and the typical motion associated with them, learning what body-parts should be involved and how. The experimental results demonstrate that our Markov Chain Monte Carlo (MCMC) based learning algorithm automatically discovers semantically meaningful interactive affordance from RGB-D videos, which allows us to generate appropriate full body motion for an agent.
BibTeX
@inproceedings{shu2016learning,
  author = {Tianmin Shu and Michael S. Ryoo and Song-Chun Zhu},
  title = {{Learning Social Affordance for Human-Robot Interaction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
  url = {https://openreview.net/forum?id=S14Ie7fdZB},
}
Figure