Learning Social Affordance for Human-Robot Interaction

Tianmin Shu, Michael S. Ryoo, Song-Chun Zhu

Abstract

In this paper, we present an approach for robot learning of social affordance from human activity videos. We consider the problem in the context of human-robot interaction: Our approach learns structural representations of human-human (and human-object-human) interactions, describing how body-parts of each agent move with respect to each other and what spatial relations they should maintain to complete each sub-event (i.e., sub-goal). This enables the robot to infer its own movement in reaction to the human body motion, allowing it to naturally replicate such interactions. We introduce the representation of social affordance and propose a generative model for its weakly supervised learning from human demonstration videos. Our approach discovers critical steps (i.e., latent sub-events) in an interaction and the typical motion associated with them, learning what body-parts should be involved and how. The experimental results demonstrate that our Markov Chain Monte Carlo (MCMC) based learning algorithm automatically discovers semantically meaningful social affordance from RGB-D videos, which allows us to generate appropriate full body motion for an agent.

Introduction: Beyond Object Manipulation

In robotics, "affordance" traditionally refers to the action possibilities an object offers to an agent (e.g., a handle affords pulling). However, most research has treated this as a unary relationship between a single robot and an object. In real-world social settings, robots must interact with humans, necessitating a "social" affordance. This involves understanding that a human’s gesture, such as an extended hand, is not merely a physical coordinate but a social cue that affords a specific response (e.g., a handshake) at a precise moment.

Social affordance concept: predicting future interaction steps from a current frame.

Modeling social affordance presents three major challenges: the intricate temporal dynamics of human interactions, the high noise and occlusion levels in consumer-grade sensor data (e.g., Microsoft Kinect), and the hierarchical nature of interactions, which consist of multiple distinct sub-events.

Methodology: Hierarchical Generative Modeling

We propose a generative model represented as a factor graph to learn these social structures. The model decomposes an interaction into three hierarchical layers:

  1. Interaction Label: The high-level goal (e.g., "High-Five").
  2. Latent Sub-events: The chronological steps required to reach the goal.
  3. Joint Selection and Grouping: Identifying which body parts (joints) are relevant for a specific step.
The proposed factor graph for interaction parsing and joint selection.

Learning is performed using a Markov Chain Monte Carlo (MCMC) algorithm with two nested loops. The outer loop utilizes a Metropolis-Hastings algorithm to temporally parse the video into sub-events. The inner loop employs Gibbs sampling and a Chinese Restaurant Process (CRP) to automatically group joints that move together or maintain specific spatial relationships. For motion synthesis, Dynamic Programming is used to infer the current sub-event in real-time based on human movement, allowing the robot to sample the most likely next joint positions to complete the social sub-goal.

Dataset and Experimental Evaluation

To evaluate the approach, we introduced the UCLA Human-Human-Object Interaction (HHOI) dataset. This benchmark includes three human-human interactions (shake hands, high-five, pull up) and two human-object-human interactions (throw and catch, hand over a cup). The dataset comprises 23.6 instances per interaction performed by 8 different actors from various angles, recorded in RGB-D format with 3D skeleton estimates.

The model was evaluated on its ability to synthesize a robot's skeleton in response to a human. Quantitatively, our model achieved an average joint distance of 0.0730m, significantly outperforming a standard Hidden Markov Model (0.2816m) and baseline versions without joint grouping. Qualitatively, a user study with 14 subjects found that for interactions like shake hands and throw and catch, the synthesized motions were rated as more natural and human-like than the original noisy Kinect data recorded from real humans.

Key Insights and Discussion

  • Affordance as a Social Function: Affordance is not just about physical capability but about appropriate social response.
  • Joint Relevance and Grouping: By using the CRP, the model learns that not all joints are relevant at all times. For instance, during a handshake, the right hand is critical while the ankles may be assigned to a "Null" group.
  • Sub-goals over Raw Trajectories: Instead of simple mimicry, the robot learns "spatial potentials"—the relative distance and orientation body parts should maintain to achieve a sub-goal.
  • Robustness to Sensor Noise: Because the model learns the intent of the movement, it can effectively "hallucinate" correct joint positions even when physical sensors are confused by occlusions or overlapping limbs.

Limitations and Future Work

A primary limitation is the current humanoid assumption, which requires the robot to share a similar skeletal structure with humans. Future work will focus on motion transfer to non-humanoid robots with different mechanical constraints. Additionally, future iterations will aim to recognize the interaction category on the fly rather than assuming it is known a priori.

Summary updated

Abstract

In this paper, we present an approach for robot learning of social affordance from human activity videos. We consider the problem in the context of human-robot interaction: Our approach learns structural representations of human-human (and human-object-human) interactions, describing how body-parts of each agent move with respect to each other and what spatial relations they should maintain to complete each sub-event (i.e., sub-goal). This enables the robot to infer its own movement in reaction to the human body motion, allowing it to naturally replicate such interactions. We introduce the representation of social affordance and propose a generative model for its weakly supervised learning from human demonstration videos. Our approach discovers critical steps (i.e., latent sub-events) in an interaction and the typical motion associated with them, learning what body-parts should be involved and how. The experimental results demonstrate that our Markov Chain Monte Carlo (MCMC) based learning algorithm automatically discovers semantically meaningful social affordance from RGB-D videos, which allows us to generate appropriate full body motion for an agent.
BibTeX
@inproceedings{shu2016learning,
  author = {Tianmin Shu and Michael S. Ryoo and Song-Chun Zhu},
  title = {{Learning Social Affordance for Human-Robot Interaction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
  url = {https://openreview.net/forum?id=S14Ie7fdZB},
}
Figure