Face Analysis and Body Language Understanding from Egocentric Cameras (short paper)

Ronja Möller, Antonino Furnari, Sebastiano Battiato, Aki Härmä, Giovanni Maria Farinella

Abstract

This paper presents a modular framework for a mobile assistive robot that utilizes egocentric (first-person) vision to analyze human behavior in real-time. By integrating deep reinforcement learning (DRL) for social navigation with specialized computer vision modules for face, body, and action recognition, the system aims to create a socially aware agent capable of supporting users in dynamic, unmapped home environments.

Introduction

While Human-Robot Interaction (HRI) has seen significant progress, many current systems are limited to constrained environments or abstract world representations. A critical gap exists in developing mobile agents that can function effectively in the messy, dynamic reality of a home. For a robot to be a truly useful assistant—particularly in healthcare or elderly care—it must be "socially aware." This requires the ability to interpret subtle social cues, such as facial expressions and body language, from its own moving perspective without obstructing the user's path or appearing intrusive.

System Architecture

The proposed system is built upon a modular software architecture deployed on a Segway Loomo robot. This modularity allows for the independent development and upgrading of specific components as computer vision and AI technologies evolve.

Overview of the proposed modular architecture for the Segway Loomo assistive robot, illustrating the central communication hub and specialized analysis modules.

The architecture consists of several semi-independent modules:

  • Intelligent Navigation: Uses DRL to achieve "social navigation," where the robot finds the optimal position to observe the user while maintaining a comfortable distance.
  • Face and Body Analysis: Extracts "soft biometrics" (age, gender, mood) and social cues (attention, emotional state) from the user’s physical posture and facial expressions.
  • Action Recognition: Infers current activities and predicts future actions based on joint positions and orientation.
  • Speech Analysis: Serves as the primary natural language interface between the user and the robot.
  • Communication Hub: Manages the data flow between the robot's sensors (fish-eye camera, Intel RealSense, IMUs) and an external PC used for heavy computational inference.

Key Insights: Navigation as Information Gathering

A core innovation of this framework is the conceptualization of navigation as a tool for information gathering. In traditional robotics, navigation is typically a means to reach a destination. In this context, the robot's movement is driven by the need to maximize the quality of visual data collected from the user. By utilizing an egocentric perspective, the mobile robot can capture nuances of interaction that static smart-home sensors might miss.

Challenges and Future Work

As a position paper, this work identifies several hurdles for future implementation:

  • Sim-to-Real Gap: Training DRL agents requires massive datasets, necessitating simulation platforms like AI Habitat or Gibson. However, transferring these learned behaviors to real-world home environments remains a significant challenge.
  • Computational and Privacy Constraints: The limited onboard processing of mobile robots like the Loomo requires external computation, which introduces latency. Furthermore, the collection of sensitive data necessitates robust privacy frameworks, potentially involving localized, personalized processing.

The immediate roadmap focuses on implementing the DRL-based navigation and core action recognition modules before integrating more nuanced social analysis features.

Summary updated

BibTeX
@inproceedings{moller2020face,
  author = {Ronja Möller and Antonino Furnari and Sebastiano Battiato and Aki Härmä and Giovanni Maria Farinella},
  title = {{Face Analysis and Body Language Understanding from Egocentric Cameras (short paper)}},
  booktitle = {SmartPhil@IUI},
  year = {2020},
}
Figure