MuMMER: Socially Intelligent Human-Robot Interaction in Public Spaces

Mary Ellen Foster, Bart Craenen, Amol Deshmukh, Oliver Lemon, Elena Bastianelli, Christian Dondrup, Ioannis Papaioannou, Andrea Vanzo, Jean‐Marc Odobez, Olivier Canévet, Yuanzhouhan Cao, Weipeng He, Ángel Martínez-González, Petr Motlícek, Rémy Siegfried, Rachid Alami, Kathleen Belhassein, Guilhem Buisan, Aurélie Clodic, Amandine Mayima, Yoan Sallami, Guillaume Sarthou, Phani Teja Singamaneni, Jules Waldhart, Alexandre Mazel, Maxime Caniot, Marketta Niemelä, Päivi Heikkilä, Hanna Lammi, Antti TammelaENLIGHTEN 2019 · OpenAlex

Abstract

In the EU-funded MuMMER project, we have developed a social robot designed to interact naturally and flexibly with users in public spaces such as a shopping mall. We present the latest version of the robot system developed during the project. This system encompasses audio-visual sensing, social signal processing, conversational interaction, perspective taking, geometric reasoning, and motion planning. It successfully combines all these components in an overarching framework using the Robot Operating System (ROS) and has been deployed to a shopping mall in Finland interacting with customers. In this paper, we describe the system components, their interplay, and the resulting robot behaviours and scenarios provided at the shopping mall.

Introduction

Most robots excel in controlled lab environments or at specific, repetitive tasks. However, "the wild"—specifically public spaces like shopping malls—presents a chaotic mix of noise, crowds, and social expectations.

A robot in a mall shouldn't just be a mobile kiosk; it needs to understand social context. This includes identifying who wants to talk (vs. who is just walking by), maintaining a conversation in a noisy environment, and navigating through crowds without being intrusive. The MuMMER (Multi-modal Mall Entertainment Robot) project tackles the challenge of building a fully integrated system that combines high-level reasoning with low-level sensing to provide a natural, engaging user experience.

The MuMMER robot interacting with a customer at the Ideapark shopping mall in Finland.

System Architecture and Methods

The researchers built a complex, modular architecture using the Robot Operating System (ROS), centered on a SoftBank Pepper robot customized with an Intel D435 camera and an NVIDIA Jetson TX2.

Audio-Visual Sensing and Social State Estimation

The system uses Convolutional Pose Machines (CPM) for body joint tracking and OpenHeadPose to estimate where a person is looking. Sound localization is handled by a multi-task neural network that filters speech from background mall noise using a 4-channel microphone array. Instead of just interacting with the closest person, the robot calculates an "attention probability" based on head pose, distance, and interaction history to decide who to engage with.

Visual tracking and voice localization system estimating user attention (VFOA) and sound direction.

Conversational AI and Shared Visual Perspective (SVP)

The system uses the Alana framework to combine social dialogue with task-oriented "recipes" using a custom NLU module called HERMIT. A standout feature is the Shared Visual Perspective (SVP). The robot uses geometric reasoning to calculate "visibility grids." If a user asks for directions to a shop that is currently blocked by a pillar from their perspective, the robot will actually navigate to a better vantage point before pointing.

Social Motion Planning

Navigation is managed by Social TEB (Timed Elastic Band), a planner that respects social constraints, ensuring the robot doesn't get too close to people or cut them off during movement.

Key Insights and Results

  • Perspective-Taking is Critical: The SVP planner prevents the "pointing at a wall" problem common in simpler systems by ensuring the robot understands what the human can see.
  • Fusing Social and Task Dialogue: By wrapping task-specific bots inside a social chatbot, the robot can handle interruptions or "small talk" during a task without losing its place.
  • Real-World Performance: The system was successfully deployed for a three-month long-term deployment at the Ideapark mall in Finland. The perception pipeline delivers output at 10 fps, which is sufficient for real-time social interaction.
  • Multi-modal Success: The robot successfully performed complex "Dialogue + Pointing" scenarios, where it navigated to a location, checked the human's visibility, and then explained the route while pointing at landmarks.

Limitations and Future Work

The authors acknowledge a "lag" between human speech and the robot's reply, partly due to heavy deep learning computation being offloaded to a remote laptop. Future iterations aim to move more processing directly onto the robot. Additionally, while the system supports both English and Finnish, the Finnish social interaction was less flexible due to the complexities of NLU translation for social chit-chat.

Summary updated

Abstract

In the EU-funded MuMMER project, we have developed a social robot designed to interact naturally and flexibly with users in public spaces such as a shopping mall. We present the latest version of the robot system developed during the project. This system encompasses audio-visual sensing, social signal processing, conversational interaction, perspective taking, geometric reasoning, and motion planning. It successfully combines all these components in an overarching framework using the Robot Operating System (ROS) and has been deployed to a shopping mall in Finland interacting with customers. In this paper, we describe the system components, their interplay, and the resulting robot behaviours and scenarios provided at the shopping mall.
BibTeX
@inproceedings{foster2019mummer,
  author = {Mary Ellen Foster and Bart Craenen and Amol Deshmukh and Oliver Lemon and Elena Bastianelli and Christian Dondrup and Ioannis Papaioannou and Andrea Vanzo and Jean‐Marc Odobez and Olivier Canévet and Yuanzhouhan Cao and Weipeng He and Ángel Martínez-González and Petr Motlícek and Rémy Siegfried and Rachid Alami and Kathleen Belhassein and Guilhem Buisan and Aurélie Clodic and Amandine Mayima and Yoan Sallami and Guillaume Sarthou and Phani Teja Singamaneni and Jules Waldhart and Alexandre Mazel and Maxime Caniot and Marketta Niemelä and Päivi Heikkilä and Hanna Lammi and Antti Tammela},
  title = {{MuMMER: Socially Intelligent Human-Robot Interaction in Public Spaces}},
  booktitle = {ENLIGHTEN},
  year = {2019},
}
Figure