Abstract
In the EU-funded MuMMER project, we have developed a social robot designed to interact naturally and flexibly with users in public spaces such as a shopping mall. We present the latest version of the robot system developed during the project. This system encompasses audio-visual sensing, social signal processing, conversational interaction, perspective taking, geometric reasoning, and motion planning. It successfully combines all these components in an overarching framework using the Robot Operating System (ROS) and has been deployed to a shopping mall in Finland interacting with customers. In this paper, we describe the system components, their interplay, and the resulting robot behaviours and scenarios provided at the shopping mall.
Introduction
Most robots excel in controlled lab environments or at specific, repetitive tasks. However, "the wild"—specifically public spaces like shopping malls—presents a chaotic mix of noise, crowds, and social expectations.
A robot in a mall shouldn't just be a mobile kiosk; it needs to understand social context. This includes identifying who wants to talk (vs. who is just walking by), maintaining a conversation in a noisy environment, and navigating through crowds without being intrusive. The MuMMER (Multi-modal Mall Entertainment Robot) project tackles the challenge of building a fully integrated system that combines high-level reasoning with low-level sensing to provide a natural, engaging user experience.
System Architecture and Methods
The researchers built a complex, modular architecture using the Robot Operating System (ROS), centered on a SoftBank Pepper robot customized with an Intel D435 camera and an NVIDIA Jetson TX2.
Audio-Visual Sensing and Social State Estimation
The system uses Convolutional Pose Machines (CPM) for body joint tracking and OpenHeadPose to estimate where a person is looking. Sound localization is handled by a multi-task neural network that filters speech from background mall noise using a 4-channel microphone array. Instead of just interacting with the closest person, the robot calculates an "attention probability" based on head pose, distance, and interaction history to decide who to engage with.
Conversational AI and Shared Visual Perspective (SVP)
The system uses the Alana framework to combine social dialogue with task-oriented "recipes" using a custom NLU module called HERMIT. A standout feature is the Shared Visual Perspective (SVP). The robot uses geometric reasoning to calculate "visibility grids." If a user asks for directions to a shop that is currently blocked by a pillar from their perspective, the robot will actually navigate to a better vantage point before pointing.
Social Motion Planning
Navigation is managed by Social TEB (Timed Elastic Band), a planner that respects social constraints, ensuring the robot doesn't get too close to people or cut them off during movement.
Key Insights and Results
- Perspective-Taking is Critical: The SVP planner prevents the "pointing at a wall" problem common in simpler systems by ensuring the robot understands what the human can see.
- Fusing Social and Task Dialogue: By wrapping task-specific bots inside a social chatbot, the robot can handle interruptions or "small talk" during a task without losing its place.
- Real-World Performance: The system was successfully deployed for a three-month long-term deployment at the Ideapark mall in Finland. The perception pipeline delivers output at 10 fps, which is sufficient for real-time social interaction.
- Multi-modal Success: The robot successfully performed complex "Dialogue + Pointing" scenarios, where it navigated to a location, checked the human's visibility, and then explained the route while pointing at landmarks.
Limitations and Future Work
The authors acknowledge a "lag" between human speech and the robot's reply, partly due to heavy deep learning computation being offloaded to a remote laptop. Future iterations aim to move more processing directly onto the robot. Additionally, while the system supports both English and Finnish, the Finnish social interaction was less flexible due to the complexities of NLU translation for social chit-chat.
Summary updated