Semantics for Robotic Mapping, Perception and Interaction: A Survey

Sourav Garg, Niko Sünderhauf, Feras Dayoub, Douglas Morrison, Akansel Cosgun, Gustavo Carneiro, Qi Wu, Tat-Jun Chin, Ian D. Reid, Stephen Gould, Peter Corke, Michael Milford

TL;DR

This comprehensive survey provides a unifying roadmap for how robots move beyond simple geometry to understand the "meaning" of their world. By synthesizing over 900 research works, the authors categorize a decade of progress in semantic mapping, perception, and interaction, highlighting the shift from purely metric maps to "Spatial AI" that understands objects, relationships, and human intent.

What Problem Does This Paper Address?

For years, mobile robots have been excellent at geometry but poor at "meaning." A standard robot can build a precise 3D map of a room to avoid hitting a wall, but it often has no idea that the wall belongs to a "kitchen" or that the cylinder on the floor is a "full coffee cup" that shouldn't be knocked over. This discrepancy is known as the semantic gap.

As robots move out of controlled factory floors and into cluttered, human-centric environments (like hospitals, homes, and city streets), they need more than just coordinates. They need semantics: the ability to categorize places, recognize objects, understand their "affordances" (what can be done with them), and communicate using natural language. This paper addresses the fragmented nature of this research, providing a unified taxonomy to help researchers navigate the intersection of computer vision and physical robotics.

Methods

The authors organize the vast landscape of semantic robotics into a four-category taxonomy:

  1. Static and Un-embodied Scene Understanding: This covers the "Computer Vision 101" fundamentals—image classification, object detection, and dense pixel-wise segmentation. It focuses on how we extract meaning from static images or videos before a robot even starts moving.
  2. Dynamic Environment Understanding and Mapping: This bridges semantics with SLAM (Simultaneous Localization and Mapping). It explores how robots build 3D semantic maps, use Scene Graphs to represent hierarchical relationships (e.g., the plate is on the table in the dining room), and process 3D point clouds.
  3. Interacting with Humans and the World: This moves from "seeing" to "doing." Key concepts include Object Affordances (recognizing that a handle is for "grasping") and Vision-and-Language Navigation (following instructions like "bring me the mug from the kitchen").
  4. Improving Task Capability: This section looks at how semantics can actually make core robotic tasks better—for example, using recognized landmarks to improve localization in extreme weather or changing light conditions where traditional visual features fail.

Datasets

The paper provides a vital reference for benchmark datasets (summarized in Table I), which are the lifeblood of modern deep learning. Notable mentions include:

  • NYUv2 & SUN RGB-D: Standard for indoor scene understanding.
  • Cityscapes & A2D2: Massive datasets for autonomous driving and urban semantic segmentation.
  • ScanNet & SceneNN: Focused on 3D meshes and indoor reconstructions.
  • Matterport3D: Large-scale RGB-D data for holistic indoor environment learning.

The authors note a significant trend: the field is moving from 2D image datasets toward large-scale 3D and multi-modal datasets (incorporating thermal, infrared, and depth data).

Key Insights and Ideas

  • From Labels to Affordances: For a robot, knowing an object is a "pitcher" is less useful than knowing it is "pourable," "heavy," and "graspable." The survey emphasizes that semantic perception must be task-driven.
  • Semantics as a SLAM Booster: Semantics aren't just an output of mapping; they can be an input. By recognizing which objects are dynamic (like cars or pedestrians) and which are static (like buildings), robots can build much more robust maps.
  • The Rise of Scene Graphs: Instead of a "flat" list of objects, researchers are increasingly using graph-based representations to capture the topology and logic of a space, which is essential for high-level reasoning.
  • Spatial AI: The authors highlight the emergence of "Spatial AI," where the co-design of algorithms and specialized hardware (like FPSP chips or FPGAs) allows for real-time semantic processing on power-limited drones and mobile platforms.

Key Results

While a survey paper doesn't present new experimental data, it highlights major milestones in the field:

  • Real-time Performance: Systems like SemanticFusion have demonstrated that we can now perform dense 3D semantic mapping at 25Hz, making it viable for real-world robotic deployment.
  • Robustness: Semantic-based localization (VPR) has shown the ability to match locations across extreme "appearance changes," such as identifying the same street corner in a summer day versus a snowy night.
  • Zero-Shot Learning: Recent progress allows robots to categorize "unseen" objects by mapping visual features to semantic word embeddings, a crucial step for robots operating in "the wild."

Limitations and Future Work

The authors are candid about the hurdles remaining:

  • Generality vs. Specificity: Most semantic systems are "overfit" to specific domains (like self-driving). Creating a "general-purpose" semantic engine for robotics remains an open challenge.
  • The Trust Gap: As robots learn their own semantic concepts via deep learning, those concepts might not align with human logic. This creates issues for explainability and safety.
  • Active Semantics: Most current systems are passive. Future work needs to focus on "active" perception—where a robot moves its camera specifically to resolve a semantic ambiguity (e.g., "Is that a person or a mannequin?").
  • Closed-Loop Performance: There is still a disconnect between achieving high accuracy on a dataset and achieving high reliability on a physical robot interacting with a dynamic world.

Summary updated

Abstract

For robots to navigate and interact more richly with the world around them, they will likely require a deeper understanding of the world in which they operate. In robotics and related research fields, the study of understanding is often referred to as semantics, which dictates what does the world "mean" to a robot, and is strongly tied to the question of how to represent that meaning. With humans and robots increasingly operating in the same world, the prospects of human-robot interaction also bring semantics and ontology of natural language into the picture. Driven by need, as well as by enablers like increasing availability of training data and computational resources, semantics is a rapidly growing research area in robotics. The field has received significant attention in the research literature to date, but most reviews and surveys have focused on particular aspects of the topic: the technical research issues regarding its use in specific robotic topics like mapping or segmentation, or its relevance to one particular application domain like autonomous driving. A new treatment is therefore required, and is also timely because so much relevant research has occurred since many of the key surveys were published. This survey therefore provides an overarching snapshot of where semantics in robotics stands today. We establish a taxonomy for semantics research in or relevant to robotics, split into four broad categories of activity, in which semantics are extracted, used, or both. Within these broad categories we survey dozens of major topics including fundamentals from the computer vision field and key robotics research areas utilizing semantics, including mapping, navigation and interaction with the world. The survey also covers key practical considerations, including enablers like increased data availability and improved computational hardware, and major application areas where...
BibTeX
@inproceedings{garg2020semantics,
  author = {Sourav Garg and Niko Sünderhauf and Feras Dayoub and Douglas Morrison and Akansel Cosgun and Gustavo Carneiro and Qi Wu and Tat-Jun Chin and Ian D. Reid and Stephen Gould and Peter Corke and Michael Milford},
  title = {{Semantics for Robotic Mapping, Perception and Interaction: A Survey}},
  booktitle = {Found. Trends Robotics},
  year = {2020},
  url = {https://openreview.net/forum?id=NyVX2vRRkY6},
}

Loading related content...

Figure