Abstract
For robots to navigate and interact more richly with the world around them, they will likely require a deeper understanding of the world in which they operate. In robotics and related research fields, the study of understanding is often referred to as semantics, which dictates what does the world "mean" to a robot, and is strongly tied to the question of how to represent that meaning. With humans and robots increasingly operating in the same world, the prospects of human-robot interaction also bring semantics and ontology of natural language into the picture. Driven by need, as well as by enablers like increasing availability of training data and computational resources, semantics is a rapidly growing research area in robotics. This survey provides an overarching snapshot of where semantics in robotics stands today. We establish a taxonomy for semantics research in or relevant to robotics, split into four broad categories of activity, in which semantics are extracted, used, or both. Within these broad categories we survey dozens of major topics including fundamentals from the computer vision field and key robotics research areas utilizing semantics, including mapping, navigation and interaction with the world.
Introduction: Bridging the Semantic Gap
For years, mobile robots have been excellent at geometry but poor at "meaning." A standard robot can build a precise 3D map of a room to avoid hitting a wall, but it often has no idea that the wall belongs to a "kitchen" or that the cylinder on the floor is a "full coffee cup" that shouldn't be knocked over. This discrepancy is known as the semantic gap.
As robots move out of controlled factory floors and into cluttered, human-centric environments like hospitals, homes, and city streets, they need more than just coordinates. They require semantics: the ability to categorize places, recognize objects, understand their "affordances" (what can be done with them), and communicate using natural language. This survey synthesizes over 900 research works to provide a unified taxonomy for this intersection of computer vision and physical robotics.
A Taxonomy for Semantics in Robotics
The vast landscape of semantic robotics can be organized into a four-category taxonomy based on whether semantic information is being extracted from the environment, used to perform a task, or both.
1. Static and Un-embodied Scene Understanding
This covers the fundamentals of computer vision—image classification, object detection, and dense pixel-wise segmentation. It focuses on how meaning is extracted from static images or videos, often serving as the prerequisite for more complex robotic behaviors.
2. Dynamic Environment Understanding and Mapping
This bridges semantics with Simultaneous Localization and Mapping (SLAM). It explores how robots build 3D semantic maps and use Scene Graphs to represent hierarchical relationships (e.g., the plate is on the table in the dining room). This allows robots to move beyond simple point clouds to structured, logical representations of space.
3. Interacting with Humans and the World
This moves from "seeing" to "doing." Key concepts include Object Affordances (recognizing that a handle is for "grasping") and Vision-and-Language Navigation (following instructions like "bring me the mug from the kitchen"). Here, semantics are used to facilitate physical interaction and human-robot collaboration.
4. Improving Task Capability
Semantics can significantly enhance core robotic tasks. For example, using recognized landmarks to improve localization in extreme weather or changing light conditions where traditional visual features fail. By understanding which objects are dynamic (cars) versus static (buildings), robots can build more robust and reliable maps.
Key Insights and Practical Enablers
From Labels to Affordances
For a robot, knowing an object is a "pitcher" is less useful than knowing it is "pourable," "heavy," and "graspable." The survey emphasizes that semantic perception must be task-driven. This shift is supported by the rise of Spatial AI, where the co-design of algorithms and specialized hardware (like FPGAs) allows for real-time semantic processing on power-limited mobile platforms.
The Role of Datasets
Modern deep learning relies on benchmark datasets. The field is currently moving from 2D image datasets (like NYUv2) toward large-scale 3D and multi-modal datasets (like ScanNet and Matterport3D) that incorporate thermal, infrared, and depth data, providing the rich information needed for holistic environment learning.
Challenges and Future Directions
While real-time performance is becoming viable—with systems demonstrating dense 3D semantic mapping at 25Hz—several hurdles remain:
- Generality: Most systems are "overfit" to specific domains like self-driving. Creating a general-purpose semantic engine remains an open challenge.
- The Trust Gap: As robots learn concepts via deep learning, their logic might not align with human reasoning, creating issues for explainability and safety.
- Active Semantics: Future work must move toward "active" perception, where a robot intentionally moves its sensors to resolve semantic ambiguities (e.g., distinguishing a person from a mannequin).
- Closed-Loop Reliability: There remains a disconnect between achieving high accuracy on a static dataset and achieving high reliability on a physical robot interacting with a dynamic, unpredictable world.
Summary updated