Semantic Filtering for Dynamic SLAM

18 concepts12 papers1 proposal

Simultaneous Localization and Mapping (SLAM) algorithms traditionally operate under a convenient but flawed assumption: the world is static. Classical visual and LiDAR SLAM frameworks track geometric primitives—such as corners, lines, or point clouds—and assume that any change in their relative position is caused by the movement of the robot itself.

When a robot operates in the real world, this assumption breaks down. If a vehicle drives past a camera, or a pedestrian walks across a LiDAR sensor's field of view, a purely geometric SLAM system faces a dilemma. It may attempt to align its current frame with these moving points, mistakenly concluding that the robot is drifting or turning. This geometric blindspot leads to tracking failures, accumulated localization errors, and maps corrupted by "ghosting" artifacts—smear marks left behind by transient objects.

To solve this, a major research direction has emerged: Semantic Filtering. By fusing deep learning-based semantic segmentation with traditional geometric pipelines, modern SLAM systems can understand what they are seeing before deciding where they are.


The Intuition: Semantic Priors as a Filter

The core insight behind semantic filtering is that geometry alone cannot reliably distinguish between a permanent structure (like a wall) and a temporarily stationary object (like a parked car). However, a semantic label provides an immediate a priori cue about an object's likelihood of movement.

As noted in a comprehensive survey by Garg et al., moving from purely metric maps to "Spatial AI" allows robots to utilize the meaning of their surroundings to filter out transient features. If a neural network flags a region of an image or a cluster of points as a "pedestrian," "vehicle," or "cyclist," the SLAM system can proactively down-weight or entirely discard those features during pose estimation, even if the object is currently motionless.

[Input Frame] ---> [Semantic Segmentation Network] ---> [Identify Dynamic Classes]
                                                                   |
                                                                   v
[Geometric Features] ------------------------------------> [Filter / De-weight]
                                                                   |
                                                                   v
                                                      [Robust Tracking & Mapping]

Without this capability, relocalization frameworks frequently fail in crowded or changing environments. Xue et al. observe that the absence of semantic understanding causes systems to mistake moving entities for stable landmarks, corrupting the global pose graph. Purely geometric approaches lack the contextual awareness required to make this distinction (Lv et al.).


How It Works: Front-End and Back-End Integration

Semantic information is typically injected into two distinct stages of the SLAM pipeline: real-time tracking (the front-end) and long-term map optimization (the back-end).

1. Front-End Feature Masking

In the front-end, semantic segmentation acts as a gatekeeper for feature selection. For example, Kang et al. integrate a U-Net-based segmentation module into an indoor visual SLAM system to identify and mask out dynamic objects in real time. By preventing the system from extracting keypoints on moving people or chairs, the frame-to-frame tracking relies solely on rigid structures.

Similarly, Ruicheng Gao demonstrates that combining semantic cues with depth estimation allows a monocular system to filter unreliable features from dynamic objects and apply object-level constraints, stabilizing localization in highly unpredictable environments.

2. Back-End Map Cleaning

In the back-end, semantic filtering prevents transient data from contaminating the permanent environmental model. When a LiDAR or camera sweeps an area containing moving traffic, a standard mapping pipeline integrates those points into the final map, creating "ghost" geometry that does not exist in reality.

Wang et al. address this in LiDAR-inertial odometry by using semantic labels to eliminate these artifacts, yielding cleaner, structurally accurate maps. For visual systems, Shen et al. filter out dynamic semantic classes during the construction of sparse 3D reference maps. This map-cleaning process ensures that when a robot returns to the same location hours or days later, the reference map contains only permanent structures, preventing transient obstacles from introducing localization drift (Huang et al.).


Generalizing Beyond Urban Environments

While vehicles and pedestrians are the most common targets for semantic filtering, the underlying mechanism generalizes across diverse and specialized domains:

  • Marine Environments: In underwater robotic reef surveys, Sauder et al. use semantic masks to identify and de-weight transient objects like fish or divers. By forcing the 3D reconstruction algorithm to focus exclusively on static substrate classes (such as coral or rock), the mapping process becomes robust to marine life movement.
  • Visually-Degraded Spaces: In dark or smoke-filled environments where distinct geometric shapes are sparse, the authors of SIGNAV use semantic filtering to remove non-rigid objects and non-Lambertian (highly reflective or translucent) surfaces. This focuses the SLAM optimization strictly on rigid, reliable features, significantly improving loop-closure accuracy.

Why the Combined Approach Succeeds

As detailed in comprehensive reviews by Chen et al. and Liu et al., the paradigm shift from purely geometric SLAM to semantically-informed SLAM succeeds because it bridges the gap between low-level sensing and high-level understanding. Geometry provides the precise mathematical constraints needed to calculate exact coordinates, while semantics provides the contextual veto power needed to validate those constraints. By filtering out the noise of a changing world, semantic SLAM achieves the long-term stability required for truly autonomous navigation.

Key Foundational Work

Semantics for Robotic Mapping, Perception and Interaction: A Survey (2021) — contributed 2 concept elements to this direction.
Visual Localization Using Sparse Semantic 3D Map (2019) — contributed 2 concept elements to this direction.
LIO-CSI: LiDAR inertial odometry with loop closure combined with semantic information (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dynamic_static_feature_filtering_2fa238c8,
  title = {dynamic-static feature filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-static-feature-filtering/rdr_2fa238c88aee4a6d8b388476d7b97154},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_dynamic_static_feature_filtering_2fa238c8_v1,
  title = {dynamic-static feature filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-static-feature-filtering/rdr_2fa238c88aee4a6d8b388476d7b97154/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake