Khronos: A Unified Approach for Spatio-Temporal Metric-Semantic SLAM in Dynamic Environments

Lukas Schmid, Marcus Abate, Yun Chang, Luca Carlone

Abstract

Perceiving and understanding highly dynamic and changing environments is a crucial capability for robot autonomy. While large strides have been made towards developing dynamic SLAM approaches that estimate the robot pose accurately, a lesser emphasis has been put on the construction of dense spatio-temporal representations of the robot environment. A detailed understanding of the scene and its evolution through time is crucial for long-term robot autonomy and essential to tasks that require long-term reasoning, such as operating effectively in environments shared with humans and other agents and thus are subject to short and long-term dynamics. To address this challenge, this work defines the Spatio-temporal Metric-semantic SLAM (SMS) problem, and presents a framework to factorize and solve it efficiently. We show that the proposed factorization suggests a natural organization of a spatio-temporal perception system, where a fast process tracks short-term dynamics in an active temporal window, while a slower process reasons over long-term changes in the environment using a factor graph formulation. We provide an efficient implementation of the proposed spatio-temporal perception approach, that we call Khronos, and show that it unifies exiting interpretations of short-term and long-term dynamics and is able to construct a dense spatio-temporal map in real-time. We provide simulated and real results, showing that the spatio-temporal maps built by Khronos are an accurate reflection of a 3D scene over time and that Khronos outperforms baselines across multiple metrics. We further validate our approach on two heterogeneous robots in challenging, large-scale real-world environments.

Introduction

Most contemporary SLAM systems operate under a "static world" assumption, treating dynamic elements as noise to be filtered out. When dynamics are addressed, they are typically bifurcated into two disconnected categories: short-term dynamics (e.g., tracking a walking person) and long-term changes (e.g., detecting moved furniture between mapping sessions). This fragmentation prevents robots from achieving a continuous, dense understanding of how human-centric spaces evolve over time.

Khronos addresses this gap by formalizing the Spatio-temporal Metric-semantic SLAM (SMS) problem. It provides a unified framework that treats all dynamics—from fast-moving agents to slow environmental shifts—within a single, consistent 4D representation. This allows robots to reason not just about where objects are, but where they were and when they moved.

The Khronos Framework

Khronos achieves real-time performance through a mathematical factorization based on spatio-temporal local consistency. The system is divided into a high-speed frontend and a robust backend.

Object Fragments and the Active Window

Instead of tracking an object's entire trajectory as a single global variable, Khronos breaks observations into "object fragments." These are short sequences of locally consistent observations that decouple local sensing noise from global estimation drift. The frontend "Active Window" performs TSDF (Truncated Signed Distance Function) fusion for the background while using greedy volumetric IoU (Intersection over Union) to track these object fragments in real-time.

Khronos tracks short-term dynamics, such as a person walking, by factorizing observations into spatio-temporal object fragments.

Global Optimization and Evidence of Absence

To maintain global consistency, Khronos employs a deformation graph that ties together robot poses, background mesh points, and fragment poses. This graph is optimized using Graduated Non-Convexity (GNC) to handle loop closures robustly.

A key innovation in Khronos is the "Evidence of Absence" mechanism. While standard maps only record occupied space, Khronos maintains a "library of rays" representing the history of free space. By ray-tracing through the current map, the system can prove an object has been removed if the robot can now see "through" the space it previously occupied. This allows the system to reconcile the map and update the status of objects that have moved or disappeared.

The system reasons over long-term changes by maintaining a history of free space to detect the removal or movement of objects, such as a cooler being relocated.

Experimental Evaluation

The researchers evaluated Khronos using the TESSE simulation engine and real-world hardware, including a Clearpath Jackal and a Boston Dynamics Spot quadruped.

In simulated benchmarks (Apartment and Office), Khronos significantly outperformed state-of-the-art baselines like Hydra and Panoptic Mapping. In the Office scenario, which featured significant odometry drift, Khronos achieved a 62.0% F1 score for change detection, nearly doubling the 38.6% achieved by Panoptic Mapping. The system maintained a processing speed of approximately 22 FPS in the Active Window, demonstrating its suitability for real-time mobile robot deployment.

Conclusion and Future Work

Khronos demonstrates that short-term and long-term dynamics are part of the same mathematical spectrum. By treating time as a core dimension, the system builds dense 4D maps that are essential for long-term robot autonomy in shared human spaces. Future work will focus on improving fragment association through 6DOF registration and implementing memory management strategies for lifelong mapping applications.

Summary updated

Abstract

Perceiving and understanding highly dynamic and changing environments is a crucial capability for robot autonomy. While large strides have been made towards developing dynamic SLAM approaches that estimate the robot pose accurately, a lesser emphasis has been put on the construction of dense spatio-temporal representations of the robot environment. A detailed understanding of the scene and its evolution through time is crucial for long-term robot autonomy and essential to tasks that require long-term reasoning, such as operating effectively in environments shared with humans and other agents and thus are subject to short and long-term dynamics. To address this challenge, this work defines the Spatio-temporal Metric-semantic SLAM (SMS) problem, and presents a framework to factorize and solve it efficiently. We show that the proposed factorization suggests a natural organization of a spatio-temporal perception system, where a fast process tracks short-term dynamics in an active temporal window, while a slower process reasons over long-term changes in the environment using a factor graph formulation. We provide an efficient implementation of the proposed spatio-temporal perception approach, that we call Khronos, and show that it unifies exiting interpretations of short-term and long-term dynamics and is able to construct a dense spatio-temporal map in real-time. We provide simulated and real results, showing that the spatio-temporal maps built by Khronos are an accurate reflection of a 3D scene over time and that Khronos outperforms baselines across multiple metrics. We further validate our approach on two heterogeneous robots in challenging, large-scale real-world environments.
BibTeX
@inproceedings{schmid2024khronos,
  author = {Lukas Schmid and Marcus Abate and Yun Chang and Luca Carlone},
  title = {{Khronos: A Unified Approach for Spatio-Temporal Metric-Semantic SLAM in Dynamic Environments}},
  booktitle = {Robotics: Science and Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=JmbgbhVrdi},
}
Figure