Nothing Stands Still: A Spatiotemporal Benchmark on 3D Point Cloud Registration Under Large Geometric and Temporal Change

Tao Sun, Yan Hao, Shengyu Huang, Silvio Savarese, Konrad Schindler, Marc Pollefeys, Iro Armeni

In the fields of computer vision and robotics, 3D maps are frequently treated as static snapshots of the world. However, the built environment is inherently dynamic: buildings undergo construction, renovation, and repurposing. A new research effort from Stanford, ETH Zurich, and Microsoft Research, titled "Nothing Stands Still (NSS)," introduces a benchmark designed to push 3D registration algorithms beyond the "static world" assumption and into the evolving reality of large-scale structural change.

The Challenge of Evolving Environments

3D point cloud registration—the process of aligning multiple 3D scans into a single coordinate system—is a foundational task for autonomous robots and digital twins. Most existing benchmarks, such as 3DMatch or 3RScan, focus on "low-dynamic" changes, such as moving furniture or opening doors.

A comparison of the NSS benchmark against existing datasets (Replica, ReScan, 3RScan), highlighting the increased scale and complexity of structural changes.

These datasets fail to capture radical geometric and topological changes. In a construction site, the very structure of the scene evolves. Current algorithms often rely on finding matching local geometric features; if those features disappear (e.g., a wall skeleton is covered by plasterboard), the algorithms fail. The NSS benchmark addresses the need for spatiotemporal maps that can link scans taken months apart, supporting the "circular economy" by tracking materials through a building's lifecycle.

The NSS Dataset and Methodology

The core of this research is the NSS Dataset, which features six large-scale indoor areas averaging 2,500 m² each. These areas were captured across two to six stages over several months during active construction. To overcome proprietary data limitations, the authors used high-fidelity synthetic fragments from real-world scans, incorporating realistic sensor noise models.

The NSS dataset captures large-scale environments across multiple temporal stages of construction, illustrating significant geometric evolution.

The researchers formalize two primary tasks:

  1. Pairwise Spatiotemporal Registration: Aligning two partial fragments captured at different times or from different views.
  2. Multi-way Registration: Aligning a collection of fragments from various stages into a coherent 4D (3D + time) map.

To quantify difficulty, they introduced the Temporal Change Ratio (TCR). Unlike the standard overlap ratio, TCR measures the percentage of shared space that has actually changed geometrically, providing a more rigorous test for registration robustness.

Key Insights and Performance

The study evaluated several state-of-the-art models, including Predator, GeoTransformer, and D3Feat. Several key findings emerged:

  • The "Attention" Advantage: Methods using cross-attention mechanisms (like Predator) performed significantly better. Attention allows models to ignore transient "noise" (like construction debris) and focus on stable structural anchors.
  • The Performance Cliff: There is a massive performance gap between "same-stage" and "different-stage" registration. Even top-tier models saw recall drop by 40-50% when bridging temporal gaps.
  • Generalization Bottlenecks: Models performed worst in "Cross-area" tests, suggesting that registration networks often over-fit to specific geometric styles, limiting their reliability for "in-the-wild" deployment in new construction sites.
  • Failure of Traditional Methods: Hand-crafted features like FPFH were almost entirely ineffective for temporal registration, with success rates near 1% for different-stage pairs.

Conclusion and Future Directions

The NSS benchmark reveals that while modern deep learning methods show promise, they are still highly sensitive to radical geometric changes. Future work must address initialization sensitivity in multi-way registration and explore the integration of semantic knowledge to help algorithms bridge temporal gaps more intelligently. By moving beyond the static assumption, the NSS benchmark sets a new standard for sustainable, long-term spatial computing.

Summary updated

Abstract

Building 3D geometric maps of man-made spaces is a well-established and active field that is fundamental to computer vision and robotics. However, considering the evolving nature of built environments, it is essential to question the capabilities of current mapping efforts in handling temporal changes. In addition, spatiotemporal mapping holds significant potential for achieving sustainability and circularity goals. Existing mapping approaches focus on small changes, such as object relocation or self-driving car operation; in all cases where the main structure of the scene remains fixed. Consequently, these approaches fail to address more radical changes in the structure of the built environment, such as geometry and topology. To this end, we introduce the Nothing Stands Still (NSS) benchmark, which focuses on the spatiotemporal registration of 3D scenes undergoing large spatial and temporal change, ultimately creating one coherent spatiotemporal map. Specifically, the benchmark involves registering two or more partial 3D point clouds (fragments) from the same scene but captured from different spatiotemporal views. In addition to the standard pairwise registration, we assess the multi-way registration of multiple fragments that belong to any temporal stage. As part of NSS, we introduce a dataset of 3D point clouds recurrently captured in large-scale building indoor environments that are under construction or renovation. The NSS benchmark presents three scenarios of increasing difficulty, to quantify the generalization ability of point cloud registration methods over space (within one building and across buildings) and time. We conduct extensive evaluations of state-of-the-art methods on NSS. The results demonstrate the necessity for novel methods specifically designed to handle large spatiotemporal changes. The homepage of our benchmark is at http://nothing-stands-still.com.
BibTeX
@misc{sun2023nothing,
  author = {Tao Sun and Yan Hao and Shengyu Huang and Silvio Savarese and Konrad Schindler and Marc Pollefeys and Iro Armeni},
  title = {{Nothing Stands Still: A Spatiotemporal Benchmark on 3D Point Cloud Registration Under Large Geometric and Temporal Change}},
  year = {2023},
  eprint = {2311.09346},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.09346},
}
Figure