Traditional 3D computer vision often operates in "batch mode": a robot or drone scans an entire room, the data is uploaded to a powerful server, and minutes later, a complete 3D mesh or semantic map is produced. While accurate, this workflow is a non-starter for embodied agents. A robot cannot wait for a post-processing phase to decide if the object in front of it is a chair or a cliff.
A growing body of research is shifting toward online incremental scene understanding. The goal is to build a rich, semantically labeled 3D world model frame-by-frame, as the sensor moves, allowing for immediate interaction with unknown environments.
The Shift from Static Scans to Streaming Representations
The fundamental challenge of online understanding is that the system only sees a small slice of the world at any given moment. To make sense of this stream, researchers are moving away from heavy, global optimizations toward architectures that can update their internal "memory" of a scene on the fly.
Xiuwei Xu and colleagues address this by using memory-based adapters. In their work on Memory-based Adapters for Online 3D Scene Perception, they demonstrate that instead of retraining models for online use, one can insert "plug-and-play" memory modules that cache and aggregate features from streaming RGB-D video. This allows a robot to maintain a consistent understanding of an object even as it moves out of view and back again.
Similarly, the MCPNet framework by Yong K. Cho and Zsolt Kira uses "Multi-view Context Pooling" to integrate information from previous viewpoints. This incremental approach achieves accuracy within 6% of offline batch processing while running at a real-time 30Hz, proving that "incomplete" geometry doesn't have to mean "inaccurate" labeling.
Lifting 2D Foundation Models into 3D Space
A key catalyst for this field is the emergence of 2D Vision-Language Models (VLMs) like SAM (Segment Anything Model) and CLIP. These models have vast "open-vocabulary" knowledge but no inherent sense of 3D space. The research direction now focuses on "lifting" these 2D priors into 3D queries.
EmbodiedSAM, developed by Xiuwei Xu and Ziwei Wang, achieves this by transforming 2D masks into learnable 3D-aware queries. Instead of re-segmenting the whole scene every frame, the system merges new 2D observations into existing 3D instances. This "lift-and-merge" strategy is significantly faster—up to 20x—than traditional methods, enabling robots to identify novel objects (like a specific toy or a rare tool) in real-time.
Jiazhao Zhang and Yijie Tang take this further with OnlineAnySeg, which uses an efficient voxel-hashing data structure to merge 2D masks into a consistent 3D world model at 15 FPS. This allows for "zero-shot" segmentation, where the robot can understand objects it has never seen in its training data by leveraging the broad knowledge of the underlying 2D foundation models.
From Points to Graphs: Structured Scene Understanding
For a robot to follow a command like "go to the fridge in the kitchen," it needs more than just a cloud of points; it needs to understand relationships. This has led to the development of Incremental 3D Semantic Scene Graphs (SSGs).
Shun-Cheng Wu and Nassir Navab have pioneered methods like SceneGraphFusion and Incremental 3D Semantic Scene Graph Prediction. These systems use Graph Neural Networks (GNNs) to predict relationships (e.g., "cup on table") as the robot explores. By using confidence-based fusion, the graph is refined over time, becoming more stable as more viewpoints are captured.
The recent FROSS framework by Chun-Yi Lee pushes this to the extreme, achieving 144 FPS by replacing heavy point cloud processing with 3D Gaussian Splatting primitives.
Unifying Mapping and Understanding with Gaussian Splatting
The most recent breakthrough in this direction is the integration of 3D Gaussian Splatting (3DGS) into the SLAM (Simultaneous Localization and Mapping) pipeline.
Traditional SLAM focuses on geometry (where are the walls?), while semantic segmentation focuses on labels (what is this object?). Gaussian Splatting SLAM, introduced by Hidenobu Matsuki and Andrew Davison, unifies these tasks. Because 3DGS is differentiable and fast to render, it allows the system to discover scene geometry and camera poses simultaneously and incrementally.
Systems like OnlinePG by Qi Zhang and Dan Xu leverage this for open-vocabulary mapping. By attaching language embeddings directly to the 3D Gaussians, the map itself becomes "queryable." A robot doesn't just build a map; it builds a searchable database of the physical world in real-time.
Why It Works: The Intuition of Local Consistency
The reason these online methods are finally rivaling offline ones is a shift in philosophy: local consistency leads to global accuracy.
Instead of trying to solve the entire scene's geometry at once (which is computationally expensive), these models focus on making the current frame consistent with the immediate past. By using techniques like:
- Spatio-temporal attention (as seen in ALSTER by Francis Engelmann),
- Knowledge distillation to prevent forgetting (as seen in IL-NeRF), and
- Frustum-based processing to ignore irrelevant distant data (as seen in SaLon3R),
these systems manage the "memory bottleneck" of real-time 3D vision. They treat the world not as a static file to be processed, but as a continuous conversation between the sensor and the internal model.