Modern computer vision has a "viewpoint problem." While foundation models like the Segment Anything Model (SAM) can identify objects in a single image with incredible accuracy, they are "view-agnostic." If you move the camera slightly, the model might assign a different ID to the same chair, or decide that a table is one object in the first frame but three separate parts in the next.
This phenomenon, often called index inconsistency or multi-view identity inconsistency, is the primary hurdle in turning 2D image intelligence into 3D scene understanding. To build a 3D map where objects can be moved, deleted, or tracked, researchers are developing methods to "lift" these flickering 2D predictions into stable, unified 3D representations.
The Challenge of Index Inconsistency
When we reconstruct a 3D scene using Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS), we aggregate information from many 2D images. If we simply project 2D segmentation masks into this 3D space, the conflicting labels clash. As noted in Panoptic Compositional Feature Field by Xinhua Cheng and colleagues, this lack of cross-view coherence prevents standard training from establishing a stable identity for objects.
Without a reconciliation strategy, a single physical object becomes a fragmented "soup" of different IDs in 3D. To solve this, the field has converged on three main strategies: 3D anchoring, contrastive lifting, and memory-based association.
Strategy 1: Anchoring via 3D Geometry
The most intuitive way to fix 2D inconsistency is to force the 2D model to look at the same 3D point every time. Instead of letting SAM pick its own segments in every frame, researchers "anchor" the segmentation process to the 3D world.
In SAMPro3D, Mutian Xu and team project 3D points back into 2D views to serve as "prompts" for SAM. Because the prompt is derived from a fixed 3D location, SAM is much more likely to produce a mask that refers to the same physical entity across different angles. Similarly, Instance-NeRF by Yichen Liu uses coarse 3D object proposals to unify inconsistent 2D labels, effectively using the 3D geometry as a "source of truth" that overrides 2D noise.
Strategy 2: Contrastive Lifting and Metric Learning
Rather than forcing consistency at the input (prompts), another approach is to learn a 3D "feature field" where points belonging to the same object are pushed together in a high-dimensional space.
Contrastive Lift, developed by Yash Bhalgat and Iro Laina, uses a "slow-fast" contrastive fusion. It treats 2D masks as noisy supervision; by comparing features across different views, the model learns to ignore the inconsistent "flicker" of the 2D labels and focus on the underlying 3D structure. UniC-Lift takes this further by mapping learnable embeddings directly to discrete labels, eliminating the need for slow post-processing clustering.
The intuition here is that while 2D labels are inconsistent, the errors are often random, while the object boundaries are tied to the consistent 3D geometry. Over many views, the consistent signal wins out.
Strategy 3: Memory Banks and Global Codebooks
A third emerging direction involves maintaining a "global registry" of objects seen so far. As the system processes new views, it tries to match the current 2D masks against a 3D memory bank.
Gaga, proposed by Weijie Lyu and Abhijit Kundu, uses a 3D-aware memory bank to associate masks. If a new mask appears, the system checks if it matches a known 3D entity in the bank. This allows the model to remain robust even when views are sparse or taken from drastically different angles. A similar concept appears in Unified-Lift, which introduces a "global object codebook" to manage hundreds of objects in a scene simultaneously without brittle matching steps.
Why This Works: Geometry as a Regularizer
The reason these methods succeed where 2D-only methods fail is that 3D geometry acts as a powerful regularizer. In 2D, an object's appearance changes completely as you walk around it. In 3D, that object occupies a fixed set of coordinates.
By "lifting" 2D information into representations like ClusteringSDF or DiscoNeRF, researchers like Corentin Dumery can apply spatial regularization. They can penalize a 3D point for having a different label than its immediate neighbor, or use epipolar geometry—the mathematical relationship between two camera views—to "vote" on the correct object boundary, as seen in iSegMan.
The end result is a shift from "seeing pixels" to "understanding entities." Once a scene is decomposed into consistent 3D objects, tasks like robotics manipulation and virtual scene editing become possible without any manual human labeling.