TL;DR
InterRSGaussian introduces a novel framework for interactive instance segmentation in 3D remote sensing scenes by leveraging 3D Gaussian Splatting. By combining user point-clicks with foundation models like SAM and DINO, the system generates highly consistent segmentation masks across hundreds of viewpoints and 3D point clouds without requiring any manual training labels.
What Problem Does This Paper Address?
Traditional remote sensing image segmentation has largely been a 2D affair. While interactive tools like the Segment Anything Model (SAM) allow users to extract objects with a few clicks, they operate on single images. In complex 3D environments—like a city block captured from multiple drone angles—single-view methods struggle. They lack "view consistency," meaning a building segmented in one photo might look different or be missed entirely in another.
Furthermore, annotating 3D remote sensing data is prohibitively expensive and labor-intensive. There is a clear need for a system that can take a 2D user prompt (like a click on a roof) and automatically understand that object's full 3D extent across all available perspectives.
Methods
The authors propose InterRSGaussian, which moves the segmentation task from the 2D pixel grid into a 3D spatial representation using 3D Gaussian Splatting (3DGS).
1. 3D Spatial Representation
The scene is first reconstructed into thousands of 3D Gaussians. Each Gaussian possesses spatial attributes (position, rotation, scale, opacity) and color information.
2. Prompt Propagation & Pseudo-labeling
When a user clicks an object in one "reference" view, the system uses the underlying 3D geometry to project that click into all other "target" views. These propagated clicks are fed into SAM to generate "pseudo-labels"—initial, noisy 2D masks for every viewpoint.
3. Identity Feature Learning
This is the core innovation. Each 3D Gaussian is assigned a 16-dimensional identity feature. These features are rendered into 2D views using alpha blending (similar to how colors are rendered in 3DGS) and passed through a shallow decoder. The system is optimized by minimizing the difference between these rendered masks and the SAM pseudo-labels. Because the identity is tied to the 3D Gaussian, the resulting segmentation is inherently consistent across all views.
4. DINO Distillation for Initialization
To speed up convergence and handle noisy SAM labels, the authors introduce a prior distillation network. They take features from DINO (a foundation model known for strong unsupervised object awareness) and distill them into the 3D Gaussians. This provides a "warm start," allowing the model to distinguish between objects even before the optimization process begins.
Datasets
The authors found existing datasets lacking for this specific task, so they constructed a new multi-view interactive instance segmentation dataset:
- Five distinct scenes: Three simulated using the CARLA autonomous driving platform and two real-world scenes captured via Google Maps.
- Scale: Each scene includes approximately 100 different viewpoints.
- Ground Truth: Manually annotated for 2-3 target objects per scene to allow for rigorous quantitative evaluation.
Key Insights and Ideas
- 3D as a Consistency Anchor: By optimizing features in 3D space rather than 2D pixels, the model naturally resolves occlusions. If an object is hidden in one view but visible in another, the 3D Gaussians maintain the object's "identity" regardless of the camera angle.
- Foundation Models as Teachers: The framework is essentially "unsupervised" in the sense that it requires no manual training data. It uses the general knowledge of SAM (for boundaries) and DINO (for semantic features) to teach the 3D representation how to group itself.
- Efficiency via Distillation: Using DINO for initialization isn't just a minor tweak; it significantly improves performance in complex scenes (like "sys #3" in the paper) where large-scale objects and heavy occlusions make standard optimization difficult.
Key Results
The researchers compared InterRSGaussian against vanilla SAM and a state-of-the-art baseline, Gaussian Grouping.
- Superior Accuracy: In complex simulated scenes (sys #3), InterRSGaussian achieved an mIoU of 76.62%, vastly outperforming SAM's 50.62%.
- Speed and Convergence: Under a 1,000-iteration optimization setting ("Ours-1k"), the method outperformed the baseline by over 5 percentage points in challenging scenarios.
- 3D Versatility: The multi-view 2D masks can be projected back into 3D space, providing high-quality 3D point cloud segmentation as a direct byproduct of the 2D user interaction.
Limitations and Future Work
- Domain Gap: Foundation models like SAM and DINO are trained on natural "ground-level" images. Their performance on top-down or oblique remote sensing imagery is good but not perfect. The authors suggest using Parameter-Efficient Fine-Tuning (PEFT) in the future to bridge this gap.
- Boundary Precision: Handling extremely thin objects or ambiguous boundaries remains a challenge when relying on noisy pseudo-labels.
- Dataset Size: While the 5-scene dataset is a start, the authors acknowledge the need to expand it to cover more diverse urban textures and object types.
For more details, see the full paper: "InterRSGaussian: Interactive Multi-view Instance Segmentation for 3D Remote Sensing Scenes" by Qi et al.
Summary updated