InterRSGaussian: Interactive Multi-view Instance Segmentation for 3D Remote Sensing Scenes

Zipeng Qi, Zhengxia Zou, Zhenwei ShiarXiv 2026 · arXiv

TL;DR

InterRSGaussian introduces a novel framework for interactive instance segmentation in 3D remote sensing scenes by leveraging 3D Gaussian Splatting. By combining user point-clicks with foundation models like SAM and DINO, the system generates highly consistent segmentation masks across hundreds of viewpoints and 3D point clouds without requiring any manual training labels.

What Problem Does This Paper Address?

Traditional remote sensing image segmentation has largely been a 2D affair. While interactive tools like the Segment Anything Model (SAM) allow users to extract objects with a few clicks, they operate on single images. In complex 3D environments—like a city block captured from multiple drone angles—single-view methods struggle. They lack "view consistency," meaning a building segmented in one photo might look different or be missed entirely in another.

Furthermore, annotating 3D remote sensing data is prohibitively expensive and labor-intensive. There is a clear need for a system that can take a 2D user prompt (like a click on a roof) and automatically understand that object's full 3D extent across all available perspectives.

Methods

The authors propose InterRSGaussian, which moves the segmentation task from the 2D pixel grid into a 3D spatial representation using 3D Gaussian Splatting (3DGS).

1. 3D Spatial Representation

The scene is first reconstructed into thousands of 3D Gaussians. Each Gaussian possesses spatial attributes (position, rotation, scale, opacity) and color information.

2. Prompt Propagation & Pseudo-labeling

When a user clicks an object in one "reference" view, the system uses the underlying 3D geometry to project that click into all other "target" views. These propagated clicks are fed into SAM to generate "pseudo-labels"—initial, noisy 2D masks for every viewpoint.

3. Identity Feature Learning

This is the core innovation. Each 3D Gaussian is assigned a 16-dimensional identity feature. These features are rendered into 2D views using alpha blending (similar to how colors are rendered in 3DGS) and passed through a shallow decoder. The system is optimized by minimizing the difference between these rendered masks and the SAM pseudo-labels. Because the identity is tied to the 3D Gaussian, the resulting segmentation is inherently consistent across all views.

4. DINO Distillation for Initialization

To speed up convergence and handle noisy SAM labels, the authors introduce a prior distillation network. They take features from DINO (a foundation model known for strong unsupervised object awareness) and distill them into the 3D Gaussians. This provides a "warm start," allowing the model to distinguish between objects even before the optimization process begins.

Datasets

The authors found existing datasets lacking for this specific task, so they constructed a new multi-view interactive instance segmentation dataset:

  • Five distinct scenes: Three simulated using the CARLA autonomous driving platform and two real-world scenes captured via Google Maps.
  • Scale: Each scene includes approximately 100 different viewpoints.
  • Ground Truth: Manually annotated for 2-3 target objects per scene to allow for rigorous quantitative evaluation.

Key Insights and Ideas

  • 3D as a Consistency Anchor: By optimizing features in 3D space rather than 2D pixels, the model naturally resolves occlusions. If an object is hidden in one view but visible in another, the 3D Gaussians maintain the object's "identity" regardless of the camera angle.
  • Foundation Models as Teachers: The framework is essentially "unsupervised" in the sense that it requires no manual training data. It uses the general knowledge of SAM (for boundaries) and DINO (for semantic features) to teach the 3D representation how to group itself.
  • Efficiency via Distillation: Using DINO for initialization isn't just a minor tweak; it significantly improves performance in complex scenes (like "sys #3" in the paper) where large-scale objects and heavy occlusions make standard optimization difficult.

Key Results

The researchers compared InterRSGaussian against vanilla SAM and a state-of-the-art baseline, Gaussian Grouping.

  • Superior Accuracy: In complex simulated scenes (sys #3), InterRSGaussian achieved an mIoU of 76.62%, vastly outperforming SAM's 50.62%.
  • Speed and Convergence: Under a 1,000-iteration optimization setting ("Ours-1k"), the method outperformed the baseline by over 5 percentage points in challenging scenarios.
  • 3D Versatility: The multi-view 2D masks can be projected back into 3D space, providing high-quality 3D point cloud segmentation as a direct byproduct of the 2D user interaction.

Limitations and Future Work

  • Domain Gap: Foundation models like SAM and DINO are trained on natural "ground-level" images. Their performance on top-down or oblique remote sensing imagery is good but not perfect. The authors suggest using Parameter-Efficient Fine-Tuning (PEFT) in the future to bridge this gap.
  • Boundary Precision: Handling extremely thin objects or ambiguous boundaries remains a challenge when relying on noisy pseudo-labels.
  • Dataset Size: While the 5-scene dataset is a start, the authors acknowledge the need to expand it to cover more diverse urban textures and object types.

For more details, see the full paper: "InterRSGaussian: Interactive Multi-view Instance Segmentation for 3D Remote Sensing Scenes" by Qi et al.

Summary updated

Abstract

Remote sensing image segmentation plays a critical role in various applications. Interactive instance segmentation, a specialized task within image segmentation, aims to generate precise segmentation results for target objects based on user input prompts. However, previous studies on interactive instance segmentation primarily focus on single-view images, limiting their ability to comprehensively understand complex 3D remote sensing scenes. In this study, we extend interactive instance segmentation from a single-view to a multi-view framework, which can also be easily converted to a 3D segmentation format, thus achieving more accurate and comprehensive 3D remote sensing scene perception across multiple views. First, we construct a multi-view interactive instance segmentation dataset for remote sensing, comprising five distinct scenes, each captured from about one hundred different views. Second, we propose InterRSGaussian, a novel instance segmentation method based on 3D Gaussians. InterRSGaussian enables users to provide point clicks on target objects from any arbitrary view and generates view-consistent instance segmentation results across all input views without requiring manual annotations. The resulting multi-view segmentation can further be projected onto 3D point clouds for 3D segmentation. To improve training efficiency and segmentation performance, we introduce a prior distillation network for initialization. Overall, this study extends traditional 2D remote sensing scene perception into a 3D multi-view framework, enabling a more comprehensive understanding of target objects in complex environments with user interactivity.
BibTeX
@misc{qi2026interrsgaussian,
  author = {Zipeng Qi and Zhengxia Zou and Zhenwei Shi},
  title = {{InterRSGaussian: Interactive Multi-view Instance Segmentation for 3D Remote Sensing Scenes}},
  year = {2026},
  eprint = {2611.20425},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20425},
}
Figure