Abstract
The Segment Anything Model (SAM) has demonstrated remarkable zero-shot segmentation capabilities but often suffers from over-segmentation, identifying object parts rather than whole entities in its automatic mode. This "granularity" issue poses significant challenges for robotics and indoor scene understanding. We present the Segment Any Object Model (SAOM), which employs a novel "Real-to-Simulation" fine-tuning strategy. By training on synthetic data from the Ai2Thor simulator using a unique point-assignment method, SAOM learns to prioritize whole-object masks. Our approach achieves a 28% absolute improvement in mIoU over vanilla SAM and demonstrates strong zero-shot generalization to real-world indoor environments without requiring real-world training labels.
Introduction
Meta’s Segment Anything Model (SAM) is a foundational tool in computer vision, yet its "everything" mode frequently produces fragmented masks. For instance, instead of a single "chair" mask, SAM might output separate segments for legs, cushions, and backrests. For autonomous agents requiring whole-object interaction, such as domestic robots, this lack of structural coherence is problematic.
Collecting large-scale, high-quality real-world datasets for whole-object segmentation is labor-intensive. To address this, we propose SAOM, a model fine-tuned exclusively on simulated data to bridge the gap between part-based and object-based segmentation.
Methodology: The Real-to-Simulation Strategy
The core of SAOM is a fine-tuning pipeline that leverages the structural consistency of simulated environments to teach SAM the concept of a "whole object."
Nearest Neighbour Assignment
A key technical challenge in fine-tuning SAM's automatic mode is mapping ground-truth masks to the point-grid prompts SAM uses. We introduce a Nearest Neighbour Assignment method. During training, ground-truth masks are mapped to the closest point in a predefined grid, allowing the model to learn which point embeddings should trigger a comprehensive "whole object" mask rather than a localized part mask.
Model Architecture
Beyond mask generation, SAOM incorporates a dedicated semantic classifier head to categorize objects into 54 distinct classes. The model is trained to select the most representative mask from SAM's multi-mask output by maximizing the Intersection over Union (IoU) relative to the whole-object ground truth.
Experimental Setup
Datasets
- Ai2Thor Simulation Dataset: We generated a benchmark comprising 33,638 images and 303,937 object masks across 120 indoor scenes. This dataset covers 54 object classes, including openable and pickupable items.
- Real-Life Test Set: To evaluate sim-to-real transfer, we curated 151 real-world images of indoor scenes with varying object states and complex backgrounds.
Results and Discussion
SAOM significantly outperforms vanilla SAM in identifying whole objects. Quantitatively, the mean Intersection over Union (mIoU) rose from 10.82% to 39.17%, and mean Accuracy (mAcc) improved from 40.63% to 65.62%.
Qualitative Performance
Qualitative analysis confirms that SAOM successfully consolidates fragments into single object masks. While vanilla SAM and variants like Semantic-SAM often over-segment complex furniture, SAOM maintains object integrity.
Efficiency and Generalization
By focusing on whole objects, SAOM reduces "mask clutter," producing approximately 81.6% fewer masks than the original SAM. This reduction is critical for downstream tasks like robotic path planning. Remarkably, the model generalizes to real-world textures and lighting despite its purely synthetic training.
Limitations and Future Work
Current limitations include a bias toward foreground objects and sensitivity to heavy occlusion (e.g., a table covered in small items). Future research will focus on domain randomization—varying lighting and textures in simulation—to further enhance the robustness of the sim-to-real transfer.
Summary updated