Abstract
Referring Object Removal (ROR) is a complex task requiring a model to identify a specific object in an image via a natural language prompt, remove it, and seamlessly inpaint the background. Traditional approaches often rely on two-stage pipelines—segmentation followed by inpainting—which are prone to error accumulation and high latency. This paper introduces the Syntax-Aware Hybrid Mapping Network (SAHM), an end-to-end architecture that treats segmentation and inpainting as a unified goal. To facilitate this research, we present ComCOCO, a large-scale synthetic dataset. Our results show that SAHM significantly outperforms state-of-the-art diffusion models and two-stage pipelines in both visual quality and real-time performance, achieving 136 FPS.
Introduction
While traditional image inpainting has advanced significantly, it typically requires a manually provided mask. Without such guidance, models struggle to interpret the user's intent—for instance, whether to modify an object or remove it entirely. Existing text-based editing tools, such as those based on Stable Diffusion, often lack the local precision needed to target specific objects (e.g., "the second teddy bear from the left") and may inadvertently alter the style of the entire image.
Current "two-stage" solutions, which first segment an object and then inpaint the resulting hole, frequently suffer from accumulated errors. A slight misalignment in the generated mask can lead to "ghosting" artifacts or messy boundaries in the final output.
The SAHM Architecture
The proposed Syntax-Aware Hybrid Mapping Network (SAHM) addresses these issues by unifying segmentation and inpainting within a single framework. This end-to-end approach allows the inpainting process to provide feedback that helps refine the segmentation boundaries.
Syntax-Aware Visual Attention (SVA)
Rather than treating the text prompt as a monolithic input, SAHM employs a BiLSTM-CRF model to parse sentences into Identity words (e.g., "train") and Attribute words (e.g., "yellow"). Three distinct attention heads then fuse these linguistic features with visual features at various encoding stages. Notably, the authors found that using cosine similarity in these attention mechanisms prevented specific attribute words from dominating the attention scores, leading to a more balanced multi-modal understanding.
Hybrid Mapping Filling (HMF)
During the decoding phase, the model generates two parallel outputs: a segmentation mask and an inpainting map. To ensure high-level semantic features align perfectly with the sharp boundaries of the target object, the network utilizes deformable convolutions. This prevents the "blurry edge" problem common in standard multi-scale networks.
The ComCOCO Dataset
To support the ROR task, the authors introduced ComCOCO, the first dedicated dataset for semantic object removal. It contains 136,495 referring expressions for 34,615 objects across approximately 24,000 image pairs.
The dataset was constructed using a rigorous three-step synthetic pipeline:
- Scene Matching: Ensures objects are placed in contextually appropriate environments (e.g., a refrigerator in a kitchen).
- Object Placement: Uses GracoNet for realistic spatial positioning within the scene.
- Style Consistency: Employs a "Harmonizer" to match the lighting and texture of the inserted object with the background, minimizing synthetic artifacts.
Unlike previous datasets, ComCOCO focuses on medium and large objects, forcing models to learn complex background reconstruction rather than simple texture filling.
Experimental Results
SAHM sets a new benchmark for the ROR task across several metrics:
- Visual Quality: On the ComCOCO test set, SAHM achieved a PSNR of 23.29, significantly outperforming the best two-stage method (CRIS+SCAT at 21.74) and doubling the performance of diffusion-based models like InstructPix2Pix (11.23).
- Efficiency: SAHM operates at 136 FPS on a V100 GPU, making it suitable for real-time applications. In contrast, diffusion-based models often struggle to reach 0.2 FPS.
- Robustness: Qualitative evaluations demonstrate SAHM's ability to remove specific objects from cluttered real-world scenes without leaving visible artifacts or affecting non-target items.
Conclusion and Future Work
The SAHM model demonstrates the power of end-to-end training for complex multi-modal tasks. By explicitly modeling linguistic syntax and unifying the segmentation-inpainting pipeline, it achieves superior precision and speed. While ROR remains an "ill-posed" task with multiple valid solutions for background reconstruction, SAHM provides a robust foundation. Future research may explore combining the structural precision of SAHM with the high-fidelity generative capabilities of diffusion models to further enhance visual realism.
Summary updated