Abstract
Referring Image Segmentation (RIS) aims to generate a pixel-level mask for an object described by a natural language expression. Traditional methods often rely on late fusion, where linguistic and visual features are combined only at the decoding stage. This paper introduces the Encoder Fusion Network (EFN), which shifts multi-modal fusion into the encoder to produce language-aware visual features. Combined with a parallel co-attention mechanism and a Boundary Enhancement Module (BEM), the proposed CEFNet achieves state-of-the-art performance across multiple benchmarks while maintaining a real-time processing speed of 50 FPS.
Introduction
The task of Referring Image Segmentation (RIS) requires a deep understanding of both visual context and linguistic nuances. Despite recent progress, existing approaches suffer from three primary limitations:
- Late Fusion Bottlenecks: Most models extract features independently and fuse them late in the decoder, missing the opportunity for language to guide early visual feature extraction.
- Serial Update Delays: Attention mechanisms that update modalities sequentially can lead to semantic misalignment.
- Detail Loss: Standard CNN architectures often lose fine-grained spatial information due to pooling operations, resulting in blurred object boundaries.
Proposed Method: CEFNet
To address these issues, the authors propose the Encoder Fusion Network with Co-Attention Embedding (CEFNet).
Encoder Fusion Network (EFN)
The core innovation is the EFN, which integrates linguistic features directly into the ResNet-101 backbone. By injecting language cues after the third, fourth, and fifth convolutional blocks, the network learns to extract visual features that are inherently conditioned on the query.
Co-Attention and Boundary Enhancement
To maintain alignment between modalities, CEFNet employs a parallel co-attention strategy. The authors introduce an Asymmetric Co-attention (ACM) module that uses a Pyramid Pooling Module (PPM) to sample features at multiple scales. This significantly reduces the computational overhead of the affinity matrix while preserving contextual richness.
Furthermore, a Boundary Enhancement Module (BEM) is integrated into the decoder. By utilizing a Spatial Transformer Network (STN) to isolate high-level abstractions and subtracting them from the feature maps, the BEM highlights fine-grained boundary cues, leading to sharper segmentation masks.
Experimental Evaluation
The model was rigorously tested on four benchmark datasets: UNC, UNC+, Google-Ref (G-Ref), and ReferIt.
Key Results
- Performance: CEFNet established new state-of-the-art results. On the complex G-Ref dataset, it achieved an Overall IoU of 51.93%, a significant improvement over previous benchmarks.
- Efficiency: The model operates at 50 FPS on an NVIDIA GTX 1080Ti, demonstrating its viability for real-time applications.
- Ablation Study: Experiments confirmed that moving fusion to the encoder (EFN) provides a substantial performance boost (~3% IoU) compared to traditional decoder-only fusion.
Conclusion and Future Work
CEFNet demonstrates that treating language as a primary guide during the encoding process is crucial for effective multi-modal alignment. The use of parallel co-attention and boundary-specific modules allows for both semantic accuracy and spatial precision. While the model shows some sensitivity to ambiguous queries or rare vocabulary, its speed and accuracy mark a significant step forward. Future research may explore extending this framework to unified one-stage grounding tasks.
Summary updated