Abstract
Referring Image Segmentation (RIS) requires a precise understanding of both visual scenes and natural language queries to generate accurate object masks. This paper introduces Cascaded Multi-modal Fusion (CMF), a novel architecture designed to enhance the alignment between linguistic descriptions and image pixels. By integrating multi-scale contextual information directly into the fusion process and utilizing a bi-directional recurrent unit to merge features across various network depths, the CMF model achieves state-of-the-art performance on four major benchmarks: UNC, UNC+, G-Ref, and ReferIt.
The Challenge of Referring Image Segmentation
Referring Image Segmentation (RIS) involves generating a binary mask for a specific object described by a natural language expression, such as "the kid on the left in the blue shirt." Unlike standard semantic segmentation, which identifies all instances of a category, RIS must resolve linguistic nuances including attributes, actions, and spatial relationships to pick out a single target.
The primary difficulty lies in multi-modal alignment. Existing methods often struggle because they either fuse visual and linguistic features too simplistically or only consider multi-scale context after the modalities have been combined. This leads to failures when objects vary in size or when descriptions involve fine-grained details that require simultaneous consideration of local and global context.
Proposed Method: Cascaded Multi-modal Fusion
The authors propose the CMF architecture to bridge the gap between vision and language through three key innovations:
1. Cascaded Multi-modal Fusion (CMF) Module
Inspired by Atrous Spatial Pyramid Pooling (ASPP), the CMF module utilizes multiple parallel convolutional layers with varying dilation rates. This allows the filters to capture a wide range of receptive fields without losing spatial resolution. Unlike standard ASPP, the CMF module introduces a cascaded branch that iteratively fuses visual and linguistic features. This progressive alignment ensures the model is robust to objects of different scales by refining the text-image interaction across multiple levels of detail.
2. Image-to-Word Attention
To reduce linguistic noise, the model employs an attention mechanism before the fusion stage. This determines which specific words in a query (e.g., "blue," "left") are most relevant to particular regions of the image, ensuring the fusion process focuses on the most informative parts of the description.
3. Bi-directional Convolutional GRU (Bi-ConvGRU)
The model leverages a ResNet-101 backbone, extracting features from multiple levels (Res2 through Res5). A Bi-ConvGRU is used to fuse these multi-level features. The top-down path injects high-level semantic meaning into lower layers, while the bottom-up path preserves sharp spatial details necessary for generating a precise final mask.
Experimental Evaluation
The CMF model was evaluated on four standard RIS benchmarks, each presenting unique challenges:
- UNC & UNC+: Large-scale datasets where UNC+ excludes location words, forcing a reliance on visual appearance.
- G-Ref (Google Refexp): Contains complex descriptions averaging 8.4 words.
- ReferIt: Includes a mix of objects and background "stuff" like sky or walls.
Key Results
The CMF model established new state-of-the-art records across all datasets:
- UNC: Achieved an IoU of 62.03% (val set), surpassing the previous best (BRINet) by 0.68%.
- UNC+: Achieved an IoU of 49.62% (val set), a 1.05% improvement, demonstrating high effectiveness in aligning visual attributes even without spatial keywords.
- Ablation Studies: The CMF module alone provided a 10.92% improvement over a baseline fusion model, confirming that cascaded multi-scale fusion is the primary driver of the model's success.
Conclusion
The CMF architecture demonstrates that modeling multi-scale context during the vision-language fusion process is critical for RIS. By avoiding the pitfalls of late-stage context modeling and utilizing a bi-directional approach to feature fusion, the model achieves superior robustness and accuracy. Future work may explore aggregating even longer-range context through Transformers to further refine the alignment of complex natural language queries with visual data.
Summary updated