CMF: Cascaded Multi-Model Fusion For Referring Image Segmentation

Jianhua Yang, Yan Huang, Zhanyu Ma, Liang Wang

Abstract

Referring Image Segmentation (RIS) requires a precise understanding of both visual scenes and natural language queries to generate accurate object masks. This paper introduces Cascaded Multi-modal Fusion (CMF), a novel architecture designed to enhance the alignment between linguistic descriptions and image pixels. By integrating multi-scale contextual information directly into the fusion process and utilizing a bi-directional recurrent unit to merge features across various network depths, the CMF model achieves state-of-the-art performance on four major benchmarks: UNC, UNC+, G-Ref, and ReferIt.

The Challenge of Referring Image Segmentation

Referring Image Segmentation (RIS) involves generating a binary mask for a specific object described by a natural language expression, such as "the kid on the left in the blue shirt." Unlike standard semantic segmentation, which identifies all instances of a category, RIS must resolve linguistic nuances including attributes, actions, and spatial relationships to pick out a single target.

Comparison between semantic segmentation and referring image segmentation.

The primary difficulty lies in multi-modal alignment. Existing methods often struggle because they either fuse visual and linguistic features too simplistically or only consider multi-scale context after the modalities have been combined. This leads to failures when objects vary in size or when descriptions involve fine-grained details that require simultaneous consideration of local and global context.

Proposed Method: Cascaded Multi-modal Fusion

The authors propose the CMF architecture to bridge the gap between vision and language through three key innovations:

1. Cascaded Multi-modal Fusion (CMF) Module

Inspired by Atrous Spatial Pyramid Pooling (ASPP), the CMF module utilizes multiple parallel convolutional layers with varying dilation rates. This allows the filters to capture a wide range of receptive fields without losing spatial resolution. Unlike standard ASPP, the CMF module introduces a cascaded branch that iteratively fuses visual and linguistic features. This progressive alignment ensures the model is robust to objects of different scales by refining the text-image interaction across multiple levels of detail.

2. Image-to-Word Attention

To reduce linguistic noise, the model employs an attention mechanism before the fusion stage. This determines which specific words in a query (e.g., "blue," "left") are most relevant to particular regions of the image, ensuring the fusion process focuses on the most informative parts of the description.

3. Bi-directional Convolutional GRU (Bi-ConvGRU)

The model leverages a ResNet-101 backbone, extracting features from multiple levels (Res2 through Res5). A Bi-ConvGRU is used to fuse these multi-level features. The top-down path injects high-level semantic meaning into lower layers, while the bottom-up path preserves sharp spatial details necessary for generating a precise final mask.

The overall architecture of the CMF model showing the flow from input image and expression to the final prediction.

Experimental Evaluation

The CMF model was evaluated on four standard RIS benchmarks, each presenting unique challenges:

  • UNC & UNC+: Large-scale datasets where UNC+ excludes location words, forcing a reliance on visual appearance.
  • G-Ref (Google Refexp): Contains complex descriptions averaging 8.4 words.
  • ReferIt: Includes a mix of objects and background "stuff" like sky or walls.

Key Results

The CMF model established new state-of-the-art records across all datasets:

  • UNC: Achieved an IoU of 62.03% (val set), surpassing the previous best (BRINet) by 0.68%.
  • UNC+: Achieved an IoU of 49.62% (val set), a 1.05% improvement, demonstrating high effectiveness in aligning visual attributes even without spatial keywords.
  • Ablation Studies: The CMF module alone provided a 10.92% improvement over a baseline fusion model, confirming that cascaded multi-scale fusion is the primary driver of the model's success.

Conclusion

The CMF architecture demonstrates that modeling multi-scale context during the vision-language fusion process is critical for RIS. By avoiding the pitfalls of late-stage context modeling and utilizing a bi-directional approach to feature fusion, the model achieves superior robustness and accuracy. Future work may explore aggregating even longer-range context through Transformers to further refine the alignment of complex natural language queries with visual data.

Summary updated

BibTeX
@inproceedings{yang2021cmf,
  author = {Jianhua Yang and Yan Huang and Zhanyu Ma and Liang Wang},
  title = {{CMF: Cascaded Multi-Model Fusion For Referring Image Segmentation}},
  booktitle = {ICIP},
  year = {2021},
  doi = {10.1109/icip42928.2021.9506483},
}
Figure