Cross-aware Early Fusion with Stage-divided Vision and Language Transformer Encoders for Referring Image Segmentation

Yubin Cho, Hyunwoo Yu, Suk-Ju Kang

Abstract

Referring segmentation aims to segment a target object related to a natural language expression. Key challenges of this task are understanding the meaning of complex and ambiguous language expressions and determining the relevant regions in the image with multiple objects by referring to the expression. Recent models have focused on the early fusion with the language features at the intermediate stage of the vision encoder, but these approaches have a limitation that the language features cannot refer to the visual information. To address this issue, this paper proposes a novel architecture, Cross-aware early fusion with stage-divided Vision and Language Transformer encoders (CrossVLT), which allows both language and vision encoders to perform the early fusion for improving the ability of the cross-modal context modeling. Unlike previous methods, our method enables the vision and language features to refer to each other's information at each stage to mutually enhance the robustness of both encoders. Furthermore, unlike the conventional scheme that relies solely on the high-level features for the cross-modal alignment, we introduce a feature-based alignment scheme that enables the low-level to high-level features of the vision and language encoders to engage in the cross-modal alignment. By aligning the intermediate cross-modal features in all encoder stages, this scheme leads to effective cross-modal fusion. In this way, the proposed approach is simple but effective for referring image segmentation, and it outperforms the previous state-of-the-art methods on three public benchmarks.

Introduction: The Challenge of Referring Image Segmentation

Referring Image Segmentation (RIS) involves generating a precise binary mask for an object described by a natural language expression, such as "the person kneeling down." This task is inherently difficult as it requires simultaneous, deep understanding of both visual context and linguistic nuances.

Existing research has identified two major bottlenecks in standard RIS models:

  1. Unidirectional Fusion: Most early fusion models are one-sided. While language features help the vision encoder locate objects, the language encoder remains "blind" to the visual scene, making it difficult to resolve linguistic ambiguities that require visual grounding.
  2. Late-Stage Alignment: Conventional methods often align vision and language features only at the final encoding stage, neglecting the rich multi-scale information—such as syntactic structures in text or edges in images—present in intermediate layers.
Comparison of fusion approaches: (a) Late fusion, (b) Previous early fusion, and (c) the proposed CrossVLT.

Proposed Method: CrossVLT

To overcome these limitations, the authors introduce CrossVLT (Cross-aware early fusion with stage-divided Vision and Language Transformer encoders). The architecture is built upon three core innovations:

1. Stage-divided Language Encoder

Unlike standard BERT models that process text in a single block, CrossVLT divides the language encoder into four distinct stages (using a [6, 2, 2, 2] layer split). This hierarchical structure mirrors that of vision transformers (like Swin), facilitating stage-by-stage cross-modal interaction.

2. Cross-aware Early Fusion

At each of the four stages, CrossVLT performs bidirectional fusion. Vision features query language features for linguistic context, while language features simultaneously query vision features to ground text in the visual scene. This mutual enhancement improves the robustness of both encoders.

Detailed architecture of CrossVLT showing bidirectional cross-aware early fusion between vision and language stages.

3. Feature-based Alignment

Prior to fusion at each stage, a text-to-pixel contrastive loss is applied. This forces the model to map vision and language features into a shared embedding space early in the process, ensuring that subsequent fusion operates on compatible multi-modal data.

Experimental Results

CrossVLT was evaluated on three major benchmarks: RefCOCO, RefCOCO+, and G-Ref. The model achieved new state-of-the-art (SOTA) performance across all datasets:

  • RefCOCO: Reached an overall Intersection-over-Union (oIoU) of 73.44%, surpassing the previous leader, LAVT (72.73%).
  • G-Ref: On this dataset featuring longer, more complex expressions, CrossVLT achieved 63.75% oIoU, a significant improvement over LAVT’s 62.09%.
  • Qualitative Analysis: t-SNE visualizations demonstrate that CrossVLT clusters relevant pixels much closer to language tokens in the embedding space than late-fusion models, resulting in cleaner and more accurate segmentation masks, particularly in crowded scenes.

Conclusion

CrossVLT demonstrates that bidirectional communication between vision and language encoders is superior to unidirectional approaches. By aligning features at every stage, the model captures a hierarchy of correlations—from low-level textures to high-level semantic concepts. While the current stage-splitting strategy is empirical, the success of CrossVLT highlights the potential of multi-scale, bidirectional fusion for complex multi-modal tasks.

Summary updated

Abstract

Referring segmentation aims to segment a target object related to a natural language expression. Key challenges of this task are understanding the meaning of complex and ambiguous language expressions and determining the relevant regions in the image with multiple objects by referring to the expression. Recent models have focused on the early fusion with the language features at the intermediate stage of the vision encoder, but these approaches have a limitation that the language features cannot refer to the visual information. To address this issue, this paper proposes a novel architecture, Cross-aware early fusion with stage-divided Vision and Language Transformer encoders (CrossVLT), which allows both language and vision encoders to perform the early fusion for improving the ability of the cross-modal context modeling. Unlike previous methods, our method enables the vision and language features to refer to each other's information at each stage to mutually enhance the robustness of both encoders. Furthermore, unlike the conventional scheme that relies solely on the high-level features for the cross-modal alignment, we introduce a feature-based alignment scheme that enables the low-level to high-level features of the vision and language encoders to engage in the cross-modal alignment. By aligning the intermediate cross-modal features in all encoder stages, this scheme leads to effective cross-modal fusion. In this way, the proposed approach is simple but effective for referring image segmentation, and it outperforms the previous state-of-the-art methods on three public benchmarks.
BibTeX
@article{cho2024cross,
  author = {Yubin Cho and Hyunwoo Yu and Suk-Ju Kang},
  title = {{Cross-aware Early Fusion with Stage-divided Vision and Language Transformer Encoders for Referring Image Segmentation}},
  journal = {IEEE Trans. Multimedia},
  year = {2024},
  url = {https://openreview.net/forum?id=P9Qv74t2KZ},
}
Figure