TFANet: three-stage image-text feature alignment network for robust referring image segmentation

Qianqi Lu, Yi Xie, Jing Zhang, Shiwei Zou, Tao Li, Xidao LuanInternational Journal of Multimedia Information Retrieval 2025 · arXiv

TL;DR

This paper introduces TFANet, a novel three-stage network designed to improve referring image segmentation (RIS) by systematically aligning visual and textual features. TFANet tackles the challenges of multimodal misalignment and semantic loss, leading to more accurate segmentation of objects described by language, especially in complex scenes.

What Problem Does This Paper Address?

Referring Image Segmentation (RIS) is a crucial task that involves identifying and segmenting specific objects in an image based on a natural language description. While existing methods have made progress, they often struggle with two main issues:

  1. Multimodal Misalignment: The model fails to accurately align the visual features of the image with the semantic meaning of the text. This can lead to incorrect attention being paid to irrelevant parts of the image or text, resulting in mislocalized or incomplete segmentations. This is particularly problematic in scenes with multiple visually similar objects.
  2. Semantic Loss: During the process of encoding and decoding visual and textual information, some of the crucial linguistic meaning can be lost. This loss of information further exacerbates the segmentation errors, making it difficult for the model to distinguish between subtly different objects.

These challenges are amplified in complex scenes where descriptions might be ambiguous or refer to objects with similar visual characteristics.

Methods

TFANet proposes a hierarchical, three-stage approach to address these challenges:

  • Knowledge Plus Stage (KPS): This stage focuses on enhancing the interaction between abstract textual knowledge and concrete visual knowledge. It introduces the Multiscale Linear Cross-Attention Module (MLAM). MLAM facilitates bidirectional, multi-scale semantic exchange across different granularities of visual and textual features. It achieves this by using cross-attention at multiple scales (pixel-to-word, region-to-word, etc.) and employs an X-Norm operation and multiplicative union law to reduce computational complexity to linear, making it efficient.
  • Knowledge Fusion Stage (KFS): This stage aims to improve global coherence and alignment accuracy. It uses the Cross-Modal Feature Scanning Module (CFSM), which selectively scans features across both channel and spatial dimensions. This allows the model to capture long-range dependencies and create a unified multimodal representation, essential for understanding complex relationships between text and image.
  • Knowledge Intensification Stage (KIS): This final stage addresses the loss of semantic information during decoding. It employs the Word-level Linguistic Feature-guided Semantic Deepening Module (WFDM). WFDM progressively injects word-level linguistic cues into the mask generation process, ensuring cross-modal consistency and enabling more precise segmentation, especially for distinguishing similar objects.

The overall architecture uses VMamba as a visual encoder and CLIP for text feature extraction.

Datasets

The authors evaluated TFANet on three widely used benchmark datasets for referring image segmentation:

  • RefCOCO: Contains approximately 19,994 images and 142,210 referring expressions.
  • RefCOCO+: Similar to RefCOCO but prohibits positional words, focusing more on appearance-based descriptions.
  • G-Ref: Features longer and more descriptive text, with 26,711 images and 54,822 annotated objects.

Key Insights and Ideas

  • Hierarchical Alignment is Key: The paper strongly advocates for a staged approach to feature alignment, arguing that a single-stage fusion is insufficient for complex RIS tasks. Each stage builds upon the previous one, progressively refining the multimodal understanding.
  • MLAM's Efficiency and Effectiveness: The MLAM module is a significant contribution, offering a way to achieve rich multimodal interaction at multiple scales with linear computational complexity, a crucial advantage over quadratic attention mechanisms.
  • CFSM for Long-Range Dependencies: The CFSM module's ability to scan features across channels and spatial dimensions is vital for capturing global context and relationships, which is often missed by local attention mechanisms.
  • WFDM for Robust Decoding: Injecting word-level linguistic information during the decoding phase is shown to be critical for mitigating semantic loss and improving the final segmentation quality, especially in challenging cases.
  • Addressing Semantic Loss: The paper directly tackles the issue of semantic information loss during encoding/decoding, a common pitfall in previous RIS models.

Key Results

TFANet achieved state-of-the-art results on the RefCOCO, RefCOCO+, and G-Ref datasets, demonstrating significant improvements in mean Intersection over Union (mIoU):

  • RefCOCO: +1.84% mIoU improvement.
  • RefCOCO+: +1.52% mIoU improvement.
  • G-Ref: +2.29% mIoU improvement.

Qualitative results showed TFANet producing more precise segmentations, particularly in edge refinement and object boundary delineation, outperforming existing models like ReMamber. Ablation studies confirmed the effectiveness of each proposed module (KPS, KFS, KIS, MLAM, CFSM, WFDM), showing progressive improvements as components were added. TFANet also demonstrated strong performance on stricter metrics like [email protected], indicating its ability to accurately localize targets.

Limitations and Future Work

  • Performance on Appearance-Centric Subsets: While TFANet excels in visually complex scenarios, the paper notes that it slightly underperforms ASDA on appearance-centric subsets of RefCOCO+ (validation and testA). This is attributed to ASDA's specialized text-adaptive feature selection mechanism.
  • Future Directions: The authors suggest future research could focus on integrating color-shape sensitive modules in KFS and developing attention mechanisms tailored for short-text linguistic structures to further enhance performance on appearance-centric subsets. They also mention that while TFANet has a higher theoretical FLOP count, its modular design allows for better hardware parallelism and runtime adaptability compared to some monolithic SOTA models.

Summary updated

Abstract

Referring Image Segmentation (RIS) is a task that segments image regions based on language expressions, requiring fine-grained alignment between two modalities. However, existing methods often struggle with multimodal misalignment and language semantic loss, especially in complex scenes containing multiple visually similar objects, where uniquely described targets are frequently mislocalized or incompletely segmented. To tackle these challenges, this paper proposes TFANet, a Three-stage Image-Text Feature Alignment Network that systematically enhances multimodal alignment through a hierarchical framework comprising three stages: Knowledge Plus Stage (KPS), Knowledge Fusion Stage (KFS), and Knowledge Intensification Stage (KIS). In the first stage, we design the Multiscale Linear Cross-Attention Module (MLAM), which facilitates bidirectional semantic exchange between visual features and textual representations across multiple scales. This establishes rich and efficient alignment between image regions and different granularities of linguistic descriptions. Subsequently, the KFS further strengthens feature alignment through the Cross-modal Feature Scanning Module (CFSM), which applies multimodal selective scanning to capture long-range dependencies and construct a unified multimodal representation. This is essential for modeling long-range cross-modal dependencies and enhancing alignment accuracy in complex scenes. Finally, in the KIS, we propose the Word-level Linguistic Feature-guided Semantic Deepening Module (WFDM) to compensate for semantic degradation introduced in earlier stages.
BibTeX
@article{lu2025tfanet,
  author = {Qianqi Lu and Yi Xie and Jing Zhang and Shiwei Zou and Tao Li and Xidao Luan},
  title = {{TFANet: three-stage image-text feature alignment network for robust referring image segmentation}},
  journal = {International Journal of Multimedia Information Retrieval},
  year = {2025},
}
Figure