Federated Bimodal Graph Neural Networks for Text-Image Retrieval

Xueming Yan, Chuyue Wang, Yaochu JinInternational Journal of Network Dynamics and Intelligence 2025 · DOI

Abstract

Text-image retrieval is a key challenge in computer vision and natural language processing, aiming to retrieve the most semantically relevant image or text given a query in the opposite modality. However, growing privacy and security concerns make traditional centralized learning approaches increasingly unsuitable for handling sensitive multimodal data. In this paper, we propose FedBi-GNNs, a federated learning framework for bimodal graph neural networks, which enables collaborative training across decentralized clients without sharing private data. Each client independently constructs heterogeneous graphs from local text and image data and learns correspondences via bimodal graph matching. These local representations are then aggregated at a central server using a heterogeneous federated aggregation scheme. Empirical results on the MSCOCO benchmark demonstrate that FedBi-GNNs significantly outperform existing state-of-the-art methods, offering improved retrieval accuracy, enhanced privacy preservation, and greater robustness to data heterogeneity across clients.

Introduction

Text-image retrieval—the task of finding the most relevant image for a text query or vice versa—is a cornerstone of modern AI. Traditionally, state-of-the-art performance has relied on centralized learning, where all data is pooled on a single server. This paradigm creates significant privacy risks and security concerns, particularly when handling sensitive user data.

Comparison between centralized learning and the proposed federated learning approach for text-image retrieval.

While Federated Learning (FL) offers a solution by keeping data on local devices, previous attempts at federated text-image retrieval often struggled with accuracy. These earlier models typically relied on "global feature vectors," which fail to capture the complex, fine-grained relationships in complex scenes. FedBi-GNNs aims to bridge this gap by combining privacy-preserving decentralized learning with high-accuracy, fine-grained semantic matching via graph neural networks.

Methodology

The FedBi-GNNs framework breaks the retrieval task into local graph construction and global parameter aggregation, ensuring that raw data never leaves the client device.

Detailed architecture of FedBi-GNNs, illustrating local graph construction, bimodal matching, and the federated aggregation process.

1. Heterogeneous Graph Construction

Clients independently convert their local data into graph structures to preserve spatial and semantic context:

  • Images: Each image is converted into a graph of 36 nodes representing salient regions (detected via Faster R-CNN). Spatial relationships are modeled using polar coordinates as edge weights.
  • Text: Sentences are processed via a Bi-GRU to create word-level nodes in a fully connected textual graph.

2. Bimodal Graph Matching

Instead of a simple dot product, the model uses a two-stage matching process:

  • Node-Level: A cross-attention mechanism aligns specific image regions with specific words.
  • Structure-Level: Graph Convolutional Networks (GCNs) propagate these node alignments to their neighbors, allowing the model to understand "phrases" and "sub-scenes" rather than just isolated objects.

3. Heterogeneous Federated Aggregation

The central server assigns weights to client updates based on both the quantity of data and the quality of local training results ($r_i$ score). This ensures that clients with more diverse or higher-quality data have a proportional influence on the global model.

Experimental Setup and Datasets

The model was evaluated on the MSCOCO benchmark, containing over 123,000 images. To simulate a realistic federated environment, the researchers introduced quantity skew across three clients (100%, 80%, and 60% of the training set respectively). This setup tests the model's robustness to the non-IID (Independent and Identically Distributed) data distributions typical of real-world federated settings.

Key Results

FedBi-GNNs demonstrated significant improvements over both centralized and federated baselines:

  • State-of-the-Art Performance: It outperformed SGRAF, the previous centralized state-of-the-art, by 0.6% on R@1 for image-to-text and 1% for text-to-image retrieval.
  • Federated Advantage: The federated approach achieved a 3.6% improvement in R@1 and an 8.1-point increase in rSum compared to training on a single client's dataset. This suggests that aggregating knowledge from diverse local distributions acts as a powerful regularizer.
  • Privacy without Performance Loss: By using GCNs to capture structural dependencies locally, the gradients sent to the server are highly informative for training but contain no raw data, maintaining high accuracy while preserving privacy.

Conclusion

FedBi-GNNs demonstrates that graph-based reasoning is particularly well-suited for federated settings. By matching local structures rather than global embeddings, the model is more robust to data heterogeneity. Future work will focus on scaling the framework to thousands of clients and optimizing the computational cost of region detection for mobile devices.

Summary updated

Abstract

Text-image retrieval is a key challenge in computer vision and natural language processing, aiming to retrieve the most semantically relevant image or text given a query in the opposite modality. However, growing privacy and security concerns make traditional centralized learning approaches increasingly unsuitable for handling sensitive multimodal data. In this paper, we propose FedBi-GNNs, a federated learning framework for bimodal graph neural networks, which enables collaborative training across decentralized clients without sharing private data. Each client independently constructs heterogeneous graphs from local text and image data and learns correspondences via bimodal graph matching. These local representations are then aggregated at a central server using a heterogeneous federated aggregation scheme. Empirical results on the MSCOCO benchmark demonstrate that FedBi-GNNs significantly outperform existing state-of-the-art methods, offering improved retrieval accuracy, enhanced privacy preservation, and greater robustness to data heterogeneity across clients.
BibTeX
@article{yan2025federated,
  author = {Xueming Yan and Chuyue Wang and Yaochu Jin},
  title = {{Federated Bimodal Graph Neural Networks for Text-Image Retrieval}},
  journal = {International Journal of Network Dynamics and Intelligence},
  year = {2025},
  doi = {10.53941/ijndi.2025.100009},
}
Figure