Federated Learning for Vision-and-Language Grounding Problems

Fenglin Liu, Xian Wu, Shen Ge, Wei Fan, Yuexian Zou

Introduction

Vision-and-Language (V&L) grounding tasks, such as Image Captioning and Visual Question Answering (VQA), are fundamental to developing AI that can interpret the visual world through natural language. Traditionally, these models are trained in isolation or via centralized multi-task learning, which requires pooling massive datasets into a single location. However, this approach faces two significant challenges: privacy concerns regarding proprietary data and the logistical difficulty of aligning disparate datasets that may not share the same images or labels.

This paper, originally presented at AAAI-20, proposes a novel intersection of V&L grounding and Federated Learning (FL). The researchers introduce aimNet, a neural network architecture designed to bridge different vision-and-language tasks within a federated framework. By sharing learned "fine-grained image representations" rather than raw data, the framework enables collaborative learning while maintaining data privacy.

The aimNet Architecture

The core contribution of this work is the Aligning, Integrating, and Mapping Network (aimNet). Unlike standard feature extractors that provide generic visual data, aimNet is designed to produce task-agnostic, fine-grained representations that capture the semantic nuances of an image.

The aimNet architecture consists of Aligning, Integrating, and Mapping modules to produce fine-grained image representations from original visual features.

The architecture is composed of three primary modules:

  1. Aligning Module: This module utilizes a mutual attention mechanism to correlate visual regions (extracted via Faster R-CNN) with textual "semantic concepts" (e.g., "dog," "sitting," or "red"). This process establishes "semantic anchors," ensuring the model focuses on the most relevant parts of an image for linguistic description.
  2. Integrating Module: Using self-attention, this module explores the spatial and relational connections between different image regions. It learns to group related objects—such as a "person" and a "bike"—to form a holistic scene representation rather than a collection of isolated objects.
  3. Mapping Module: Recognizing that different tasks (like captioning vs. VQA) require different feature formats, this module uses non-linear layers to project the integrated representations into the specific format required by a task's decoder.

Federated Learning Frameworks

The authors evaluate aimNet across three distinct flavors of Federated Learning, demonstrating its versatility in real-world scenarios:

  • Horizontal FL: Applied when clients perform the same task but possess different datasets (e.g., two different organizations training image captioning models on their own private image sets).
  • Vertical FL: Used for different tasks on overlapping data (e.g., one client has captions for a set of images, while another has VQA pairs for the same set).
  • Federated Transfer Learning: The most flexible approach, where different tasks are performed on entirely different datasets. This allows a model to "transfer" visual wisdom from a large dataset to a smaller, unrelated one.

Key Results and Insights

The aimNet-based federated framework demonstrated "universal improvements" across all tested benchmarks, including MSCOCO, Flickr30k, and VQA v2.0.

One of the most striking findings was the "Small Dataset Advantage." The smaller Flickr30k dataset saw a 14% relative gain in the CIDEr metric and a 13% gain in SPICE when paired with the larger MSCOCO dataset via federated learning. This suggests that FL allows smaller datasets to effectively "borrow" the robust visual representations learned from larger, more diverse data pools without needing direct access to those images.

For VQA, the framework boosted the performance of state-of-the-art baselines (such as BUTD and BAN) by up to 3%. The results indicate that the semantic grounding provided by the Aligning Module is particularly critical for VQA, where the model must pinpoint specific details to answer questions accurately.

Qualitative comparison showing how individual aimNet modules contribute to more accurate captioning and VQA results compared to baselines.

As shown in the qualitative analysis, the inclusion of the Aligning and Integrating modules allows the model to identify missing elements (like a "pineapple") or correct misidentifications (changing "cup" to "red cups") that the baseline model misses. The "Full Model" consistently produces the most detailed and accurate descriptions and answers.

Conclusion

By moving the focus from raw data sharing to representation sharing, the aimNet framework provides a powerful solution for collaborative AI development. It achieves significant performance boosts while addressing the critical hurdles of data privacy and dataset mismatch. Future work may explore the computational efficiency of these updates and the application of this framework to other complex V&L tasks like image-text retrieval or visual storytelling.

Summary updated

Abstract

Recently, vision-and-language grounding problems, e.g., image captioning and visual question answering (VQA), has attracted extensive interests from both academic and industrial worlds. However, given the similarity of these tasks, the efforts to obtain better results by combining the merits of their algorithms are not well studied. Inspired by the recent success of federated learning, we propose a federated learning framework to obtain various types of image representations from different tasks, which are then fused together to form fine-grained image representations. The representations merge useful features from different vision-and-language grounding problems, and are thus much more powerful than the original representations alone in individual tasks. To learn such image representations, we propose the Aligning, Integrating and Mapping Network (aimNet). The aimNet is validated on three federated learning settings, which include horizontal federated learning, vertical federated learning, and federated transfer learning. Experiments of aimNet-based federated learning framework on two representative tasks, i.e., image captioning and VQA, demonstrate the effective and universal improvements of all metrics over the baselines. In image captioning, we are able to get 14% and 13% relative gain on the task-specific metrics CIDEr and SPICE, respectively. In VQA, we could also boost the performance of strong baselines by up to 3%.
BibTeX
@inproceedings{liu2020federated,
  author = {Fenglin Liu and Xian Wu and Shen Ge and Wei Fan and Yuexian Zou},
  title = {{Federated Learning for Vision-and-Language Grounding Problems}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6824},
}
Figure