Bidirectional Contrastive Split Learning for Visual Question Answering

Yuwei Sun, Hideya OchiaiAAAI 2024 · DOI

Abstract

Visual Question Answering (VQA) typically relies on massive, centralized datasets, which poses significant privacy and security risks. This paper introduces Bidirectional Contrastive Split Learning (BiCSL), a novel decentralized framework designed to protect user privacy by partitioning model components between clients and servers. By leveraging self-supervised contrastive learning and semantic answer projections, BiCSL achieves performance competitive with centralized training while offering significantly higher resilience against adversarial "backdoor" attacks. Furthermore, its bidirectional architecture addresses the efficiency bottlenecks inherent in traditional split learning paradigms.

Introduction

Visual Question Answering (VQA) is a complex multi-modal task that requires the integration of visual and textual information. Traditionally, training these models requires centralizing vast amounts of user data, raising serious privacy concerns. While Decentralized Machine Learning, such as Federated Learning (FL), keeps data on local devices, it often requires sharing the entire model architecture with clients, making the system vulnerable to adversarial "backdoor" or Trojan attacks. Additionally, the high computational cost of large-scale multi-modal models often exceeds the resource constraints of mobile or IoT client devices.

Methods: The BiCSL Framework

The authors propose Bidirectional Contrastive Split Learning (BiCSL) to decouple the VQA model into three primary segments, effectively creating a privacy barrier between the user and the server.

The BiCSL framework architecture, illustrating the division between client-side representation modules and server-side adapter networks, along with the bidirectional flow of activations and gradients.

Architectural Components

  • Client-Side Components: These include local representation modules such as Vision Encoders, Text Encoders, and Multi-Head Attention (MHA) mechanisms. These modules process raw images and questions locally, ensuring that sensitive raw data never leaves the client device.
  • Server-Side Components: Adapter networks, specifically the Nonlinear Head Adapter (NHA) and Linear Tail Adapter (LTA), handle the computationally intensive task of high-level feature alignment.
  • Answer Projection Network (APN): A key innovation of BiCSL is the APN. Rather than treating answers as discrete numeric IDs, the APN uses GloVe embeddings to project lexical answers into a semantic feature space. This allows the model to understand linguistic nuances, recognizing that semantically similar answers (like "sofa" and "couch") should have similar representations.

Bidirectional Innovation and Contrastive Learning

Unlike traditional Split Learning, which often suffers from sequential processing bottlenecks where the client and server must wait for each other, BiCSL enables more concurrent processing. It utilizes a bidirectional flow where all layer activations are sent to the server and gradients are returned to the client simultaneously.

A conceptual comparison between traditional Split Learning (a), which is often sequential, and the proposed BiCSL framework (b), which incorporates contrastive learning and semantic notions.

The framework employs InfoNCE (Information Noise Contrastive Estimation) loss. Instead of a standard classification task, the model learns to maximize the similarity between "positive" pairs (the correct image-question-answer triplet) and minimize it for "negative" pairs within a batch. This self-supervised approach removes the need for clients to share explicit labels in a traditional supervised manner.

Datasets and Evaluation

The researchers evaluated BiCSL using the VQA-v2 dataset, a standard benchmark containing 82.8k images and 443.8k questions for training, with images sourced from the COCO dataset. To simulate a decentralized environment, the training set was partitioned into non-overlapping subsets distributed across simulated clients.

Key Insights and Results

BiCSL demonstrates that decentralization does not have to come at the cost of performance or security:

  • Competitive Accuracy: BiCSL achieved an overall accuracy of 49.89%, remarkably close to the 53.82% achieved by a centralized version of the same model. Statistical testing confirmed that this minor gap does not represent a significant degradation in performance.
  • Superior Adversarial Robustness: Under a sophisticated "dual-key Trojan attack," centralized models and standard Split Learning saw their accuracy plummet toward zero. In contrast, BiCSL maintained approximately 38.5% overall accuracy. This resilience is attributed to the split architecture; because no single entity has access to the full model, it is significantly harder for an attacker to generate effective triggers.
  • Efficiency: The bidirectional flow addresses the "idle time" bottleneck of standard Split Learning, making the framework more practical for real-world IoT and mobile deployments.

Conclusion

BiCSL represents a significant step forward in secure, decentralized multi-modal learning. By combining model splitting with contrastive learning and semantic answer projections, it provides a robust defense against adversarial attacks while maintaining high utility. Future work will focus on narrowing the remaining performance gap and incorporating Differential Privacy to further protect the gradients shared during training.

Summary updated

BibTeX
@inproceedings{sun2024bidirectional,
  author = {Yuwei Sun and Hideya Ochiai},
  title = {{Bidirectional Contrastive Split Learning for Visual Question Answering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i19.30158},
}
Figure