Collaborating Heterogeneous Natural Language Processing Tasks via Federated Learning

Chenhe Dong, Yuexiang Xie, Bolin Ding, Ying Shen, Yaliang Li

Federated Learning (FL) typically assumes that all participating clients share a common task objective. However, in real-world Natural Language Processing (NLP) scenarios, clients often possess diverse tasks ranging from text classification to document summarization. This paper introduces the ASSIGN-THEN-CONTRAST (ATC) framework, a novel approach designed to bridge "isolated data islands" characterized by task heterogeneity. ATC employs a two-stage training process: first, a unified self-supervised "assignment" stage to build a shared linguistic foundation, followed by a "contrastive" stage that aligns task-specific representations using synthetic data.

Introduction

Standard Federated Learning frameworks, such as FedAvg, are designed for scenarios where clients collaborate on a single global model for a uniform task. In practice, NLP applications are highly heterogeneous; one client may focus on sentiment analysis (NLU) while another performs question generation (NLG). Forcing these disparate objectives into a single global model often leads to performance degradation due to gradient conflicts and task-specific nuances. The ATC framework addresses this by allowing clients to maintain private task-specific decoders while collaboratively training a shared encoder backbone.

The ASSIGN-THEN-CONTRAST (ATC) Framework

The ATC framework utilizes an encoder-decoder architecture (e.g., BERT2BERT) and operates in two distinct phases to facilitate cross-task knowledge transfer.

Overview of the ATC framework stages.
  1. The ASSIGN Stage: To establish a common feature space, the server assigns unified self-supervised tasks—Masked Language Modeling (MLM) and Denoising Reconstruction (DR)—to all clients. This phase ignores local task labels to focus on extracting general linguistic patterns. To mitigate the effects of non-IID data distributions, the server employs agglomerative clustering, aggregating model updates only among clients with similar data characteristics.
  2. The CONTRAST Stage: After building a shared foundation, clients return to their original private tasks. To align their internal representations without sharing raw data, the framework introduces a contrastive learning mechanism.

Contrastive Learning via Synthetic Data

A key challenge in heterogeneous FL is facilitating representation alignment without compromising data privacy. ATC solves this by generating synthetic features at the server level.

Mechanism for synthetic data generation and contrastive alignment.

As shown in the figure above, the server synthesizes "feature-averaged" tokens by aggregating encoder outputs from multiple clients. These synthetic features serve as a privacy-preserving proxy for real data. During the contrastive stage, clients "pull" their model representations closer to "positive" neighbors (clients with consistent updates) and "push" away from "negative" ones. This process ensures that despite having different downstream objectives, the encoders learn to map similar linguistic concepts to similar regions in the latent space.

Evaluation and Results

The framework was evaluated using a suite of NLU tasks (IMDB, AGNews, SQuAD, NewsQA) and NLG tasks (CNN/DM, MSQG). Key findings include:

  • Performance Gains: ATC achieved an average improvement of 3.66% over isolated local training across all tasks.
  • Robustness to Heterogeneity: ATC outperformed personalized FL methods like Ditto and FedBN, which typically struggle when task objectives (not just data distributions) differ.
  • NLG Excellence: Significant improvements were noted in generation tasks, particularly MSQG, where the collaborative encoder provided a more robust structural understanding of language.
  • Ablation Study: Removing either the ASSIGN or CONTRAST stage resulted in a performance drop of approximately 0.7-0.8%, validating the necessity of both general foundation building and specific representation alignment.

Conclusion

The ATC framework represents a significant step toward flexible Federated Learning for NLP. By decoupling the shared linguistic encoder from private task-specific decoders and using a two-stage "assign-then-contrast" strategy, it enables effective collaboration across diverse applications. This approach not only improves model performance but also respects the privacy of local task objectives and data.

Summary updated

Abstract

The increasing privacy concerns regarding personal private data promote the development of federated learning (FL) in recent years. However, the existing studies on applying FL in Natural Language Processing (NLP) are not suitable for coordinating participants with heterogeneous or private learning objectives. In this study, we further broaden the application scope of FL in NLP by proposing an Assign-Then-Contrast (ATC) framework, which enables clients with heterogeneous NLP tasks to construct an FL course and learn useful knowledge from each other. Specifically, clients are suggested to first perform local training with the unified tasks assigned by the server rather than using their own learning objectives, which is called the Assign training stage. After that, in the Contrast training stage, clients train with different local learning objectives and exchange knowledge with other clients who contribute consistent and useful model updates. We conduct extensive experiments on six widely-used datasets covering both Natural Language Understanding and Natural Language Generation tasks, showing that ATC framework achieves significant improvements compared to several baseline methods. We will release the source code for promoting further research.
BibTeX
@misc{dong2022collaborating,
  author = {Chenhe Dong and Yuexiang Xie and Bolin Ding and Ying Shen and Yaliang Li},
  title = {{Collaborating Heterogeneous Natural Language Processing Tasks via Federated Learning}},
  year = {2022},
  eprint = {2212.05789},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.05789},
}
Figure