Collaborating Heterogeneous Natural Language Processing Tasks via Federated Learning

Chenhe Dong, Yuexiang Xie, Bolin Ding, Ying Shen, Yaliang LiSubmission to ICLR 2024 · Rejected

TL;DR

Standard federated learning (FL) assumes all clients share the exact same learning task—an unrealistic assumption when institutions have different downstream goals or want to keep their objectives proprietary. The Assign-Then-Contrast (ATC) framework breaks this constraint by pairing unified self-supervised pre-training with a privacy-preserving synthetic contrastive learning stage, enabling clients with completely heterogeneous NLP tasks (e.g., text classification and abstractive summarization) to collaborate effectively.

What Problem Does This Paper Address?

In traditional horizontal FL for NLP, participants must agree on a single, shared learning objective (such as sentiment analysis or machine translation). However, real-world data owners often have completely different applications for their text data (e.g., one firm extracts events from financial news, while another performs sentiment classification). Furthermore, competitive or privacy-sensitive entities (like hedge funds) cannot disclose their exact downstream objectives.

Standard FL algorithms like FedAvg collapse when applied across vastly different task objectives and loss functions, often performing worse than isolated, purely local training. ATC seeks to enable meaningful knowledge sharing among clients holding heterogeneous or secret NLP objectives without requiring centralized data pooling.

Methods

ATC uses a shared Transformer encoder-decoder architecture (initialized via BERT2BERT) as the common backbone across all clients. Clients with Natural Language Understanding (NLU) tasks keep private prediction heads on the encoder, while clients with Natural Language Generation (NLG) tasks use the full Seq2Seq pipeline.

Training proceeds in two complementary stages:

[Stage 1: ASSIGN]
Server assigns unified self-supervised tasks (MLM / Denoising Reconstruction)
           ↓
Agglomerative clustering on gradient updates groups clients by domain
           ↓
[Stage 2: CONTRAST]
Clients train on private local tasks + Contrastive loss on synthetic data
           ↓
Local encoder updates aggregated with top-K nearest gradient neighbors
  1. Assign Stage (Domain Knowledge Sharing):
  2. The central server assigns task-agnostic, self-supervised objectives—specifically Masked Language Modeling (MLM) and Denoising Reconstruction (DR).
  3. Because clients annotate their own data without external supervision, domain and vocabulary knowledge are integrated into the shared model without revealing private downstream tasks.
  4. The server performs hierarchical agglomerative clustering on model updates to group clients with similar text distributions before averaging.
  5. Contrast Stage (Task Alignment via Contrastive Learning):
  6. Clients switch to optimizing their own private downstream tasks.
  7. To align representations without an external public dataset, the server constructs a synthetic dataset: clients send mixed-up latent vectors from their encoders' top layers, which the server reconstructs into tokens using an MLM head.
  8. Clients pass this synthetic data through decoders and small MLP projection heads to yield summary representations ($h$).
  9. A contrastive loss pulls a client's representations closer to its $K$-nearest neighbors (measured by cosine similarity of model gradients) and pushes dissimilar clients away.
  10. During aggregation, clients only send encoder updates to their $K$-nearest neighbors, protecting decoders and task-specific heads from task leakage.

Datasets

The authors evaluate ATC using six standard benchmarks spanning both NLU and NLG across multiple domains:

  • Text Classification (NLU): IMDB (movie reviews, 1 client) and AGNews (news articles, 3 clients).
  • Question Answering (NLU): SQuAD (Wikipedia, 3 clients) and NewsQA (news, 2 clients).
  • Text Generation (NLG): CNN/DailyMail (abstractive summarization, 5 clients) and MSQG (question generation, 4 clients).

Across all 18 clients participating in a single federated learning course, evaluations use task-appropriate metrics: Classification Accuracy, Exact Match (EM) & F1 for QA, and ROUGE-1/2/L, BLEU-4, and METEOR for generation. No new benchmarks were introduced.

Key Insights and Ideas

  • Decoupling domain learning from task objectives: Self-supervised pre-training tasks serve as an effective "lingua franca" in FL, allowing models to learn vocabulary and syntax across domains before specializing on private objectives.
  • Privacy-preserving feature mixup replaces public proxy data: Contrastive FL typically requires a curated public dataset, which rarely matches client distributions. Generating synthetic probe datasets by mixing hidden encoder states circumvented the need for external data.
  • Task clustering emerges naturally: When contrastive learning was applied, NLG clients ranked other NLG clients among their top-10 nearest neighbors significantly more often than NLU clients, confirming that gradient similarity effectively discovers shared task mechanics without explicit labels.

Key Results

  • Consistent Superiority: ATC achieved an average score of 49.04% across all tasks, outperforming isolated local training (45.38%) by +3.66% and beating the strongest personalized FL baseline (PerCFL at 47.59%).
  • Outperforming Task-Separated FL: ATC outperformed FedAvg-ST (47.19%), where FL was artificially segregated into clients sharing the same task. This proves that cross-task transfer provided real value beyond training tasks in isolation.
  • Ablation Evidence:
  • Removing the Assign stage (w/o ASSIGN) dropped average performance to 48.26%.
  • Replacing the Contrast stage with standard FedAvg (w/o CONTRAST) reduced performance to 48.38%.
  • Combining both MLM and DR with client clustering provided cumulative gains (+2.43% over baseline).

Limitations and Future Work

  • Model Scale: Experiments were conducted using a compact backbone (BERT$_{\text{Tiny}}$). Scaling encoder-decoder architectures to modern LLMs (e.g., LLaMA, T5-Large) poses substantial memory and communication constraints.
  • Synthetic Data Reconstruction Quality: Relying on simple token reconstruction from mixed encoder states works for compact vocabularies, but could struggle with complex domain-specific terminology or long document contexts.
  • Privacy Trade-offs: While latent representation sharing is safer than raw text sharing, differential privacy guarantees or formal defenses against gradient inversion on the synthetic generation step were discussed conceptually rather than evaluated quantitatively.

Summary updated

Abstract

The increasing privacy concerns regarding personal private data promote the development of federated learning (FL) in recent years. However, the existing studies on applying FL in Natural Language Processing (NLP) are not suitable for coordinating participants with heterogeneous or private learning objectives. In this study, we further broaden the application scope of FL in NLP by proposing an Assign-Then-Contrast (ATC) framework, which enables clients with heterogeneous NLP tasks to construct an FL course and learn useful knowledge from each other. Specifically, clients are suggested to first perform local training with the unified tasks assigned by the server rather than using their own learning objectives, which is called the Assign training stage. After that, in the Contrast training stage, clients train with different local learning objectives and exchange knowledge with other clients who contribute consistent and useful model updates. We conduct extensive experiments on six widely-used datasets covering both Natural Language Understanding and Natural Language Generation tasks, showing that ATC framework achieves significant improvements compared to several baseline methods. We will release the source code for promoting further research.
BibTeX
@misc{dong2022collaborating,
  author = {Chenhe Dong and Yuexiang Xie and Bolin Ding and Ying Shen and Yaliang Li},
  title = {{Collaborating Heterogeneous Natural Language Processing Tasks via Federated Learning}},
  year = {2022},
  eprint = {2212.05789},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.05789},
}
Figure