Introduction
In the evolving landscape of artificial intelligence, data remains the primary fuel for innovation. However, for high-security institutions such as banks, hospitals, and government agencies, this data is often siloed within secure environments. While Federated Learning (FL) offers a paradigm for training models on decentralized data without direct movement, most existing research focuses on "cross-device" scenarios involving millions of edge devices.
This study shifts the focus to "cross-silo" or institutional FL, where a small number of organizations collaborate to build shared Natural Language Processing (NLP) models. The research specifically investigates the application of sample-level Differential Privacy (DP) to TextCNN models within this institutional framework.
Problem Statement
Standard FL is typically optimized for Business-to-Consumer (B2C) applications. Transitioning to Business-to-Business (B2B) cooperation introduces unique challenges:
- Privacy Risks: Gradient updates can inadvertently leak sensitive information about the underlying training data.
- Quantifiable Evaluation: There is a lack of empirical data on how privacy-preserving mechanisms, such as Differential Privacy, impact the performance of NLP models.
- Data Imbalance: Unlike the relatively uniform data distribution across mobile users, institutions often possess datasets of vastly different sizes, leading to potential model bias.
Methodology
The researchers implemented a TextCNN architecture on a GPU-enabled server cluster using the Federated Averaging (FedAvg) algorithm.
The technical core involves sample-level $(\epsilon, \delta)$-Differential Privacy. Unlike client-level privacy, which masks an institution's participation, sample-level privacy protects individual records within each silo. A Privacy Accountant was employed to monitor the cumulative privacy budget ($\epsilon$). If the loss exceeds a set threshold, training terminates to prevent leakage. Communication security was further bolstered using RSA encryption.
The study utilized the TREC question dataset for intent classification, containing 5,452 training samples across six categories. This allowed for controlled simulations of various silo configurations (2, 3, or 4 clients) and data distribution imbalances.
Empirical Analysis
The empirical findings highlight several critical factors in institutional FL:
- Sampling Ratio Impact: The sampling ratio ($q$)—the fraction of data used per training batch—emerged as a dominant factor. High sampling ratios led to a significant 38.4% drop in accuracy because they exhaust the privacy budget faster, forcing premature termination of training.
- Noise Robustness: Models demonstrated surprising resilience to the noise multiplier ($\sigma$). Variations in noise levels resulted in less than a 3% variance in test accuracy, suggesting that the model can tolerate gradient "fuzziness" better than it can tolerate shortened training durations.
- Data Imbalance: Non-uniform data distribution (e.g., a 1:2:4 ratio) caused performance drops of up to 10%. Smaller datasets reach convergence quickly and cease contributing, biasing the global model toward larger participants.
Key Results
The study established a centralized baseline accuracy of 91.2%. In a federated setting without privacy constraints, accuracy decreased slightly as the number of clients increased (90.0% for 2 clients vs. 86.4% for 4 clients).
With Differential Privacy enabled:
- A lot size ($L$) of 256 yielded a respectable 88.2% accuracy.
- Increasing the lot size to 1024 caused accuracy to plummet to 45.6%, as the privacy budget was depleted in only 27 communication rounds.
Conclusion
This research provides an empirical roadmap for institutional FL in NLP. While the study used the relatively small TREC dataset and TextCNN architecture, it identifies the sampling ratio and data imbalance as primary hurdles for cross-silo collaboration. Future work should extend these findings to larger-scale industrial datasets and more complex architectures like Transformers (e.g., BERT) to further bridge the gap between privacy theory and practical institutional AI.
Summary updated