TL;DR
Standard federated learning shares model parameters, exposing private data to membership inference attacks and limiting models to gradient-based neural networks. Federated Co-Training (FedCT) eliminates parameter exchange by having clients share only discrete, hard pseudo-labels on a shared, public unlabeled dataset, which are aggregated via majority voting. This simple shift drastically reduces privacy vulnerability down to random-guess levels while matching standard federated learning accuracy and enabling collaborative training of non-differentiable models like decision trees and XGBoost.
What Problem Does This Paper Address?
Federated learning (FL) allows distributed clients (e.g., hospitals) to collaboratively train models without directly exchanging raw data. However, standard approaches like Federated Averaging (FedAvg) suffer from major drawbacks:
- Parameter Leakage: Sharing gradients or weights leaves models vulnerable to data reconstruction and membership inference attacks. Standard defenses like Differential Privacy (DP) via Gaussian noise often destroy model utility unless privacy guarantees are relaxed to near-meaningless levels.
- Model Inflexibility: FedAvg requires averaging parameters in a shared vector space. This excludes popular, highly interpretable non-neural architectures such as decision trees, random forests, and gradient-boosted trees (e.g., XGBoost).
- Soft-Label Vulnerabilities in Distributed Distillation: Alternative approaches that distill knowledge using public unlabeled data typically share continuous probability distributions (soft labels), which still leak sensitive information and require differentiable loss functions.
Methods
FedCT introduces an iterative consensus framework based on classical co-training and knowledge transfer:
- Hard-Label Consensus: Clients evaluate a shared, public unlabeled dataset $U$ using their current local models. Instead of sharing soft probability vectors, clients send hard discrete class predictions to the central server.
- Majority-Vote Aggregation: The server aggregates client predictions for each unlabeled sample via majority voting to generate a consensus pseudo-label.
- Local Data Augmentation: Each client combines its private labeled dataset with the newly pseudo-labeled public dataset $U$ and retrains its local model using any off-the-shelf supervised algorithm.
- Differential Privacy via the XOR-Mechanism: For formal $\epsilon$-differential privacy on binary one-hot prediction matrices, the authors utilize a matrix-valued Bernoulli XOR mechanism. They derive theoretical bounds on sensitivity based on algorithmic on-average-replace-one stability.
- Convergence Guarantees: The authors theoretically demonstrate that if local models achieve linearly increasing training accuracy over time, the consensus labels stabilize with high probability (bounded via the Hurwitz zeta function).
Datasets
The authors evaluated FedCT across multiple domains:
- Vision Benchmarks: CIFAR-10, FashionMNIST, and SVHN (Street View House Numbers).
- Medical Imaging:
- Pneumonia: Chest X-ray classification (healthy vs. pneumonia).
- Brain MRI: Scans for brain tumor detection.
- Tabular / Interpretable Model Benchmarks (UCI Repository): WineQuality, BreastCancer, AdultsIncome, Mushroom, and Covertype.
(No brand-new benchmarks were introduced; standard public sets were partitioned into private client subsets and shared unlabeled pools.)
Key Insights and Ideas
- Hard Labels Act as a Strong Privacy Filter: Discarding prediction confidence and outputting only the discrete top class strips away subtle gradient and margin signals that membership inference attacks exploit, while still conveying enough structural signal for co-training.
- Enabling True Model Heterogeneity: Because the aggregation interface is purely discrete labels over a shared dataset, clients do not need to share network architectures or even model families. One client can run an MLP, another an XGBoost classifier, and another a random forest.
- Consensus Mitigates DP Noise: In the consensus voting step, noise injected for differential privacy only flips the final label if it flips enough client votes to overturn the majority. This collective robustness yields strong empirical accuracy even under strict formal privacy budgets ($\epsilon \le 0.1$).
Key Results
- Privacy Defense: In membership inference attacks measured via ROC AUC vulnerability (where 0.5 is random guessing and 1.0 is full exposure), FedCT achieved 0.51–0.53 across all benchmark and medical image datasets. In contrast, FedAvg hovered between 0.71 and 0.76, and Distributed Distillation (DD) scored between 0.59 and 0.63.
- Competitive Accuracy: On CIFAR-10, FedCT matched FedAvg at 77% accuracy (DD achieved 67%). On FashionMNIST, FedCT reached 82% (vs. 83% for FedAvg and 69% for DP-FedAvg).
- Communication Efficiency: Communicating discrete labels over $|U| = 10,000$ samples reduced data transmission by approximately 67$\times$ compared to transferring the 669k parameters of a CNN in FedAvg.
- Interpretable Models: FedCT trained decision trees, RuleFit ensembles, XGBoost, and random forests across distributed clients, achieving accuracies matching centralized pooled training (e.g., XGBoost on WineQuality: 0.94 FedCT vs. 0.94 Centralized; BreastCancer: 0.93 FedCT vs. 0.94 Centralized).
Limitations and Future Work
- Reliance on Public Auxiliary Data: FedCT requires a sufficiently large, relevant unlabeled dataset $U$ accessible to all parties (or reliable synthetic data generators).
- Vulnerability to Pathological Non-IID Skew: While FedCT handles moderate label skew, performance degrades significantly under severe non-IID distributions (e.g., Dirichlet $\alpha = 0.01$, where clients observe only a small subset of classes). In this extreme case, accuracy dropped to 36.6% compared to 73.4% for FedAvg, because clients cannot form a coherent majority vote.
- Task Scope: The paper focuses on classification; extending FedCT to regression tasks will require continuous consensus mechanisms (e.g., medians, trimmed means) and revised privacy analyses.
Summary updated