TL;DR
FedWQ-CP is a simple, one-shot federated uncertainty quantification (UQ) method designed to handle "dual heterogeneity"—the messy reality where agents have both different data distributions and different model architectures. By aggregating local quantile thresholds through a weighted average, it provides reliable prediction coverage across all agents while producing significantly smaller and more useful prediction sets than existing state-of-the-art methods.
What Problem Does This Paper Address?
In high-stakes fields like medical diagnostics, knowing when a model is unsure is just as important as the prediction itself. Federated Learning (FL) allows multiple institutions (like hospitals) to train models collaboratively without sharing sensitive data. However, FL faces a major hurdle in Uncertainty Quantification (UQ): Dual Heterogeneity.
- Data Heterogeneity: Different hospitals see different patient populations (distribution shift) and have varying amounts of data.
- Model Heterogeneity: Not every agent has the same hardware. A large research hospital might run a deep ResNet, while a rural clinic might use a shallow CNN.
Existing UQ methods usually tackle one of these in isolation. When both are present, models often become overconfident on "weak" agents (those with less data or smaller models), leading to "silent failures"—where the global system looks fine on paper, but individual agents are making dangerously confident mistakes.
Methods
The authors introduce FedWQ-CP (Federated Weighted Quantile Conformal Prediction). It builds on Conformal Prediction (CP), a framework that guarantees a model's prediction set will contain the true label with a user-defined probability (e.g., 95%).
The workflow is elegantly simple and requires only a single round of communication:
- Local Calibration: Each agent computes "non-conformity scores" (how "surprised" the model is by new data) on a local calibration set. They then calculate a local quantile threshold ($q_k$) that satisfies the coverage requirement for their specific model and data.
- One-Shot Communication: Each agent sends only two scalars to the server: their local threshold ($q_k$) and their calibration sample size ($n_k$). No raw data or model weights are ever shared.
- Weighted Aggregation: The server computes a global threshold ($\hat{q}$) by taking a weighted average of the local thresholds based on sample sizes: $\hat{q} = \sum (n_k/N) q_k$.
- Global Deployment: This global threshold is sent back to all agents to define their prediction sets or intervals.
Datasets
The researchers evaluated the method across seven diverse public datasets to simulate real-world variety:
- Standard Vision: MNIST (digits), FashionMNIST (apparel), and CIFAR-10 (natural images).
- Medical Imaging: DermaMNIST (skin lesions), BloodMNIST (blood cells), TissueMNIST (kidney cortex), and RetinaMNIST (retinal images).
To stress-test the system, they used a Dirichlet partition to induce label skew (data heterogeneity) and assigned agents different architectures—LargeCNN for "strong" agents and VeryWeakLinear models for "weak" agents—to simulate model heterogeneity.
Key Insights and Ideas
- Quantiles as a "Common Currency": Different model architectures produce raw scores on different scales (different "temperatures"). By converting these into quantiles locally, agents translate their internal uncertainty into a standardized rank-based scale that can be compared and aggregated.
- The Power of Weighting: The "aha moment" is that simple unweighted averaging fails. Weak agents with small datasets produce noisy, high-variance thresholds. Weighting by sample size ($n_k$) ensures the global boundary isn't disproportionately skewed by these unreliable local estimates, providing a robust "safety margin."
- Efficiency is King: In UQ, "coverage" (hitting the 95% target) is the bare minimum. The real goal is efficiency—producing the smallest possible prediction set that still meets that target. A model that says "the answer is either A, B, or C" is much more useful than one that says "the answer is anything from A to Z."
Key Results
- Reliable Coverage: FedWQ-CP consistently maintained agent-wise and global coverage near the 95% target across all seven datasets, whereas baselines like DP-FedCP often suffered from severe under-coverage.
- Superior Efficiency: FedWQ-CP produced the most compact prediction sets. For example, on the MNIST dataset, it achieved a prediction set size of 3.00, compared to 6.88 for SplitCP and 9.59 for FedCP-QQ.
- Computational Speed: Because it is a one-shot method, its runtime is comparable to the fastest existing baselines and significantly faster than iterative optimization-based approaches.
Limitations and Future Work
The authors acknowledge that because the quantile functional is non-linear, a weighted average of quantiles is technically a surrogate (heuristic) for the true mixture quantile, not a perfect mathematical match. While it performs exceptionally well empirically, there is a theoretical gap between this heuristic and a formal finite-sample guarantee under extreme heterogeneity.
Future work could explore more sophisticated aggregation functions or investigate how to adapt the threshold dynamically if the global population shifts significantly over time.
Summary updated