Uncertainty-Based Extensible Codebook for Discrete Federated Learning in Heterogeneous Data Silos

Tianyi Zhang, Yu Cao, Dianbo Liu

Federated Learning (FL) enables multiple parties—or "silos"—to collaboratively train a shared model without exchanging raw data. However, a significant hurdle remains: data heterogeneity. When data distributions vary significantly across silos (e.g., medical images from different hospitals with varying equipment), a standard global model often struggles to generalize, resulting in a "jack of all trades, master of none" scenario. This paper introduces UEFL (Uncertainty-Based Extensible-Codebook Federated Learning), a framework that dynamically adapts its feature representation to handle diverse data distributions.

Methodology: The Extensible Codebook

UEFL addresses heterogeneity by treating the model’s feature "vocabulary" as a dynamic entity. Instead of a static set of discrete representations, the framework utilizes an Extensible Codebook. The system monitors prediction uncertainty to identify when the model encounters unfamiliar data distributions, triggering the addition of specialized codewords.

The UEFL architecture: Distributed data is processed through an encoder, mapped to shared or extended codebooks based on uncertainty, and then classified. An uncertainty evaluator provides feedback to trigger codebook expansion.

Key Components

  1. Uncertainty Evaluation via MC Dropout: UEFL uses Monte Carlo (MC) Dropout to measure prediction variance (entropy). High entropy signals that the model is "confused" by a silo's specific distribution, prompting a codebook expansion.
  2. K-means Initialization: To ensure rapid convergence, new codewords are initialized using K-means clustering on features from the current encoder, rather than being assigned random values.
  3. Segmented Codebooks: To handle complex datasets like CIFAR-100, UEFL divides feature vectors into smaller segments, each mapped to its own codeword. This exponentially increases representational capacity with minimal computational overhead.

Key Results and Insights

The researchers evaluated UEFL across several benchmarks, including MNIST, CIFAR-10/100, and GTSRB, intentionally introducing heterogeneity through image rotations. UEFL demonstrated substantial improvements over standard baselines like FedAvg.

  • Accuracy and Uncertainty: UEFL achieved accuracy boosts ranging from 3% to 22.1%. On CIFAR-100, it reached a top-1 accuracy of 0.326, compared to FedAvg’s 0.088. Simultaneously, it reduced mean entropy (uncertainty) by 38.83% to 96.24%.
  • Ablation Studies: The authors confirmed that starting with a small, growing codebook is superior to starting with a large static one, as large static codebooks introduce noise that hinders early training. Furthermore, the K-means initialization was shown to be critical for performance.
Ablation studies showing (a) the benefit of discretization (VQ-FedAvg), (b) the superiority of extensible codebooks over static ones, and (c) the significant impact of proper codebook initialization on accuracy and perplexity.

Conclusion and Future Work

UEFL provides a robust mechanism for privacy-preserving specialization in federated environments. By using uncertainty as a functional trigger for model growth, it maintains high performance across heterogeneous sites without leaking raw data. Future research will explore the framework's sensitivity to the expansion threshold ($\gamma$) and its applicability to other data modalities, such as Natural Language Processing (NLP) and time-series data.

Summary updated

BibTeX
@inproceedings{zhang2025uncertainty,
  author = {Tianyi Zhang and Yu Cao and Dianbo Liu},
  title = {{Uncertainty-Based Extensible Codebook for Discrete Federated Learning in Heterogeneous Data Silos}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=EU5lci90fF},
}
Figure