Uncertainty-Based Extensible Codebook for Discrete Federated Learning in Heterogeneous Data Silos

Tianyi Zhang, Yu Cao, Dianbo Liu

TL;DR

UEFL (Uncertainty-Based Extensible-Codebook Federated Learning) introduces a dynamic approach to handling data heterogeneity in federated environments. By utilizing predictive uncertainty as a signal to "expand" a model's vocabulary (codebook) of discrete features, UEFL achieves up to a 22.1% boost in accuracy and a massive reduction in prediction instability compared to traditional federated learning methods.

The Challenge of Heterogeneous Data Silos

Federated Learning (FL) allows multiple parties (silos) to train a shared model collaboratively without ever exchanging raw, sensitive data. However, it faces a major hurdle known as the "Non-IID" (Non-Independent and Identically Distributed) problem. Data in one silo (e.g., a hospital in New York) might exhibit a vastly different distribution than data in another (e.g., a hospital in Tokyo).

When a global model attempts to aggregate updates from these diverse sources, it often performs poorly across the board—a classic "jack of all trades, master of none" scenario. Previous attempts to solve this utilized discrete representations, mapping complex continuous features to a fixed "dictionary" or codebook to help models generalize. Unfortunately, these fixed dictionaries struggle when they encounter a brand-new data distribution unrepresented in the original codebook, often requiring the entire model to be retrained from scratch.

Introducing UEFL: Growing the Model's Knowledge Base

To address the limitations of static codebooks, the authors propose UEFL, a framework that treats a model's "knowledge base" as an extensible entity that grows as it encounters novel data distributions.

UEFL Architecture

The UEFL framework operates on several key principles:

  1. Discrete Codebooks: Instead of passing continuous, noisy features directly through the network, UEFL maps them to the nearest "codewords" in a trainable dictionary. This acts as a filter, helping the model focus on essential, generalized patterns.
  2. Uncertainty as a Sensor: The system employs Monte Carlo (MC) Dropout to measure "predictive entropy." If a local silo's data produces high uncertainty, the system recognizes that the current codebook is insufficient for that specific data distribution.
  3. Extensible Growth: When high uncertainty is detected, UEFL does not simply retrain the existing parameters; it adds new codewords to the dictionary to accommodate the novel features.
  4. Smart Initialization (K-means): Rather than adding random new codewords, UEFL uses K-means clustering on the features of the "confusing" data to initialize the new codewords. This ensures the expanded dictionary is immediately relevant to the new data.
  5. Segmented Codebooks: To handle high-dimensional data without causing a massive computational explosion, the authors split feature vectors into smaller segments, each utilizing its own codeword mapping.

Datasets and Evaluation

The researchers evaluated UEFL across several standard computer vision benchmarks, modified to simulate "siloed" heterogeneity (e.g., using image rotations like 0°, -50°, and 120°):

  • MNIST & FMNIST: Basic grayscale digit and clothing images.
  • CIFAR-10 & CIFAR-100: More complex color object recognition.
  • GTSRB: German Traffic Sign Recognition Benchmark.
  • PACS & Rotated MNIST: Used specifically for Domain Generalization (DG) tasks, where the model is tested on entirely unseen styles (e.g., training on "Photos" and testing on "Sketches").

Key Insights and Results

A core philosophy of UEFL is that uncertainty is a feature, not a bug. While most researchers attempt to strictly minimize uncertainty, this paper leverages it as a triggering mechanism to decide exactly when and where the model needs to expand its capacity.

Static vs. Extensible Codebook Performance
  • The "Small Start" Advantage: The authors discovered that starting with a small codebook and growing it iteratively yields better results than starting with a massive, static one. A large static codebook often suffers from "codeword collapse," where many entries are never used or become redundant. The extensible approach ensures high utilization and better accuracy.
  • Significant Accuracy Boost: UEFL outperformed state-of-the-art methods (like DisTrans and FedAvg) by 3% to 22.1% across various datasets.
  • Enhanced Stability: Prediction uncertainty was slashed by 38.8% to 96.2%. This indicates the model is not only more accurate but also much more confident and consistent in its predictions.
  • Domain Generalization: In "leave-one-out" tests (training on some domains and testing on a completely new one), UEFL set new benchmarks, proving it adapts better to "unseen" data than existing algorithms like FedSR or FedIIR.
  • Efficiency: Despite the dynamic growth, the computational overhead remains surprisingly low—incurring only about a 3.3% increase in parameters and a 3.6% increase in runtime compared to standard FedAvg.

Limitations and Future Work

While highly effective, UEFL has a few areas ripe for future exploration:

  • Threshold Tuning: Performance heavily depends on the hyperparameter $\gamma$ (the uncertainty threshold). Setting it too low might cause the codebook to grow unnecessarily large, increasing computational costs.
  • Iteration Limits: The authors manually capped the number of expansion iterations (at 5). Developing an automated "stopping criterion" based on convergence would be a logical next step.
  • Label Heterogeneity: The current paper focuses heavily on feature heterogeneity (how the data looks). Further research is needed to evaluate how this extensible approach handles extreme label heterogeneity, where different silos contain entirely different classes of data.

Summary updated

Abstract

Federated learning (FL), aimed at leveraging vast distributed datasets, confronts a crucial challenge: the heterogeneity of data across different silos. While previous studies have explored discrete representations to enhance model generalization across minor distributional shifts, these approaches often struggle to adapt to new data silos with significantly divergent distributions. In response, we have identified that models derived from FL exhibit markedly increased uncertainty when applied to data silos with unfamiliar distributions. Consequently, we propose an innovative yet straightforward iterative framework, termed \emph{Uncertainty-Based Extensible-Codebook Federated Learning (UEFL)}. This framework dynamically maps latent features to trainable discrete vectors, assesses the uncertainty, and specifically extends the discretization dictionary or codebook for silos exhibiting high uncertainty. Our approach aims to simultaneously enhance accuracy and reduce uncertainty by explicitly addressing the diversity of data distributions, all while maintaining minimal computational overhead in environments characterized by heterogeneous data silos. Extensive experiments across multiple datasets demonstrate that UEFL outperforms state-of-the-art methods, achieving significant improvements in accuracy (by 3\%--22.1\%) and uncertainty reduction (by 38.83\%--96.24\%). The source code is available at https://github.com/destiny301/uefl.
BibTeX
@inproceedings{zhang2025uncertainty,
  author = {Tianyi Zhang and Yu Cao and Dianbo Liu},
  title = {{Uncertainty-Based Extensible Codebook for Discrete Federated Learning in Heterogeneous Data Silos}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=EU5lci90fF},
}
Figure