FedTMOS: Efficient One-Shot Federated Learning with Tsetlin Machine

Shannon How Shi Qi, Jagmohan Chauhan, Geoff V. Merrett, Jonathon HareICLR 2025 · OpenReview

Overview

FedTMOS is a novel "one-shot" federated learning (OFL) framework that replaces resource-heavy Deep Neural Networks (DNNs) with logic-based Tsetlin Machines (TMs). By leveraging bitwise operations and a clever weight-reassignment strategy, it eliminates the need for expensive server-side training and public datasets. The result is a highly efficient system that achieves a 75x reduction in server latency and significantly better accuracy on heterogeneous data compared to current state-of-the-art methods.

The Bottleneck in One-Shot Federated Learning

Standard Federated Learning (FL) typically requires hundreds of communication rounds between a central server and decentralized edge devices (like phones or IoT sensors). This iterative process is slow, bandwidth-intensive, and risky in unstable network conditions.

One-Shot Federated Learning (OFL) aims to solve this by training models in a single communication round. However, existing OFL methods face two major bottlenecks:

  1. Data Dependency: Many rely on Knowledge Distillation (KD), which requires a "public" surrogate dataset at the server to help merge client models. In many real-world scenarios, such a dataset doesn't exist or poses severe privacy risks.
  2. Computational Cost: Methods that avoid public data often require heavy server-side training (such as generative models) or struggle to align models when clients have highly skewed, non-IID (Independent and Identically Distributed) data.

Furthermore, most FL research assumes clients can run power-hungry DNNs, which is often impractical for tiny, battery-powered edge devices.

The FedTMOS Approach

The authors introduce FedTMOS (One-Shot Federated Learning with Tsetlin Machine). Unlike DNNs that rely on complex calculus and floating-point weights, TMs use propositional logic and bitwise operations, making them incredibly efficient for hardware implementation.

The FedTMOS pipeline: (1) Client upload, (2) Weight clustering, and (3) Inter-class maximization to form global models.

The FedTMOS pipeline operates in three main steps to aggregate models without server-side training:

  1. Local Training & Gini Scaling: Clients train a TM locally for one round. Along with their model weights, they upload a Gini Index, which quantifies how skewed their local data distribution is. The server uses this index to rescale weights, ensuring that "majority" classes don't drown out "minority" classes during aggregation.
  2. Weight Clustering: The server uses k-means clustering to group similar weights from different clients. This reduces computational complexity and prevents large disparities in class weights.
  3. Inter-Class Maximization: Instead of simple averaging, the server greedily reassigns weights to form a set of global models. It specifically selects weights that maximize the distance between different classes. This creates "well-separated" models with distinct class boundaries, which is crucial for handling diverse, non-IID data.

Key Insights: Logic Over Calculus

By using Tsetlin Machines, the authors move the "intelligence" of the model from heavy matrix multiplications to simple bitwise logic. This represents a paradigm shift for FL on low-power devices.

Architecture of the Multi-Class Weighted Tsetlin Machine, highlighting the shift from floating-point calculus to boolean logic.

Because TMs have a transparent, class-specific structure, the server can easily "unzip" client models and "re-stitch" them based on class performance. This class-adaptive aggregation is much harder to achieve with the entangled, "black box" weights of a standard neural network. Consequently, FedTMOS achieves high performance without needing a surrogate public dataset or a generative model, making it highly practical for private or niche applications.

Performance and Results

The researchers evaluated FedTMOS on standard image benchmarks (MNIST, F-MNIST, SVHN, CIFAR-10) as well as more complex datasets (CIFAR-100, Tiny-ImageNet), simulating real-world difficulty using Dirichlet distributions to create highly heterogeneous data splits.

  • Accuracy: FedTMOS outperformed leading state-of-the-art OFL baselines (like Distilled-FedOV) by an average of 7.22% across various settings. It also beat its own "ensemble" version by 6.16%, proving the efficacy of the inter-class maximization strategy.
  • Latency: By avoiding backpropagation at the server, the "training" happens almost instantly through weight reassignment, achieving a staggering 75x reduction in server latency compared to methods requiring server-side training.
  • Communication: Upload costs were reduced by at least 2.3x compared to CNN-based methods, thanks to the compact, bit-based architecture of Tsetlin Machines.
  • Scalability: The system remained robust even as the number of clients scaled up to 80, maintaining a significant lead over traditional FedAvg and other one-shot methods.

Limitations and Future Directions

While FedTMOS is a significant leap forward for FL efficiency, the authors note a few hurdles. Currently, Tsetlin Machines struggle with high-resolution, multi-channel images compared to deep CNNs, as the "booleanization" process (converting raw images to bits) can lose fine-grained detail. Additionally, the current model uses fixed hyperparameters for the number of server models and clusters; future iterations could benefit from data-driven, dynamic sizing. Finally, unlike DNNs which leverage massive pre-trained foundation models, TMs are an emerging field with fewer pre-trained resources available.

Summary updated

Abstract

One-Shot Federated Learning (OFL) is a promising approach that reduce communication to a single round, minimizing latency and resource consumption. However, existing OFL methods often rely on Knowledge Distillation, which introduce server-side training, increasing latency. While neuron matching and model fusion techniques bypass server-side training, they struggle with alignment when heterogeneous data is present. To address these challenges, we proposed One-Shot Federated Learning with Tsetlin Machine (FedTMOS), a novel data-free OFL framework built upon the low-complexity and class-adaptive properties of the Tsetlin Machine. FedTMOS first clusters then reassigns class-specific weights to form models using an inter-class maximization approach, efficiently generating balanced server models without requiring additional training. Our extensive experiments demonstrate that FedTMOS significantly outperforms its ensemble counterpart by an average of $6.16$%, and the leading state-of-the-art OFL baselines by $7.22$% across various OFL settings. Moreover, FedTMOS achieves at least a $2.3\times$ reduction in upload communication costs and a $75\times$ reduction in server latency compared to methods requiring server-side training. These results establish FedTMOS as a highly efficient and practical solution for OFL scenarios.
BibTeX
@inproceedings{qi2025fedtmos,
  author = {Shannon How Shi Qi and Jagmohan Chauhan and Geoff V. Merrett and Jonathon Hare},
  title = {{FedTMOS: Efficient One-Shot Federated Learning with Tsetlin Machine}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=44hcrfzydU},
}
Figure