Federated Learning for Inference at Anytime and Anywhere

Zicheng Liu, Da Li, Javier Fernandez-Marques, Stefanos Laskaridis, Yan Gao, Łukasz Dudziak, Stan Z. Li, Shell Xu Hu, Timothy Hospedales

TL;DR

This paper introduces the Accumulator, a lightweight, parameter-efficient adapter that seamlessly integrates pre-trained Transformer foundation models into federated learning (FL). By training only this small module, the framework drastically slashes communication overhead and handles severe data heterogeneity while enabling flexible, "anytime" early-exit inference for resource-constrained edge devices.

What Problem Does This Paper Address?

Federated Learning (FL) allows decentralized devices to collaboratively train machine learning models without sharing their private local data. However, traditional FL faces three massive bottlenecks:

  1. Communication Cost: Exchanging millions of weights between clients and a central server over wireless networks is slow and expensive.
  2. Data Heterogeneity (Non-IID Data): Different users have vastly different data distributions, which often degrades global model convergence and accuracy.
  3. System Heterogeneity: Edge devices have highly diverse hardware capabilities; a weak smartphone or a device with a low battery cannot run the same heavy model as a high-end tablet.

While leveraging large, centrally pre-trained foundation models (like Vision Transformers) can mitigate data heterogeneity, fine-tuning the entire model on-device remains prohibitively expensive. This paper tackles the challenge of adapting these massive foundation models to FL efficiently, ensuring they can run anytime (under varying compute/battery budgets) and anywhere (regardless of data distributions).

Methods

Instead of fine-tuning the entire pre-trained Transformer, the authors freeze the foundation model and inject a novel, lightweight adapter called the Accumulator.

Figure 2: Model architecture of the proposed Accumulator. Given a frozen Transformer FM (in grey), the Accumulator aggregates the information from the history of the CLS tokens through the transformation, which yields a better feature representation for early exits at different layers. The early exit slots are numbered (there are three exits in this example), and the corresponding path- ways are denoted by different line types (e.g., dotted for exit 0 and dashed for exit 1). A residual connection is introduced to each early exit between the CLS token at the exit and the output of the Accumulator. A special CLS token (or referred to as Client token) is introduced to be the first token of the Accumulator, which can be tuned for personalization while freezing the other parts.

Key components of the architecture include:

  • The Accumulator Module: A tiny, single-block Transformer that tracks and aggregates the history of the classification (CLS) tokens across all self-attention blocks. Instead of treating each layer independently, it uses past token representations to refine the current layer's features.
  • Early Exits (Anytime Inference): A shared Multi-Layer Perceptron (MLP) head is attached across layers. If a device is running low on battery or has weak compute power, it can halt processing at an earlier layer and output a prediction immediately.
  • Client Token Personalization: The authors introduce a unique Client token ($z_{client}$) at the beginning of the Accumulator sequence. For local personalization, clients only need to tune this single token while keeping the rest of the network frozen.

Datasets

The authors evaluated their approach on three standard FL benchmarks adapted for downstream tasks:

  • CIFAR-100: Evaluated under both Independent and Identically Distributed (IID) and highly non-IID settings using Latent Dirichlet Allocation (LDA) partitioning ($\alpha = 1.0, 0.1$).
  • FEMNIST: A handwritten character dataset containing 80,000 grayscale images across 62 classes, partitioned into IID and non-IID client splits.
  • SpeechCommandsv2: An audio dataset consisting of 1-second clips of spoken words, utilizing a 1-class setup to test the framework on non-visual modalities.
  • CIFAR-100-C & Corrupted SpeechCommandsv2: Used specifically to evaluate model personalization against real-world data corruptions and background noise.

Key Insights and Ideas

  • Leveraging Token History: Traditional early-exit networks append independent classifiers to intermediate layers, which often perform poorly because early layers lack high-level semantic context. The Accumulator fixes this by passing historical token information forward, significantly boosting early-exit accuracy.
  • Ultra-Lean Personalization: Instead of updating millions of parameters to personalize a model for a specific user, updating just the Client token (a mere 0.38K parameters) yields superior robustness against local data corruptions.
  • Complementary Efficiency: The Accumulator works synergistically with existing parameter-efficient methods like Parallel Adapters (PA), allowing the system to match or even exceed full-model fine-tuning accuracy at a fraction of the cost.

Key Results

  • Drastic Communication Savings: As shown in the convergence plots, the Accumulator reaches target accuracy with orders of magnitude less communication volume than full fine-tuning or standard layer-wise MLPs.
Figure 1: The effectiveness of Accumulator-based federated FM adaptation under anytime and any- where setting in terms of communication cost and classification performance. The experiments are conducted on CIFAR-100. Full details can be found in Section 5. Each point corresponds to an evaluation during FL, where the cumulative communication cost measures the communication of gradients w.r.t. model parameters between clients and the server. We would like to emphasize that our Accumulator a) converges faster (less communication cost) than the baselines regardless of the data heterogenity condition and b) performs as well as the upper bound – full-model fine-tuning.
  • Maintained Accuracy: On CIFAR-100 (non-IID, $\alpha=0.1$), the Accumulator combined with Parallel Adapters achieved 84.02% accuracy, virtually matching full fine-tuning (84.25%), despite updating only a tiny fraction of the parameters (3.17M vs 30.62M).
  • Superior Anytime Performance: In multi-tier system heterogeneity tests (where clients are forced to exit early due to hardware constraints), the Accumulator consistently outperformed standard early-exit baselines by 20% to 30% absolute accuracy in earlier layers.
  • Personalization Champion: When faced with corrupted data (CIFAR-100-C), tuning only the Client Token achieved an accuracy of 45.38% (IID) and 47.02% (Non-IID), significantly outperforming full-model fine-tuning (34.30% and 37.75% respectively).

Limitations and Future Work

  • Architecture Dependency: The Accumulator relies heavily on the presence of a explicit CLS token pathway, making it natively suited for Transformers (like ViT, DeiT, and AST). Adapting this specific mechanism to pure Convolutional Neural Networks (CNNs) remains an open question.
  • Alignment with Pre-training: The ablation studies reveal that performance depends heavily on the foundation model having strong, pre-existing feature representations (e.g., pre-trained on ImageNet). If the downstream FL task is completely decoupled from the foundation model's pre-training domain, the performance gains might decrease.

Summary updated

Abstract

Federated learning has been predominantly concerned with collaborative training of deep networks from scratch, and especially the many challenges that arise, such as communication cost, robustness to heterogeneous data, and support for diverse device capabilities. However, there is no unified framework that addresses all these problems together. This paper studies the challenges and opportunities of exploiting pre-trained Transformer models in FL. In particular, we propose to efficiently adapt such pre-trained models by injecting a novel attention-based adapter module at each transformer block that both modulates the forward pass and makes an early prediction. Training only the lightweight adapter by FL leads to fast and communication-efficient learning even in the presence of heterogeneous data and devices. Extensive experiments on standard FL benchmarks, including CIFAR-100, FEMNIST and SpeechCommandsv2 demonstrate that this simple framework provides fast and accurate FL while supporting heterogenous device capabilities, efficient personalization, and scalable-cost anytime inference.
BibTeX
@misc{liu2022federated,
  author = {Zicheng Liu and Da Li and Javier Fernandez-Marques and Stefanos Laskaridis and Yan Gao and Łukasz Dudziak and Stan Z. Li and Shell Xu Hu and Timothy Hospedales},
  title = {{Federated Learning for Inference at Anytime and Anywhere}},
  year = {2022},
  eprint = {2212.04084},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.04084},
}
Figure