Towards Instance-adaptive Inference for Federated Learning

Chun-Mei Feng, Kai Yu, Nian Liu, Xinxing Xu, Salman Khan, Wangmeng Zuo

Abstract

Federated Learning (FL) aims to train robust global models across decentralized devices while maintaining data privacy. While significant research has focused on inter-client heterogeneity (the variance of data distributions between users), the challenge of intra-client heterogeneity—where a single user's data contains multiple, diverse sub-domains—remains largely unaddressed. This paper introduces FedIns, a novel framework designed for Federated Instance-adaptive inference. By utilizing a pool of lightweight "Scale and Shift" (SSF) parameters and a dynamic routing mechanism, FedIns allows models to adapt to individual data instances on the fly. Experimental results demonstrate that FedIns achieves up to a 7.77% accuracy improvement over traditional FL methods while reducing communication bandwidth requirements by 85%.

1. Introduction

In standard Federated Learning (FL) paradigms, the primary objective is to mitigate the performance degradation caused by non-I.I.D. (independent and identically distributed) data across different clients. However, real-world data collection often introduces "intra-client heterogeneity." For instance, a single medical imaging device may process scans from various manufacturers, or a mobile photo gallery may contain a mixture of sketches, photographs, and digital art.

Traditional FL algorithms typically attempt to learn a single global model or a single personalized model per client. These approaches often fail when a client's local data is highly diverse, as the model essentially tries to "average" across conflicting data distributions. To solve this, we propose FedIns, which shifts the focus from client-level adaptation to instance-level adaptation.

2. The FedIns Framework

FedIns introduces a parameter-efficient approach to handle complex, mixed-style data within individual devices. The framework is built upon two core components: Parameter-Efficient Fine-Tuning (PEFT) and an Instance-Adaptive Inference mechanism.

2.1 Parameter-Efficient Fine-Tuning via SSF

Rather than updating the entire weight matrix of a deep neural network—which is computationally expensive and communication-heavy—FedIns employs Scale and Shift deep Features (SSF). SSF modulates the features of a pre-trained backbone model using a minimal set of parameters. This allows the model to be "nudged" toward specific data distributions without the need for full-model fine-tuning.

2.2 The SSF Pool and Dynamic Routing

Instead of maintaining a single set of SSF parameters, each client in FedIns manages an SSF Pool. When the model encounters a new input instance, it performs the following steps:

  1. Query Generation: The pre-trained backbone generates a unique query for the specific input image.
  2. Key Matching: This query is compared against a set of "keys" associated with the SSF subsets in the pool.
  3. Instance-Adaptive Aggregation: The model selects and aggregates the best-matched SSF subsets to create a custom, instance-specific model for that single inference task.

This dynamic routing ensures that a "sketch" image is processed by parameters optimized for sketches, while a "photograph" in the same dataset is processed by parameters optimized for realism.

3. Experimental Evaluation

3.1 Datasets and Setup

The authors evaluated FedIns against three major benchmarks modified to simulate both inter- and intra-client diversity:

  • CIFAR-100 & Tiny-ImageNet: Used to evaluate performance under "Label Shift," where class distributions vary across clients.
  • DomainNet: A large-scale dataset featuring six distinct domains (Clipart, Infograph, Painting, Quickdraw, Real, and Sketch), used to simulate "Feature Shift."

3.2 Performance and Efficiency

FedIns was compared against state-of-the-art FL algorithms, including FedAvg, FedProx, and FedBN. The results highlight three key advantages:

  • Superior Accuracy: On the Tiny-ImageNet benchmark, FedIns achieved a 7.77% accuracy boost over the baseline FedAvg. It consistently outperformed competitors across all tested levels of data heterogeneity.
  • Communication Efficiency: By only transmitting the lightweight SSF parameters (approximately 5.35 million parameters) rather than the full model (85.8 million parameters), FedIns reduces communication costs by over 85%.
  • Robustness: As data became increasingly heterogeneous, the performance of traditional algorithms declined sharply. In contrast, FedIns maintained stable performance, proving its resilience to "messy" real-world data.

4. Discussion and Limitations

While FedIns represents a significant step forward, certain considerations remain for deployment. The performance of the system is sensitive to hyperparameters, specifically the size of the SSF pool ($M$) and the number of subsets selected for aggregation ($C$). Additionally, while communication costs are low, the instance-specific matching process introduces a small amount of computational overhead during inference, which may be a factor for ultra-low-power edge devices.

5. Conclusion

FedIns addresses the neglected challenge of intra-client heterogeneity in Federated Learning. By moving toward instance-adaptive inference, the framework provides a scalable, efficient, and highly accurate solution for decentralized learning on diverse datasets. Future work will focus on extending this instance-adaptive logic to other domains, such as Natural Language Processing and time-series analysis.

Summary updated

Abstract

Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. However, the performance of the global model is often hampered by non-i.i.d. distribution among the clients, requiring extensive efforts to mitigate inter-client data heterogeneity. Going beyond inter-client data heterogeneity, we note that intra-client heterogeneity can also be observed on complex real-world data and seriously deteriorate FL performance. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Instead of huge instance-adaptive models, we resort to a parameter-efficient fine-tuning method, i.e., scale and shift deep features (SSF), upon a pre-trained model. Specifically, we first train an SSF pool for each client, and aggregate these SSF pools on the server side, thus still maintaining a low communication cost. To enable instance-adaptive inference, for a given instance, we dynamically find the best-matched SSF subsets from the pool and aggregate them to generate an adaptive SSF specified for the instance, thereby reducing the intra-client as well as the inter-client heterogeneity. Extensive experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
BibTeX
@inproceedings{feng2023towards,
  author = {Chun-Mei Feng and Kai Yu and Nian Liu and Xinxing Xu and Salman Khan and Wangmeng Zuo},
  title = {{Towards Instance-adaptive Inference for Federated Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure