FedSM: Robust Semantics-Guided Feature Mixup for Bias Reduction in Federated Learning with Long-Tail Data

Jingrui Zhang, Yimeng Xu, Shujie Li, Feng Liang, Haihan Duan, Yanjie Dong, Victor C. M. Leung, Xiping Hu

TL;DR

Federated Learning often struggles with long-tailed (highly imbalanced) data distributed across clients, leading to severely biased global models. FedSM solves this by leveraging a pretrained image-text foundation model (like CLIP) to guide feature-space data augmentation, mixing only semantically related classes to generate high-quality pseudo-features. This client-centric approach corrects classifier bias locally, outperforming state-of-the-art methods with minimal computational and communication overhead.

What Problem Does This Paper Address?

Federated Learning (FL) allows multiple decentralized clients to collaboratively train a shared machine learning model without exposing their raw private data. However, real-world data is rarely distributed evenly. Instead, it typically follows a long-tail distribution, where a few "head" classes contain the vast majority of the samples, while many "tail" classes are severely underrepresented or entirely missing on certain clients. This data imbalance heavily biases the global model's classifier toward the dominant head classes.

To fix class imbalances in centralized learning, researchers often use Mixup—a technique that interpolates between training samples to smooth out decision boundaries. However, blindly applying Mixup in a federated setting can backfire. If a client indiscriminately mixes features from completely unrelated categories (e.g., a squirrel and a train) simply because those are the samples it has on hand, it generates unrealistic pseudo-features. These misleading samples distort decision boundaries rather than refining them.

Figure 1: Illustration of Mixup problems. (a) Random Mixup ignores semantic relevance between categories and may blend unrelated samples, such as squirrel and train, across boundaries, producing synthetic data that misguides bound- ary refinement. (b) When the mountain category has signif- icantly fewer samples or the pine tree category is absent, random Mixup has a higher chance of generating unrepre- sentative or even misleading synthetic samples.

Methods

The authors introduce FedSM (Robust Semantics-Guided Feature Mixup), a client-centric framework designed to address classifier bias locally without incurring heavy server overhead or risking data privacy. The method operates in three distinct phases on the client side:

  1. Local Training with Knowledge Distillation: Clients train their local models normally. To enhance the feature extractor's ability to align image features with broader concepts, FedSM introduces a teacher-student framework using knowledge distillation from a frozen, pretrained image-text model (such as CLIP).
  2. Semantics-Guided Feature Mixup: Instead of randomly pairing features, FedSM uses the foundation model's text encoder to calculate semantic similarity scores between class labels. To generate a pseudo-feature for a rare tail class, it mixes the global prototype of that class with a local feature from a semantically relevant class (e.g., mixing a squirrel with a mountain background rather than a train). It employs a probabilistic category selection strategy to introduce controlled randomness, which boosts feature diversity and protects against domain shifts.
  3. Lightweight Classifier Retraining: Because classification bias primarily resides in the final classifier layer rather than the feature extractor, FedSM retrains only the classifier locally using the balanced, semantically consistent pseudo-features. Crucially, this retraining is deferred to the final few communication rounds of FL, drastically reducing computational costs.
Figure 2: Overview of the FedSM framework. The client side consists of three key phases: a) local training, b) label relevance-guided feature mixup, and c) classifier retraining.

Datasets

The authors evaluated FedSM on three widely recognized long-tail benchmarks:

  • CIFAR-10-LT & CIFAR-100-LT: Created by downsampling standard CIFAR datasets using varying Imbalance Factors (IF = 100, 50, and 10), where an IF of 100 means the most frequent class has 100x more samples than the rarest.
  • ImageNet-LT: A large-scale long-tailed subset of ImageNet containing 115.8K images across 1,000 categories.

To simulate real-world data heterogeneity across decentralized clients, the datasets were distributed using Dirichlet partitioning ($\alpha = 0.5$ for CIFAR, $\alpha = 0.1$ for ImageNet).

Key Insights and Ideas

  • Semantic Proximity Prevents Boundary Distortion: Guiding feature mixup via text-derived semantic relevance ensures that synthetic features land in meaningful regions of the feature space, preventing the generation of confusing "hybrid" anomalies.
  • The Power of Probabilistic Selection: Instead of always pairing a class with its absolute closest semantic match (deterministic selection), picking pairs probabilistically expands feature diversity. This simple twist makes the framework highly robust to domain shifts between the pretrained foundation model and the local target data.
  • Late-Stage Calibration Saves Compute: Unlike previous state-of-the-art methods that run expensive optimization or retraining steps during every single federated round, FedSM proves that calibrating the classifier only in the final 50 rounds is sufficient to achieve superior accuracy.

Key Results

  • CIFAR Benchmarks: FedSM consistently outperformed all baseline algorithms across all imbalance factors. On CIFAR-100-LT, it achieved a top-1 accuracy improvement of up to 1.9 percentage points over the next-best method, benefiting from CIFAR-100's finer-grained labels which maximize the utility of semantic guidance.
  • ImageNet-LT Performance: On the highly complex ImageNet-LT dataset, FedSM achieved an overall top-1 accuracy of 30.9%, outperforming the previous state-of-the-art method (CLIP2FL at 27.5%) by 3.4 percentage points.
  • Robustness to Domain Shift: When replacing CLIP with MetaCLIP (simulating a shift in the underlying foundation model), FedSM maintained highly competitive performance, proving it doesn't over-rely on a single pretrained model's distribution.

Limitations and Future Work

While FedSM is highly efficient during the classifier retraining phase, it still requires clients to host a pretrained image-text foundation model (like CLIP) locally to perform knowledge distillation and extract text semantics. For extremely resource-constrained edge devices, the memory footprint of these foundation models could pose a challenge.

For future work, the authors suggest exploring more advanced, adaptive fine-tuning strategies for the pretrained encoders to help them adjust even faster to highly specialized, out-of-domain downstream tasks.

Summary updated

Abstract

Federated Learning (FL) enables collaborative model training across decentralized clients without sharing private data. However, FL suffers from biased global models due to non-IID and long-tail data distributions. We propose \textbf{FedSM}, a novel client-centric framework that mitigates this bias through semantics-guided feature mixup and lightweight classifier retraining. FedSM uses a pretrained image-text-aligned model to compute category-level semantic relevance, guiding the category selection of local features to mix-up with global prototypes to generate class-consistent pseudo-features. These features correct classifier bias, especially when data are heavily skewed. To address the concern of potential domain shift between the pretrained model and the data, we propose probabilistic category selection, enhancing feature diversity to effectively mitigate biases. All computations are performed locally, requiring minimal server overhead. Extensive experiments on long-tail datasets with various imbalanced levels demonstrate that FedSM consistently outperforms state-of-the-art methods in accuracy, with high robustness to domain shift and computational efficiency.
BibTeX
@article{zhang2026fedsm,
  author = {Jingrui Zhang and Yimeng Xu and Shujie Li and Feng Liang and Haihan Duan and Yanjie Dong and Victor C. M. Leung and Xiping Hu},
  title = {{FedSM: Robust Semantics-Guided Feature Mixup for Bias Reduction in Federated Learning with Long-Tail Data}},
  journal = {IEEE Internet of Things Journal},
  year = {2026},
  doi = {10.1109/jiot.2026.3652363},
}
Figure