Inconsistency-Based Federated Active Learning

Chen-Chen Zong, Tong Jin, Sheng-Jun HuangIJCAI 2025 · DOI

Abstract

Federated Learning (FL) enables collaborative model training across distributed clients without centralizing private data. However, the practical application of FL is often hindered by the high cost of data labeling. Federated Active Learning (FAL) seeks to minimize this cost by selectively querying the most informative samples. We introduce Inconsistency-Based Federated Active Learning (IFAL), a novel framework designed to overcome the challenges of data heterogeneity and noisy data. By measuring inconsistencies between local, global, and distilled models, IFAL effectively identifies high-value samples for labeling while filtering out uninformative noise.

Introduction

In a typical Federated Learning scenario, clients such as mobile devices or medical institutions contribute to a shared global model. While FL research often assumes pre-labeled data, real-world data is frequently unlabeled, making manual annotation a significant bottleneck. Federated Active Learning (FAL) addresses this by allowing clients to request labels only for data points that maximize model improvement.

Existing FAL methods face two primary obstacles. First, data heterogeneity (Non-IID) causes local models to develop biases, leading them to overvalue samples simply because they belong to classes underrepresented in their local training set. Second, noisy data often mimics the uncertainty of informative samples, causing traditional active learning algorithms to waste labeling budgets on corrupted data. IFAL is specifically engineered to navigate these "messy" environments.

The IFAL Framework

IFAL employs a multi-faceted approach to sample selection based on three distinct types of inconsistency checks:

1. Data-Driven (Structural) Probability

Standard active learning relies on a model's softmax output to gauge uncertainty. However, in heterogeneous FL, softmax outputs are often skewed. IFAL introduces a structural probability based on Reverse K-Nearest Neighbors (rKNN) in the feature space. By examining whether an unlabeled sample resides in a neighborhood of samples associated with a specific class, the framework gains an objective, geometry-based view of the data distribution.

Conceptual overview of IFAL comparing learning-based and data-driven probabilities across global and local models.

2. Inter-Model Inconsistency (The Noise Filter)

To distinguish between truly informative "hard" samples and useless noise, IFAL utilizes a third, intermediate model created by distilling knowledge from the global model.

  • Noise Detection: If the local, global, and distilled models exhibit extreme disagreement, the sample is likely noise and is discarded.
  • Knowledge Gap Identification: If the models disagree in a pattern suggesting a discrepancy between local and global knowledge, the sample is prioritized for labeling.

3. Intra-Model Inconsistency

This metric measures the divergence between the model’s internal prediction (learning-based probability) and the data’s neighborhood-based suggestion (structural probability). High divergence indicates that the model's decision boundary is poorly aligned with the actual data structure and requires refinement.

Experimental Evaluation

The researchers evaluated IFAL on three standard computer vision benchmarks: CIFAR-10, CIFAR-100, and Tiny-Imagenet. To simulate realistic federated environments, they utilized a Dirichlet distribution ($\alpha=0.1$) to create highly unbalanced class distributions across 10 to 20 clients.

Key Results and Insights

  • Superior Performance: IFAL consistently outperformed nine state-of-the-art baselines, including LoGo and KAFAL. On CIFAR-100, it achieved significantly higher test accuracy for the same labeling budget.
  • Robustness to Heterogeneity: The framework maintained its performance edge even as data became more non-IID or as the number of clients increased.
  • Effective Noise Mitigation: Qualitative analysis confirmed that the inter-model inconsistency check successfully avoided querying "garbage" samples, focusing instead on informative data that bridged the gap between local and global perspectives.
  • Wasserstein Distance: The authors found that using Wasserstein distance to measure inconsistency was more effective than KL divergence, particularly when comparing non-overlapping distributions in early training stages.

Conclusion and Future Work

IFAL represents a significant step toward making Federated Active Learning viable in complex, real-world settings. By grounding model uncertainty in the structural geometry of the data and using model distillation as a noise filter, it ensures that labeling efforts are spent on the most impactful samples.

While the current framework introduces some computational overhead due to rKNN searches and distilled model training, future work will focus on optimizing these processes for resource-constrained edge devices. Additionally, exploring IFAL's performance in asynchronous federated learning environments remains a promising direction for further research.

Summary updated

BibTeX
@inproceedings{zong2025inconsistency,
  author = {Chen-Chen Zong and Tong Jin and Sheng-Jun Huang},
  title = {{Inconsistency-Based Federated Active Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/812},
}
Figure