Diffusion-Based Active Learning for Distributed Client Manifolds

Kwang In KimAAAI 2025 · DOI

Introduction

In modern machine learning, data is increasingly siloed across thousands of distributed clients (e.g., mobile devices or hospitals) due to privacy constraints. While Federated Learning allows training on this data, Active Learning (AL)—the process of identifying which data points are most informative to label—remains a significant challenge. Traditional AL requires a central server to inspect raw data, which is impossible in privacy-preserving distributed environments.

This paper introduces a novel framework for client-level selection. Instead of asking which images to label, the server must decide which clients should be tasked with labeling their local data. The goal is to maximize the global model's performance while minimizing labeling costs and maintaining strict data privacy.

Methodology: The Client Manifold

The authors propose a two-step approach to model the relationships between clients without ever accessing their raw local data:

1. Pseudo-Parameters as Distribution Fingerprints

Since the server cannot see local data, it asks clients to perform a local update using pseudo-labels (the model's current predictions). These resulting parameter updates serve as a "fingerprint" for the client's data distribution. Clients with similar updates are likely to have similar underlying data distributions, allowing the server to map them onto a "parameter manifold."

2. Anisotropic Diffusion for Impact Assessment

The algorithm constructs a graph where each node represents a client. It then simulates an anisotropic diffusion process on this graph to measure the "global impact" of a candidate client.

Conceptual illustration of the client manifold and the diffusion process used to identify representative nodes.

By "suppressing" the uncertainty at a specific node and observing how that reduction spreads to its neighbors, the server can identify clients in high-density regions of the manifold. Labeling these "representative" clients provides the most significant benefit to the entire network, effectively maximizing the coverage of uncertainty reduction.

Experimental Results

The researchers evaluated their approach on five standard computer vision benchmarks: CIFAR-10, CIFAR-100, Fashion-MNIST, EMNIST, and CINIC-10. To simulate real-world complexity, they utilized two non-IID (non-independent and identically distributed) settings: Dirichlet (varying class distributions) and Shard (limited classes per client).

Performance comparison across five benchmarks under different non-IID settings, demonstrating the consistent superiority of the proposed diffusion-based approach (purple line) over standard baselines.

Key Findings

  • Superior Efficiency: The proposed algorithm was statistically significantly better than random selection in 93.6% of all tested cases.
  • Consistency: Unlike simple uncertainty sampling, which can underperform in early rounds, the diffusion-based approach provided consistent gains from the very first labeling round.
  • Diversity: By modeling the manifold geometry, the algorithm naturally avoids redundant selections, ensuring the labeled set is diverse and representative of the global population.

Discussion and Insights

One of the paper's most critical insights is the importance of Unnormalized Laplacians. While normalized graph Laplacians are common in spectral clustering, the authors found they often over-promote isolated outlier clients. The unnormalized version correctly prioritizes clients in dense regions, which is essential for maximizing the impact of limited labeling budgets.

Limitations

Despite its success, the method faces a "cold start" problem: it relies on the model having enough initial predictive power to generate meaningful pseudo-labels. Additionally, the computational overhead of simulating diffusion across millions of clients and the sensitivity to hyperparameters (like neighborhood size $K$) remain areas for future optimization.

Conclusion

By treating distributed active learning as a graph diffusion problem on a parameter manifold, this framework provides a privacy-preserving way to identify the most impactful clients. This shift from point-based to manifold-based selection marks a significant step forward for efficient and private distributed machine learning.

Summary updated

BibTeX
@inproceedings{kim2025diffusion,
  author = {Kwang In Kim},
  title = {{Diffusion-Based Active Learning for Distributed Client Manifolds}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i17.33969},
}
Figure