In an ideal machine learning world, data is centralized, uniformly distributed, and easily accessible. In the real world, data is siloed across thousands of client devices, highly heterogeneous (non-IID), and strictly protected by privacy constraints.
To train effective models in these distributed environments, a central coordinator often needs to answer a fundamental question: How similar are the data distributions of two decentralized entities? Knowing this similarity allows a system to cluster similar clients together, personalize global models, or select the most relevant public datasets to warm up a model. However, because raw data cannot be shared or directly inspected, traditional statistical distance metrics fail.
To bypass this barrier, a powerful research direction has emerged: Privacy-Preserving Similarity Estimation. Instead of comparing raw data points, these techniques use mathematical proxies—such as model parameters, gradients, or differentially private statistical summaries—to safely quantify the alignment between distributed data distributions.
The Model Parameter as a Distribution Proxy
The core intuition behind this field is that a model's parameters are a direct reflection of its training data. When a neural network optimizes its weights on a specific dataset, its loss landscape and resulting parameter trajectories are uniquely shaped by that data's underlying distribution. Therefore, if two clients train models on similar data, their parameters or gradients should point in similar directions.
[Client A Data] ---> [Local Training] ---> [Model Parameters A] \
+---> Cosine Similarity / Alignment
[Client B Data] ---> [Local Training] ---> [Model Parameters B] /
Several researchers leverage this geometric alignment to coordinate collaboration without data leakage:
- Gradient and Parameter Cosine Similarity: Ma et al. et al. in their work on PFedAtt demonstrate that calculating the similarity between model parameters or gradients allows a central server to identify related client tasks. Similarly, Li et al. et al. (FEDACS) use the cosine similarity of model parameters as a reliable proxy for data similarity, enabling targeted peer-to-peer collaboration in data-scarce scenarios.
- Classifier Layer Distances: Rather than looking at the entire network, some approaches isolate specific layers. Wu et al. et al. (PFedCS) show that the distance between local classifier parameters (the final layer) serves as an effective proxy for identifying clients with similar label distributions. Shaik et al. et al. (Clustered FedStack) similarly cluster local models based on their output layer weights to automatically determine optimal client groupings.
- Parameter Masks and Manifolds: Wu et al. et al. (FedCAC) look at the spatial overlap of critical parameter masks, using this structural intersection to cluster similar participants. Taking this a step further, Kim demonstrates that parameter updates derived from pseudo-labels can embed clients into a continuous "parameter manifold," allowing the system to map the global geometry of distributed data without ever accessing it.
Compact Statistical Summaries and Optimal Transport
While raw parameters are highly expressive, they can sometimes be noisy or tightly coupled with the optimization path. An alternative approach is to compute compact, mathematically rigorous statistical summaries of the data distribution that can be safely shared, often protected by cryptographic or differential privacy guarantees.
Differentially Private Fréchet Distance
When developers want to pre-train a model on public data before deploying it to private users, they must ensure the public data matches the users' actual distribution. Hou et al. et al. solved this with FreD (Federated Private Fréchet Distance) in Privately Customizing Prefinetuning to Better Match User Data in Federated Learning. Instead of sharing raw samples, clients compute local mean and covariance statistics. By injecting differential privacy noise and using secure aggregation, the central server calculates a global Fréchet distance while only ever observing noisy, aggregated statistics.
Optimal Transport and Wasserstein Metrics
Optimal Transport (OT) provides a powerful framework for measuring the distance between probability distributions by calculating the minimum "cost" to morph one distribution into another.
- Rakotomamonjy et al. et al. introduce the Federated Wasserstein Distance, allowing a central server to quantify distribution overlap across decentralized devices without accessing raw data.
- At the edge, Mehra et al. et al. (Test-time Assessment of a Model's Performance on Unseen Domains via Optimal Transport) demonstrate that OT can operate on compact statistical summaries of a source distribution to assess model reliability under distribution shift, bypassing the need to store or transmit the original training data.
Kernel Mean Embeddings
Rather than relying on explicit distance metrics, Fermanian et al. et al. (Adaptive Personalized Federated Learning via Multi-task Averaging of Kernel Mean Embeddings) map client distributions into a high-dimensional reproducing kernel Hilbert space using Kernel Mean Embeddings (KME). This allows the system to treat personalization as a high-dimensional mean estimation problem, automatically learning exactly how much to weight data from other agents with rigorous statistical guarantees.
Why It Works: Downstream Applications
Estimating similarity is rarely the end goal; instead, it serves as the foundational layer for coordinating complex distributed systems.
[Raw Data Silos]
│
▼ (Privacy-Preserving Similarity Estimation)
[Similarity Matrix / Affinities]
│
┌─────┼──────────────────────────┐
▼ ▼ ▼
[Client Selection] [Targeted Clustering] [Ecosystem Matching]
1. Discrepancy-Aware Aggregation and Client Selection
Standard federated learning averages updates based on dataset size, which can degrade performance if data is highly skewed. Ye et al. et al. (FedDisco) incorporate the discrepancy between local and global category distributions to reward more representative clients with higher influence. Similarly, Wang et al. et al. (Federated Analytics Informed Distributed Industrial IoT Learning With Non-IID Data) quantify client data skewness in a privacy-preserving manner to intelligently prioritize nodes that contribute most effectively to global training.
2. Graph and Structural Clustering
When data structures are complex, such as graphs, calculating affinity becomes even harder. Liu et al. et al. (PERFECT) overcome this by sharing privacy-preserving representative samples, allowing a central server to calculate client affinities and coordinate personalized graph-level clustering without exposing raw graph topologies.
3. Ecosystem Stability and Market Matching
Privacy-preserving similarity also governs how models interact within larger ecosystems:
- Federated Unlearning: When a client invokes the "right to be forgotten," removing their data can degrade performance for remaining clients with similar data. Shao et al. et al. (Beyond Right to be Forgotten) use similarity metrics to calculate how a client's withdrawal cascades through a network, allowing operators to strategically incentivize crucial nodes to remain.
- Learnware Markets: In a model-sharing marketplace where different users have heterogeneous feature spaces (e.g., different medical metrics), Tan et al. et al. (Towards enabling learnware to handle heterogeneous feature spaces) demonstrate that distribution alignment can be measured under strict privacy constraints to match a user's requirements with a pre-trained model's expertise.
Open Challenges
While parameter and summary-based proxies unlock distributed coordination, they are not entirely immune to risks. As highlighted by Lu et al. et al. (Federated Learning with Uncertainty-Based Client Clustering for Fleet-Wide Fault Diagnosis), many current clustering methods still lack robust privacy-hardening techniques. Sophisticated adversaries can potentially reverse-engineer client characteristics from the shared similarity proxies themselves. The frontier of this research direction lies in tightening these privacy guarantees—ensuring that the metrics used to protect data do not inadvertently become side-channels for leakage.
Key Foundational Work
FedAP: Adaptive Personalization in Federated Learning for Non-IID Data (2022) — contributed 1 concept elements to this direction.
PFedAtt: Attention-based Personalized Federated Learning on Heterogeneous Clients (2021) — contributed 1 concept elements to this direction.
On the Reusability of Machine Learning Models in Edge Computing: A Statistical Learning Approach (2022) — contributed 1 concept elements to this direction.