Statistical Model Aggregation via Parameter Matching

Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan H. Greenewald, Trong Nghia Hoang

Introduction

In the modern data landscape, information is often "sequestered" in disparate silos due to privacy concerns, massive data volumes that are too expensive to move, or limited communication bandwidth. While Federated Learning and Hierarchical Bayesian models aim to learn from these distributed sources, they often rely on bespoke architectures designed for specific models (like neural networks) and incur prohibitive computational and communication costs.

To address this, researchers have developed SPAHM (Statistical Parameter Aggregation via Heterogeneous Matching), a new framework for "model fusion." SPAHM combines multiple machine learning models trained on separate, private data silos into one global model. By treating local model parameters as noisy observations of global latent parameters and leveraging Bayesian nonparametrics, SPAHM offers a fast, model-agnostic method that can be up to 1,400x faster than traditional approaches.

The SPAHM Framework

Instead of modeling the raw data, SPAHM operates on a "meta-modeling" principle: it models the parameters of the locally trained models.

  • The Generative Process: The framework assumes there is a set of "global parameters" of an unknown size. Each local dataset utilizes only a subset of these global parameters.
  • Beta-Bernoulli Process (BBP): To handle the unknown and potentially infinite number of global parameters, SPAHM employs a BBP. This allows the global model to dynamically "grow" or "shrink" as it processes more local models (e.g., merging two Hidden Markov Models with different numbers of states).
  • The Subtraction Trick: The mathematical breakthrough of the paper is reformulating the complex Maximum A Posteriori (MAP) inference for this model into a series of small, efficient linear sum assignment problems.
  • The Hungarian Algorithm: Thanks to the subtraction trick, the parameter matching problem can be solved using the classic Hungarian algorithm, which is extremely fast and computationally lightweight.

Because the framework uses general exponential family distributions, it can be applied to any model whose parameters can be described this way, making it highly versatile.

Key Insights

The most powerful idea behind SPAHM is treating model parameters as "noisy realizations" of a global truth, which completely decouples the aggregation step from the initial training step. Unlike previous work focused solely on neural networks, SPAHM is model agnostic, working seamlessly for Topic Models, Gaussian Processes, and HMMs using the same underlying math.

Furthermore, SPAHM handles heterogeneity gracefully. It does not assume every local model is identical; it can match a 9-state HMM from one subject with a 13-state HMM from another, identifying which states are shared and which are unique. Because it only requires the final parameters of local models, it avoids the "chatty," communication-heavy nature of traditional federated learning.

Experimental Results

The authors evaluated SPAHM across four distinct scenarios: simulated Gaussian mixture models, text summarization (Gutenberg Dataset), temperature forecasting (NOAA weather data), and human movement analysis (CMU Motion Capture).

Speed and Quality in Text Analysis: For Gaussian Topic Models trained on 40 books, SPAHM was over 1,400 times faster than the standard Gibbs sampler (taking 40 seconds versus 16 hours). It also achieved a significantly better UCI coherence score (-2.1) compared to the baseline (-4.6).

Topic matching across different books using SPAHM.

Generalization in Weather Forecasting: In the weather forecasting task using Sparse Gaussian Processes, local models trained on specific states performed poorly when evaluated on a national level. SPAHM-fused models, however, generalized across the entire US while maintaining high accuracy on local data.

RMSE comparison showing SPAHM's superior generalization across the USA.

Discovery in Motion Capture: In Motion Capture analysis, SPAHM correctly identified shared activities (like "arm circles") across different subjects while keeping unique activities (like "up-downs" performed by only one subject) as separate global states.

Limitations and Future Work

While powerful, SPAHM does have limitations. The user must still choose an appropriate "base measure" (e.g., deciding that the parameters follow a Gaussian distribution); a poor choice will degrade the fusion quality. Additionally, SPAHM assumes the local models are "good" representations of their data. If a local model is poorly trained or overfitted, it provides noisy input to the meta-model.

Currently, the framework treats local models as a static group. Future work could incorporate temporal dynamics, allowing the global model to evolve dynamically as local data streams change over time.

Summary updated

Abstract

We consider the problem of aggregating models learned from sequestered, possibly heterogeneous datasets. Exploiting tools from Bayesian nonparametrics, we develop a general meta-modeling framework that learns shared global latent structures by identifying correspondences among local model parameterizations. Our proposed framework is model-independent and is applicable to a wide range of model types. After verifying our approach on simulated data, we demonstrate its utility in aggregating Gaussian topic models, hierarchical Dirichlet process based hidden Markov models, and sparse Gaussian processes with applications spanning text summarization, motion capture analysis, and temperature forecasting.
BibTeX
@inproceedings{yurochkin2019statistical,
  author = {Mikhail Yurochkin and Mayank Agarwal and Soumya Ghosh and Kristjan H. Greenewald and Trong Nghia Hoang},
  title = {{Statistical Model Aggregation via Parameter Matching}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=fzDl8t-yeFl},
}
Figure