A Unified Framework for Training-Dataset Aggregation and Exact Machine Unlearning via Gibbs Probability Measures

Yaiza Bermudez, Samir M. Perlaza, Iñaki EsnaolaarXiv 2026 · arXiv

TL;DR

This paper introduces a unified mathematical framework using Gibbs probability measures to aggregate datasets and perform "exact machine unlearning." By defining three fundamental operations—renormalization, nesting, and log-linear combination—the authors demonstrate how to achieve centralized-equivalent performance in distributed systems and remove the influence of specific data points without the need for costly retraining from scratch.

What Problem Does This Paper Address?

In modern machine learning, two major challenges are becoming increasingly prominent: distributed learning and machine unlearning.

  1. Aggregation: In decentralized or federated learning, data is siloed across different clients. The goal is to produce a model that performs as well as one trained on a single, massive "centralized" dataset, but without actually moving the raw data.
  2. Unlearning: With regulations like GDPR, users have a "right to be forgotten." If a user requests their data be removed, a service provider must ensure the model no longer carries the influence of that data. "Exact unlearning" is the gold standard here—it requires the updated model to be statistically identical to a model that was never trained on the deleted data in the first place.

The authors identify a gap in the theoretical understanding of how to combine models (represented as probability measures) such that the resulting model remains a valid solution to a well-defined optimization problem.

Methods

The paper centers on Gibbs probability measures, which are defined by an objective function (like empirical risk), a reference measure (a prior), and a regularization factor ($\lambda$). The authors study three core operations:

  • Renormalization: Normalizing a power of the density of a measure (related to "escort distributions" in physics).
  • Normalized Log-Linear Combination: Combining multiple Gibbs measures by multiplying powers of their densities and re-normalizing.
  • Nesting: A sequential process where one Gibbs measure serves as the reference (prior) for the next.

What’s new: The authors prove that these operations always result in a new Gibbs measure. They provide explicit formulas for the new objective functions and regularization factors. Crucially, they establish a mathematical equivalence between the sequential "nesting" approach (common in decentralized peer-to-peer systems) and the parallel "log-linear" approach (common in federated client-server systems). They also provide a geometric interpretation of these operations using exponential geodesics and Pythagorean identities within the space of probability measures.

Datasets

This is a theoretical paper focused on establishing a mathematical framework and proving theorems. Consequently, no specific empirical datasets (like MNIST or ImageNet) were used for training or evaluation. The "datasets" discussed are abstract sets of patterns and labels used to define the empirical risk functions within the proofs.

Key Insights and Ideas

  • Aggregation as Reweighting: The authors show that aggregating models is essentially a way of controlling the "weight" of each data point in the final empirical risk. By manipulating the parameters of their three operations, they can precisely tune how much influence a specific local dataset has on the global model.
  • The "Aha!" of Unlearning: Exact unlearning is achieved by setting the weight of the data to be forgotten to exactly zero. The paper provides the specific parameter settings ($\lambda$ and $\alpha$) required to "cancel out" the contribution of a dataset from a previously trained Gibbs measure.
  • Centralized Equivalence: They prove that distributed systems (decentralized or federated) can perfectly replicate the performance of a centralized system. A sequence of clients passing models (nesting) or a server combining models (log-linear) can produce the exact same Gibbs measure as if all data were in one place.
  • Geometric Unity: The paper links these statistical operations to information geometry, showing that moving between these measures follows "straight lines" (geodesics) in the space of distributions, which helps visualize how models evolve during aggregation or unlearning.

Key Results

  • Theorem 4 & 7: Provide the explicit Radon-Nikodym derivatives for renormalized and combined measures, proving they remain in the Gibbs family.
  • Corollary 13: Formally proves the equivalence between nesting and log-linear combinations under specific conditions.
  • Exact Unlearning Formulas: The paper provides concrete corollaries (27, 34, and 37) that act as "recipes" for exact unlearning in centralized, decentralized, and federated settings, respectively.
  • Centralized-Equivalent Guarantees: Corollaries 30 and 35 establish the specific regularization choices clients must make to ensure the final global model is identical to a centralized one.

Limitations and Future Work

  • Gibbs Assumption: The framework relies on the assumption that the learning algorithm can be represented as a Gibbs measure. While this covers many stochastic-gradient-based algorithms and regularized empirical risk minimization, it may not directly apply to all model types (e.g., hard-margin SVMs or non-regularized deep nets).
  • Computational Complexity: While the exact unlearning is mathematically elegant, Gibbs measures often require calculating a "partition function" (the denominator in the Gibbs formula), which can be computationally expensive for high-dimensional model spaces.
  • Parameter Optimization: The authors note that the global regularization parameter $\lambda_0$ remains a free variable. Future work could focus on how to optimize this parameter to improve the generalization properties of the resulting "unlearned" or "aggregated" models.

Summary updated

Abstract

In this paper, three operations on Gibbs probability measures are studied: renormalization, normalized log-linear combination, and nesting. The resulting measures are shown to be Gibbs probability measures with explicit expressions for their objective functions, reference measures, and regularization factors. These measures are further characterized as unique solutions to optimization problems involving linear combinations of the original objective functions, subject to either relative entropy regularization or relative entropy constraints. When the reference measures are probability measures, geometric interpretations of these three operations are established involving Pythagorean identities and exponential geodesics.A central result is the equivalence between nesting and normalized log-linear combinations of Gibbs probability measures. These results are then applied to supervised machine learning with Gibbs algorithms, which are represented by Gibbs probability measures. In this context, either through nesting or log-linear combinations, Gibbs algorithms trained on local datasets are used to form new Gibbs algorithms trained upon the aggregation of all local training datasets. Interestingly, the parameters of such operations, determine the weights of each training data point on the aggregated dataset. %Operations on local Gibbs probability measures are shown to control the contributions of training data. This observation is leveraged for achieving: (a) exact machine unlearning in centralized, decentralized, and federated learning systems; and (b) centralized-equivalent performance, in the sense of producing algorithms trained on the aggregation of all local datasets in decentralized and federated learning .
BibTeX
@misc{bermudez2026unified,
  author = {Yaiza Bermudez and Samir M. Perlaza and Iñaki Esnaola},
  title = {{A Unified Framework for Training-Dataset Aggregation and Exact Machine Unlearning via Gibbs Probability Measures}},
  year = {2026},
  eprint = {2611.05544},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05544},
}
Figure