Marginal Distribution Adaptation for Discrete Sets via Module-Oriented Divergence Minimization

Hanjun Dai, Mengjiao Yang, Yuan Xue, Dale Schuurmans, Bo DaiICML 2022 · OpenReview

Generative models excel at capturing how items co-occur in discrete sets—think of products in a shopping cart or diagnosis codes in a medical record. For instance, a model might learn that people who buy cereal are highly likely to buy milk.

However, the real world is dynamic. Seasonality (buying more sunscreen in summer) or new product launches (a new smartphone release) cause "distribution shifts." While the frequency of individual items changes, the underlying correlations (the "vibe" or structural logic of the set) often remain the same.

The standard solution to this shift is retraining the model from scratch on new data. This approach is computationally expensive, requires massive amounts of new data, and risks "forgetting" the high-order correlations learned previously. This raises a critical question: How can we efficiently nudge an existing model to match new target frequencies while keeping its learned correlations intact?

The MODEM Framework

To address this, researchers introduce MODEM (Module-Oriented DivErgence Minimization), a framework that treats adaptation as a constrained optimization problem. The goal is to find a new distribution that is as close as possible to the original distribution (minimizing KL divergence) while strictly satisfying new marginal constraints (the new item frequencies).

The core innovation of MODEM is Module Reusable Parametrization. Instead of updating every weight in a neural network, MODEM identifies and updates only the specific "modules" responsible for item frequency:

MODEM Framework Overview

As illustrated above, MODEM applies surgical updates depending on the model architecture:

  • Latent Variable Models (LVMs): The decoder (which handles how items relate to latent themes) is frozen. Only the prior distribution over those themes is updated.
  • Autoregressive Models: The conditional probabilities of subsequent items are frozen. Only the distribution of the first item in the sequence is updated.
  • Energy-Based Models (EBMs): Using mathematical duality, the optimal adaptation involves simply adding a learnable linear "bias" term to the existing energy function, transforming a complex alignment task into a straightforward Maximum Likelihood Estimation (MLE) problem.

Experimental Validation

The researchers evaluated MODEM on synthetic datasets and diverse real-world datasets, including retail (Instacart, Groceries, Market-Basket) and healthcare (MIMIC3 clinical diagnosis codes).

Key Results

  • Superior Alignment: Across all datasets, MODEM significantly reduced the Marginal RMSE (a measure of how well the model matches target frequencies) compared to "No Adapt" baselines and standard reweighting methods.
  • Correlation Preservation: Using the Pairwise-F1 metric, the adapted models successfully maintained original co-occurrence patterns. In some cases, adaptation even improved the correlation metrics.
  • Massive Speedups: Because MODEM only touches a tiny fraction of the model's parameters (e.g., updating just 512 parameters instead of 1 million in an LVM), it typically converged in about 10% of the steps required for full retraining. On the Market-Basket dataset, it updated fewer than 0.1% of the original parameters while achieving better results.
Performance Comparison: RMSE and F1 Score

The results clearly demonstrate that surgical precision beats brute force. You don't need to re-teach a model "what goes with what" just because "how often things appear" has changed. By isolating the modules that control marginals, MODEM preserves the structural knowledge of the model while adapting to new realities.

Limitations and Future Work

While highly efficient, MODEM's optimization for EBMs and LVMs can become challenging as the domain size (number of unique items) scales into the tens of thousands. Additionally, the current framework focuses on single-item marginals (e.g., "how many apples?"). Future work could extend this to more complex, multi-dimensional constraints. Finally, for autoregressive models, exploring a more native set-based approach could yield even better results by fully addressing permutation invariance.

Summary updated

Abstract

Distributions over discrete sets capture the essential statistics including the high-order correlation among elements. Such information provides powerful insight for decision making across various ...
BibTeX
@inproceedings{dai2022marginal,
  author = {Hanjun Dai and Mengjiao Yang and Yuan Xue and Dale Schuurmans and Bo Dai},
  title = {{Marginal Distribution Adaptation for Discrete Sets via Module-Oriented Divergence Minimization}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=rpBjBmbCxoX},
}
Figure