Compositional Generative Modeling: A Single Model is Not All You Need

Yilun Du, Leslie Pack Kaelbling

Abstract

Large monolithic generative models trained on massive amounts of data have become an increasingly dominant approach in AI research. In this paper, we argue that we should instead construct large generative systems by composing smaller generative models together. We show how such a compositional generative approach enables us to learn distributions in a more data-efficient manner, enabling generalization to parts of the data distribution unseen at training time. We further show how this enables us to program and construct new generative models for tasks completely unseen at training. Finally, we show that in many cases, we can discover separate compositional components from data.

Introduction: The Limits of Monolithic Scaling

The current AI paradigm is defined by scaling: more parameters, more data, and more compute. However, this "monolithic" approach hits several walls. Training costs are tripling every year, with top-tier models now costing hundreds of millions of dollars. Furthermore, these models are often data-inefficient; learning a joint distribution of many variables requires exponentially more data than learning individual factors. Finally, large models often struggle with simple compositionality—like understanding spatial relations they haven't seen exactly in training—and are prohibitively expensive to fine-tune for new tasks.

The exponential growth of training compute (FLOPs) for state-of-the-art monolithic models over time.

The central question of this work is whether we can achieve the same (or better) intelligence by snapping together simpler "expert" models.

Methodology: Factorization and Compositional Sampling

The core idea is Factorization. Instead of one model learning everything, multiple models learn subsets of variables or specific constraints.

Energy-Based Models (EBMs)

The authors represent probability distributions as energy functions ($e^{-E(x)}$). In this framework, composing models becomes simple addition: to combine two models, you just add their energy functions. This allows for flexible "programming" of distributions.

Visualizing how compositional models reconstruct complex distributions by combining individual factors, compared to monolithic approaches.

Compositional Sampling

To generate data from these combined models, the authors utilize Markov Chain Monte Carlo (MCMC) and Diffusion models. A key mathematical insight provided is that a standard reverse diffusion step is essentially an MCMC step (specifically, the Unadjusted Langevin Algorithm). This allows existing diffusion hardware to be used for sampling from composed "energy landscapes."

Experimental Evaluation

The approach was validated across a diverse range of domains:

  • Reasoning (MATH Dataset): By composing five instances of GPT-3.5, the authors achieved 58.0% accuracy, outperforming a single, much larger GPT-4 (55.0%).
  • Trajectory Modeling (Maze2D): Factorized models learned to simulate dynamics with significantly fewer agent episodes and training iterations than monolithic counterparts.
  • Visual Synthesis: The system successfully generated "Image Tapestries"—complex, high-resolution scenes constructed by composing models constrained to different spatial regions.
  • Physics Discovery: In an unsupervised manner, the system identified whether particles were governed by springs or electrical charges, then "recombined" those laws to simulate hybrid physical systems.

Discussion: The Society of Models

This research suggests a move toward a "Society of Models"—a decentralized architecture where different models (text, video, action) communicate via inference. This approach offers several advantages:

  • Zero-Shot Programming: New generative models can be "programmed" without new training data by composing existing models (e.g., a dynamics model plus a goal-seeking model creates a planner).
  • Data Efficiency: Learning individual factors is linearly difficult ($O(K)$), while learning their joint combination is exponentially difficult ($O(K^N)$). Composition bypasses this "curse of dimensionality."
  • Generalization: Compositional models can handle novel combinations (e.g., "pink sky over a blue mountain") even if the specific combination was never seen during training.

Limitations and Future Work

While promising, several challenges remain. Currently, the "recipe" for composition is often pre-specified; future work should focus on the automatic discovery of optimal compositional structures. Additionally, the mathematical assumptions of independence between components may be violated by "spurious correlations" in real-world data. Finally, inference speed is a concern, as sampling from composed models via MCMC can be slower than a single forward pass of a monolithic model.

Summary updated

Abstract

Large monolithic generative models trained on massive amounts of data have become an increasingly dominant approach in AI research. In this paper, we argue that we should instead construct large generative systems by composing smaller generative models together. We show how such a compositional generative approach enables us to learn distributions in a more data-efficient manner, enabling generalization to parts of the data distribution unseen at training time. We further show how this enables us to program and construct new generative models for tasks completely unseen at training. Finally, we show that in many cases, we can discover separate compositional components from data.
BibTeX
@inproceedings{du2024compositional,
  author = {Yilun Du and Leslie Pack Kaelbling},
  title = {{Compositional Generative Modeling: A Single Model is Not All You Need}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=SoNexFx8qz},
}
Figure