Position: Compositional Generative Modeling: A Single Model is Not All You Need

Yilun Du, Leslie Pack Kaelbling

TL;DR

This position paper argues that the current AI trend of building ever-larger monolithic generative models is hitting a wall of diminishing returns in cost and data efficiency. Instead, the authors propose Compositional Generative Modeling: constructing complex systems by combining smaller, specialized models. This approach leads to better generalization, lower training costs, and the ability to "program" new behaviors without retraining.

The Problem with Monolithic Models

The dominant paradigm in AI research is "scaling up"—training massive monolithic models like GPT-4 or Gemini on nearly all available internet data. However, this approach faces three major hurdles:

  1. Exponential Costs: Training costs are tripling every year, with single runs now costing hundreds of millions of dollars. The computational requirements (FLOPs) for state-of-the-art models are growing at an unsustainable rate.
The rising computational cost of training monolithic models.
  1. Data Inefficiency: Learning a joint distribution over many variables (like pixels in an image or tokens in a sentence) requires a number of samples that grows exponentially with the number of variables.
  2. Failure of Compositionality: Despite their massive size, these models often fail at simple logical or spatial tasks. For example, they might struggle to accurately depict a specific spatial relationship (like a blue cup below a red bowl) because they rely heavily on statistical biases in their training data rather than underlying compositional rules.

The Compositional Approach

The authors advocate for factorizing probability distributions. Instead of learning one giant joint distribution $p(x)$, we can learn smaller components $p_i(x)$ that represent specific factors—such as objects, relations, or physics—and compose them.

  • Energy-Based Models (EBMs): The core technical framework relies on EBMs, where a distribution is represented as $p(x) \propto e^{-E(x)}$. Composing models becomes mathematically straightforward by adding their energy functions: $E_{total}(x) = \sum E_i(x)$.
  • Sampling via MCMC: To generate data from these composed models, the authors use Markov Chain Monte Carlo (MCMC) methods like Langevin dynamics.
  • Diffusion Connection: Modern diffusion models can be re-interpreted as a sequence of EBMs, allowing them to be composed using the same mathematical principles.
  • Unsupervised Discovery: The paper highlights techniques to discover these components directly from raw data, such as "slot-based" models that learn to identify individual objects in a scene without explicit labels.

Key Insights and Advantages

The "Empty Space" Problem and Generalization

Monolithic models only learn from the specific combinations of data they see during training. Compositional models, however, can generalize to "unseen" parts of the distribution. If a model learns the independent factors of "red car" and "blue bird," it can compose them to generate a "red bird" even if it never encountered one in the training data.

Compositional models generalize to unseen data combinations better than monolithic models.

Data Efficiency and Zero-Shot Programmability

Learning individual factors is mathematically much easier than learning a massive joint distribution. A factorized model can reach higher accuracy with significantly less data. Furthermore, this approach enables zero-shot programmability: you can create a "new" model for a task never seen at training by simply composing existing ones. For example, a robot's "dynamics model" can be composed with a "goal model" to perform on-the-fly planning.

Key Empirical Results

The paper synthesizes results across several domains to prove the efficacy of this approach:

  • Language Reasoning: Composing five instances of a smaller GPT-3.5 model achieved 58.0% accuracy on the MATH dataset, outperforming a single, much larger GPT-4 model (55.0%).
  • Trajectory Modeling: In Maze2D environments, factorized models simulated dynamics more accurately and trained in fewer iterations than their monolithic counterparts.
  • Robotic Manipulation: Compositional models successfully solved complex object arrangement tasks by treating constraints (like "avoid collisions") as individual factors to be satisfied simultaneously.

Limitations and Future Work

While promising, compositional generative modeling faces several challenges:

  • Structural Assumptions: Most current work assumes we already know how to decompose the problem (e.g., knowing to look for distinct "objects"). Automatically discovering the optimal compositional structure remains an open problem.
  • Spurious Correlations: In the real world, variables aren't always perfectly independent. Handling "messy" correlations while maintaining the benefits of factorization is difficult.
  • Sampling Speed: MCMC sampling for composed models can be slower than the single-pass inference of a monolithic transformer. Improving the efficiency of compositional sampling is a critical direction for future research.

Summary updated

Abstract

Large monolithic generative models trained on massive amounts of data have become an increasingly dominant approach in AI research. In this paper, we argue that we should instead construct large generative systems by composing smaller generative models together. We show how such a compositional generative approach enables us to learn distributions in a more data-efficient manner, enabling generalization to parts of the data distribution unseen at training time. We further show how this enables us to program and construct new generative models for tasks completely unseen at training. Finally, we show that in many cases, we can discover separate compositional components from data.
BibTeX
@inproceedings{du2024position,
  author = {Yilun Du and Leslie Pack Kaelbling},
  title = {{Position: Compositional Generative Modeling: A Single Model is Not All You Need}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=SoNexFx8qz},
}
Figure