TL;DR
This position paper argues that the current AI trend of building ever-larger monolithic generative models is hitting a wall of diminishing returns in cost and data efficiency. Instead, the authors propose Compositional Generative Modeling: constructing complex systems by combining smaller, specialized models. This approach leads to better generalization, lower training costs, and the ability to "program" new behaviors without retraining.
The Problem with Monolithic Models
The dominant paradigm in AI research is "scaling up"—training massive monolithic models like GPT-4 or Gemini on nearly all available internet data. However, this approach faces three major hurdles:
- Exponential Costs: Training costs are tripling every year, with single runs now costing hundreds of millions of dollars. The computational requirements (FLOPs) for state-of-the-art models are growing at an unsustainable rate.
- Data Inefficiency: Learning a joint distribution over many variables (like pixels in an image or tokens in a sentence) requires a number of samples that grows exponentially with the number of variables.
- Failure of Compositionality: Despite their massive size, these models often fail at simple logical or spatial tasks. For example, they might struggle to accurately depict a specific spatial relationship (like a blue cup below a red bowl) because they rely heavily on statistical biases in their training data rather than underlying compositional rules.
The Compositional Approach
The authors advocate for factorizing probability distributions. Instead of learning one giant joint distribution $p(x)$, we can learn smaller components $p_i(x)$ that represent specific factors—such as objects, relations, or physics—and compose them.
- Energy-Based Models (EBMs): The core technical framework relies on EBMs, where a distribution is represented as $p(x) \propto e^{-E(x)}$. Composing models becomes mathematically straightforward by adding their energy functions: $E_{total}(x) = \sum E_i(x)$.
- Sampling via MCMC: To generate data from these composed models, the authors use Markov Chain Monte Carlo (MCMC) methods like Langevin dynamics.
- Diffusion Connection: Modern diffusion models can be re-interpreted as a sequence of EBMs, allowing them to be composed using the same mathematical principles.
- Unsupervised Discovery: The paper highlights techniques to discover these components directly from raw data, such as "slot-based" models that learn to identify individual objects in a scene without explicit labels.
Key Insights and Advantages
The "Empty Space" Problem and Generalization
Monolithic models only learn from the specific combinations of data they see during training. Compositional models, however, can generalize to "unseen" parts of the distribution. If a model learns the independent factors of "red car" and "blue bird," it can compose them to generate a "red bird" even if it never encountered one in the training data.
Data Efficiency and Zero-Shot Programmability
Learning individual factors is mathematically much easier than learning a massive joint distribution. A factorized model can reach higher accuracy with significantly less data. Furthermore, this approach enables zero-shot programmability: you can create a "new" model for a task never seen at training by simply composing existing ones. For example, a robot's "dynamics model" can be composed with a "goal model" to perform on-the-fly planning.
Key Empirical Results
The paper synthesizes results across several domains to prove the efficacy of this approach:
- Language Reasoning: Composing five instances of a smaller GPT-3.5 model achieved 58.0% accuracy on the MATH dataset, outperforming a single, much larger GPT-4 model (55.0%).
- Trajectory Modeling: In Maze2D environments, factorized models simulated dynamics more accurately and trained in fewer iterations than their monolithic counterparts.
- Robotic Manipulation: Compositional models successfully solved complex object arrangement tasks by treating constraints (like "avoid collisions") as individual factors to be satisfied simultaneously.
Limitations and Future Work
While promising, compositional generative modeling faces several challenges:
- Structural Assumptions: Most current work assumes we already know how to decompose the problem (e.g., knowing to look for distinct "objects"). Automatically discovering the optimal compositional structure remains an open problem.
- Spurious Correlations: In the real world, variables aren't always perfectly independent. Handling "messy" correlations while maintaining the benefits of factorization is difficult.
- Sampling Speed: MCMC sampling for composed models can be slower than the single-pass inference of a monolithic transformer. Improving the efficiency of compositional sampling is a critical direction for future research.
Summary updated