Composing Ensembles of Pre-trained Models via Iterative Consensus

Shuang Li, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, Igor Mordatch

Introduction

We are currently in the era of "Foundation Models"—massive neural networks trained on internet-scale data. However, these models are often specialists. Large Language Models (LLMs) like GPT-3 excel at complex textual reasoning but lack visual grounding. Conversely, Vision-Language Models (VLMs) like CLIP or DALL-E understand images but struggle with complex logic.

The standard approach to combining these strengths is either to finetune them together, which is computationally expensive, or to cascade them, where one model passes its output to the next. The problem with cascading—often called "Socratic Models"—is that it is a one-way street. If the first model makes a mistake, the second model is forced to work with flawed input, leading to compounding errors. There is no mechanism for the downstream model to provide corrective feedback.

To address this, researchers propose Pre-trained Iterative Consensus (PIC), a unified framework that allows specialized AI models to work together as a committee. By using a "generator" to propose solutions and an "ensemble of scorers" to provide iterative feedback, the system achieves impressive zero-shot performance across vision, language, and robotics tasks without any additional training.

The Pre-trained Iterative Consensus (PIC) Framework

PIC treats model composition as a closed-loop optimization problem. The architecture relies on two primary roles:

  1. The Generator ($G$): A model that constructs initial proposals. For example, this could be a Diffusion model for generating images or a language model like GPT-2 for text.
  2. The Scorers ($E$): An ensemble of "expert" models that evaluate the generator's proposals. For instance, CLIP might check if an image matches a text prompt, or a math classifier might verify a logical step.

The core innovation is Iterative Consensus. Instead of a single feed-forward pass, the generator produces a proposal, and the scorers provide feedback. This feedback iteratively refines the proposal. The authors employ two types of refinement depending on the task:

  • Continuous Refinement: Uses the gradients of the scorers to "nudge" the generator’s output, typically applied in continuous domains like image generation.
  • Discrete Refinement: Samples multiple candidates and uses scorer feedback to guide the next step of generation, which is ideal for text and robotics.

By ensembling multiple scorers, the system leverages the "wisdom of the crowds." The strengths of one model (e.g., a VLM's spatial awareness) can compensate for the weaknesses of another (e.g., an LLM's lack of visual grounding).

Key Insights and Zero-Shot Performance

The researchers evaluated PIC across four diverse multimodal domains, revealing several critical insights about model composition:

  • Closed-Loop Over Open-Loop: Bidirectional communication is essential. Allowing scorers to propagate information back to the generator enables the system to self-correct errors that would be fatal in a traditional, open-loop pipeline.
  • The Wisdom of the Crowds: An ensemble of scorers (e.g., three different versions of CLIP) consistently outperforms any single scorer. Because different models have different biases, ensembling allows these biases to cancel out, resulting in a more robust consensus.
  • Zero-Shot Superiority: Remarkably, PIC often outperforms models specifically finetuned for a task, suggesting that the necessary "knowledge" already exists within pre-trained models; we simply need better frameworks to extract it.

Empirical Results

The PIC framework demonstrated significant improvements across all tested domains:

  • Mathematical Reasoning: Evaluated on the GSM8K benchmark, PIC improved accuracy on grade school math problems by 7.5% over baselines without any model finetuning. In fact, PIC using a frozen GPT-2 outperformed a version of GPT-2 explicitly finetuned on the math dataset.
  • Image Generation: On ImageNet (64x64), composing a Diffusion generator with an ensemble of three scorers (CLIP, an Image Classifier, and Classifier-free guidance) raised the Inception Score from ~25 to 34.95, marking a massive jump in image quality and diversity.
  • Video Question Answering: On ActivityNet-QA, where the model must describe video content, PIC achieved an accuracy of 61.1%. This significantly outperformed the zero-shot "JustAsk" baseline (50.6%) and approached the performance of fully finetuned models.
  • Robot Manipulation: Evaluated in the Ravens simulator, the system was tasked with arranging objects based on textual or visual goals. In complex tasks involving three object relations, using an ensemble of multi-view scorers increased the success rate from 45% to 65.3%.
Robot manipulation task in the Ravens simulator

Limitations and Future Directions

While highly effective, the PIC framework has a few notable limitations. First is inference latency: because the process is iterative and requires multiple passes through the models to reach consensus, it is significantly slower than a single-pass feed-forward model. Second, there is a dependency on pre-training: the system is only as capable as its experts. If all scorers in the ensemble share a fundamental blind spot, the resulting consensus will still be incorrect.

Future work may explore how to automatically select the best "experts" for a specific task from a large library of models, paving the way for dynamic, task-specific AI committees that can tackle increasingly complex multimodal challenges.

Summary updated

Abstract

Large pre-trained models exhibit distinct and complementary capabilities dependent on the data they are trained on. Language models such as GPT-3 are capable of textual reasoning but cannot understand visual information, while vision models such as DALL-E can generate photorealistic photos but fail to understand complex language descriptions. In this work, we propose a unified framework for composing ensembles of different pre-trained models -- combining the strengths of each individual model to solve various multimodal problems in a zero-shot manner. We use pre-trained models as "generators" or "scorers" and compose them via closed-loop iterative consensus optimization. The generator constructs proposals and the scorers iteratively provide feedback to refine the generated result. Such closed-loop communication enables models to correct errors caused by other models, significantly boosting performance on downstream tasks, e.g. improving accuracy on grade school math problems by 7.5%, without requiring any model finetuning. We demonstrate that consensus achieved by an ensemble of scorers outperforms the feedback of a single scorer, by leveraging the strengths of each expert model. Results show that the proposed method can be used as a general purpose framework for a wide range of zero-shot multimodal tasks, such as image generation, video question answering, mathematical reasoning, and robotic manipulation.
BibTeX
@inproceedings{li2023composing,
  author = {Shuang Li and Yilun Du and Joshua B. Tenenbaum and Antonio Torralba and Igor Mordatch},
  title = {{Composing Ensembles of Pre-trained Models via Iterative Consensus}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=gmwDKo-4cY},
}
Figure