Introduction
The rapid advancement of large-scale pre-trained models has yielded "experts" in specific domains: GPT-3 for text, CLIP for vision-language alignment, and DALL-E for image synthesis. However, solving complex multimodal tasks often requires the seamless integration of these disparate capabilities. Traditional approaches typically rely on "Socratic" chains—one-way pipelines where one model's output serves as another's input—or expensive joint fine-tuning on massive datasets. The former is prone to error propagation, while the latter is computationally prohibitive.
This paper introduces PIC (Pre-trained Iterative Consensus), a framework that enables specialized models to collaborate through a closed-loop communication cycle. By treating models as an ensemble of generators and scorers, PIC allows them to iteratively refine solutions until a consensus is reached, effectively correcting errors in a zero-shot manner.
The PIC Framework
The core of PIC is the division of labor between two roles: the Generator ($G$) and the Scorers ($E$).
- The Generator ($G$): This model proposes candidate solutions. Depending on the task, this could be a diffusion model for images, a language model for text or math, or a world model for robotic actions.
- The Scorers ($E$): An ensemble of models evaluates the generator's proposals. Each scorer provides a "feedback score" or energy value representing how well the proposal aligns with the task constraints (e.g., CLIP scoring the alignment between a generated image and a text prompt).
Iterative Optimization
Unlike single-pass methods, PIC employs a closed-loop cycle. The generator produces a proposal, the scorers provide feedback, and the generator then updates its output to minimize the total "energy" (disagreement) across the ensemble. This optimization happens in two ways:
- Continuous Optimization: For image generation or text latents, gradients are used to nudge representations toward higher consensus.
- Discrete Optimization: For tasks like robotics or math, the system samples multiple actions or steps and selects the one that maximizes the consensus score.
Experimental Results
The authors evaluated PIC across four diverse domains, demonstrating its versatility without any task-specific fine-tuning.
Image Generation and Video QA
In image generation, PIC combined a diffusion generator with an ensemble of scorers (CLIP and standard classifiers). This ensemble approach achieved an Inception Score (IS) of 34.95 on ImageNet, a nearly 40% improvement over using a single CLIP scorer. For Video Question Answering on ActivityNet-QA, PIC reached 61.1% accuracy, significantly outperforming previous zero-shot methods like JustAsk (50.6%).
Mathematical Reasoning
Using the GSM8K dataset, PIC improved the accuracy of standard GPT-2 baselines by 7.5%. The iterative feedback allowed the model to "re-think" logical steps, preventing the accumulation of errors common in chain-of-thought reasoning without verification.
Robot Manipulation
In the Ravens simulation environment, PIC addressed spatial reasoning tasks (e.g., "place the orange mug on the orange bowl"). By incorporating scorers from multiple camera perspectives, the success rate jumped from 35% to 67.5%, highlighting the benefit of multi-view consensus in physical tasks.
Discussion and Limitations
PIC demonstrates that the "wisdom of the crowds" applies to pre-trained models: an ensemble of scorers often compensates for the individual biases of its members. Furthermore, the framework's zero-shot nature makes it highly scalable, allowing researchers to swap in newer, more powerful models as they become available.
However, the iterative nature of PIC introduces higher computational costs at inference time compared to single-pass models. Additionally, the system's performance remains bounded by the fundamental capabilities of its constituent models; if the underlying models cannot grasp a concept like counting, the ensemble will likely struggle as well. Future work may explore using these consensus-driven solutions as a signal for self-supervised fine-tuning to improve base model performance over time.
Summary updated