Iterated Learning Improves Compositionality in Large Vision-Language Models

Chenhao Zheng, Jieyu Zhang, Aniruddha Kembhavi, Ranjay Krishna

Abstract

Large vision-language models (VLMs) like CLIP often exhibit a "bag-of-words" behavior, excelling at object recognition but failing to understand how concepts combine to form complex meanings. This paper introduces an Iterated Learning (IL) algorithm that simulates "cultural transmission" by periodically resetting the weights of the language agent during training. This forces the model to develop representations that are easier to learn and inherently more compositional. Our approach, IL-CLIP, demonstrates significant improvements on compositionality benchmarks like SugarCrepe and CREPE without sacrificing general recognition performance.

Introduction

Despite their impressive zero-shot capabilities, state-of-the-art VLMs are notoriously poor at compositionality—the ability to understand how individual components (objects, attributes, relations) combine. Recent research suggests that simply scaling up models or data does not solve this issue; compositionality does not appear to be an emergent property of scale.

Drawing from cognitive science, specifically the theory of cultural transmission, we hypothesize that human language became compositional because it had to be "teachable" to new generations. A language where every unique concept has a unique, unrelated symbol is impossible to learn; a compositional language, which reuses a limited set of symbols in structured ways, is much easier to transmit. We translate this principle into a neural training pipeline to induce structural pressures in VLMs.

Methodology: Neural Iterated Learning

We reframe standard vision-language contrastive learning as a Lewis Signaling Game between a vision agent (VA) and a language agent (LA). To induce compositionality, we implement an Iterated Learning pipeline:

  1. Shared Codebook: A learnable codebook acts as a representational bottleneck, forcing agents to communicate via a restricted set of shared "symbols."
  2. Spawning New Agents: Periodically during training, the language agent’s weights are randomly re-initialized to simulate a "new generation" entering the game.
  3. Distillation Phase: The new language agent undergoes a short distillation phase to adapt to the existing codebook structure while the vision agent and codebook are frozen.
  4. Interaction Phase: Finally, all components are unfrozen, and the agents train together using standard contrastive loss.
The Iterated Learning training cycle: spawning, distilling, and interacting.

The core intuition is that the vision agent (the "teacher") is pressured to produce representations that the new language agent (the "student") can pick up quickly. This pressure favors structured, compositional representations over idiosyncratic, complex ones.

Key Insights

Learnability and Smoothness

Our analysis reveals that Iterated Learning acts as a form of implicit smoothness regularization. By measuring the Lipschitz constant of the models, we discovered that IL models have much smoother functional spaces than standard CLIP, making them easier to optimize and more generalizable.

The "Knowledge Gap"

By resetting one agent, we create a "knowledge gap." The vision agent must "teach" the new agent to align cross-modal representations. This pressure encourages the development of "easy-to-learn" features, which we show are inherently more compositional. Interestingly, resetting the language agent works better than resetting the vision agent, likely because text is already a high-level human abstraction.

Experimental Results

IL-CLIP was trained on datasets including CC3M, CC12M, and DataComp-10M to ensure robustness across scales.

Compositional Benchmarks

IL-CLIP consistently outperformed standard CLIP and other compositionality-focused baselines like NegCLIP. On the SugarCrepe benchmark, it improved over standard CLIP by 4.7% on CC3M and 4.0% on CC12M. It also showed significant gains in both systematicity and productivity on the CREPE benchmark.

Performance comparison on SugarCrepe: IL-CLIP shows superior compositional accuracy compared to normal CLIP throughout training.

Recognition and Efficiency

Unlike some methods that improve compositionality at the cost of raw accuracy, IL-CLIP maintained or slightly improved performance on 18 standard image classification datasets, including ImageNet-1k. Furthermore, new agents paired with an IL-trained vision teacher reached high accuracy much faster than those paired with a standard teacher, confirming the "easy-to-learn" hypothesis.

Conclusion

Iterated Learning provides a biologically-inspired path toward more compositional and interpretable vision-language models. While the multi-phase cycle adds some complexity to the training pipeline, the resulting models are more robust and better at understanding the structured nature of the visual world. Future work will focus on stabilizing the generational shifts and scaling the approach to billions of samples.

Summary updated

Abstract

A fundamental characteristic common to both human vision and natural language is their compositional nature. Yet, despite the performance gains contributed by large vision and language pretraining, recent investigations find that most-if not all-our state-of-the-art vision-language models struggle at compositionality. They are unable to distinguish between images of " a girl in white facing a man in black" and "a girl in black facing a man in white". Moreover, prior work suggests that compositionality doesn't arise with scale: larger model sizes or training data don't help. This paper develops a new iterated training algorithm that incentivizes compositionality. We draw on decades of cognitive science research that identifies cultural transmission-the need to teach a new generation-as a necessary inductive prior that incentivizes humans to develop compositional languages. Specifically, we reframe vision-language contrastive learning as the Lewis Signaling Game between a vision agent and a language agent, and operationalize cultural transmission by iteratively resetting one of the agent's weights during training. After every iteration, this training paradigm induces representations that become "easier to learn", a property of compositional languages: e.g. our model trained on CC3M and CC12M improves standard CLIP by 4.7%, 4.0% respectfully in the SugarCrepe benchmark.
BibTeX
@inproceedings{zheng2024iterated,
  author = {Chenhao Zheng and Jieyu Zhang and Aniruddha Kembhavi and Ranjay Krishna},
  title = {{Iterated Learning Improves Compositionality in Large Vision-Language Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure