Abstract
Large vision-language models (VLMs) like CLIP often exhibit a "bag-of-words" behavior, excelling at object recognition but failing to understand how concepts combine to form complex meanings. This paper introduces an Iterated Learning (IL) algorithm that simulates "cultural transmission" by periodically resetting the weights of the language agent during training. This forces the model to develop representations that are easier to learn and inherently more compositional. Our approach, IL-CLIP, demonstrates significant improvements on compositionality benchmarks like SugarCrepe and CREPE without sacrificing general recognition performance.
Introduction
Despite their impressive zero-shot capabilities, state-of-the-art VLMs are notoriously poor at compositionality—the ability to understand how individual components (objects, attributes, relations) combine. Recent research suggests that simply scaling up models or data does not solve this issue; compositionality does not appear to be an emergent property of scale.
Drawing from cognitive science, specifically the theory of cultural transmission, we hypothesize that human language became compositional because it had to be "teachable" to new generations. A language where every unique concept has a unique, unrelated symbol is impossible to learn; a compositional language, which reuses a limited set of symbols in structured ways, is much easier to transmit. We translate this principle into a neural training pipeline to induce structural pressures in VLMs.
Methodology: Neural Iterated Learning
We reframe standard vision-language contrastive learning as a Lewis Signaling Game between a vision agent (VA) and a language agent (LA). To induce compositionality, we implement an Iterated Learning pipeline:
- Shared Codebook: A learnable codebook acts as a representational bottleneck, forcing agents to communicate via a restricted set of shared "symbols."
- Spawning New Agents: Periodically during training, the language agent’s weights are randomly re-initialized to simulate a "new generation" entering the game.
- Distillation Phase: The new language agent undergoes a short distillation phase to adapt to the existing codebook structure while the vision agent and codebook are frozen.
- Interaction Phase: Finally, all components are unfrozen, and the agents train together using standard contrastive loss.
The core intuition is that the vision agent (the "teacher") is pressured to produce representations that the new language agent (the "student") can pick up quickly. This pressure favors structured, compositional representations over idiosyncratic, complex ones.
Key Insights
Learnability and Smoothness
Our analysis reveals that Iterated Learning acts as a form of implicit smoothness regularization. By measuring the Lipschitz constant of the models, we discovered that IL models have much smoother functional spaces than standard CLIP, making them easier to optimize and more generalizable.
The "Knowledge Gap"
By resetting one agent, we create a "knowledge gap." The vision agent must "teach" the new agent to align cross-modal representations. This pressure encourages the development of "easy-to-learn" features, which we show are inherently more compositional. Interestingly, resetting the language agent works better than resetting the vision agent, likely because text is already a high-level human abstraction.
Experimental Results
IL-CLIP was trained on datasets including CC3M, CC12M, and DataComp-10M to ensure robustness across scales.
Compositional Benchmarks
IL-CLIP consistently outperformed standard CLIP and other compositionality-focused baselines like NegCLIP. On the SugarCrepe benchmark, it improved over standard CLIP by 4.7% on CC3M and 4.0% on CC12M. It also showed significant gains in both systematicity and productivity on the CREPE benchmark.
Recognition and Efficiency
Unlike some methods that improve compositionality at the cost of raw accuracy, IL-CLIP maintained or slightly improved performance on 18 standard image classification datasets, including ImageNet-1k. Furthermore, new agents paired with an IL-trained vision teacher reached high accuracy much faster than those paired with a standard teacher, confirming the "easy-to-learn" hypothesis.
Conclusion
Iterated Learning provides a biologically-inspired path toward more compositional and interpretable vision-language models. While the multi-phase cycle adds some complexity to the training pipeline, the resulting models are more robust and better at understanding the structured nature of the visual world. Future work will focus on stabilizing the generational shifts and scaling the approach to billions of samples.
Summary updated