Spontaneous Emerging Preference in Two-tower Language Model

Zhengqi He, Taro ToyoizumiarXiv 2022 · DOI · arXiv

Introduction

As AI models continue to scale, the computational and environmental costs of training massive foundation models have become significant concerns. One potential path toward efficiency is modularity, such as the Mixture-of-Experts (MoE) approach, where specialized components handle specific tasks. However, a fundamental question remains: is natural language inherently modular? This research investigates whether language possesses natural "seams" by observing if specialization spontaneously emerges when a model is given the opportunity to delegate processing between two identical components.

Methods

The researchers introduced a Two-tower Language Model architecture. Instead of a single large encoder, the system utilizes two identical encoders (towers) trained in parallel.

The Two-tower Language Model architecture where two identical encoders process the same input to predict the next word.

Both towers receive identical word (position) embeddings. Their hidden representations are concatenated and passed to a final output layer for word prediction. To quantify the specialization of each tower, the researchers froze the trained towers and retrained separate output heads for each. A Preference Score was then calculated based on the difference in average log probability of correct word prediction between the two towers.

The models—utilizing ELMo, GPT, and BERT backbones—were trained on a standard corpus of English Wikipedia and BookCorpus, totaling approximately 3 billion words with a 30,000-token WordPiece vocabulary.

Results and Discussion

The study revealed that specialization is not a random occurrence. Despite identical initializations, the preference patterns were remarkably consistent across different random seeds and even across different architectures. The correlation of preference scores between separate runs of the same model reached as high as 0.94, while the correlation between different architectures like ELMo and GPT remained significant at 0.57.

The Syntax-Semantics Split

The most significant finding was the nature of the division of labor. The towers consistently split processing along the lines of syntax and semantics. One "primary" tower typically specialized in functional and syntactic tokens, such as verbs, adverbs, and auxiliaries. In contrast, the "secondary" tower focused on semantic and referential tokens, particularly proper nouns (PROPN).

Distribution of preference scores (Average Perp Diff) across different Part-of-Speech (POS) tags for BERT-large, ELMo, and GPT models.

This consistent emergence across LSTMs and Transformers suggests that this modularity is an intrinsic property of natural language rather than an architectural artifact. The researchers noted parallels to biological systems, such as cerebral dominance in the human brain, where different hemispheres and neural pathways specialize in distinct functions like "what" versus "where/how" processing.

Conclusion and Future Work

This research demonstrates that language is naturally divisible into modular components. While the current two-tower setup increases parameter counts, these findings provide a foundation for designing more efficient, asymmetrical models where components are pre-optimized for specific linguistic roles (e.g., a smaller tower for syntax). Future work will explore the causal mechanisms that trigger one tower to become "primary" and investigate how to leverage these natural linguistic fault lines to build more interpretable and modular AI architectures.

Summary updated

Abstract

The ever-growing size of the foundation language model has brought significant performance gains in various types of downstream tasks. With the existence of side-effects brought about by the large size of the foundation language model such as deployment cost, availability issues, and environmental cost, there is some interest in exploring other possible directions, such as a divide-and-conquer scheme. In this paper, we are asking a basic question: are language processes naturally dividable? We study this problem with a simple two-tower language model setting, where two language models with identical configurations are trained side-by-side cooperatively. With this setting, we discover the spontaneous emerging preference phenomenon, where some of the tokens are consistently better predicted by one tower while others by another tower. This phenomenon is qualitatively stable, regardless of model configuration and type, suggesting this as an intrinsic property of natural language. This study suggests that interesting properties of natural language are still waiting to be discovered, which may aid the future development of natural language processing techniques.
BibTeX
@misc{he2022spontaneous,
  author = {Zhengqi He and Taro Toyoizumi},
  title = {{Spontaneous Emerging Preference in Two-tower Language Model}},
  year = {2022},
  eprint = {2210.07041},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.07041},
}
Figure