Gaussian mixture models as a proxy for interacting language models

Edward L. Wang, Tianyu Wang, Avanti Athreya, Vince Lyzinski, Carey E. PriebearXiv 2025 · DOI · arXiv

Abstract

Large Language Models (LLMs) are increasingly utilized to simulate social dynamics, yet their high computational requirements and "black box" nature present significant barriers to large-scale research. This paper demonstrates that Gaussian Mixture Models (GMMs) can serve as an efficient, interpretable proxy for interacting LLMs. By representing agent perspectives as Gaussian distributions and interactions as Expectation-Maximization (EM) updates, the GMM framework replicates key social phenomena—such as silo formation and consensus building—at a fraction of the computational cost.

Introduction

Social scientists are increasingly leveraging Large Language Models (LLMs) to model complex human behaviors, including the emergence of echo chambers ("silos") and the process of social alignment. However, LLM-based simulations face three primary challenges: opacity, as LLMs are difficult to interpret; high computational cost, requiring significant energy and hardware; and inconsistency across different model architectures. This research asks whether these complex dynamics can be captured using a simpler, mathematically tractable model: the Gaussian Mixture Model (GMM).

Methodology

The researchers compared a baseline LLM simulation, based on prior work by McGuinness et al., against a proposed interacting GMM framework.

LLM Baseline

The baseline simulation involved multiple LLaMA-2 agents equipped with Retrieval-Augmented Generation (RAG) databases. These agents were seeded with initial "memory" sentences generated by ChatGPT regarding the beauty of various flowers. During the simulation, agents "talked" to their neighbors, updated their RAG databases with the received opinions (typically by replacing the most distant existing memory), and gradually shifted their preferences.

GMM Proxy

In the GMM framework, each agent is represented as a mixture of Gaussians in $\mathbb{R}^m$, where $m$ represents distinct possible perspectives.

  • Perspectives: Represented by the means ($\mu$) of the Gaussians.
  • Beliefs: Represented by the weights ($w$) assigned to each Gaussian component.
  • Interaction: When two GMM agents interact, one samples a "thought" (a vector) from the other's distribution. The receiving agent then uses the Expectation-Maximization (EM) algorithm to update its internal weights.

The study systematically varied parameters such as $k$ (the number of neighbors an agent interacts with) and $p$ (the probability an agent "mirrors" or interacts with its own distribution) to observe their effects on social siloing.

Key Results and Insights

Fidelity of Social Dynamics

The GMM proxy successfully replicated the three primary patterns observed in LLM social simulations: stable silos (fixed groups), unstable silos (agents jumping between views), and decaying silos (temporary groups that eventually merge). As shown in Figure 1, the qualitative behavior of GMM agents—including interaction trajectories and population stability—closely mirrors that of LLM agents.

Comparison of GMM and LLM social dynamics, illustrating interaction trajectories, silo populations, and stability metrics over time.

One notable difference is the "stubbornness" of GMM agents. While LLMs might shift their entire output based on a single piece of new context in a RAG prompt, the EM update in GMMs is incremental, integrating new data into an existing statistical pool. This results in slower oscillations between perspectives compared to LLMs.

Structural Similarity and Connectivity

A central finding is the "inverse relationship" between network connectivity ($k$) and the number of silos. As agents interact with a larger number of neighbors, the population more readily reaches a global consensus. This relationship, depicted in Figure 2, is consistent across both LLM and GMM systems, suggesting that GMMs capture the fundamental structural dynamics of social alignment.

The number of silos in the population decreases as the number of neighbors (k) increases, demonstrating the impact of connectivity on social alignment.

Efficiency and Mathematical Clarity

GMMs offer a "glass box" alternative to the "black box" of LLMs. Because GMMs are governed by clear statistical rules, researchers could demonstrate that increasing the "distance" between different perspectives ($\Delta\mu$) accelerates the time to reach consensus. Furthermore, GMM simulations are orders of magnitude more efficient; while specific FLOP reductions were not quantified, the qualitative difference between EM updates and 7B-parameter model inference is vast.

Limitations and Future Work

While promising, the current GMM proxy uses simplified one-dimensional perspectives with fixed means. Real-world human beliefs are multivariate and evolve over time. Future research could incorporate more sophisticated clustering for silo identification and more realistic memory models, such as temporal decay or importance weighting, to better reflect human cognitive processes.

Conclusion

Gaussian Mixture Models provide a robust, low-cost, and interpretable proxy for studying the macro-level social dynamics of interacting language models. By replicating essential phenomena like siloing and alignment through simple statistical updates, GMMs offer a scalable foundation for future research in digital social simulation.

Summary updated

Abstract

Large language models (LLMs) are powerful tools that, in a number of settings, overlap with the results of human pattern recognition and reasoning. Retrieval-augmented generation (RAG) further allows LLMs to produce tailored output depending on the contents of their RAG databases. However, LLMs depend on complex, computationally expensive algorithms. In this paper, we introduce interacting Gaussian mixture models (GMMs) as a proxy for interacting LLMs. We construct a model of interacting GMMs, complete with an analogue to RAG updating, under which GMMs can generate, exchange, and update data and parameters. We show that this interacting system of Gaussian mixture models, which can be implemented at minimal computational cost, mimics certain aspects of experimental simulations of interacting LLMs whose iterative responses depend on feedback from other LLMs. We build a Markov chain from this system of interacting GMMs; formalize and interpret the notion of polarization for such a chain; and prove lower bounds on the probability of polarization. This provides theoretical insight into the use of interacting Gaussian mixture models as a computationally efficient proxy for interacting large language models.
BibTeX
@misc{wang2025gaussian,
  author = {Edward L. Wang and Tianyu Wang and Avanti Athreya and Vince Lyzinski and Carey E. Priebe},
  title = {{Gaussian mixture models as a proxy for interacting language models}},
  year = {2025},
  eprint = {2506.00077},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.00077},
}
Figure