On the Diversity of Analogy Making in Large Language Models

Yuanhao Shen, Daniel Xavier de Sousa, Caio César Sifuentes Barcelos, Hongyu Guo, Xiaodan ZhuarXiv 2026 · arXiv

TL;DR

While Large Language Models (LLMs) excel at generating high-quality text, they suffer from severe "domain homogeneity" when making analogies, repeatedly drawing from a narrow, predictable set of target domains. This paper systematically evaluates ten state-of-the-art LLMs, uncovering a fundamental trade-off between analogy diversity and quality, and reveals through causal analysis that the internal model regions governing analogical reasoning vary drastically across different model architectures.

What Problem Does This Paper Address?

Analogy making is a cornerstone of human intelligence and a primary driver of scientific innovation. A classic example is James Clerk Maxwell, who formulated his theory of electromagnetism by drawing structural analogies from fluid dynamics and mechanical engineering. For AI to truly accelerate scientific discovery (AI-for-Science, or AI4S), LLMs must be able to connect conceptually distant domains.

However, prior research has almost exclusively focused on the quality and correctness of LLM analogies rather than their diversity. If an LLM is asked to create an analogy for a deceptive person and always defaults to "a book you shouldn't judge by its cover," its creative utility is severely limited. This paper addresses this gap by investigating whether current LLMs can generate diverse, cross-domain analogies, and why existing diversity-enhancing techniques fall short.

Methods

The authors establish a comprehensive evaluation framework to analyze both open-source and closed-source models.

Figure 2. Overview of experiments and evaluation methods on LLM’s analogy making diversity. The model is given the input of the format [Instruction, source] along with different inference time methods in the left column. We then cluster the output generations into different target domains and conduct multi-faceted evaluations on those generations.

To push the boundaries of LLM diversity, they test three main categories of inference-time interventions:

  1. Diversity Prompting: Explicitly instructing the model to be creative or to preserve logical structures across different domains.
  2. Probabilistic and Logits Steering: Standard decoding strategies like Top-$k$, Top-$p$, Min-$p$, Top-$\eta\sigma$, and G2 that resample token distributions.
  3. Entropy-gated Steering: An adaptive method that triggers resampling only when next-token entropy crosses a specific threshold during domain word generation.

Outputs are evaluated using a weighted combination of semantic diversity (via SentenceTransformer embeddings), lexical diversity (Self-BLEU), and vocabulary breadth. Crucially, they use an LLM-as-a-judge setup to score analogy quality and employ clustering to count how many unique semantic domains a model actually visits. Finally, they perform a causal perturbation analysis—injecting noise into hidden states via looped inference—to locate the exact layers responsible for analogy generation.

Datasets

The experiments are conducted across three distinct analogy and metaphor datasets:

  • AnaloBench (AB): A benchmark containing 340 pairs of human-annotated, story-level analogies.
  • Metaphoric Analogies (MA): 203 samples focused on structured, four-term analogical mappings extracted from literary metaphors.
  • Metaphor Understanding Challenge (MUNCH): A dataset testing cross-domain mappings through paraphrasing (1K randomly sampled instances used).

Key Insights and Ideas

  • The Domain Homogeneity Trap: LLMs are incredibly repetitive in their analogical targets. When prompted to generate 10 distinct analogies, most models only manage to explore an average of two semantic clusters.
  • The Diversity-Quality Trade-Off: There is no free lunch. Methods that successfully force the model to choose rarer, more diverse target domains mechanically degrade the logical coherence and overall quality of the analogy.
  • Anatomical Misalignment: Why do standard logit-steering methods (like Top-$p$) perform erratically across different models? The authors' causal analysis reveals that the internal "analogy-making engine" shifts depending on the architecture. In Llama and Mistral, the sensitive regions are spread widely across early-to-mid layers (20th to 70th percentile). In Phi and Qwen, they are highly concentrated in the final layers. Because standard steering methods only manipulate the final layer's logits, they miss the upstream internal computations in models with distributed representations.

Key Results

  • Limited Baseline Diversity: Even the best-performing closed-source model under diversity prompting, Gemini 3.1 Pro, only achieved an average intra-output diversity score of 0.49.
  • High Inter-Model Overlap: Different model families often duplicate each other's ideas. Comparing the generation distributions of Grok-4.5 and Gemini-3.1-Pro yielded a remarkably high MAUVE similarity score of 0.72, indicating an overwhelming overlap in their choice of target domains.
Figure 3. (a) Comparison of sentence diversity, domain diversity, and output quality across prompting strategies. Solid lines denote sentence diversity, dashed lines denote domain diversity, and square markers indicate quality scores for open-source and closed-source models. (b) Inter-domain diversity using average linkage. Bars denote average linkage; dashed line indicates mean clusters per sample.
  • Unstable Steering Performance: While Top-$p$ and G2 steering successfully boosted the diversity score of Llama-3.1-8B to 0.61, the exact same methods completely failed when applied to Qwen-3-8B, yielding stagnant scores of 0.28 and 0.29.

Limitations and Future Work

The study primarily focuses on inference-time interventions (prompting and decoding modifications). It does not explore how fine-tuning or reinforcement learning with diversity-targeted reward functions might alleviate domain homogeneity. The authors suggest that future work should focus on designing "diversity-aware" architectures and alignment strategies that directly target the specific internal layers where analogical reasoning emerges, rather than relying on surface-level logit manipulation at the final layer.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated remarkable potential for analogy making, a core cognitive capability that drives novelty and creativity. While prior research has extensively investigated the applications and underlying mechanisms of LLM-based analogy making, its output diversity remains largely unexplored, despite being essential for broadening cross-domain connections and fostering scientific innovation. In this work, we present a comprehensive evaluation of analogy diversity across ten state-of-the-art open- and closed-source LLMs. Our findings highlight a concerning issue of domain homogeneity, a prevalent tendency for LLMs to generate analogies from a narrow set of target domains, limiting both inter-query and intra-model diversity. Furthermore, our analysis reveals a fundamental trade-off in existing LLM diversity-enhancement methods: increasing output diversity often comes at the expense of output quality. Finally, our causal analysis of LLM information flow reveals substantial differences in the model-sensitive regions governing analogy diversity across LLMs, suggesting a potential mechanism for the observed diversity-quality trade-off. To our knowledge, this is among the first studies to systematically investigate output diversity in LLM-based analogy making.
BibTeX
@misc{shen2026diversity,
  author = {Yuanhao Shen and Daniel Xavier de Sousa and Caio César Sifuentes Barcelos and Hongyu Guo and Xiaodan Zhu},
  title = {{On the Diversity of Analogy Making in Large Language Models}},
  year = {2026},
  eprint = {2608.03233},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.03233},
}
Figure