TL;DR
While Large Language Models (LLMs) excel at generating high-quality text, they suffer from severe "domain homogeneity" when making analogies, repeatedly drawing from a narrow, predictable set of target domains. This paper systematically evaluates ten state-of-the-art LLMs, uncovering a fundamental trade-off between analogy diversity and quality, and reveals through causal analysis that the internal model regions governing analogical reasoning vary drastically across different model architectures.
What Problem Does This Paper Address?
Analogy making is a cornerstone of human intelligence and a primary driver of scientific innovation. A classic example is James Clerk Maxwell, who formulated his theory of electromagnetism by drawing structural analogies from fluid dynamics and mechanical engineering. For AI to truly accelerate scientific discovery (AI-for-Science, or AI4S), LLMs must be able to connect conceptually distant domains.
However, prior research has almost exclusively focused on the quality and correctness of LLM analogies rather than their diversity. If an LLM is asked to create an analogy for a deceptive person and always defaults to "a book you shouldn't judge by its cover," its creative utility is severely limited. This paper addresses this gap by investigating whether current LLMs can generate diverse, cross-domain analogies, and why existing diversity-enhancing techniques fall short.
Methods
The authors establish a comprehensive evaluation framework to analyze both open-source and closed-source models.
To push the boundaries of LLM diversity, they test three main categories of inference-time interventions:
- Diversity Prompting: Explicitly instructing the model to be creative or to preserve logical structures across different domains.
- Probabilistic and Logits Steering: Standard decoding strategies like Top-$k$, Top-$p$, Min-$p$, Top-$\eta\sigma$, and G2 that resample token distributions.
- Entropy-gated Steering: An adaptive method that triggers resampling only when next-token entropy crosses a specific threshold during domain word generation.
Outputs are evaluated using a weighted combination of semantic diversity (via SentenceTransformer embeddings), lexical diversity (Self-BLEU), and vocabulary breadth. Crucially, they use an LLM-as-a-judge setup to score analogy quality and employ clustering to count how many unique semantic domains a model actually visits. Finally, they perform a causal perturbation analysis—injecting noise into hidden states via looped inference—to locate the exact layers responsible for analogy generation.
Datasets
The experiments are conducted across three distinct analogy and metaphor datasets:
- AnaloBench (AB): A benchmark containing 340 pairs of human-annotated, story-level analogies.
- Metaphoric Analogies (MA): 203 samples focused on structured, four-term analogical mappings extracted from literary metaphors.
- Metaphor Understanding Challenge (MUNCH): A dataset testing cross-domain mappings through paraphrasing (1K randomly sampled instances used).
Key Insights and Ideas
- The Domain Homogeneity Trap: LLMs are incredibly repetitive in their analogical targets. When prompted to generate 10 distinct analogies, most models only manage to explore an average of two semantic clusters.
- The Diversity-Quality Trade-Off: There is no free lunch. Methods that successfully force the model to choose rarer, more diverse target domains mechanically degrade the logical coherence and overall quality of the analogy.
- Anatomical Misalignment: Why do standard logit-steering methods (like Top-$p$) perform erratically across different models? The authors' causal analysis reveals that the internal "analogy-making engine" shifts depending on the architecture. In Llama and Mistral, the sensitive regions are spread widely across early-to-mid layers (20th to 70th percentile). In Phi and Qwen, they are highly concentrated in the final layers. Because standard steering methods only manipulate the final layer's logits, they miss the upstream internal computations in models with distributed representations.
Key Results
- Limited Baseline Diversity: Even the best-performing closed-source model under diversity prompting, Gemini 3.1 Pro, only achieved an average intra-output diversity score of 0.49.
- High Inter-Model Overlap: Different model families often duplicate each other's ideas. Comparing the generation distributions of Grok-4.5 and Gemini-3.1-Pro yielded a remarkably high MAUVE similarity score of 0.72, indicating an overwhelming overlap in their choice of target domains.
- Unstable Steering Performance: While Top-$p$ and G2 steering successfully boosted the diversity score of Llama-3.1-8B to 0.61, the exact same methods completely failed when applied to Qwen-3-8B, yielding stagnant scores of 0.28 and 0.29.
Limitations and Future Work
The study primarily focuses on inference-time interventions (prompting and decoding modifications). It does not explore how fine-tuning or reinforcement learning with diversity-targeted reward functions might alleviate domain homogeneity. The authors suggest that future work should focus on designing "diversity-aware" architectures and alignment strategies that directly target the specific internal layers where analogical reasoning emerges, rather than relying on surface-level logit manipulation at the final layer.
Summary updated