The LALA Benchmark: Evaluating Literary and Linguistic Analysis in LLMs

Michael Foster, Barbara McGillivray, Gard B. Jenset, Jakub Mácha, Jason Harding, Marcus Perlman, Devyani Sharma, Patrick Skeels, Jana Willer-Gold, Jeannette LittlemorearXiv 2026 · arXiv

TL;DR

The LALA (Literature and Linguistics Analysis) benchmark introduces 236 expert-level, open-ended questions designed to test whether LLMs can perform nuanced linguistic and literary analysis. The study finds that while post-training is essential for making linguistic knowledge "usable," it adds competencies in a fragmented, "piecemeal" fashion rather than fostering a general linguistic intelligence capable of handling ambiguity or complex literary style.

What Problem Does This Paper Address?

Most existing linguistic benchmarks for LLMs focus on "minimal pairs"—simple grammatical distinctions that can be tested via probability or multiple-choice (e.g., The cat sit vs. The cat sits). While useful, these don't capture the depth of human expertise in linguistics or literary studies.

The authors argue there is a gap in evaluating how models handle explicit understanding: can a model explain why a sentence is ambiguous? Can it identify an allusion or a shift in literary voice? Furthermore, the paper investigates a fundamental architectural question: does the ability to analyze language emerge naturally with scale (parameter count), or is it a product of specific post-training (instruction tuning/RLHF)?

Methods

The researchers developed the LALA Benchmark, consisting of 236 questions across three broad domains: Linguistics, Logic, and Textuality.

Key methodological features include:

  • Expert Gold Standards: Each question was written by a linguist, philosopher, or literary scholar who also provided an "ideal" answer and specific keywords for grading.
  • Multi-Faceted Evaluation: 17 models were tested. Responses were evaluated by human experts and two "automated judges" (GPT-5.6 Sol and Qwen3 14B) using a 0–2 rubric for accuracy, relevance, and completeness.
  • ReliableEval Integration: To ensure results weren't just a fluke of specific phrasing, the authors used the ReliableEval method, generating over 2,000 reworded variants of the questions to test for consistency.
  • Cross-Family Comparisons: They compared base models (e.g., DeepSeek V3 Base) directly against their post-trained versions (e.g., DeepSeek R1) to isolate the effects of training stages.

Datasets

  • LALA Benchmark: The primary contribution, featuring 236 open-ended questions tagged with 298 keywords across 27 subcategories (e.g., Morphosyntax, Epistemic Logic, Philology).
  • ReliableEval Variants: A supplementary set of 2,131 prompt variants derived from the original LALA questions to check for model robustness.

Key Insights and Ideas

  • The Post-Training Transformation: Base models were almost entirely incapable of performing the analysis required, scoring near zero. Post-training doesn't just refine a model; it appears to be the primary mechanism that makes latent linguistic knowledge accessible and articulable.
  • Piecemeal Competence: The most striking finding is that linguistic gains are not holistic. A model might become an expert at Propositional Logic but remain completely baffled by a simple literary metaphor or a misleading assumption in a prompt. This suggests models are learning specific "response policies" rather than a generalized "linguistic intelligence."
  • Scale vs. Policy: Surprisingly, parameter count showed no statistically significant association with a model's ability to earn partial credit. A smaller, well-tuned model (like Gemma 4 31B) could outperform much larger models, reinforcing the idea that how a model is trained matters more than its size for these specific tasks.
  • The "Jagged Frontier": Models performed best on logical relationships and discourse-level tasks but consistently struggled with ambiguity, reference, and literary style. They often "hallucinated" single interpretations for intentionally ambiguous prompts.

Key Results

  • Top Performers: Proprietary models led the pack. Claude Fable 5 ranked first (72.3% human score), followed by Gemini 3.1 Pro (64.4%) and GPT-5.6 Sol (59.9%).
  • Base vs. Instruct Gaps: The performance jump was massive. For example, DeepSeek V3 Base scored 2.7%, while the post-trained DeepSeek V3 jumped to 40.5%.
  • Judge Reliability: The automated judge Sol (GPT-5.6) showed nearly perfect correlation with human rankings (Kendall’s W = 0.994), suggesting that high-end LLMs are becoming viable proxies for expert human assessment in complex domains.
  • Domain Strengths: Logic questions saw the highest average scores (45.3%), while Textuality (literary analysis) was the most difficult for models (36.9%).

Limitations and Future Work

  • Uneven Category Representation: Some subcategories (like Cognitive Pragmatics) only had a few questions, making it hard to draw broad conclusions for those specific niches.
  • Safety Guardrail Interference: Several models (notably Claude and Llama) refused to answer certain questions due to over-active safety filters (e.g., misinterpreting a literary analysis of Othello or a folk song metaphor as "harmful content").
  • Subjectivity of "Ideal" Answers: About 50 questions had "ideal" answers that were just one of many possible interpretations, which might have introduced bias in scoring.
  • Future Directions: The authors suggest expanding the dataset to include more categories and using "blind" evaluations where judges don't know the name of the model they are scoring to eliminate potential brand bias.

The complete LALA dataset is available on Hugging Face for researchers to test their own models.

Summary updated

Abstract

We introduce the Literature and Linguistics Analysis Benchmark (LALA), comprising 236 open-ended questions written by experts in linguistics and literary studies, together with expert-provided ideal answers. By comparing models within families and at similar training scales using human and LLM judges, LALA identifies differences in linguistic competence associated with post-training, offering a framework for evaluating future models and guiding targeted improvements. We administered questions to 17 language models and evaluated their responses using both human expert assessment and two LLM evaluators, Sol and Qwen. Base models displayed little of the linguistic competence assessed by the benchmark, whereas their post-trained versions performed markedly better, suggesting that post-training plays a central role in making linguistic knowledge accessible and usable. Parameter count, by contrast, showed no statistically significant association with partial-credit performance, and competencies remained unevenly distributed across linguistic domains. We interpret this pattern as evidence that post-training adds linguistic competencies piecemeal, without producing a general linguistic intelligence capable of extending them beyond the learned policies that support them. Models performed best on logical relations between statements and selected discourse-level tasks, while even the strongest post-trained models continued to struggle with ambiguity, literary style, reference, and misleading assumptions. Human and Sol evaluations produced nearly identical model rankings in both the original and ReliableEval runs, supporting the replicability of the comparative results.
BibTeX
@misc{foster2026lala,
  author = {Michael Foster and Barbara McGillivray and Gard B. Jenset and Jakub Mácha and Jason Harding and Marcus Perlman and Devyani Sharma and Patrick Skeels and Jana Willer-Gold and Jeannette Littlemore},
  title = {{The LALA Benchmark: Evaluating Literary and Linguistic Analysis in LLMs}},
  year = {2026},
  eprint = {2610.34080},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34080},
}
Figure