TL;DR
The authors introduce dhAIBench, an open-source Python framework and web dashboard designed to benchmark Large Language Models (LLMs) specifically for linguistic annotation. By evaluating 48 models across nine complex tasks—ranging from Old English lemmatization to modern sentiment analysis—they demonstrate that while LLMs can match expert human performance, their success is highly task-dependent and requires rigorous, reproducible evaluation.
What Problem Does This Paper Address?
Linguistic research often hinges on high-quality annotation: identifying parts of speech, semantic roles, or discourse functions within a corpus. While traditional NLP tools are efficient, they are often "brittle"—struggling with non-standardized historical dialects or nuanced contextual interpretations.
LLMs offer a flexible alternative through natural language prompting, but they introduce new headaches:
- Reproducibility: Research papers often omit crucial details like exact prompts, model versions, or temperature settings.
- Generalization Gap: Popular leaderboards (like MMLU) don't reflect how a model performs on niche tasks like Middle English noun disambiguation.
- The "Black Box" Nature: It is difficult to know if a model is truly "understanding" the linguistic rules or just hallucinating plausible-sounding tags.
Methods
The authors developed dhAIBench to standardize the evaluation workflow. The tool follows a structured five-step process:
- Input & Setup: Researchers provide data in a keyword-in-context (KWIC) format.
- Prompting: The system separates "system prompts" (general instructions) from "user prompts" (specific data items), allowing for efficient API caching.
- Validation: A novel feature is the use of external deterministic validators. For example, in lemmatization tasks, a Python script checks the LLM's output against a known dictionary. If the model hallucinates a non-existent word, the validator triggers a retry with a list of valid candidates.
- Metrics: Beyond simple accuracy, the tool tracks Macro F1, Cohen’s kappa (for repeatability), Brier scores (for confidence calibration), and estimated monetary cost.
- Analysis: A web-based dashboard allows researchers to compare models using radar charts and scatterplots.
Datasets
The study evaluates models on nine tasks (T1–T9), focusing heavily on English varieties that are typically underserved by standard NLP:
- Historical Varieties: Old English lemmatization and noun number identification; Middle English semantic disambiguation; Early Modern English spelling normalization.
- Non-Standard English: Discourse-pragmatic uses of the word "like" in spoken British English; error correction in L2 learner English.
- Syntax & Semantics: Adverbial -ing clauses in news writing; fine-grained Part-of-Speech tagging for 18th-century correspondence.
- Baseline: Standard movie review sentiment analysis.
Key Insights and Ideas
- The Gemini Dominance: In this specific suite of linguistic tasks, the Google Gemini family (particularly Gemini-4 Argon and Gemini-3.5 Flash) consistently outperformed GPT-4 and Claude variants.
- Validators are Force Multipliers: External validators significantly boosted performance. In Old English lemmatization, the validator corrected thousands of initially wrong predictions, raising the median accuracy by over 6 percentage points.
- Cost vs. Performance: There is a staggering 14,000x difference in price between the cheapest and most expensive models tested. Interestingly, some mid-tier models provide "good enough" accuracy for a fraction of the cost of frontier models.
- Calibration Matters: Some models are "overconfident"—they provide high confidence scores even when they are wrong. dhAIBench helps identify these "confidently incorrect" runs, which is vital for researchers who might otherwise trust automated tags blindly.
Key Results
- Saturation: Several tasks, such as Middle English disambiguation and Early Modern English normalization, reached >95% accuracy, suggesting these tasks are nearly "solved" for current-generation LLMs.
- Expert Parity: In Middle English noun disambiguation, the best models reached 99% accuracy, matching the performance of a professor in the field and exceeding that of a postgraduate student (93%).
- Temperature Sensitivity: Higher temperatures generally reduced repeatability (consistency between runs) without necessarily improving accuracy. For annotation, "colder" models (Temp ≈ 0) are generally preferred.
Limitations and Future Work
The authors acknowledge that LLMs still struggle with sub-token processing. Tasks requiring word-internal segmentation (like identifying morphemes) are difficult because of how LLMs tokenize text.
Future versions of dhAIBench aim to:
- Support more complex input/output formats like XML/TEI, which are standard in digital humanities.
- Expand benchmarks to other languages (Spanish modal periphrases are already in progress).
- Test model performance on alliteration and morphological segmentation in historical texts.
The tool and current benchmarks are publicly available at dhaibench.silent3.ff.cuni.cz.
Summary updated