Expert Centric Evaluation of Specialized Language Models

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2339 papers386 conceptsfrozen 2026-10-04

Large Language Models (LLMs) have reached a level of linguistic fluency that makes them appear authoritative across almost any topic. However, in specialized domains like medicine, law, and structural engineering, this fluency often masks a lack of technical veracity. Researchers have identified a "familiarity paradox": models are so well-trained on general data that they confidently inject plausible-sounding but incorrect information into technical summaries.

To bridge this gap, a growing body of research advocates for Expert-in-the-Loop (EITL) frameworks. This direction moves beyond general-purpose benchmarks and crowdsourced labels, instead positioning domain experts as the primary architects, annotators, and judges of model performance.

The Credibility Gap in Specialized Domains

The fundamental problem with evaluating LLMs in technical fields is that they are often "credible without credit." As Denis Peskoff and Brandon M. Stewart demonstrate in Credible without Credit: Domain Experts Assess Generative Language Models, models can pass a "vibe check" by being coherent and concise while failing catastrophically on factual accuracy and citation reliability.

In high-stakes environments, a model that sounds professional but provides "dangerous advice"—a phenomenon noted by Senthujan Senkaiahliyan and Augustin Toma in their assessment of GPT-4V(ision)—is worse than a model that admits ignorance. Generalist annotators cannot catch these errors because they lack the "ground truth" knowledge required to see through the model's authoritative tone.

Experts as Architects of Ground Truth

To solve the hallucination problem, researchers are shifting from passive data scraping to active, expert-driven curation. Instead of relying on what is available on the web, projects like ExpertQA, led by Chaitanya Malaviya and Dan Roth, utilize specialists to curate challenging, professional-grade questions and verify the resulting answers.

This expert-centric approach works because it captures three things general data cannot:

  1. Conditional Logic: In legal or medical contexts, the "correct" answer often depends on subtle nuances that Abhilasha Ravichander and Alan W. Black highlight in PRIVACYQA.
  2. Fine-Grained Distinction: In fields like aircraft identification or botany, structural differences are so subtle that only enthusiast or professional communities can provide accurate labels, as seen in the FGVC-Aircraft dataset by Subhransu Maji and Andrea Vedaldi.
  3. Multi-step Reasoning: Benchmarks like MMMU, developed by Xiang Yue and Kai Zhang, use college-level problems from textbooks to ensure models are performing actual reasoning rather than pattern matching.

From Generators to Discriminators

A key intuition in this research is that experts are most effective when they act as discriminators rather than generators. Asking a lawyer to write 1,000 training examples is inefficient; asking them to validate a set of "labeling rules" or heuristics is highly scalable.

Benedikt Boecking and Artur Dubrawski formalize this in Interactive Weak Supervision. By having experts provide feedback on the rules used to label data, researchers can inject domain knowledge into a system with significantly less human effort. This shift reduces cognitive load while maintaining the high precision required for specialized fields.

The Expert Productivity Multiplier

Ultimately, this research direction reframes the role of the LLM. As Amos Azaria and Rina Azoulay argue, ChatGPT is a Remarkable Tool - For Experts. Because the models are prone to subtle hallucinations, they serve as productivity multipliers only for those who already possess the knowledge to verify, debug, and refine the output.

By building benchmarks like LegalBench (Neel Guha et al.) and PLLaMa (Junfeng Gao et al.), the field is moving toward a future where AI is not a replacement for expertise, but a specialized instrument that requires an expert's hand to operate safely.

Papers in this version

Showing 60 of 339 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 181 papers
  2. Version 1 · 2026-10-04 · 339 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_domain_expert_annotation_43ca1e4c,
  title = {domain-expert annotation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/domain-expert-annotation/rdr_43ca1e4c452841a9909be293b749f73b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_expert_centric_evaluation_of_specialized_43ca1e4c_v1,
  title = {Expert Centric Evaluation of Specialized Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expert-centric-evaluation-of-specialized-language-models/rdr_43ca1e4c452841a9909be293b749f73b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}