Modeling Heterogeneous Expertise for Optimal Task Routing

32 concepts26 papers

In traditional crowdsourcing and human-in-the-loop systems, participants are often treated as interchangeable units of labor. We typically assign a single "reliability" score to an annotator or a model, assuming that if they are accurate on one task, they will be similarly accurate on another. However, as Yan et al. argue, expertise is rarely universal; an individual’s accuracy is not a static trait but a function of the specific data they are evaluating.

Recent research has shifted toward heterogeneous expertise modeling, a framework that treats the pool of available agents—whether humans, Large Language Models (LLMs), or specialized algorithms—as a collection of diverse, non-overlapping domains of competence. By learning "who knows what," these systems can route tasks to the specific expert most likely to provide a high-reward outcome, significantly improving accuracy and efficiency.

The Fallacy of the Uniform Expert

The core problem with traditional aggregation models, such as simple majority voting, is the assumption of uniform skill levels. Pfeiffer et al. note that failing to account for variations in individual proficiency leads to suboptimal results where the insights of a true expert are diluted by less informed contributors.

This variation is not just about general intelligence but about specialized knowledge. For instance, Mandal et al. demonstrate that worker reliability rankings often fail to remain consistent across different task types; a worker who excels at "easy" tasks may not be the best choice for "hard" ones. Similarly, in mobile crowdsensing, Wu et al. show that a user might be highly proficient in traffic monitoring but completely unreliable for noise sensing. Treating these workers as generalists ignores the "Who Knows What" bottleneck identified by Zhao et al..

Learning Task-Dependent Reliability

To move beyond global reliability metrics, researchers have developed mechanisms to profile expertise at a fine-grained level. The goal is to map task features to agent performance.

  • Contextual Bandits: Zhang et al. propose using a bandit framework to balance exploration (learning a worker's specific skills) with exploitation (assigning tasks to the best-known expert). This allows the system to adaptively learn task-specific reliability within a limited budget.
  • Transfer Learning: When data is scarce in a new domain, Han et al. demonstrate that knowledge patterns can be transferred from related source domains to infer worker expertise more quickly.
  • Latent Variable Modeling: To handle subjective tasks, Jin introduces models that disentangle question difficulty from subjectivity, recognizing that disagreements often stem from different "schools of thought" rather than simple incompetence.

Strategic Routing and Deferral

Once a system can model individual competencies, the challenge becomes routing: deciding in real-time whether to use an automated model, a specific human expert, or a combination of both.

In high-stakes environments like medicine, Saar-Tsechansky et al. introduce the HAI-BLBF framework, which learns to route cases to the most qualified human or rely on AI when no human expert is suitable. This "learning to defer" logic is further refined by Vernade et al., who model the decision to involve a costly human expert as a knapsack problem, optimizing for the best use of a limited human budget.

A critical nuance in routing is the presence of "hidden" information. Yin et al. argue that human experts often possess specialized intuition or access to non-digitized context that an AI lacks. Their ConfHAI framework uses causal sensitivity analysis to ensure that deferral policies are robust even when past decisions were influenced by unobserved factors.

Collaborative Synergy

The most advanced iterations of this research move from simple hand-offs to true collaboration. The CoAnnotating framework by Li et al. uses LLM uncertainty (cross-prompt entropy) to identify which data points are "easy" for AI and which require human judgment.

Rather than viewing humans and AI as competitors, the authors of the LECODU framework (Zhang et al.) unify "learning to defer" with "learning to complement." By explicitly modeling collaboration costs and individual expert competencies, the system can decide when the AI should work alone, when it should ask for a human's help, and which specific human is best suited for the task.

By treating expertise as a heterogeneous, multidimensional landscape, these research directions allow for systems that are more than the sum of their parts—optimizing for accuracy, cost, and fairness by matching every problem to its most capable solver.

Key Foundational Work

Modeling annotator expertise: Learning when everybody knows a bit of something (2010) — contributed 1 concept elements to this direction.
Adaptive Polling for Information Aggregation (2012) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_domain_specific_expertise_modeling_94b56245,
  title = {domain-specific expertise modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/domain-specific-expertise-modeling/rdr_94b562459ab64481a305e5b64654468f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_domain_specific_expertise_modeling_94b56245_v1,
  title = {domain-specific expertise modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/domain-specific-expertise-modeling/rdr_94b562459ab64481a305e5b64654468f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake