Hierarchical Error Taxonomies for Diagnostic AI Evaluation

This direction is merged into another direction: hallucination taxonomy. Its last version stays readable and citable here.

Version 1 of 128 papers28 conceptsfrozen 2026-10-04

Standard performance metrics like accuracy and F1 scores are "black boxes": they indicate that a system failed, but not why. A growing body of research advocates for hierarchical error taxonomies—structured classification systems that decompose failures into granular, multi-tiered categories. By moving beyond aggregate scores, these frameworks allow researchers to distinguish between superficial procedural slips and fundamental conceptual gaps.

The Anatomy of a Failure

The core intuition behind this approach is that not all errors are created equal. A model might fail a task because it misunderstood the instructions, hallucinated a fact, or suffered a logical breakdown during a multi-step process.

Researchers like Zishan Xu and Shuyi Xie have formalized this through models like MisAttributionLLM, which uses a 21-category framework to attribute failures to specific domains such as reasoning or instruction following. Similarly, Dingling Zhang and JinCheng Ren introduced the DEFT taxonomy for research agents, which separates failures into retrieval, reasoning, and generation. This granularity is essential because the "fix" for a retrieval error (better search) is entirely different from the fix for a reasoning error (better logic).

Distinguishing Process from Outcome

A recurring theme in this research is the "unsound reasoning" problem: models often reach the correct conclusion through invalid steps. Venkatesh Mishra and Bimsara Pathiraja demonstrate this in legal contexts, where they use a structured hierarchy to separate premise-level flaws (misinterpreting a rule) from conclusion-level flaws (invalid logic).

This process-oriented diagnosis is particularly critical in multi-step tasks:

  • Mathematical Reasoning: Yupeng Su and Songmiao Wang show that quantization-induced math failures are often just early-step procedural slips rather than a loss of "intelligence."
  • Multi-Hop Analysis: Anushka Yadav and Isha Nalawade identify "overthinking" as a systematic failure mode, where models meander into irrelevant steps, losing logical fidelity over time.

From Diagnosis to Correction

Taxonomies are not just for evaluation; they are blueprints for improvement. By categorizing errors, systems can apply targeted interventions:

  • Agentic Self-Correction: The SAMULE framework by Yubin Ge and Jason Cai uses a multi-level taxonomy (micro, meso, and macro) to help agents reflect on past failures and internalize reusable principles.
  • Fault-Tolerant Interpretation: In educational settings, Helmut Horacek and Magdalena Wolska use a hierarchy of structural, type-based, and logical errors to "read between the lines" of student formulas, correcting minor typos without derailing the learning process.

Cleaning the Ground Truth

Finally, these taxonomies reveal that the "gold standard" benchmarks themselves are often flawed. Aryo Pradipta Gema and colleagues used a hierarchical decision tree to audit the MMLU benchmark, discovering that nearly 6.5% of questions contained structural issues or incorrect ground truths. By systematically categorizing these benchmark errors, researchers can ensure that model evaluations reflect actual capability rather than the ability to guess a "buggy" answer.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hierarchical_error_taxonomies_for_diagno_32596a31,
  title = {Hierarchical Error Taxonomies for Diagnostic AI Evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-error-taxonomies-for-diagnostic-ai-evaluation/rdr_32596a31f1574f099714f7be3ad4dc3e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_hierarchical_error_taxonomies_for_diagno_32596a31_v1,
  title = {Hierarchical Error Taxonomies for Diagnostic AI Evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-error-taxonomies-for-diagnostic-ai-evaluation/rdr_32596a31f1574f099714f7be3ad4dc3e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}