Hierarchical Task Evaluation for Large Models

836 concepts643 papers39 proposals

When evaluating large language and multimodal models, traditional benchmarks often rely on a binary success metric: did the model output the correct final answer? However, this "flat" evaluation paradigm creates a significant diagnostic blind spot. If a model fails to solve a complex multi-step problem, a single accuracy score cannot reveal why it failed. Did it fail because it couldn't perceive the basic elements of the prompt, or because it failed to synthesize those elements into a coherent logical chain?

As noted in the design of From Generation to Selection: Findings of Converting Analogical Problem-Solving into Multiple-Choice Questions, targeting only the highest levels of cognitive processing without assessing intermediate stages prevents researchers from pinpointing exactly where a model's reasoning chain breaks down. To build better models, we need to move beyond binary success metrics toward diagnostic frameworks that map out the boundaries of machine intelligence.

The Core Mechanism: Progressive Task Hierarchies

To address this, a growing body of research advocates for hierarchical task evaluation. Instead of testing a capability in isolation, these frameworks decompose complex reasoning domains into a structured pyramid or multi-tiered progression of tasks. The evaluation moves systematically from low-level pattern recognition or data retrieval up to high-level abstraction, composition, and strategic reasoning.

By testing models across these explicit tiers, researchers can isolate the exact transition point where a model's capabilities collapse. For instance, in multimodal video understanding, Tianhao Peng et al. introduce a progressive hierarchy in MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs that separates fundamental perceptual abilities (like object counting) from higher-order skills (such as in-context learning across multiple video inputs). This structured approach transforms evaluation from a simple report card into a precise diagnostic tool.

       ▲
      / \
     /   \     Level 3: Composition & Synthesis (e.g., Multi-step Generation)
    /     \
   /-------\
  /         \  Level 2: Relational Interpretation (e.g., Spatial/Temporal Links)
 /-----------\
/             \ Level 1: Perception & Retrieval (e.g., Element Identification)
---------------

Deconstructing the Layers: From Perception to Abstraction

Hierarchical frameworks generally structure their evaluation tiers around three core cognitive stages:

  1. Perception and Retrieval: Verifying if the model can identify individual components, tokens, or visual elements.
  2. Relational and Contextual Interpretation: Testing if the model understands how these components interact or relate to one another spatially, temporally, or logically.
  3. Composition and Synthesis: Requiring the model to execute multi-step reasoning or combine multiple rules to generate a final solution.

This architecture is highly versatile and has been applied across diverse domains:

Exposing the Performance Cliff

The primary value of hierarchical task design is its ability to expose "performance cliffs"—phenomena where a model maintains high accuracy on lower-level tasks but suffers a catastrophic drop the moment composition or abstraction is required.

For example, in spatio-temporal reasoning, Carolin Holtermann et al. demonstrate via their hierarchical task design that models can successfully verify individual geographical or temporal facts, yet fail dramatically when forced to execute multi-step joint reasoning over those same facts (Around the World in 24 Hours: Probing LLM Knowledge of Time and Place).

By systematically scaling the cognitive depth or context scope—as also seen in manga narrative understanding (MangaUB: A Manga Understanding Benchmark for Large Multimodal Models) and economic rationality testing (STEER: Assessing the Economic Rationality of Large Language Models)—hierarchical evaluation proves that high performance on surface-level patterns frequently masks a fundamental inability to handle underlying structural abstractions. This granular visibility allows developers to see precisely where the cognitive chain breaks, providing a clear roadmap for targeted model improvements.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 603 papers
  2. Version 1 · 2026-10-04 · 886 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_hierarchical_task_taxonomy_0adee763,
  title = {hierarchical task taxonomy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-task-taxonomy/rdr_0adee763a5b44d7faafc813e376e389a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_hierarchical_task_taxonomy_0adee763_v2,
  title = {hierarchical task taxonomy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-task-taxonomy/rdr_0adee763a5b44d7faafc813e376e389a/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake