From Cases to Criteria: Reusing Evaluation Knowledge for Task-Specific Rubric Generation

Qiyan Deng, Long-Kai Huang, Kangfei ZhaoarXiv 2026 · arXiv

TL;DR

RubricDuo is a novel framework that generates high-quality, task-specific evaluation rubrics for LLMs by organizing historical evaluation knowledge into a structured Knowledge Graph. By shifting from simple "query similarity" to "conceptual requirement" reuse, it significantly improves the alignment of machine-generated rubrics with human standards and boosts the accuracy of LLM-as-a-judge systems.

What Problem Does This Paper Address?

As we increasingly rely on Large Language Models (LLMs) to act as judges for complex, open-ended tasks, we face a transparency crisis. A judge that simply outputs a "7/10" or "Response A is better" provides no actionable feedback. Rubric-based evaluation solves this by decomposing requirements into specific, checkable criteria.

However, writing custom rubrics for every new query is labor-intensive. Existing automated methods typically try to reuse historical rubrics in one of two ways:

  1. Case-based retrieval: Finding a similar past query and copying its rubric. This often carries over irrelevant details (e.g., a "per night" hotel limit when the new query asks for a "total" stay cost).
  2. Pool-based distillation: Aggregating criteria into a flat list. This loses the vital context of why a specific check was necessary in the first place.

The authors argue that what actually transfers across tasks isn't the literal wording of a criterion, but the underlying evaluation requirement (e.g., "budget compliance" or "safety protocol").

Figure 1: Similarity- and requirement-based rubric reuse. (a) A similar hotel case carries over an incorrect nightly limit and misses taxes and mandatory fees. (b) A shared concept links criteria from different domains to their source query contexts; applicable checks are adapted to verify the $300 two-night total, including fees and taxes.

Methods

The authors introduce RubricDuo, which operates in two distinct phases:

1. Offline: Building the Rubric Knowledge Graph (RKG)

RubricDuo organizes historical query-rubric pairs into a multi-level graph. It extracts "local concepts" from criteria and groups them into "shared concepts" (like Budget Compliance or Family Suitability) across six general dimensions: Domain, Task, Response Protocol, Constraint, General Quality, and Safety & Policy. This structure preserves the "provenance" of evaluation knowledge—linking abstract concepts back to the specific criteria and queries that spawned them.

2. Online: Concept-Guided Generation

When a new query arrives, RubricDuo follows a two-step "Draft-then-Refine" strategy:

  • Step I: Query-level Drafting. It identifies relevant shared concepts from the RKG and retrieves similar historical cases to create a broad "draft" rubric that covers the necessary evaluation scope.
  • Step II: Criterion-level Refinement. It revisits each draft criterion, retrieves specific historical evidence linked to the same concept (even from unrelated topics), and adapts that evidence to the target query's specific constraints.
Figure 2: The RubricDuo workflow, showing RKG construction (top) and the two-step generation process (bottom).

Datasets

The framework was evaluated on two major benchmarks:

  • RubricBench: A general-purpose benchmark containing 1,147 queries across chat, coding, STEM, and safety tasks, featuring human-authored atomic rubrics.
  • HealthBench: A specialized benchmark curated with 262 physicians, featuring realistic healthcare conversations. This was used to test how well the system handles highly technical, domain-specific requirements.

Key Insights and Ideas

  • Cross-Domain Synergy: A core "aha" moment is that a rubric for a restaurant reservation (checking for service charges) can provide the "evidence" needed to refine a hotel rubric (checking for mandatory fees). Both share the concept of "total cost calculation," even if the topics differ.
  • Structure Over Similarity: Simply retrieving "similar" queries is often insufficient. The paper shows that graph-guided retrieval outperforms flat dense or hybrid retrieval because it allows the model to combine complementary evidence from different parts of the knowledge base.
  • The Power of Refinement: The two-step process is crucial. The drafting phase ensures broad coverage, while the refinement phase ensures the criteria are concrete and consistent with the target query's specific constraints.

Key Results

RubricDuo achieved the best overall performance across all evaluated automatic methods:

  • Alignment: On RubricBench, it achieved a 43.64% F1 score, significantly outperforming baselines like RubricRAG (38.12%) and Direct-Q (39.08%).
  • Utility: In downstream LLM judging, RubricDuo improved accuracy to 53.34%, closing 22% of the performance gap between automated rubrics and human-written ones.
  • Cross-Benchmark Transfer: When using a graph built from general RubricBench data to generate rubrics for medical HealthBench queries, RubricDuo maintained superior performance, proving that structured evaluation knowledge is highly transferable.
Table 1: Main results on RubricBench and HealthBench. Values are percentages: mean (sample SD) over three runs.

Limitations and Future Work

  • The Human Gap: Despite the improvements, a ~22-point gap remains between RubricDuo and human-authored rubrics on RubricBench. Automated systems still struggle with the extreme nuance experts provide.
  • RKG Dependency: The system's performance is tied to the quality and breadth of the initial Rubric Knowledge Graph.
  • Future Directions: The authors suggest exploring more sophisticated adaptation logic during the refinement step and expanding the set of evaluation dimensions to capture even more fine-grained requirements.

Summary updated

Abstract

Query-specific rubrics make LLM evaluation more fine-grained and transparent, and generating high-quality rubrics requires both identifying the target query's evaluation scope and specifying concrete checks. Historical rubrics provide reusable evaluation experiences, yet existing approaches reuse either complete query-rubric cases or aggregated criteria and dimensions without explicitly connecting recurring evaluation requirements to their criterion-level evidence and source-query context. This can preserve irrelevant task details or lose the broader evaluation structure. In this paper, we introduce RubricDuo, which organizes historical query-rubric pairs into a Rubric Knowledge Graph linking evaluation dimensions, shared concepts, local concepts, criteria, and queries, where reusable evaluation requirements across queries are abstracted by shared concepts. Given a target query, RubricDuo first combines concepts relevant to the target query with retrieved historical cases to draft a rubric with broad coverage. It then revisits each draft criterion, retrieves concept-linked historical criteria and their source queries, and adapts applicable evidence to refine the criterion. Experiments on RubricBench and HealthBench demonstrate that RubricDuo achieves the overall best performance among the evaluated automatic methods in both alignment with human-authored rubrics and downstream LLM judging.
BibTeX
@misc{deng2026from,
  author = {Qiyan Deng and Long-Kai Huang and Kangfei Zhao},
  title = {{From Cases to Criteria: Reusing Evaluation Knowledge for Task-Specific Rubric Generation}},
  year = {2026},
  eprint = {2611.25903},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25903},
}
Figure