TL;DR
RubricDuo is a novel framework that generates high-quality, task-specific evaluation rubrics for LLMs by organizing historical evaluation knowledge into a structured Knowledge Graph. By shifting from simple "query similarity" to "conceptual requirement" reuse, it significantly improves the alignment of machine-generated rubrics with human standards and boosts the accuracy of LLM-as-a-judge systems.
What Problem Does This Paper Address?
As we increasingly rely on Large Language Models (LLMs) to act as judges for complex, open-ended tasks, we face a transparency crisis. A judge that simply outputs a "7/10" or "Response A is better" provides no actionable feedback. Rubric-based evaluation solves this by decomposing requirements into specific, checkable criteria.
However, writing custom rubrics for every new query is labor-intensive. Existing automated methods typically try to reuse historical rubrics in one of two ways:
- Case-based retrieval: Finding a similar past query and copying its rubric. This often carries over irrelevant details (e.g., a "per night" hotel limit when the new query asks for a "total" stay cost).
- Pool-based distillation: Aggregating criteria into a flat list. This loses the vital context of why a specific check was necessary in the first place.
The authors argue that what actually transfers across tasks isn't the literal wording of a criterion, but the underlying evaluation requirement (e.g., "budget compliance" or "safety protocol").
Methods
The authors introduce RubricDuo, which operates in two distinct phases:
1. Offline: Building the Rubric Knowledge Graph (RKG)
RubricDuo organizes historical query-rubric pairs into a multi-level graph. It extracts "local concepts" from criteria and groups them into "shared concepts" (like Budget Compliance or Family Suitability) across six general dimensions: Domain, Task, Response Protocol, Constraint, General Quality, and Safety & Policy. This structure preserves the "provenance" of evaluation knowledge—linking abstract concepts back to the specific criteria and queries that spawned them.
2. Online: Concept-Guided Generation
When a new query arrives, RubricDuo follows a two-step "Draft-then-Refine" strategy:
- Step I: Query-level Drafting. It identifies relevant shared concepts from the RKG and retrieves similar historical cases to create a broad "draft" rubric that covers the necessary evaluation scope.
- Step II: Criterion-level Refinement. It revisits each draft criterion, retrieves specific historical evidence linked to the same concept (even from unrelated topics), and adapts that evidence to the target query's specific constraints.
Datasets
The framework was evaluated on two major benchmarks:
- RubricBench: A general-purpose benchmark containing 1,147 queries across chat, coding, STEM, and safety tasks, featuring human-authored atomic rubrics.
- HealthBench: A specialized benchmark curated with 262 physicians, featuring realistic healthcare conversations. This was used to test how well the system handles highly technical, domain-specific requirements.
Key Insights and Ideas
- Cross-Domain Synergy: A core "aha" moment is that a rubric for a restaurant reservation (checking for service charges) can provide the "evidence" needed to refine a hotel rubric (checking for mandatory fees). Both share the concept of "total cost calculation," even if the topics differ.
- Structure Over Similarity: Simply retrieving "similar" queries is often insufficient. The paper shows that graph-guided retrieval outperforms flat dense or hybrid retrieval because it allows the model to combine complementary evidence from different parts of the knowledge base.
- The Power of Refinement: The two-step process is crucial. The drafting phase ensures broad coverage, while the refinement phase ensures the criteria are concrete and consistent with the target query's specific constraints.
Key Results
RubricDuo achieved the best overall performance across all evaluated automatic methods:
- Alignment: On RubricBench, it achieved a 43.64% F1 score, significantly outperforming baselines like RubricRAG (38.12%) and Direct-Q (39.08%).
- Utility: In downstream LLM judging, RubricDuo improved accuracy to 53.34%, closing 22% of the performance gap between automated rubrics and human-written ones.
- Cross-Benchmark Transfer: When using a graph built from general RubricBench data to generate rubrics for medical HealthBench queries, RubricDuo maintained superior performance, proving that structured evaluation knowledge is highly transferable.
Limitations and Future Work
- The Human Gap: Despite the improvements, a ~22-point gap remains between RubricDuo and human-authored rubrics on RubricBench. Automated systems still struggle with the extreme nuance experts provide.
- RKG Dependency: The system's performance is tied to the quality and breadth of the initial Rubric Knowledge Graph.
- Future Directions: The authors suggest exploring more sophisticated adaptation logic during the refinement step and expanding the set of evaluation dimensions to capture even more fine-grained requirements.
Summary updated