TL;DR
Evaluating the factual completeness (recall) of open-ended LLM generation typically forces a compromise between expressive but ungradable structured rubrics and reliable but oversimplified flat checklists. This paper introduces a two-level meta-rubric framework that authors complex, structured criteria and mechanically compiles them into flat binary checklists for reliable LLM-as-a-judge scoring. The authors demonstrate its efficacy through GAMUT, a highly challenging new benchmark of 1,813 image-grounded "everyday deep research" questions.
What Problem Does This Paper Address?
Most work on long-form factuality evaluation focuses on precision—verifying whether the claims a model makes are true. Assessing recall—whether a model includes all the necessary information for a complete answer—is much harder.
Existing recall evaluations reduce factuality to a flat list of independent boolean checks. However, real-world knowledge is inherently structured. For instance, some questions are open-ended with unbounded sets of acceptable answers (e.g., "Which traditional dishes use ingredient X?"), others turn on a process where chronological order matters, and different facts carry different weights of importance. While human experts reason well with these structured rubrics, automated LLM judges are only reliable when scoring flat, binary checklists. This paper resolves the tension between expressiveness at authoring time and reliability at grading time.
Methods
The authors propose a two-level meta-rubric framework that decouples how a rubric is described from how it is scored:
- Structured Meta-Rubric (Authoring Time): Captures the rich organization of a complete answer. Knowledge is categorized into typed components: Simple Knowledge (isolated facts), Strict Lists (all items required), Flexible Lists (pools of valid options where a baseline threshold of coverage is required), Processes (ordered sequences), and Relationships. Items are also assigned to importance tiers: Answer-Critical, Valuable, or Context.
- Compiled Binary Checklist (Grading Time): Fixed, deterministic mechanical rules automatically compile the structured meta-rubric into a flat checklist of binary, machine-gradable checks. For example, a flexible list of seven items is converted into an Answer-Critical threshold check ("mention at least 2 of the following...") plus seven individual, lower-tier bonus checks.
An LLM judge evaluates the response against this compiled flat checklist, returning one of four verdicts per check: meets, partially meets, missing, or contradicts (which incurs a heavy penalty).
Datasets
The framework is instantiated as GAMUT (Grounded Assessment of Multimodal Factuality), a benchmark for everyday deep research:
- 1,813 questions grounded in real egocentric imagery from wearable devices (drawn from CRAG-MM) across 10 diverse domains (e.g., food, plants, vehicles, consumer products).
- The questions mimic natural queries a user might ask smart glasses (e.g., "What dish is this and how is it prepared?"), requiring multi-step web research to answer fully.
- Each question is paired with an evidence-backed rubric verified by expert human annotators, averaging 15.2 binary checks per question.
- A text-only variant (1,806 questions) is also provided, where the visual subject is explicitly named, isolating knowledge retrieval capabilities from visual perception.
Key Insights and Ideas
- Structure is Ubiquitous: A striking 98% of GAMUT's questions require at least one structured component (like a flexible list or process) beyond simple facts. This proves that conventional flat checklists fundamentally misrepresent the quality of knowledge-seeking answers.
- Omission is the Primary Failure Mode: Across all 14 frontier and open-weight models evaluated, the dominant error was omitting critical or valuable information rather than stating outright contradictions. What separates top-tier models from weaker ones is primarily how much of a complete answer they supply.
- Perception Acts as a Constant Tax: Switching to the text-only variant uniformly raised scores across all models by 10 to 20 points. Because the ranking of models remained identical, the benchmark demonstrates that the primary bottleneck for these models is knowledge completeness rather than visual recognition.
Key Results
- GAMUT is Highly Challenging: The best-performing model, Gemini 3.1 Pro, achieved a headline GAMUT score of only 58.7%, leaving substantial headroom for future models.
- Clear Proprietary vs. Open-Weight Gap: Proprietary models heavily outscored open-weight alternatives. The strongest open-weight model, Qwen3-VL 235B, scored 33.0%, while smaller models like Llama 3.2 11B struggled significantly, scoring just 5.1%.
- Robust to the Choice of Judge: The evaluation framework is highly stable. Replacing the default Gemini 3.1 Pro judge with Claude Opus 4.8 resulted in an identical model ranking, proving the evaluation is robust against judge-specific biases.
Limitations and Future Work
- Pipeline Overhead: While the compilation from meta-rubric to binary checklist is automated, generating the initial high-quality meta-rubric requires a multi-round pipeline involving frontier LLMs and expert human annotators, making it resource-intensive to scale to new domains.
- Scope of Depth: GAMUT targets "everyday deep research" rather than specialized academic or highly technical report writing. Testing the framework's expressiveness on professional or scientific domains remains future work.
Summary updated