When training and evaluating Large Language Models (LLMs), we are constantly at the mercy of Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure.
In reinforcement learning from human feedback (RLHF), this manifests as reward hacking. If an automated judge or reward model uses a static, universal rubric to score responses, generative models quickly learn to exploit its systematic flaws. They discover that adding excessive bullet points, adopting an overly polite tone, or inflating response length can artificially boost scores without adding real substance. As Gupta et al. observe in Unveiling Context-Aware Criteria in Self-Assessing LLMs, static rubrics fail because they reward superficial correlates of quality rather than task-relevant correctness.
The root of the problem is that "quality" in natural language is highly context-dependent and fundamentally underspecified. As Pitis argues in Improving Context-Aware Preference Modeling for Language Models, what constitutes an excellent response changes completely based on user intent. A universal yardstick cannot accurately judge both a creative poem and a Python script; a model prioritizing creative flair will misjudge code optimization, and vice versa.
To solve this, a new paradigm has emerged: Dynamic Criteria Generation. Instead of evaluating outputs against a fixed, global reward function, these systems use an auxiliary model to analyze the user's specific prompt and dynamically synthesize a tailored evaluation rubric on the fly.
The Core Architecture: Instance-Level Rubrics
The transition from static evaluation to dynamic criteria generation shifts the reward function from a rigid mapping—$R(\text{prompt}, \text{response}) \rightarrow \text{score}$—to a two-stage contextual process:
- Criteria Generation: $\text{prompt} \rightarrow \text{custom rubric}$
- Contextual Evaluation: $(\text{prompt}, \text{response}, \text{custom rubric}) \rightarrow \text{score}$
By generating unique evaluation dimensions for every individual query, the system anchors its judgment to the exact constraints of the task.
[ User Prompt ] ───► [ Meta-Evaluator / LLM ] ───► [ Task-Specific Rubric ]
│
▼
[ LLM Response ] ────────────────────────────────► [ Contextual Judge ] ───► [ Final Score ]
Several frameworks implement this pipeline to improve alignment. For instance, Jin et al. introduced Praetor: A Fine-Grained Generative LLM Evaluator with Instance-Level Customizable Evaluation Criteria, which bypasses broad category standards to generate fine-grained, prompt-specific grading dimensions at inference time. Similarly, Saha et al. developed EvalPlanner in Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge, which explicitly forces the evaluator to formulate a custom evaluation "plan" before inspecting or scoring the candidate response.
This approach allows open-source evaluators like Prometheus 2, developed by Kim et al., to achieve high correlation with human judgment across highly diverse domains—such as medical accuracy or legal formatting—simply by conditioning the judge on custom, user-defined rubrics within the input prompt.
Concretization: Turning Abstract Goals into Verifiable Checks
A major challenge with dynamic rubrics is that an LLM might simply generate a new set of vague, abstract criteria (e.g., "The response should be high quality"). To make these rubrics actionable, frameworks must perform dynamic criteria concretization—decomposing high-level goals into specific, verifiable questions.
In CAMIEval: Enhancing NLG Evaluation through Multidimensional Comparative Instruction-Following Analysis, Fan et al. introduce a mechanism called ConcreteCoT. Instead of allowing the judge to rely on generic quality metrics, the system translates abstract criteria into a checklist of prompt-specific, binary questions. For example, if a prompt asks for a short story about a scientist, the abstract criterion of "adherence to constraints" is concretized into explicit checks: Did the response mention a laboratory? Is the word count under 500 words?
A parallel approach is found in AgentEval, a framework proposed by Arabzadeh et al. in Assessing and Verifying Task Utility in LLM-Powered Applications. AgentEval utilizes a multi-agent system (comprising a Critic, Quantifier, and Verifier) to analyze task descriptions alongside examples of successful and failed executions. It dynamically synthesizes multi-dimensional utility criteria tailored to that specific application, ensuring that the judge measures actual utility rather than superficial correctness.
Dynamic Weighting and Gating Networks
Not all criteria are created equal for every prompt. In a mathematical query, computational precision should dominate the score, while length or style should be heavily penalized or ignored. Conversely, for a creative writing prompt, stylistic nuance is paramount. Dynamic evaluation frameworks handle this by adaptively weighting sub-metrics based on the input context.
The mathematical formulation often leverages an input-dependent gating mechanism or hierarchical optimization:
Analytic Hierarchy Process (AHP)
In DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models, Xie et al. use LLMs combined with the Analytic Hierarchy Process to dynamically weight sub-metrics like semantic coherence, edit level, and fluency based on sentence context. This prevents the evaluation from tanking a score due to a minor, stylistically appropriate variation.
Gating Networks for Length Bias
To mitigate length bias without sacrificing detail when it is genuinely requested, Huo et al. designed an adaptive length bias mitigation framework (Beyond Excess and Deficiency: Adaptive Length Bias Mitigation in Reward Models for RLHF). An input-dependent gating mechanism analyzes the prompt to determine whether the user requires a concise summary or an exhaustive explanation, dynamically scaling the length reward component accordingly.
Scenario-Aware Arbitration
Feng et al. demonstrate in SaMer: A Scenario-aware Multi-dimensional Evaluator for Large Language Models that automatically recognizing the scenario and dynamically activating/weighting a subset of a predefined criteria library yields judgments that align much more closely with human intuition. This concept extends directly to safety alignment: Wen et al. show in Pragma-VL that assigning dynamic, context-sensitive weights to safety versus helpfulness objectives prevents the common pitfall of over-refusing benign but complex queries.
Why It Works: Shifting the Optimization Landscape
To understand why dynamic criteria generation works so effectively, it helps to look at the optimization landscape from the perspective of the generation policy being optimized.
When a reward model is static, it defines a fixed mathematical surface over the space of all possible text generations. Because natural language is incredibly high-dimensional, this fixed surface inevitably contains "loopholes"—directions where the model can increase the reward score without increasing true quality. The policy model exploits these loopholes during RLHF training, resulting in the classic symptoms of reward hacking (e.g., extreme verbosity).
Static Reward Landscape:
Reward Score
▲
│ ======► [Loophole: Verbosity/Formatting]
│ /
│ /\ / /\
│ / \/ / \
└─┴──┴──┴──┴──► Generation Space (Fixed Surface)
Dynamic Reward Landscape:
Reward Score
▲
│ [Surface morphs based on Prompt Intent]
│ /\ /\
│ / \____/ \ ◄── Loopholes are closed because the criteria
│/ \ shift to match the specific task constraints.
└─┴──┴──┴──┴──► Generation Space (Fluid Surface)
By introducing dynamic criteria generation, we effectively morph the reward surface for every unique prompt. As Singla et al. demonstrate in Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models, analyzing the specific intent of a query to select and prioritize criteria provides a highly precise feedback signal.
When the reward model is forced to justify its score based on context-grounded constraints—such as logical flow for a proof or stylistic constraint for a poem—the superficial shortcuts disappear. The policy model can no longer game the system by simply being wordy or formatting its output into neat bullet points, because the dynamic judge has explicitly been instructed to ignore or penalize those features unless they serve the specific task at hand.