Context Aware and Dynamic Evaluation for Language Models

39 concepts36 papers1 proposal

When evaluating a large language model (LLM), traditional frameworks rely on static benchmarks—fixed sets of questions scored against rigid, predefined rubrics. However, as models transition from narrow task-solvers to general-purpose assistants, this "one-size-fits-all" approach breaks down. A fixed rubric cannot simultaneously judge the emotional resonance of a poem, the logical correctness of a Python script, and the safety constraints of a medical query.

As argued in Unveiling Context-Aware Criteria in Self-Assessing LLMs, any finite set of static criteria will inevitably fail to capture the full complexity of a true reward function across diverse tasks. This theoretical limitation manifests practically as reward hacking, where models exploit the blind spots of static rubrics (such as maximizing response length or overusing bullet points) to score highly without actually improving output quality.

To overcome this rigidity, a new paradigm has emerged: dynamic and context-aware evaluation. Instead of using fixed benchmarks, modern evaluation frameworks adapt their criteria, testing strategies, and reward functions on the fly based on the specific instance being evaluated.


Dynamic Criteria Generation: Tailoring the Rubric to the Prompt

Why should an automated judge use the same criteria for every query? In standard alignment and reinforcement learning from human feedback (RLHF), a single static reward model evaluates all outputs. This creates a noisy reward signal because the system cannot prioritize task-relevant constraints over general preferences.

To solve this, researchers have developed frameworks that dynamically generate instance-specific evaluation rules at runtime. In CARMO: Dynamic Criteria Generation for Context Aware Reward Modelling, Taneesh Gupta et al. propose using an LLM to analyze an incoming prompt and generate custom, context-relevant dimensions—such as "logical flow" for an essay or "depth of analysis" for a technical report—before judging the response. Similarly, the SaMer framework introduced by Kehua Feng et al. in SaMer: A Scenario-aware Multi-dimensional Evaluator for Large Language Models automatically identifies relevant evaluation dimensions and weights them appropriately based on the scenario context.

[Static Evaluation]
Prompt ──> Fixed Rubric (Grammar, Length, Tone) ──> Uniform Score (Prone to Hacking)

[Dynamic Evaluation]
Prompt ──> Context Analysis ──> Custom Rubric (e.g., Rigor for Math, Empathy for Stories) ──> Tailored Score

This approach changes the fundamental nature of automated judging. Instead of checking items off a rigid list, the evaluation system acts like a human expert. In Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge, Swarnadeep Saha et al. break evaluation down into two distinct stages: first, creating a custom evaluation "plan" tailored to the prompt, and second, executing that plan. By explicitly separating planning from execution, the judge avoids applying irrelevant standards and can scale across highly diverse domains without requiring hand-crafted rubrics for every new task.


Adaptive Testing: Moving Beyond Static Benchmarks

The rigidity of evaluation does not just affect reward modeling; it also limits how we benchmark models. Standard leaderboards treat every test sample as equally important, failing to predict real-world performance across varying domains where the cost of specific errors differs (How Robust are Model Rankings : A Leaderboard Customization Approach for Equitable Evaluation). Furthermore, static datasets cannot adapt their testing strategies based on a model's performance during a trial, making it difficult to find the exact boundaries of a model's capabilities (Automatic benchmarking of large multimodal models via iterative experiment programming).

To address this, recent frameworks turn benchmarking into a dynamic, interactive process. Consider TreeEval: Benchmark-Free Evaluation of Large Language Models through Tree Planning, where Xiang Li et al. replace fixed test sets with a tree-structured "interview." An LLM examiner generates questions on the fly based on the model's previous answers, allowing it to efficiently probe unique failure modes and distinguish between models with similar aggregate scores.

This adaptive philosophy is also applied to optimize evaluation efficiency and prevent data contamination:


Decomposing Complex Judgments: Multi-Turn and Step-by-Step Evaluation

When an evaluation framework relies on a single scalar score or a broad rubric, it often suffers from "judgment diffusion"—where the critical flaws of a complex response get washed out in a generalized rating. To counteract this, modern adaptive evaluators decompose the judging process into granular, sequential steps.

In Think Twice: Branch-and-Rethink Reasoning Reward Model, Yizhu Jiao et al. introduce a two-turn reasoning process for reward models. The model first identifies critical evaluation dimensions (branching) and then performs a targeted re-evaluation (rethinking) to isolate specific flaws.

For multi-step reasoning tasks, applying a single static metric across an entire generation is particularly ineffective. The authors of Dynamic and Generalizable Process Reward Modeling address this by constructing a hierarchical "reward tree" that dynamically selects fine-grained evaluation criteria for every individual step of a reasoning process. This ensures that a model is judged by a distinct, context-appropriate lens at each phase of a problem, significantly improving generalizability to entirely new domains.


Why It Works: The Intuition of Contextual Grounding

Dynamic evaluation works because it shifts the burden of generalization from the rubric designer to the evaluator model. A static rubric forces a human engineer to anticipate every possible edge case, task constraint, and failure mode ahead of time—an impossible feat for open-ended generation.

By allowing the evaluation framework to classify the task type first—such as distinguishing between factoid and non-factoid queries as done in MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs—the system can apply tailored scoring logic, such as precise key-point extraction for facts versus instance-aware ranking for creative tasks.

Furthermore, this paradigm allows for hybrid verification methods. For example, in Learning to Align Multi-Faceted Evaluation: A Unified and Robust Framework, Kaishuai Xu et al. combine dynamic criteria generation with Python code execution to verify objective constraints like word counts or formatting rules. In agentic workflows, frameworks like those proposed in Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models and ManipEvalAgent: Promptable and Efficient Evaluation Framework for Robotic Manipulation Policies use multi-round adaptive planning and dynamic code generation to test visual and robotic policies against open-ended user queries.

Ultimately, grounding the evaluation criteria in the immediate context of the prompt ensures a robust, un-hackable, and highly aligned signal, paving the way for more reliable model training and deployment.

Key Foundational Work

How Robust are Model Rankings : A Leaderboard Customization Approach for Equitable Evaluation (2021) — contributed 1 concept elements to this direction.
Personalized Feedback for Open-Response Mathematical Questions using Long Short-Term Memory Networks (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_static_evaluation_criteria_6822df72,
  title = {static evaluation criteria},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/static-evaluation-criteria/rdr_6822df7208ec4f98948c7a00cc4d3047},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_static_evaluation_criteria_6822df72_v1,
  title = {static evaluation criteria},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/static-evaluation-criteria/rdr_6822df7208ec4f98948c7a00cc4d3047/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake