Abstract
Ecological modeling often relies on standard statistical metrics like RMSE, which may fail to capture domain-specific nuances such as peak alignment in crop growth. This paper introduces the Adaptive Policy Evaluation Framework (APEF), which leverages Large Language Models (LLMs) to extract interpretable evaluation policies from expert pairwise comparisons. APEF bridges the gap between rigid numerical metrics and expert judgment, providing transparent, human-readable scoring rubrics that outperform traditional baselines in aligning with domain expertise.
Introduction: The Limitations of Traditional Metrics
In ecological modeling, simulating variables like greenhouse gas fluxes and hydrological cycles is critical for climate science. Traditionally, these models are evaluated using standard statistical metrics such as $R^2$ or Root Mean Square Error (RMSE). However, these metrics are often "blind" to domain-specific priorities. For instance, an agronomist may prioritize a model's ability to correctly predict the timing of a crop's peak growth over its overall average error.
As shown in Figure 1, a model with a lower overall RMSE might still be less desirable if it fails to capture the "peak period" accurately. To address this, experts often resort to manual visual inspection, which is subjective and unscalable. There is a clear need for an evaluation method that is as rigorous as a numerical metric but as "smart" and interpretable as a human expert.
Methodology: The Adaptive Policy Evaluation Framework (APEF)
The authors propose APEF, a three-stage system designed to automate expert-level evaluation while maintaining transparency.
- Base Metric Construction: APEF decomposes time-series similarity into modular components: temporal segmentation (e.g., growing seasons), peak behavior (alignment and amplitude), and derivative relationships (slopes and curvatures).
- LLM-based Weight Optimization: Using pairwise annotations (e.g., "Model A is better than Model B"), an LLM iteratively adjusts the weights of these components to reflect expert preferences.
- Policy Extraction: The LLM synthesizes a structured evaluation policy, including a named metric (e.g., "Peak Period Consistency Score"), a mathematical formula, and a decision rubric. This transforms the evaluation from a "black box" into a transparent scientific tool.
Experimental Validation
The framework was validated across three distinct datasets:
- Synthetic Data: Based on $CO_2$ flux and Gross Primary Production (GPP) data from 11 US cropland sites, researchers created 20 synthetic model predictions with known ground-truth rankings.
- Human Expert Alignment: Three domain experts provided pairwise comparisons for GPP and $CO_2$ flux to test APEF's ability to mimic real-world human intuition.
- ILAMB Benchmarking: APEF was applied to the International Land Model Benchmarking (ILAMB) system to reproduce established rankings of complex CMIP6 climate models.
Key Results and Insights
APEF demonstrated a significant improvement in aligning with expert judgment compared to traditional baselines. In "Peak Period" scenarios, APEF achieved a Spearman correlation of 0.452 for $CO_2$ flux and 0.300 for GPP, vastly outperforming $R^2$ (0.067 and 0.139, respectively).
A key insight is the resolution of the "RMSE Paradox": APEF learns to ignore "noise" in dormant periods to focus on critical segments. Furthermore, the LLM successfully "discovered" new metrics during the process, such as the Peak Period Consistency Score (PPCS), which measures the duration difference of peak periods—a metric not explicitly programmed into the base system but extracted from the data patterns.
Conclusion and Future Work
APEF provides a scalable, interpretable bridge between numerical rigor and expert nuance. While it currently relies on initial pairwise labels and involves LLM computational costs, future work aims to extend the framework to larger sets of interdependent variables and allow for natural language "assessment priorities" to guide policy extraction directly.
Summary updated