LLM-based Evaluation Policy Extraction for Ecological Modeling

Qi Cheng, Licheng Liu, Qing Zhu, Runlong Yu, Zhenong Jin, Yiqun Xie, Xiaowei JiaarXiv 2025 · DOI · arXiv

Abstract

Ecological modeling often relies on standard statistical metrics like RMSE, which may fail to capture domain-specific nuances such as peak alignment in crop growth. This paper introduces the Adaptive Policy Evaluation Framework (APEF), which leverages Large Language Models (LLMs) to extract interpretable evaluation policies from expert pairwise comparisons. APEF bridges the gap between rigid numerical metrics and expert judgment, providing transparent, human-readable scoring rubrics that outperform traditional baselines in aligning with domain expertise.

Introduction: The Limitations of Traditional Metrics

In ecological modeling, simulating variables like greenhouse gas fluxes and hydrological cycles is critical for climate science. Traditionally, these models are evaluated using standard statistical metrics such as $R^2$ or Root Mean Square Error (RMSE). However, these metrics are often "blind" to domain-specific priorities. For instance, an agronomist may prioritize a model's ability to correctly predict the timing of a crop's peak growth over its overall average error.

Traditional metrics like RMSE often conflict with expert judgment, especially during critical periods like peak productivity.

As shown in Figure 1, a model with a lower overall RMSE might still be less desirable if it fails to capture the "peak period" accurately. To address this, experts often resort to manual visual inspection, which is subjective and unscalable. There is a clear need for an evaluation method that is as rigorous as a numerical metric but as "smart" and interpretable as a human expert.

Methodology: The Adaptive Policy Evaluation Framework (APEF)

The authors propose APEF, a three-stage system designed to automate expert-level evaluation while maintaining transparency.

The APEF workflow iteratively optimizes metric weights based on expert feedback to extract a structured, human-readable evaluation policy.
  1. Base Metric Construction: APEF decomposes time-series similarity into modular components: temporal segmentation (e.g., growing seasons), peak behavior (alignment and amplitude), and derivative relationships (slopes and curvatures).
  2. LLM-based Weight Optimization: Using pairwise annotations (e.g., "Model A is better than Model B"), an LLM iteratively adjusts the weights of these components to reflect expert preferences.
  3. Policy Extraction: The LLM synthesizes a structured evaluation policy, including a named metric (e.g., "Peak Period Consistency Score"), a mathematical formula, and a decision rubric. This transforms the evaluation from a "black box" into a transparent scientific tool.

Experimental Validation

The framework was validated across three distinct datasets:

  • Synthetic Data: Based on $CO_2$ flux and Gross Primary Production (GPP) data from 11 US cropland sites, researchers created 20 synthetic model predictions with known ground-truth rankings.
  • Human Expert Alignment: Three domain experts provided pairwise comparisons for GPP and $CO_2$ flux to test APEF's ability to mimic real-world human intuition.
  • ILAMB Benchmarking: APEF was applied to the International Land Model Benchmarking (ILAMB) system to reproduce established rankings of complex CMIP6 climate models.

Key Results and Insights

APEF demonstrated a significant improvement in aligning with expert judgment compared to traditional baselines. In "Peak Period" scenarios, APEF achieved a Spearman correlation of 0.452 for $CO_2$ flux and 0.300 for GPP, vastly outperforming $R^2$ (0.067 and 0.139, respectively).

A key insight is the resolution of the "RMSE Paradox": APEF learns to ignore "noise" in dormant periods to focus on critical segments. Furthermore, the LLM successfully "discovered" new metrics during the process, such as the Peak Period Consistency Score (PPCS), which measures the duration difference of peak periods—a metric not explicitly programmed into the base system but extracted from the data patterns.

Conclusion and Future Work

APEF provides a scalable, interpretable bridge between numerical rigor and expert nuance. While it currently relies on initial pairwise labels and involves LLM computational costs, future work aims to extend the framework to larger sets of interdependent variables and allow for natural language "assessment priorities" to guide policy extraction directly.

Summary updated

Abstract

Evaluating ecological time series is critical for benchmarking model performance in many important applications, including predicting greenhouse gas fluxes, capturing carbon-nitrogen dynamics, and monitoring hydrological cycles. Traditional numerical metrics (e.g., R-squared, root mean square error) have been widely used to quantify the similarity between modeled and observed ecosystem variables, but they often fail to capture domain-specific temporal patterns critical to ecological processes. As a result, these methods are often accompanied by expert visual inspection, which requires substantial human labor and limits the applicability to large-scale evaluation. To address these challenges, we propose a novel framework that integrates metric learning with large language model (LLM)-based natural language policy extraction to develop interpretable evaluation criteria. The proposed method processes pairwise annotations and implements a policy optimization mechanism to generate and combine different assessment metrics. The results obtained on multiple datasets for evaluating the predictions of crop gross primary production and carbon dioxide flux have confirmed the effectiveness of the proposed method in capturing target assessment preferences, including both synthetically generated and expert-annotated model comparisons. The proposed framework bridges the gap between numerical metrics and expert knowledge while providing interpretable evaluation policies that accommodate the diverse needs of different ecosystem modeling studies.
BibTeX
@misc{cheng2025llm,
  author = {Qi Cheng and Licheng Liu and Qing Zhu and Runlong Yu and Zhenong Jin and Yiqun Xie and Xiaowei Jia},
  title = {{LLM-based Evaluation Policy Extraction for Ecological Modeling}},
  year = {2025},
  eprint = {2505.13794},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.13794},
}
Figure