Evaluating Time Series Models with Knowledge Discovery

Li Zhang

Introduction

In the rapidly evolving field of time series analysis—spanning critical applications in healthcare, seismology, and energy—researchers are increasingly developing deep, complex models. However, a significant challenge remains: these models are typically evaluated on "clean" benchmark datasets using standard error metrics such as Root Mean Square Error (RMSE) and Accuracy. This paper argues that these metrics are insufficient for determining whether a model has truly learned universal physical laws or has merely exploited dataset-specific noise. This discrepancy leads to a "generalization gap," where a model may appear highly performant on benchmarks but fails in real-world deployment because it is "right for the wrong reasons."

The Knowledge-Discovery Based Evaluation Framework

As a visionary "blue sky" contribution, this work proposes a conceptual shift toward a Knowledge-Discovery Based Evaluation Framework. The core idea is that models should be judged on their ability to provide "evidence-seeking explanations" that align with established domain expertise.

The proposed framework includes several key components:

  • Evidence-Seeking Explanation: Instead of evaluating only the final output, the framework requires the model to justify its decision using features that domain experts recognize as physically meaningful.
  • Human-in-the-loop Evaluation: The protocol moves away from purely automated metrics, incorporating domain experts to verify if a model's internal logic aligns with known scientific principles.
  • Invariance Testing: This involves checking whether the model's logic remains valid across different "latent configurations" (e.g., different sensor types or participant demographics) where the underlying physics should remain constant but the environmental noise changes.

Case Study: The GunPoint Classification Task

To illustrate the limitations of current evaluation methods, the paper examines the GunPoint classification task, a classic time series benchmark where models must distinguish between a person drawing a gun and merely pointing a finger based on hand-position sensors.

Comparison of Model 1 (99% accuracy) and Model 2 (93% accuracy) on the GunPoint task.

As shown in the figure above, two hypothetical models demonstrate the conflict between accuracy and knowledge discovery. Model 1 achieves a near-perfect 99% accuracy by focusing on the rising pattern of the signal (highlighted in blue). However, the author argues that this model likely relies on latent factors such as the operator's arm length or height. In contrast, Model 2 achieves a lower accuracy of 93% but identifies a specific "dip" in the signal (highlighted in red) caused by the physical weight of the gun. Despite the lower metric, Model 2 is considered superior because its logic is invariant to individual body types and captures the actual physical mechanism of the event.

Discussion: Knowledge as an Invariant

The paper uses the analogy of gravity to emphasize its point: a model trained on noisy data might miss the existence of universal laws if it prioritizes minimizing prediction error over discovering physical invariants. The author distinguishes between "explanation seeking" (understanding the why) and "evidence seeking" (detecting the what in the signal that proves a physical event occurred). True "knowledge," such as the weight of a gun causing a hand dip, is invariant to individual differences. Evaluation protocols should therefore reward models that identify these invariants, as they offer better potential for real-world generalization.

Limitations and Future Directions

While the proposed framework offers a path toward more robust models, several limitations must be addressed. Human-in-the-loop evaluation is difficult to scale, and the domain expertise required is often scarce and expensive. Furthermore, the paper serves as a call to action rather than a finalized protocol; it does not yet provide a standardized, automated method for implementation.

Future work should focus on developing a formal "knowledge-centric evaluation protocol" that encourages data holders to share not only raw data but also the domain-specific verification mechanisms needed to properly test model logic against physical reality.

Summary updated

BibTeX
@inproceedings{zhang2025evaluating,
  author = {Li Zhang},
  title = {{Evaluating Time Series Models with Knowledge Discovery}},
  booktitle = {SDM},
  year = {2025},
  doi = {10.1137/1.9781611978520.36},
}
Figure