Lazy Testing of Machine-Learning Models

Anastasia Isychev, Valentin Wüstholz, Maria ChristakisIJCAI 2025 · DOI

TL;DR

This paper introduces LAZ, a novel framework that significantly speeds up the testing of machine learning (ML) models by intelligently skipping redundant model invocations. By employing static analysis, LAZ identifies and avoids unnecessary calls to the ML model, leading to a median 33% increase in testing throughput and substantial cost savings, especially for metered APIs.

What Problem Does This Paper Address?

Testing the reliability of machine learning (ML) models is a critical but challenging task. As ML models become integral to various applications, ensuring their correctness, robustness, and fairness is paramount. Existing frameworks like NOMOS allow users to specify complex functional properties, including "hyperproperties" that involve multiple model executions. However, these tests can be computationally expensive, requiring thousands of model invocations, which translates to significant time and financial costs, particularly when using pay-per-use APIs like OpenAI's. The core problem is how to efficiently test these properties without incurring prohibitive costs.

Methods

LAZ tackles the cost of ML model testing through two primary optimizations:

  • Skipping Redundant Model Invocations: LAZ employs a technique called abstract interpretation, a form of static analysis. It analyzes the test harness code before executing it. By abstracting the program state into intervals of possible values, LAZ can determine if the outcome of a specific model invocation will affect the final test result. If the postcondition (the property being tested) can be proven to hold or fail regardless of a particular model's output, that invocation is deemed redundant and skipped. This is analogous to "lazy evaluation" in programming languages.
  • Reordering Model Invocations: To further optimize, LAZ introduces a dynamic reordering mechanism. It splits the testing campaign into an "exploration" phase and an "exploitation" phase. During exploration, LAZ tries different orders of model invocations for a small subset of tests. It then observes which order yields the highest throughput (tests per second) and the most skipped invocations. The "exploitation" phase then uses this learned best order for the remainder of the campaign. This helps prioritize skipping more expensive invocations.

Datasets

The paper evaluates LAZ on models from various domains, using pre-trained models and hyperproperties from the NOMOS framework. The specific benchmarks include:

  • Tabular Data: GermanCredit, COMPAS
  • Images: MNIST
  • Natural Language: HotelReview
  • Speech: SpeechCommand
  • Action Policies: LunarLander, BipedalWalker

The paper does not introduce new datasets but leverages existing ones commonly used in ML research.

Key Insights and Ideas

  • Static Analysis for Dynamic Testing: The core innovation is using static analysis (abstract interpretation) to optimize a dynamic testing process. This bridges the gap between compile-time analysis and runtime execution for ML models.
  • "Lazy" is Efficient: The "lazy" approach of only invoking models when their output is strictly necessary dramatically reduces computational overhead.
  • Exploration-Exploitation for Optimization: Dynamically learning the best order of model invocations based on observed performance is a clever way to adapt to the specific cost profiles of different models and properties.
  • Cost Savings are Significant: The ability to skip a substantial percentage of model invocations translates directly into reduced testing time and cost, making thorough testing more accessible.
  • Generalizability: LAZ's effectiveness is demonstrated across diverse ML domains, suggesting its broad applicability.

Key Results

  • Increased Throughput: LAZ achieves a median 33% increase in test throughput compared to NOMOS, with improvements up to 60% for some specifications.
  • Reduced Invocations: LAZ skips a median of 30% of redundant model invocations, up to 47% in some cases.
  • Comparable Bug Detection: LAZ finds the same number of bugs as NOMOS, demonstrating that skipping invocations does not compromise test effectiveness.
  • Exploration Phase Impact: The exploration phase can significantly improve throughput when the default invocation order is suboptimal, though it incurs a small overhead.
  • Analysis Overhead: The static analysis overhead is generally low, with cache hits saving a significant percentage of queries. The analysis time is often negligible compared to model invocation costs.

Limitations and Future Work

  • Analysis Overhead: While generally low, the analysis overhead can be a concern for very fast model invocations. The LAZ-FLEX variant attempts to address this by dynamically deciding whether to enable analysis in the exploitation phase.
  • Hyperparameter Sensitivity: The performance of the exploration phase is influenced by hyperparameters like explorePt (percentage of tests for exploration) and marginPt (time margin for deciding the best order). While the default settings perform well, further tuning might be beneficial for specific scenarios.
  • Future Work: The authors suggest that LAZ could complement other hyperproperty-testing techniques. The framework's ability to skip redundant invocations could theoretically be applied to any testing scenario where model invocations are expensive.

Summary updated

BibTeX
@inproceedings{isychev2025lazy,
  author = {Anastasia Isychev and Valentin Wüstholz and Maria Christakis},
  title = {{Lazy Testing of Machine-Learning Models}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/826},
}
Figure