Accounting for Precision Uncertainty of Toxicity Testing: Methods to Define Borderline Ranges and Implications for Hazard Assessment of Chemicals

Silke Gabbert, Miriam Mathea, Susanne N. Kolle, Robert LandsiedelRisk 2020 · DOI

TL;DR

This paper addresses the critical issue of uncertainty in toxicity testing by proposing and comparing methods to define "borderline ranges" (BRs). These BRs account for the inherent variability in test results, leading to more transparent hazard assessments and a better understanding of the precision of different testing methods.

What Problem Does This Paper Address?

In toxicology, continuous data from tests (whether animal or non-animal) are often converted into binary "positive" or "negative" classifications using a specific threshold (classification threshold, CT). However, experimental results are never perfectly precise due to biological and technical variability. When a test result falls close to the CT, it can be ambiguous, leading to different classifications upon retesting. This uncertainty is often overlooked. The paper aims to quantify this uncertainty by defining "borderline ranges" (BRs) around the CT, representing the zone where results are equally likely to be positive or negative. This is crucial for accurate hazard assessment and for evaluating the reliability of new testing methods, especially as the field moves towards non-animal alternatives.

Methods

The authors explore and compare several statistical methods for quantifying these borderline ranges:

  • Pooled Standard Deviation (pSD): A traditional approach using a weighted average of standard deviations from different datasets. However, this method is sensitive to outliers and non-normal data.
  • Pooled Median Absolute Deviation (MAD): A more robust alternative to pSD, using the median absolute deviation, which is less affected by extreme values.
  • Bootstrap Percentile Method: A non-parametric approach that uses resampling with replacement to estimate confidence intervals for the MAD, providing an approximate 95% confidence interval for the BR.
  • 90% Percentile of All MADs: This method aims to capture the variability of individual MADs rather than just the average, potentially leading to wider, more conservative BRs.

They also applied these methods to both log-transformed and non-transformed data to assess the impact of data transformation.

Datasets

The study utilized experimental data from nine different toxicity testing methods. These included:

  • Non-animal tests:
    • Direct Peptide Reactivity Assay (DPRA) for skin sensitization.
  • LuSens assay for skin sensitization.
  • Human Cell Line Activation Test (h-CLAT) for skin sensitization.
  • EpiDerm Skin Corrosion test (SCT) for skin corrosion.
  • EpiDerm Skin Irritation test (SIT) for skin irritation.
  • Bovine Corneal Opacity and Permeability test (BCOP) for eye irritation.
  • EpiOcular Eye Irritation test (EIT) for eye irritation.
  • Animal test:
    • Local Lymph Node Assay (LLNA) for skin sensitization.

The data were generated from routine testing in GLP and ISO 17020 certified laboratories. For one method (DPRA), experimental validation of the derived BR was also performed using a specific test substance (EGDMA).

Key Insights and Ideas

  • Precision is Paramount: The size of the borderline range is directly determined by the precision of the testing method. Less precise methods result in wider BRs, indicating greater uncertainty.
  • No Single "Perfect" Method: The study highlights that there isn't one universally superior method for calculating BRs. Different statistical approaches yield varying results, suggesting the need for consensus.
  • BRs as a Pragmatic Tool: Borderline ranges offer a practical way to acknowledge and quantify the inherent uncertainty in toxicity testing, moving beyond simple binary classifications.
  • Log Transformation's Impact: Log-transforming the data generally decreased the size of the BR, suggesting it can help in some cases to normalize data and reduce variability.
  • Experimental Validation is Key: Comparing statistically derived BRs with experimentally determined ones (as done for DPRA) is crucial for validating the methods and understanding real-world uncertainty.

Key Results

  • The study quantified BRs for various endpoints, including skin corrosion, irritation, sensitization, and eye irritation, using both non-animal and one animal test (LLNA).
  • Different statistical methods produced varying BR sizes. Generally, the pooled MAD and bootstrapped MAD methods tended to yield smaller BRs compared to the 90% percentile method, which produced wider intervals.
  • The experimental validation for the DPRA showed that the experimentally derived BR was smaller than those calculated by most statistical methods, suggesting that some statistical methods might overestimate the BR. However, one method (log pooled MAD) provided a close approximation in this specific case.
  • The paper advocates for incorporating BRs into test guidelines, leading to three possible outcomes: "positive," "negative," or "inconclusive/ambiguous" within the BR.

Limitations and Future Work

  • Limited Data Sets: The study primarily used data from a single laboratory, which might limit the generalizability of findings related to inter-laboratory variability.
  • No "Perfect" Method: As stated, the lack of a single, universally agreed-upon method for calculating BRs is a limitation.
  • Need for Consensus: The authors strongly emphasize the need for a consensus on how to define and report BRs to ensure consistency and facilitate regulatory acceptance.
  • Future Work: The authors suggest that BRs should be statistically defined within test guidelines and determined through multi-laboratory ring trials before guideline adoption. They also propose that BRs should be considered when evaluating the concordance and predictive accuracy of testing methods, especially in the context of "defined approaches" that combine multiple non-animal tests. Further work is needed to explore the implications of BRs for regulatory decision-making and to establish standardized approaches.

Summary updated

Abstract

For hazard classifications of chemicals, continuous data from animal- or nonanimal testing methods are often dichotomized into binary positive/negative outcomes by defining classification thresholds (CT). Experimental data are, however, subject to biological and technical variability. Each test method's precision is limited resulting in uncertainty of the positive/negative outcome if the experimental result is close to the CT. Borderline ranges (BR) around the CT were suggested, which represent ranges in which the study result is ambiguous, that is, positive or negative results are equally likely. The BR reflects a method's precision uncertainty. This article explores and compares different approaches to quantify the BR. Besides using the pooled standard deviation, we determine the BR by means of the median absolute deviation (MAD), with a sequential combination of both methods, and by using nonparametric bootstrapping. Furthermore, we quantify the BR for different hazardous effects, including nonanimal tests for skin corrosion, eye irritation, skin irritation, and skin sensitization as well as for an animal test on skin sensitization (the local lymph node assay, LLNA). Additionally, for one method (direct peptide reactivity assay) the BR was determined experimentally and compared to calculated BRs. Our results demonstrate that (i) the precision of the methods is determining the size of their BRs, (ii) there is no "perfect" method to derive a BR, alas, (iii) a consensus on BR is needed to account for the limited precision of testing methods.
BibTeX
@inproceedings{gabbert2020accounting,
  author = {Silke Gabbert and Miriam Mathea and Susanne N. Kolle and Robert Landsiedel},
  title = {{Accounting for Precision Uncertainty of Toxicity Testing: Methods to Define Borderline Ranges and Implications for Hazard Assessment of Chemicals}},
  booktitle = {Risk},
  year = {2020},
  doi = {10.1111/risa.13648},
}
Figure