A novel metric of reliability in pressure pain threshold measurement

Bernard X. W. Liew, Ho Yin Lee, David Rügamer, Alessandro Marco De Nunzio, Nicola R Heneghan, Deborah Falla, David EvansScientific 2021 · DOI

Abstract

The inter-session Intraclass Correlation Coefficient (ICC) is a commonly investigated and clinically important metric of reliability for pressure pain threshold (PPT) measurement. However, current investigations do not account for inter-repetition variability when calculating inter-session ICC, even though a PPT measurement taken at different sessions must also imply different repetitions. The primary aim was to evaluate and report a novel metric of reliability in PPT measurement: the inter-session-repetition ICC. One rater recorded ten repetitions of PPT measurement over the lumbar region bilaterally at two sessions in twenty healthy adults using a pressure algometer. Variance components were computed using linear mixed-models and used to construct ICCs; most notably inter-session ICC and inter-session-repetition ICC. At 70.1% of the total variance, the source of greatest variability was between subjects ($222.28 N^2$), whereas the source of least variability (1.5% total variance) was between sessions ($4.83 N^2$). Derived inter-session and inter-session-repetition ICCs were 0.88 (95%CI: 0.77 to 0.94) and 0.73 (95%CI: 0.53 to 0.84) respectively. Inter-session-repetition ICC provides a more conservative estimate of reliability than inter-session ICC, with the magnitude of difference being clinically meaningful. Quantifying individual sources of variability enables ICC construction to be reflective of individual testing protocols.

Introduction

Pressure Pain Threshold (PPT) measurement—the minimum pressure required to evoke pain—is a staple in clinical sensory testing and research. To determine if a patient’s sensitivity is changing over time or responding to treatment, researchers rely on the Intraclass Correlation Coefficient (ICC) to quantify measurement reliability.

However, a significant flaw exists in traditional reliability calculations. Standard "inter-session" ICCs often average multiple repetitions into a single score for each session before analysis. This "collapsing" of data effectively masks the inherent variability of manual pressure application. Since any measurement taken on a subsequent day is necessarily a different physical repetition than the first, repetition-level "noise" must be integrated into the reliability model. Without accounting for this trial-to-trial variability, clinicians risk overestimating the stability of their measurements and misinterpreting clinical changes.

Methods

The study recruited 20 healthy adults (8 male, 12 female; mean age 24.6 years). A single rater used a digital pressure algometer to measure PPT at the lumbar paraspinal muscles. The protocol involved 10 repetitions per side (left and right) across two separate sessions, conducted between 48 hours and 7 days apart.

Pressure pain threshold (PPT) measurements across 10 repetitions for left and right sides over two sessions.

To analyze the data, the researchers moved beyond traditional ANOVA-based ICCs to Linear Mixed-Effects Models (LMM). This approach allowed for the decomposition of total variance into five distinct components:

  1. Between-subject variance: Differences between individuals.
  2. Inter-session variance: Differences between the two testing days.
  3. Side-to-side variance: Differences between left and right lumbar regions.
  4. Session-side interaction: Changes in side-to-side differences between days.
  5. Residual (Repetition) variance: The "noise" between consecutive manual trials.

By isolating these components, the authors constructed a novel inter-session-repetition ICC, which explicitly includes repetition-level noise in the denominator, providing a more rigorous estimate of reliability.

Results

The variance decomposition revealed that the vast majority of variance (70.1%) was between subjects ($222.28 N^2$). This indicates that the PPT test is highly effective at distinguishing between different individuals' pain thresholds. Conversely, the actual difference between sessions (days) accounted for a negligible 1.5% of the total variance ($4.83 N^2$). Crucially, repetition-level noise (residual variance) accounted for a substantial 14.4% ($45.77 N^2$).

The relationship between the number of repetitions and the resulting ICC, showing the threshold for substantial reliability.

When ignoring repetition noise, the standard inter-session ICC was 0.88 (95% CI: 0.77–0.94), suggesting "substantial" reliability. However, the novel inter-session-repetition ICC dropped to 0.73 (95% CI: 0.53–0.84), which is considered "moderate." This "overoptimism gap" demonstrates that traditional metrics may lead to inflated confidence in measurement stability.

Furthermore, the LMM framework allowed the researchers to model how many repetitions are required to reach specific reliability thresholds. While reliability improves as more repetitions are averaged, the gains diminish after the first few trials. The study suggests that averaging two repetitions per side offers a clinically feasible balance between measurement accuracy and time constraints.

Conclusion

The inter-session-repetition ICC provides a more conservative and clinically realistic estimate of PPT reliability than traditional metrics. By identifying individual sources of variability, this framework allows for the optimization of testing protocols. While this study focused on a single rater and a healthy cohort, the methodology provides a foundation for future research into multi-rater reliability and the assessment of clinical populations, such as those with chronic pain, where trial-to-trial variability may be even more pronounced.

Summary updated

Abstract

The inter-session Intraclass Correlation Coefficient (ICC) is a commonly investigated and clinically important metric of reliability for pressure pain threshold (PPT) measurement. However, current investigations do not account for inter-repetition variability when calculating inter-session ICC, even though a PPT measurement taken at different sessions must also imply different repetitions. The primary aim was to evaluate and report a novel metric of reliability in PPT measurement: the inter-session-repetition ICC. One rater recorded ten repetitions of PPT measurement over the lumbar region bilaterally at two sessions in twenty healthy adults using a pressure algometer. Variance components were computed using linear mixed-models and used to construct ICCs; most notably inter-session ICC and inter-session-repetition ICC. At 70.1% of the total variance, the source of greatest variability was between subjects ([Formula: see text] = 222.28 N<sup>2</sup>), whereas the source of least variability (1.5% total variance) was between sessions ([Formula: see text] = 4.83 N<sup>2</sup>). Derived inter-session and inter-session-repetition ICCs were 0.88 (95%CI: 0.77 to 0.94) and 0.73 (95%CI: 0.53 to 0.84) respectively. Inter-session-repetition ICC provides a more conservative estimate of reliability than inter-session ICC, with the magnitude of difference being clinically meaningful. Quantifying individual sources of variability enables ICC construction to be reflective of individual testing protocols.
BibTeX
@inproceedings{liew2021novel,
  author = {Bernard X. W. Liew and Ho Yin Lee and David Rügamer and Alessandro Marco De Nunzio and Nicola R Heneghan and Deborah Falla and David Evans},
  title = {{A novel metric of reliability in pressure pain threshold measurement}},
  booktitle = {Scientific},
  year = {2021},
  doi = {10.1038/s41598-021-86344-6},
}
Figure