RECIST 1.1 assessments variability: a systematic pictorial review of blinded double reads

Antoine Iannessi, Hubert Beaumont, Christine Ojango, Anne-Sophie Bertrand, Yan LiuInsights 2024 · DOI

TL;DR

This paper provides a comprehensive visual review of the factors contributing to variability in RECIST 1.1 assessments for cancer treatment response. It highlights how inconsistencies in lesion selection, new lesion detection, and image interpretation can impact clinical trial outcomes and patient management, emphasizing the need for standardized training and adherence to guidelines.

What Problem Does This Paper Address?

In oncology, accurately assessing a patient's response to treatment is crucial for determining treatment efficacy and guiding clinical decisions. This assessment often relies on imaging criteria, with RECIST (Response Evaluation Criteria in Solid Tumors) being a widely adopted standard. However, even with standardized criteria like RECIST 1.1, significant variability exists between radiologists in interpreting these images. This inter-observer variability can lead to inconsistent assessments of disease progression or response, potentially affecting clinical trial endpoints and, ultimately, patient care. The paper aims to dissect the root causes of this variability, particularly within the context of blinded, double-read assessments common in clinical trials, and to propose ways to mitigate it.

Methods

The paper employs a systematic pictorial review approach. It breaks down the RECIST 1.1 criteria into its core components: target lesions (TLs), non-target lesions (NTLs), and new lesions (NLs). For each component, the authors illustrate common sources of variability using clinical imaging examples. They categorize these sources into several key areas:

  • Baseline Lesion Selection: This includes issues like non-representativeness, choosing different target lesions, or failing to select the largest lesions.
  • Measurement Variability: Discrepancies in how lesions are measured, including issues with lesion definition, measurement methods, and acquisition parameters.
  • Detection of New Lesions: This covers false positives (mistaking artifacts or benign findings for new lesions) and false negatives (missing new lesions).
  • Assessment of Non-Target Lesions: The qualitative nature of assessing NTLs, which can be subjective.
  • Image Quality and Contextual Information: The impact of poor image quality, lack of contrast, or missing patient clinical history on interpretation.
  • Cognitive Biases: Such as confirmation bias, where radiologists might interpret findings to align with previous assessments.

The review uses a "toolbox" approach, presenting practical actions radiologists can take to minimize variability.

Datasets

Not discussed in the paper. The paper focuses on illustrating variability using real-world clinical imaging examples from various patients, rather than a specific, curated dataset for training or evaluation.

Key Insights and Ideas

  • Baseline Selection is Paramount: The initial choice of target lesions significantly influences subsequent assessments. If the baseline selection isn't representative of the overall disease burden, it can lead to downstream discrepancies.
  • "New Lesions" are a Major Pitfall: The detection and characterization of new lesions are frequent sources of discordance, with false positives and false negatives contributing substantially to variability. This is particularly critical as the appearance of a new lesion often triggers a "progressive disease" classification, impacting treatment decisions.
  • Subjectivity Creeps In: While RECIST aims for quantification, many aspects, especially the assessment of non-target lesions and the distinction between scars and residual disease, remain qualitative and thus prone to subjective interpretation.
  • Context is King (But Often Missing): In blinded central reviews, radiologists are often deprived of crucial clinical information. This lack of context can lead to diagnostic uncertainty and variability, while conversely, having too much context can introduce bias.
  • Cognitive Biases are Real: Radiologists, like all humans, are susceptible to cognitive biases such as "satisfaction of search" or "confirmation bias," which can subtly influence their interpretations over time.
  • Small Differences Matter: Even millimetric differences in lesion measurements can lead to different response classifications (e.g., stable disease vs. progressive disease), especially when dealing with small residual lesions or patients with a low tumor burden.

Key Results

The paper doesn't present quantitative results in terms of accuracy metrics or statistical comparisons. Instead, its key findings are qualitative and illustrative:

  • Categorization of Variability Sources: The paper systematically identifies and visually demonstrates the primary causes of RECIST 1.1 variability across target lesions, non-target lesions, and new lesions.
  • Pictorial Evidence: Through numerous figures, the paper provides clear visual examples of how different interpretations of the same imaging findings can occur, leading to discordance.
  • "Toolbox" for Radiologists: A practical table (Table 2) summarizes actionable steps radiologists can take to mitigate variability, covering lesion selection, new lesion detection, NTL assessment, and target lesion measurement.
  • Emphasis on Training and Standardization: The overarching message is that adherence to RECIST guidelines and comprehensive training are essential for reducing errors and improving consistency.

Limitations and Future Work

  • Focus on Pictorial Review: While effective for illustration, this method doesn't provide quantitative data on the prevalence or impact of each variability factor.
  • Lack of Specific Dataset: The paper doesn't introduce a new benchmark dataset, which could be valuable for developing and testing automated or semi-automated tools to reduce variability.
  • Future Work Suggestions: The authors implicitly suggest that further research could focus on developing more objective or automated methods for lesion detection and measurement. They also highlight the importance of understanding how different trial phases (e.g., Phase 2 vs. Phase 3) might require different strategies for minimizing variability, with Phase 2 trials potentially needing closer monitoring of lesion measurement and selection, while Phase 3 trials might prioritize new lesion assessment. The paper also points to the need for better integration of contextual information into blinded review processes where feasible.

Summary updated

Abstract

Reader variability is intrinsic to radiologic oncology assessments, necessitating measures to enhance consistency and accuracy. RECIST 1.1 criteria play a crucial role in mitigating this variability by standardizing evaluations, aiming to establish an accepted "truth" confirmed by histology or patient survival. Clinical trials utilize Blind Independent Centralized Review (BICR) techniques to manage variability, employing double reads and adjudicators to address inter-observer discordance effectively. It is essential to dissect the root causes of variability in response assessments, with a specific focus on the factors influencing RECIST evaluations. We propose proactive measures for radiologists to address variability sources such as radiologist expertise, image quality, and accessibility of contextual information, which significantly impact interpretation and assessment precision. Adherence to standardization and RECIST guidelines is pivotal in diminishing variability and ensuring uniform results across studies. Variability factors, including lesion selection, new lesion appearance, and confirmation bias, can have profound implications on assessment accuracy and interpretation, underscoring the importance of identifying and addressing these factors. Delving into the causes of variability aids in enhancing the accuracy and consistency of response assessments in oncology, underscoring the role of standardized evaluation protocols and mitigating risk factors that contribute to variability. Access to contextual information is crucial. CRITICAL RELEVANCE STATEMENT: By understanding the causes of diagnosis variability, we can enhance the accuracy and consistency of response assessments in oncology, ultimately improving patient care and clinical outcomes. KEY POINTS: Baseline lesion selection and detection of new lesions play a major role in the occurrence of discordance. Image interpretation is influenced by contextual information, the lack of which can lead to diagnostic uncertainty. Radiologists must be trained in RECIST criteria to reduce errors and variability.
BibTeX
@inproceedings{iannessi2024recist,
  author = {Antoine Iannessi and Hubert Beaumont and Christine Ojango and Anne-Sophie Bertrand and Yan Liu},
  title = {{RECIST 1.1 assessments variability: a systematic pictorial review of blinded double reads}},
  booktitle = {Insights},
  year = {2024},
  doi = {10.1186/s13244-024-01774-w},
}
Figure