Safeguarding against spurious AI-based predictions: The case of automated verbal memory assessment

Chelsea Chandler, Peter Foltz, Alex Cohen, Terje Holmlund, Brita ElvevågNAACL 2021

TL;DR

This paper addresses the critical need for safeguards in AI-driven clinical decision support systems, particularly in mental health, to prevent erroneous predictions. The authors propose and demonstrate methods for detecting "noise" in data and evaluating model generalizability using a story recall task, aiming to build more trustworthy and reliable AI tools for patient care.

What Problem Does This Paper Address?

The increasing application of AI and natural language processing (NLP) in psychiatric research holds immense promise for improving the assessment and treatment of mental illness. However, a significant hurdle remains in translating these research findings into actual clinical decision support systems. A major concern is the risk of AI models making spurious predictions, especially when trained on limited, homogeneous datasets. This is particularly problematic in mental health, where conditions are complex, data can be noisy, and incorrect predictions can have severe consequences for patients. The paper highlights that current AI models, while potentially accurate on their training data, may not generalize well to new, unseen data encountered in real-world clinical settings. This lack of robustness can lead to unreliable diagnoses, inappropriate treatment recommendations, and ultimately, harm to patients. The authors aim to bridge this gap by developing computational safeguards to ensure AI predictions are reliable and interpretable, fostering trust among clinicians and patients.

Methods

The paper introduces two primary approaches to safeguarding AI models:

  • Attribute and Class Noise Detection: This involves identifying instances where the input data (attributes) or the predicted outcome (class) might be erroneous or misleading.
  • Outlier Detection: The authors employ the Isolation Forest algorithm to detect outliers in the data. This algorithm is particularly suited for high-dimensional behavioral data, as it efficiently isolates anomalous data points by creating shorter paths to leaves in its isolation trees. The goal is to flag data that deviates significantly from the norm, which could indicate attribute noise (e.g., unusual feature values) or class noise (e.g., incorrect labels).
  • Feature Analysis: They also analyze the relationship between predictive features. By plotting features against each other (e.g., BERTScore vs. common word types), they can identify instances where features that should be collinear show disagreement, potentially pointing to flawed feature extraction or weighting.
  • Model Generalizability Evaluation: This focuses on understanding how well a model performs on data it hasn't seen before.
  • Dataset Subsampling: The authors investigate the impact of reducing training data size on model performance. By systematically using smaller subsets of the data, they assess if the model's accuracy drops significantly, indicating instability or overfitting to the original training set.
  • Transfer Tests: The trained model is tested on entirely new, independent datasets collected from different populations (e.g., substance abuse program inpatients, healthy undergraduates). This rigorous evaluation helps determine if the model can generalize across diverse clinical populations and settings.

Datasets

The primary dataset used for training and evaluating the story recall regression model was collected via a mobile phone application called the delta Mental Status Examination (dMSE). This dataset comprised:

  • N = 25 patients with severe mental illness (schizophrenia, major depressive disorder, bipolar disorder).
  • N = 79 presumed healthy undergraduate students from Louisiana State University.

These participants completed a total of N = 118 patient sessions and N = 226 healthy participant sessions. After removing responses with no words, the final dataset contained N = 846 samples.

For transfer tests, two additional datasets were used:

  • N = 99 inpatients from a substance abuse program in Louisiana.
  • N = 124 presumed healthy undergraduates from Louisiana State University.

These combined datasets yielded N = 1254 story recalls.

Key Insights and Ideas

  • Noise is Inevitable and Harmful: The paper strongly emphasizes that noise (both attribute and class) is a common issue in mental health data and can significantly degrade AI model performance. Ignoring this noise can lead to unreliable predictions.
  • Outlier Detection is Crucial for Trust: Identifying outliers using methods like Isolation Forest is not just about finding errors; it's about flagging potentially problematic cases for human review. This "human-in-the-loop" approach is essential for building trust in AI systems.
  • Generalizability is Not Guaranteed: Even models with high accuracy on their training data can fail when applied to new populations or settings. The paper demonstrates that rigorous testing on diverse datasets is vital before clinical deployment.
  • Feature Disagreement Signals Potential Issues: Analyzing the relationships between features can reveal subtle problems, such as flawed weighting in NLP models like BERTScore, which might not be apparent from overall accuracy metrics.
  • Clinical Relevance vs. Predictive Power: High predictive power on a limited dataset does not automatically translate to clinical relevance or generalizability. Robust validation is key.

Key Results

  • The automated story recall scoring model achieved a high Pearson correlation of r = 0.91 with human expert ratings on the primary dMSE dataset.
  • The Isolation Forest algorithm identified 18 outliers in the story recall data. Of these, 9 were deemed invalid responses, and 9 were valid but showed poor performance.
  • The model's performance on outlier data was significantly lower than its overall performance.
  • When using smaller subsets of the training data (down to 25%), the model's performance (correlations for model rating, BERTScore, and common word types) showed minimal degradation, indicating good stability. For example, the average model rating correlation remained around 0.90 even with 50% of the data.
  • Transfer tests showed a strong Pearson r correlation of 0.86 when the model trained on the initial dataset was tested on new datasets, and 0.84 in the reverse direction, suggesting good generalizability across different clinical populations.

Limitations and Future Work

  • Feasibility of Ground Truth Labels: The paper acknowledges that while attribute and class noise detection is feasible in research settings with ground truth labels, this may not always be possible in eventual clinical applications.
  • Cultural Differences: The authors suggest that while the model may generalize across different clinical populations, language differences could confound transferability to differing cultural populations. They advocate for testing models on each new population before implementation.
  • Parameter Tuning: The contamination threshold for outlier detection needs careful tuning to balance detecting true outliers without including normal data. This parameter would need to be learned based on the specific application and data distribution.
  • Future Work: The authors advocate for incorporating these safeguarding approaches into all machine learning and NLP models used in clinical research and implementation. They emphasize the need for transparency, reproducibility, and clinical agreement in the association of language features to underlying illness, and call for ethics review boards to demand such rigor. The ultimate goal is a human-in-the-loop system where AI handles high-dimensional data analysis, and humans manage anomalies and interpret patient data, creating a "best of both worlds" scenario.

Summary updated

BibTeX
@inproceedings{chandler2021safeguarding,
  author = {Chelsea Chandler and Peter Foltz and Alex Cohen and Terje Holmlund and Brita Elvevåg},
  title = {{Safeguarding against spurious AI-based predictions: The case of automated verbal memory assessment}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure