Linking Symptom Inventories using Semantic Textual Similarity

Eamonn Kennedy, Shashank Vadlamani, Hannah M. Lindsey, Kelly S. Peterson, Kristen Dams OConnor, Kenton Murray, Ronak Agarwal, Houshang H. Amiri, Raeda K. Andersen, Talin Babikian, David A Baron, Erin D. Bigler, Karen Caeyenberghs, Lisa Delano-Wood, Seth G. Disner, Ekaterina Dobryakova, Blessen C. Eapen, Rachel M. Edelstein, Carrie Esopenko, Helen M. Genova, Elbert Geuze, Naomi J. Goodrich-Hunsaker, Jordan Grafman, Asta K. Håberg, Cooper B. Hodges, Kristen R. Hoskinson, Elizabeth S. Hovenden, Andrei Irimia, Neda Jahanshad, Ruchira M. Jha, Finian Keleher, Kimbra Kenney, Inga Koerte, Spencer W. Liebel, Abigail Livny, Marianne Lovstad, Sarah L. Martindale, Jeffrey E. Max, Andrew R. Mayer, Timothy B. Meier, Deleene S. Menefee, Abdalla Z. Mohamed, Stefania Mondello, Martin M. Monti, Rajendra A. Morey, Virginia Newcombe, et al.arXiv 2023 · DOI · arXiv

Introduction

In clinical research, particularly within the field of Traumatic Brain Injury (TBI), the lack of a single standardized assessment tool has created a significant barrier to data integration. Over several decades, various research groups have developed a multitude of self-reported symptom inventories, such as the Neurobehavioral Symptom Inventory (NSI), the Rivermead Post-Concussion Symptoms Questionnaire (RPQ), and the Brief Symptom Inventory-18 (BSI-18). While these instruments often target the same underlying clinical constructs—such as "blurred vision" or "dizziness"—their specific phrasing, scoring scales, and reference periods vary widely.

This fragmentation has resulted in a "Tower of Babel" effect, where data from independent studies cannot be easily compared or aggregated for meta-analysis. Traditionally, harmonizing these datasets required expensive, subjective panels of experts to manually map questions to one another. This paper presents an automated, objective, and scalable solution using Semantic Textual Similarity (STS) to link disparate clinical symptom inventories.

Methodology

The researchers developed an AI-driven pipeline that leverages Large Language Models (LLMs) to understand the underlying meaning of questionnaire items. Instead of relying on exact keyword matching, the system uses pre-trained Transformer models to convert question text into high-dimensional vectors, known as embeddings.

The STS Pipeline

The core technical approach involves three primary steps:

  1. Semantic Mapping: Using cosine similarity, the model identifies the "nearest neighbor" for a question in a target inventory. For example, it can autonomously map "Feeling dizzy" in one test to "Faintness or dizziness" in another.
  2. Scale Adjustment: To account for differences in how severity is measured, the team implemented a percentile-based stochastic sampling approach. This ensures that a specific score on a 5-point Likert scale in one inventory is aligned with the distribution of scores in the target inventory.
  3. Cross-Inventory Prediction: The final model predicts a patient's score on a target inventory using only their responses from a source inventory.

The study evaluated four specific models: a general-purpose model (MiniLMBERT) and three models pre-trained on clinical or biomedical corpora (ClinicalCovidBERT, VAClinicalDocsBERT, and VAMetadataBERT).

Experimental Design and Datasets

The study utilized a massive aggregated dataset for a "secondary mega-analysis," drawing from 16 international data sources, including ENIGMA, LIMBIC-CENC, and FITBIR.

  • Total Cohort: $n=6,607$ participants.
  • Validation Set: A subset of $n=2,056$ individuals who were "dually administered" two different inventories, providing a ground-truth for cross-inventory conversion accuracy.
  • Inventories Analyzed: The study focused on the NSI, RPQ, BSI-18, and the Symptom Checklist-90-Revised (SCL-90-R).

Key Insights

General Models vs. Clinical Models

One of the most surprising findings was that the general-purpose MiniLMBERT model consistently outperformed the clinically-trained models. The authors hypothesize that because clinical questionnaires are designed for patients, they are typically written in straightforward, "lay" language rather than the dense medical jargon found in clinical notes or academic papers. Consequently, models trained on general web text are better suited for interpreting these assessments.

Meaning over Correlation

The STS approach frequently outperformed traditional machine learning benchmarks, such as Random Forest or Gradient Boosting models, which rely purely on empirical correlations. This suggests that the semantic meaning of a symptom is a more robust and stable predictor than noisy statistical trends often found in clinical datasets.

Automated Discovery of Subsets

The AI successfully identified that the BSI-18 is essentially a semantic subset of the SCL-90-R. Without any prior human instruction, the model identified 18 near-identical item pairs between the two instruments, demonstrating its utility in identifying redundant assessments.

Results

The MiniLMBERT model achieved an overall accuracy of 74.8% across five distinct conversion tasks. On the more rigorous task of Exact Match Accuracy (EMA)—predicting the specific point on a 5-point Likert scale—the STS model achieved 54.3%. This significantly outperformed standard statistical benchmarks like Linear Regression (48.0%) and Gradient Boosting (40.6%). Furthermore, the model's performance reached levels very close to the "within-inventory" upper bound, indicating that the conversion process loses very little clinical information.

Discussion and Future Work

While the results are promising, the study notes several limitations. The current pipeline is restricted to inventories using 5-point scales; future iterations will need to address mapping between different numeric ranges (e.g., 3-point to 7-point scales). Additionally, the current study was limited to English-language datasets, and further research is required to determine how semantic similarity holds up across different cultural and linguistic contexts.

To facilitate widespread adoption and improve clinical reproducibility, the authors have released their pipeline as a free online tool and made the source code available on GitHub. This automated "crosswalk" represents a significant step toward unlocking massive legacy datasets for modern meta-analysis.

Summary updated

BibTeX
@misc{kennedy2023linking,
  author = {Eamonn Kennedy and Shashank Vadlamani and Hannah M. Lindsey and Kelly S. Peterson and Kristen Dams OConnor and Kenton Murray and Ronak Agarwal and Houshang H. Amiri and Raeda K. Andersen and Talin Babikian and David A Baron and Erin D. Bigler and Karen Caeyenberghs and Lisa Delano-Wood and Seth G. Disner and Ekaterina Dobryakova and Blessen C. Eapen and Rachel M. Edelstein and Carrie Esopenko and Helen M. Genova and Elbert Geuze and Naomi J. Goodrich-Hunsaker and Jordan Grafman and Asta K. Håberg and Cooper B. Hodges and Kristen R. Hoskinson and Elizabeth S. Hovenden and Andrei Irimia and Neda Jahanshad and Ruchira M. Jha and Finian Keleher and Kimbra Kenney and Inga Koerte and Spencer W. Liebel and Abigail Livny and Marianne Lovstad and Sarah L. Martindale and Jeffrey E. Max and Andrew R. Mayer and Timothy B. Meier and Deleene S. Menefee and Abdalla Z. Mohamed and Stefania Mondello and Martin M. Monti and Rajendra A. Morey and Virginia Newcombe and et al.},
  title = {{Linking Symptom Inventories using Semantic Textual Similarity}},
  year = {2023},
  eprint = {2309.04607},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.04607},
}
Figure