Addressing Inaccurate Nosology in Mental Health: A Multi Label Data Cleansing Approach for Detecting Label Noise from Structural Magnetic Resonance Imaging Data in Mood and Psychosis Disorders

Hooman Rokham, Godfrey D. Pearlson, Anees Abrol, Haleh Falakshahi, Sergey Plis, Vince D. CalhounbioRxiv 2020 · DOI

Abstract

Background: Mental health diagnostic approaches are seeking to identify biological markers to work alongside advanced machine learning approaches. It is difficult to identify a biological marker of disease when the traditional diagnostic labels themselves are not necessarily valid.

Methods: We worked with T1 structural magnetic resonance imaging data collected from individuals with mood and psychosis disorders from over 1400 individuals comprising healthy controls, psychosis patients and their unaffected first-degree relatives. We assumed there might be noise in the diagnostic labeling process. We detected label noise by classifying the data multiple times using a support vector machine classifier, and then we flagged those individuals in which all classifiers unanimously mislabeled those subjects. Next, we assigned a new diagnostic label to these individuals, based on the biological data (MRI), using iterative data cleansing approach.

Results: Simulation results showed our method was highly accurate in identifying label noise. Both diagnostic and Biotype categories showed about 65% and 63% respectively of noisy labels with the largest amount of relabeling occurring between the healthy control and bipolar and schizophrenia disorder individuals. The extraction of imaging features highlighted regional brain changes associated with each group.

Conclusions: This approach represents an initial step towards developing strategies that need not assume existing mental health diagnostic categories are always valid, but rather allows us to leverage this information while also acknowledging that there are misassignments.

Introduction: The Challenge of Symptom-Based Nosology

In modern medicine, diagnoses are typically confirmed through biological tests. Psychiatry, however, remains largely dependent on symptom-based nosology—the classification of diseases based on clinical observation (e.g., DSM-IV). This reliance creates a significant hurdle for researchers using Artificial Intelligence (AI) to identify biomarkers. If the clinical labels used to train machine learning models are inaccurate (label noise), the resulting models cannot accurately characterize the biological signatures of the disease. This study proposes a framework to detect and correct this noise using structural brain imaging as a biological ground truth.

Methodology and Data Cleansing Framework

The researchers utilized the Bipolar-Schizophrenia Network on Intermediate Phenotypes (B-SNIP-1) dataset, comprising 912 individuals after quality control, including healthy controls (HC), bipolar probands (BPD), schizoaffective probands (SADP), and schizophrenia probands (SZP).

The core of the methodology is a multi-label data cleansing approach that treats clinical diagnoses as potentially noisy starting points. The workflow involves four primary stages:

  1. Feature Extraction: Utilizing T1-weighted MRI scans to focus on Gray Matter Density (GMD), with ANOVA F-tests selecting the top 100 most informative features.
  2. Consensus Voting: An ensemble of Support Vector Machine (SVM) classifiers was trained. Subjects were flagged as "noisy" only if every classifier in the ensemble unanimously mislabeled them during cross-validation.
  3. Iterative Relabeling: Flagged subjects were assigned new labels based on their biological similarity to "clean" subjects.
  4. Convergence: The process was repeated until the identified noise fell below a 5% threshold.
Iterative data cleansing workflow

Results: Accuracy Gains and Biological Insights

The iterative cleansing process revealed that approximately 63-65% of traditional diagnostic labels were biologically "noisy." By refining these labels, the researchers observed a dramatic improvement in model performance. The average cross-validated accuracy of the SVM classifier rose from an initial 0.38 to 0.89 for both DSM and Biotype categories within five iterations.

Classification accuracy improvement over iterations

Beyond accuracy, the relabeling process provided deeper biological insights. Visualization of the data shifted from overlapping clusters to a clear biological gradient moving from healthy controls to severe schizophrenia. Interestingly, many "unaffected" first-degree relatives were biologically categorized as carrying signatures of Biotype 3 or SADP, despite not meeting clinical diagnostic criteria. Furthermore, cleansing the labels allowed for the identification of specific regional brain changes in the amygdala, cerebellum, and insula that were previously obscured by label noise.

Discussion and Conclusion

While this study demonstrates the power of iterative cleansing, it is limited by its reliance on a single modality (structural MRI). Future work should incorporate multi-modal data, such as functional MRI and EEG, and validate these new biological labels against long-term clinical outcomes.

Ultimately, this research argues that the search for the biological basis of mental illness requires a paradigm shift: we must allow biological data to refine and challenge traditional clinical definitions rather than assuming those definitions are immutable. This approach paves the way for a more accurate, biologically-grounded psychiatry.

Summary updated

Abstract

ABSTRACT Background Mental health diagnostic approaches are seeking to identify biological markers to work alongside advanced machine learning approaches. It is difficult to identify a biological marker of disease when the traditional diagnostic labels themselves are not necessarily valid. Methods We worked with T1 structural magnetic resonance imaging data collected from individuals with mood and psychosis disorders from over 1400 individuals comprising healthy controls, psychosis patients and their unaffected first-degree relatives including 176 bipolar probands, 134 schizoaffective probands, 240 schizophrenia proband, 581 patients relatives and 362 controls. We assumed there might be noise in the diagnostic labeling process. We detected label noise by classifying the data multiple times using a support vector machine classifier, and then we flagged those individuals in which all classifiers unanimously mislabeled those subjects. Next, we assigned a new diagnostic label to these individuals, based on the biological data (MRI), using iterative data cleansing approach. Results Simulation results showed our method was highly accurate in identifying label noise. Both diagnostic and Biotype categories showed about 65% and 63% respectively of noisy labels with the largest amount of relabeling occurring between the healthy control and bipolar and schizophrenia disorder individuals as well as in the unaffected close relatives. The extraction of imaging features highlighted regional brain changes associated with each group. Conclusions This approach represents an initial step towards developing strategies that need not assume existing mental health diagnostic categories are always valid, but rather allows us to leverage this information while also acknowledging that there are misassignments.
BibTeX
@misc{rokham2020addressing,
  author = {Hooman Rokham and Godfrey D. Pearlson and Anees Abrol and Haleh Falakshahi and Sergey Plis and Vince D. Calhoun},
  title = {{Addressing Inaccurate Nosology in Mental Health: A Multi Label Data Cleansing Approach for Detecting Label Noise from Structural Magnetic Resonance Imaging Data in Mood and Psychosis Disorders}},
  year = {2020},
  doi = {10.1101/2020.05.06.081521},
  howpublished = {bioRxiv},
}
Figure