Automatic Interpretation of Line Probe Assay Test for Tuberculosis

Jatin Agrawal, Mukul Kumar, Avtansh Tiwari, Sachin Danisetty, Soma Dhavala, Nakul Jain, Prasaanth Balraj, Niket Singh, Siddhant Shingi, Jayakrishna Kurada, Raghuram Rao, S. Anand, Nishant KumarAAAI 2024 · DOI

TL;DR

This paper introduces an automated system for interpreting Line Probe Assay (LPA) tests for Tuberculosis (TB), significantly reducing the time and labor required for diagnosis. By leveraging computer vision and machine learning, the system achieves over 95% accuracy, standardizes results, and incorporates a "human-in-the-loop" mechanism to enhance trustworthiness and handle ambiguous cases.

What Problem Does This Paper Address?

Tuberculosis (TB) remains a major global health concern, and diagnosing drug-resistant TB (DRTB) is crucial for effective treatment. Line Probe Assay (LPA) is a common diagnostic method, but its manual interpretation is time-consuming, labor-intensive, and prone to subjective errors. This process involves analyzing dark bands on special paper strips, which represent specific gene loci and drug resistance patterns. The manual interpretation can take up to an hour per patient, hindering rapid diagnosis and treatment initiation. This paper aims to automate this complex interpretation process, making it more efficient, standardized, and accessible.

Methods

The authors developed a multi-stage automated system for LPA interpretation:

  • Strip Extraction: The system first identifies and extracts individual patient strips from scanned LPA sheets. This involves landmark detection (e.g., doctor icon, reference strips) using ORB KeyPoint detection and RANSAC for registration, followed by sheet alignment and strip segmentation.
  • Band Identification: This crucial step involves two sub-tasks:
  • Band Detection: Object detection models like FRCNN and DETR were compared for locating bands. DETR, pre-trained on a simulated dataset and fine-tuned on real data, showed superior performance in detecting band bounding boxes.
  • Band Classification: A BiLSTM model is used to classify the detected bands based on their distance from reference bands and their neighboring band context. This process generates band probabilities.
  • Clinical Interpretation:
    • Deterministic Rules (DET): A set of predefined rules are applied to the band probabilities to infer drug outcomes (Sensitive, Resistive, or Invalid).
  • Hierarchical Neural Network (HNN): A custom HNN is trained to learn the interpretation logic, taking band probabilities and band intensity as input. This model is designed to mimic the deterministic rules but can also handle more complex scenarios.
  • Human in the Loop (HiL): To ensure robustness and trustworthiness, three modules are implemented:
  • Rejection Classifier (RC): Flags samples that are likely to be misclassified by the HNN.
  • Discrepancy Module (DM): Identifies strips where the DET and HNN outputs disagree.
  • Fidelity Module (FM): Helps identify specific bands that might be causing discrepancies, flagging them for expert review. This approach reduces the burden on lab technicians by only presenting them with ambiguous or novel samples.

Datasets

  • Real Annotated Dataset: The authors curated a dataset of 1200 LPA sheets collected from various TB laboratories across India. This dataset was annotated by multiple microbiologists and includes bounding boxes for bands, gene loci outcomes, and drug outcomes. It was split into training and testing sets.
  • Simulated Dataset: To overcome the labor-intensive and costly nature of creating real annotated datasets, a simulated dataset of approximately 10,000 sheets was generated. This dataset mimics real LPA sheets with added features like creases and wrinkles, and synthetically generated strips with bands. This simulated data was used for pre-training models.

Key Insights and Ideas

  • Automating a Complex Manual Process: The paper demonstrates that a complex, expert-driven diagnostic process like LPA interpretation can be effectively automated using modern AI techniques.
  • Synergy of Computer Vision and Machine Learning: The solution effectively combines object detection (for band localization) and sequence modeling (for band classification and clinical interpretation) to achieve high accuracy.
  • The Power of Simulation: Creating a large, diverse simulated dataset was crucial for pre-training models, significantly reducing the reliance on expensive manual annotation and enabling the generation of rare band configurations.
  • "Human-in-the-Loop" for Trust and Robustness: Incorporating a structured human-in-the-loop mechanism (RC, DM, FM) is a key differentiator, addressing the inherent subjectivity and potential for errors in medical diagnostics. This approach not only improves accuracy but also builds trust in the automated system.
  • Handling Label Noise: The authors acknowledge that label noise is incompressible and that the HiL approach is a plausible reason for achieving high accuracy, suggesting that even with perfect models, some level of human oversight is beneficial.

Key Results

  • The automated system achieves over 95% accuracy for all drugs on the real annotated dataset.
  • The DETR model outperformed FRCNN in band detection, providing more precise bounding boxes.
  • The HNN model showed superior performance compared to deterministic rules (DET), especially for samples with light or faint bands.
  • The Human-in-the-Loop system achieved near-perfect performance (Accuracy > 98%) on all drugs for both FL and SL tests, with only a small fraction of samples requiring human intervention (e.g., 37% of samples reviewed for RMP, but only 4% of total samples).
  • TrOCR model achieved 0.943 accuracy for Lab ID recognition, significantly reducing manual data entry time.

Limitations and Future Work

  • Generalization to Rare Configurations: The authors acknowledge that while their dataset is diverse, they cannot fully guarantee generalization to extremely rare band configurations not present in their dataset.
  • Light Bands: Light or faint bands remain a significant source of error, making interpretation difficult and potentially changing drug outcomes.
  • Low Support for Rare Drugs: For some SL drugs (e.g., resistive KAC and LLK), the number of samples in the dataset is very low, making performance estimates less reliable.
  • Future Work: The paper suggests that improving performance will require more diverse datasets, especially those with more examples of rare band configurations and light bands. The authors also highlight the potential for this system to be integrated into a designated Management Information System (MIS) for faster dissemination of results.

Summary updated

BibTeX
@inproceedings{agrawal2024automatic,
  author = {Jatin Agrawal and Mukul Kumar and Avtansh Tiwari and Sachin Danisetty and Soma Dhavala and Nakul Jain and Prasaanth Balraj and Niket Singh and Siddhant Shingi and Jayakrishna Kurada and Raghuram Rao and S. Anand and Nishant Kumar},
  title = {{Automatic Interpretation of Line Probe Assay Test for Tuberculosis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i20.30191},
}
Figure