TL;DR
This paper introduces the first decision support system for nursing and midwifery regulatory agencies to efficiently investigate complaints. Leveraging advanced NLP and machine learning, the system predicts complaint risk levels, provides explanations, identifies similar past cases, and links to relevant regulations, aiming to reduce investigation time and cost.
What Problem Does This Paper Address?
Regulatory bodies for health professionals, like nursing and midwifery boards, play a crucial role in protecting public safety by overseeing practitioner conduct. A significant challenge they face is the investigation of complaints, which is a time-consuming and costly process. These investigations have a profound impact on practitioners' careers and reputations. The sheer volume of complaints and the need for thorough, fair, and timely decision-making necessitate more efficient tools. Existing systems often lack the sophistication to handle the nuances of complaint text and provide actionable insights to investigators.
Methods
The core of the system is an ensemble machine learning model designed for risk level prediction. This ensemble combines several base classifiers, including gradient boosting, AdaBoost, CNNs, and BERT, to leverage their diverse strengths. To address data sparsity, they employ stacked generalization, where the predictions of the base classifiers are used as input for a meta-classifier (logistic regression).
Beyond risk prediction, the system offers several key functionalities:
- Explanations: LIME (Local Interpretable Model-agnostic Explanations) is used to highlight important words in the complaint text that influenced the risk prediction, helping regulators understand the reasoning.
- Similar Past Cases: To aid decision-making, the system identifies similar past cases by first computing TF-IDF cosine similarity and then using a fine-tuned BERT model for semantic similarity ranking. To improve efficiency, they experimented with summarization models (LSA-based extractive and T5-based abstractive) to reduce the computational burden of comparing cases.
- Non-Compliance to Regulations: Natural Language Inference (NLI) models, specifically RoBERTa fine-tuned on MNLI, are used to detect potential non-compliance with regulations by identifying contradictions between complaint sentences and regulatory code entries.
- Gender Debiasing: The system incorporates gender debiasing techniques, specifically gender swapping, to mitigate potential biases in the models, acknowledging that biases can be learned from historical data.
Datasets
The system was developed and evaluated using real-world complaint data from three international regulatory agencies: the UK Nursing and Midwifery Council (NMC), the US Texas Board of Nursing (TBON), and the Australian Health Practitioner Regulation Agency (AHPRA). Initially, they worked with 1.2k complaints, which later expanded to 5.7k. This dataset includes complaint text, meta-information (like case status and referrer), and investigation results (including the risk level). The paper also mentions using semantic similarity datasets like STS-B and MedSTS for fine-tuning the BERT model for similar past case identification.
Key Insights and Ideas
- Ensemble Learning for Robustness: Combining multiple models (gradient boosting, AdaBoost, CNN, BERT) in an ensemble proved more effective than individual models, especially given the limited dataset size. This highlights the benefit of diverse perspectives in complex classification tasks.
- LIME for Explainability: Using LIME to highlight influential words provides a crucial bridge between the AI's prediction and the human investigator's understanding, fostering trust and enabling better justification of decisions.
- Summarization for Efficiency: The realization that direct sentence-to-sentence comparison for finding similar cases is computationally prohibitive, and the subsequent successful application of summarization techniques (specifically LSA-based extractive summarization) to make this task tractable, is a significant practical insight.
- Gender Debiasing is Necessary but Tricky: The study confirmed that the model exhibits gender biases. While gender swapping proved effective in reducing these biases with minimal accuracy loss, the paper notes that using gender-debiased embeddings alongside other debiasing methods can sometimes degrade performance, suggesting a complex interplay between different debiasing strategies.
- Domain Expertise is Key for Nuance: The authors acknowledge that capturing the full nuance of complaints, especially for identifying truly similar past cases, remains challenging. Human experts found that the system sometimes missed context when identifying similar cases, underscoring the need for deep domain knowledge and common-sense reasoning, which are difficult to fully automate.
Key Results
- The ensemble risk prediction model achieved an accuracy of 0.71.
- The ensemble model significantly outperformed the majority baseline and other base models in terms of accuracy and macro F1 score.
- User evaluations with regulatory staff indicated high satisfaction with the system's usability and responsiveness (average scores of 4.4 and 4.2).
- Ratings for the quality of risk predictions, explanations, and relevant regulations were moderate (2.8), while similar cases received lower ratings (1.8).
- Gender debiasing methods, particularly gender swapping, reduced gender biases (FPED and FNED) with at most a two percentage point loss in accuracy.
Limitations and Future Work
- Data Sparsity: The limited amount of labelled data was a significant challenge, necessitating ensemble methods and careful model selection.
- Nuance in Similar Case Identification: Accurately identifying truly similar past cases, especially when context is crucial, remains a difficult task. The system sometimes struggled with cases where keywords were used in different contexts.
- Explainability of Low-Risk Cases: Explanations for low-risk cases were sometimes less interpretable, potentially due to the model relying on stopwords or spurious correlations.
- Future Work: The authors plan to collect more labelled data to improve domain-specific sentence similarity and NLI models. They also aim to explore active learning to continuously update the system based on human feedback and conduct further experiments in control groups to quantify the system's impact on investigation time. They also plan to investigate ethnic biases in the future.
Summary updated