Collaborative artificial intelligence system for investigation of healthcare claims compliance

Marco Luca Sbodio, Vanessa López, Thanh Lam Hoang, Theodora S. Brisimi, Gabriele Picco, Inge Vejsbjerg, Valentina Rho, Pól Mac Aonghusa, Morten Kristiansen, John Segrave-DalyScientific 2024 · DOI

TL;DR

This paper introduces Clais, a novel AI system that collaborates with human experts to identify non-compliant healthcare claims. Clais automatically extracts human-readable rules from complex policy documents, allowing experts to refine them before applying them to detect fraud, waste, and abuse with high accuracy, significantly outperforming traditional machine learning models.

What Problem Does This Paper Address?

Healthcare fraud, waste, and abuse (FWA) represent a massive societal cost, diverting billions of dollars annually. Detecting non-compliant claims is crucial but challenging. Traditional methods involve manual audits by trained professionals, which are labor-intensive and struggle with the sheer volume of claims. Existing machine learning approaches often require vast amounts of labeled data, which is scarce, and their "black box" nature makes it difficult for human experts to understand and trust their decisions. Furthermore, policy documents are complex, written in natural language, and frequently updated, making it hard to translate them into consistent, executable rules for automated analysis. This paper addresses the need for a system that can bridge the gap between complex policy language and actionable claim analysis, while keeping human experts in the loop for validation and trust.

Methods

Clais employs a multi-stage approach:

  1. Rule Extraction: It uses natural language processing (NLP) techniques and a domain ontology to automatically extract human-interpretable rules from healthcare policy documents. This process involves identifying relevant text fragments, translating them into a formal rule knowledge graph, and then normalizing these rules into an executable format.
  2. Human-AI Collaboration: A key innovation is the intuitive user interface that allows human experts (policy investigators) to review, edit, and validate the AI-extracted rules. This collaborative aspect is crucial for ensuring accuracy and building trust. Experts can also manually create new rules for policy fragments that Clais might miss.
  3. Rule Execution: Once validated, Clais executes these rules on claim records. It dynamically constructs software pipelines to process claim data and identify non-compliant claims.
  4. Explainability: A significant feature is Clais's ability to provide human-understandable explanations for why a claim is flagged as non-compliant, referencing specific policy text and rule conditions.

Datasets

  • Policy Documents: The system was evaluated using two dental policy documents from Iowa and Colorado.
  • Claims Data: A large dataset of approximately 800 million claims, including nearly 45 million dental claims, was used for rule execution. This data was sourced from the MarketScan database.
  • Ground Truth: A ground truth of 141 manually defined rules was created by domain experts to evaluate rule extraction. For rule execution, 20 ground truth algorithms were developed to label non-compliant claims.

Key Insights and Ideas

  • Human-AI Collaboration is Key: The study strongly suggests that combining AI's ability to process vast amounts of text with human expertise for validation and refinement leads to more accurate and trustworthy results than either approach alone.
  • Interpretability Drives Trust: The ability of Clais to explain its findings in human-understandable terms is critical for adoption by domain experts, moving beyond "black box" AI models.
  • Bridging Policy to Practice: Clais effectively translates complex, natural language policy documents into executable logic, a significant step towards "Rules As Code" initiatives.
  • Domain Ontology is Foundational: The use of a domain ontology is crucial for guiding the AI in understanding the semantics of policy text and building consistent rule structures.

Key Results

  • Rule Extraction: Clais achieved a median F1-score of 0.72 for automatically extracting human-interpretable rules from policy documents.
  • Rule Execution: When executing these extracted rules on claim data, Clais achieved a median F1-score of 1.0 (IQR = 0.83 to 1.0). After human curation of the extracted rules, the median F1-score was also 1.0 (IQR = 1.0 to 1.0).
  • Performance vs. Baselines: Clais significantly outperformed two baseline machine learning models (gradient boosting and deep neural network) in identifying non-compliant claims.
  • User Study: A user study with 15 professional FWA investigators indicated that Clais is perceived as useful, making their workflows simpler and more effective. Participants generally responded positively to questions about the system's usefulness.

Limitations and Future Work

  • Domain Specificity: The current implementation and ontology are tailored to the dental domain. Extending Clais to other healthcare domains will require adapting the ontology and potentially fine-tuning the extraction pipeline.
  • Ground Truth Completeness: The authors acknowledge that the developed ground truth rules may not be exhaustive, and a more extensive set would be beneficial for future evaluations.
  • Future Work: The authors plan to test Clais on policies outside the dental domain, further develop functionalities based on user feedback (e.g., query exploration, claim mapping), and enhance explanations and interpretability to further build trust in the AI system.

Summary updated

Abstract

Healthcare fraud, waste and abuse are costly problems that have huge impact on society. Traditional approaches to identify non-compliant claims rely on auditing strategies requiring trained professionals, or on machine learning methods requiring labelled data and possibly lacking interpretability. We present Clais, a collaborative artificial intelligence system for claims analysis. Clais automatically extracts human-interpretable rules from healthcare policy documents (0.72 F1-score), and it enables professionals to edit and validate the extracted rules through an intuitive user interface. Clais executes the rules on claim records to identify non-compliance: on this task Clais significantly outperforms two baseline machine learning models, and its median F1-score is 1.0 (IQR = 0.83 to 1.0) when executing the extracted rules, and 1.0 (IQR = 1.0 to 1.0) when executing the same rules after human curation. Professionals confirm through a user study the usefulness of Clais in making their workflow simpler and more effective.
BibTeX
@inproceedings{sbodio2024collaborative,
  author = {Marco Luca Sbodio and Vanessa López and Thanh Lam Hoang and Theodora S. Brisimi and Gabriele Picco and Inge Vejsbjerg and Valentina Rho and Pól Mac Aonghusa and Morten Kristiansen and John Segrave-Daly},
  title = {{Collaborative artificial intelligence system for investigation of healthcare claims compliance}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41598-024-62665-0},
}

Loading related content...

Figure