Towards Reconstructing the Provenance of Clinical Guidelines

Sara Magliacane, Paul GrothSWAT4LS 2012 · OpenReview

Abstract

Clinical guidelines are essential for evidence-based medical practice, yet the history of their development—their provenance—is often poorly documented. This paper introduces a multi-signal pipeline designed to automatically reconstruct these dependencies by combining text, metadata, and image similarity metrics. Evaluated on a real-world dataset, the system demonstrates that multi-modal signals significantly improve the recall and overall F1-score of provenance reconstruction compared to traditional text-only methods.

Introduction

Understanding the provenance of clinical guidelines is critical for transparency and reproducibility in medicine. However, when guidelines are developed in shared digital environments, the specific links between research papers, drafts, and final recommendations are frequently lost. This research aims to automate the reconstruction of these document dependencies using only the available file contents and basic filesystem metadata, treating the problem as a graph generation task.

Methodology

The proposed system utilizes a multi-modal similarity pipeline to identify relationships between documents. The process involves several distinct stages:

  1. Data Gathering & Extraction: Files and metadata are collected, with text and images extracted using tools like Apache Tika.
  2. Indexing: Content is indexed using Apache Lucene (text) and LIRE (images) to facilitate similarity comparisons.
  3. Hypothesis Generation: A preliminary graph is constructed where nodes represent files and edges represent potential dependencies based on text, metadata, and image similarity.
  4. Hypothesis Pruning: Temporal constraints are applied to remove logically impossible edges (e.g., a document cannot depend on a file created after it).
  5. Aggregation: Similarity scores are combined into a single weighted sum to produce the final dependency graph, exported in the W3C PROV-DM standard.
The multi-signal pipeline prototype for provenance reconstruction, showing stages from preprocessing to aggregation.

Experimental Evaluation

The system was tested on a real-world dataset centered on a clinical guideline for febrile neutropenia. This dataset included the guideline itself, associated biomedical publications, and Excel sheets used for reference mapping. A manually annotated ground truth served as the baseline for evaluation.

Results and Key Insights

The experimental results highlight the importance of multi-modal signals. While text similarity is a useful baseline, it fails to capture many complex dependencies.

Comparison of precision, recall, and F1-score across different system configurations.

As shown in the results, the transition from a text-only baseline to "System 2" (Text + Metadata + Inverse Lucene) resulted in a dramatic increase in recall from 0.403 to 0.717, with the F1-score rising from 0.494 to 0.706. Interestingly, the inclusion of image similarity (System 3) did not provide additional benefits in this specific domain, slightly decreasing precision.

A key finding was the power and fragility of temporal pruning. While logically sound, the reliance on filesystem "creation dates" proved problematic, as these often do not align with actual publication dates, leading to the accidental pruning of valid dependencies.

Conclusion and Future Work

This work demonstrates the feasibility of automatically reconstructing document provenance in the clinical domain. The integration of metadata and advanced indexing techniques significantly boosts the system's ability to identify dependencies. Future research will focus on integrating external bibliographic data to resolve temporal inconsistencies and incorporating explicit citation networks to further refine the precision of the generated graphs.

Summary updated

BibTeX
@inproceedings{magliacane2012towards,
  author = {Sara Magliacane and Paul Groth},
  title = {{Towards Reconstructing the Provenance of Clinical Guidelines}},
  booktitle = {SWAT4LS},
  year = {2012},
  url = {https://openreview.net/forum?id=8qgLHpROzwq},
}

Loading related content...

Figure