eEvidence: Information Seeking Support for Evidence-based Practice: An Implementation Case Study.

Jin Zhao, Min‐Yen Kan, Paula Procter, Siti Zubaidah, Wai Kin Yip, Goh Mien LiPubMed 2010

TL;DR

This paper introduces "eEvidence," a novel system designed to streamline evidence-based practice (EBP) for healthcare professionals. It tackles the challenge of information overload by automatically collecting, classifying, and extracting key information from freely available online EBP resources, offering both active and passive search interfaces to suit user needs and time constraints.

What Problem Does This Paper Address?

Evidence-based practice (EBP) is crucial in healthcare, requiring practitioners to integrate the best research evidence with clinical expertise and patient values. However, the process of finding and selecting relevant evidence is often hindered by several factors. EBP resources are frequently siloed, requiring separate searches across multiple subscription-based databases. Generic search engines lack specialized medical knowledge and proper categorization, while specialized medical search engines are often restricted in accessibility. This complexity, coupled with the time constraints faced by healthcare professionals, makes it difficult to stay current with the latest research. The paper addresses this by proposing a system that automates much of the evidence gathering and selection process, making it more efficient and accessible.

Methods

The eEvidence system is built around a three-pronged approach:

  • Periodic Crawling of EBP Resources: The system uses the Nutch crawler to periodically collect freely accessible articles from a curated set of EBP websites. This ensures a broad coverage of resources and keeps the collection up-to-date.
  • Automated Article Classification and Key Information Extraction: Machine learning classifiers are employed to categorize webpages into relevant types (e.g., abstract, full text) and filter out irrelevant content. Crucially, key information such as publication year, article type, key sentences (relevant to clinical questions), and keywords (categorized by condition, sex, age, etc.) is extracted. This extraction is performed using Maximum Entropy models trained on annotated data.
  • Dual Active/Passive User Interface: The system offers two distinct interfaces:
  • Read Interface (Passive): This interface proactively presents users with the latest relevant articles based on their stored profiles (primary and secondary keywords representing interests).
  • Search Interface (Active): This interface allows users to formulate queries using primary and secondary keywords, with options for Boolean operators and filtering by facets like publication year.

The system architecture involves four stages: crawling, classification/extraction, indexing (using Apache Lucene), and user interaction through the read or search interfaces.

Datasets

The paper describes the creation of a training dataset for their classifiers by randomly choosing and annotating 500 webpages harvested from EBP resources. This dataset was used for training and evaluating their classification and extraction models using 5-fold cross-validation. No specific public benchmark datasets are mentioned as being used for comparison, but the paper implies the creation of a novel collection of EBP resources.

Key Insights and Ideas

  • Automating the "Drudgery" of Evidence Gathering: The core idea is to offload the time-consuming and often frustrating task of searching and filtering EBP literature from busy healthcare professionals to an automated system.
  • The Power of Key Information Extraction: Extracting specific pieces of information like "key sentences" and "keywords" is highlighted as a significant aid in quickly assessing an article's relevance and validity, saving users from reading entire papers.
  • Balancing Active and Passive Information Seeking: Recognizing that users have different needs and time availability, the dual interface approach is a smart way to cater to both proactive researchers and those who prefer to be passively updated.
  • Leveraging Freely Available Resources: The focus on crawling freely accessible web content democratizes access to EBP information, bypassing the common barrier of expensive subscriptions.

Key Results

The paper reports on the performance of their classification and extraction models:

  • Webpage Type Classification: Achieved high performance with a Marco-averaged F1-score of .98 for abstract, full text, and other webpage types.
  • Key Sentence Extraction: Demonstrated good precision (.70-.92) but lower recall (.24-.56) across different information types (Patient, Intervention, Study Design, Result, Research Goal). The authors note this trend favors their interface, as limited screen real estate can effectively display a limited set of precise key sentences.
  • Keyword Extraction: Showed strong performance for categories with fixed vocabularies (e.g., Sex, F1=.99) but room for improvement in categories with diverse vocabularies (e.g., Condition, F1=.69).

Qualitatively, informal evaluations with healthcare professionals (nurses) were mostly positive. Key appreciated features included the classification/extraction for focusing on relevant content and the ability to find free full-text articles. The search history function was also noted as useful for writing systematic reviews.

Limitations and Future Work

The authors acknowledge a significant limitation: the current collection of EBP resources is still small compared to the overall pool of available information. They were only able to crawl 17 out of 94 recommended websites due to robot exclusion policies. Their current collection consists of 3371 full-text articles and 16,522 abstracts.

Future work includes:

  • Extending the Collection: Aiming to index more documents to enable more realistic and sizeable evaluations, such as full literature reviews.
  • Full-Fledged User Evaluation: Conducting a comprehensive evaluation to assess the system's usefulness and efficiency in EBP implementation.
  • Investigating Information Seeking Behaviors: Studying how these behaviors differ across various healthcare professional types to develop more user- and group-specific features.

Summary updated

Abstract

We propose to collect freely available articles from the web to build an evidence-based practice resource collection with up-to-date coverage, and then apply automated classification and key information extraction on the collected articles to provide means for sounder relevance judgments. We implement these features into a dual-interface system that allows users to choose between an active or passive information seeking process depending on the amount of time available.
BibTeX
@inproceedings{zhao2010eevidence,
  author = {Jin Zhao and Min‐Yen Kan and Paula Procter and Siti Zubaidah and Wai Kin Yip and Goh Mien Li},
  title = {{eEvidence: Information Seeking Support for Evidence-based Practice: An Implementation Case Study.}},
  booktitle = {PubMed},
  year = {2010},
}
Figure