The first step in the development of text mining technology for cancer risk assessment: identifying and organizing scientific evidence in risk assessment literature

Anna Korhonen, Ilona Silins, Lin Sun, Ulla SteniusBMC Bioinformatics 2009 · DOI

Abstract

Cancer Risk Assessment (CRA) is a critical but labor-intensive process requiring manual review of vast biomedical literature. This study presents a foundational framework to automate CRA by developing a specialized taxonomy of scientific evidence and an expert-annotated corpus. By training machine learning classifiers, the researchers demonstrate that text mining can accurately categorize complex literature, offering a path toward more efficient, data-driven risk assessment workflows.

Introduction: The Bottleneck in Cancer Risk Assessment

Determining whether chemical exposure increases cancer risk involves sifting through thousands of scientific papers to identify "Mode of Action" (MOA) and "Toxicokinetics." Currently, this is a manual task for risk assessors. While international guidelines exist, they lack the granularity needed for computational automation. There is a pressing need for machine-readable taxonomies and gold-standard datasets to train text-mining (TM) systems to assist experts in this high-stakes domain.

Methodology and Tooling

The researchers collaborated with three experienced risk assessors to build a three-part taxonomy consisting of 48 distinct classes:

  1. Carcinogenic Activity: Animal studies and cell experiments.
  2. Mode of Action (MOA): Genotoxic vs. non-genotoxic effects.
  3. Toxicokinetics: Metabolism and excretion processes.

To facilitate data collection, a custom Firefox extension called the MedLine Annotator was developed. This tool allowed experts to browse PubMed and tag keywords or document relevance directly within their natural workflow.

The MedLine Annotator tool used by experts to tag keywords and document relevance directly in the browser.

The study evaluated three machine learning models: Naive Multinomial Bayesian (NMB), Complement Naive Bayesian (CNB), and Linear Support Vector Machines (L-SVM). They also compared "Bag of Words" (BOW) against "Bag of Substrings" (BOS) feature engineering to better handle complex biomedical terminology.

The CRA Corpus and Taxonomy

The study introduced the CRA Corpus, a benchmark dataset of 1,297 MEDLINE abstracts from 15 journals (1998–2008). The corpus focuses on eight specific chemicals with varying carcinogenic profiles.

Characteristics of the chemicals included in the CRA Corpus, detailing their Mode of Action and associated health impacts.

The taxonomy serves as a bridge, translating expert "gut feeling" into a hierarchical structure machines can navigate. Substantial inter-annotator agreement (Kappa = 0.68) confirmed the robustness of this structure. A key insight was that "Bag of Substrings" (BOS) outperformed "Bag of Words" by capturing character sequences in long, complex biological terms without requiring specialized stemmers.

Key Results and Performance

The Linear SVM model combined with BOS features emerged as the top performer, achieving an overall F-measure of 0.74. Performance varied across sub-taxonomies:

  • Carcinogenic Activity: F-measure of 0.93.
  • Toxicokinetics: F-measure of 0.78.

In real-world validation using five unseen chemicals (e.g., Aflatoxin B1), the system maintained a precision of over 0.90, demonstrating its reliability in filtering irrelevant noise for human experts.

Limitations and Future Directions

While promising, several challenges remain. Some taxonomy classes, such as "DNA repair," suffered from data scarcity, leading to lower performance. Future work will focus on expanding the taxonomy to include finer-grained MOA types and integrating these models into a comprehensive workbench that supports the entire CRA reporting lifecycle.

Summary updated

BibTeX
@article{korhonen2009first,
  author = {Anna Korhonen and Ilona Silins and Lin Sun and Ulla Stenius},
  title = {{The first step in the development of text mining technology for cancer risk assessment: identifying and organizing scientific evidence in risk assessment literature}},
  journal = {BMC Bioinformatics},
  year = {2009},
  doi = {10.1186/1471-2105-10-303},
}
Figure