The Protein-Protein Interaction tasks of BioCreative III: classification/ranking of articles and linking bio-ontology concepts to full text

Martin Krallinger, Miguel Vazquez, Florian Leitner, David Salgado, Andrew Chatr-aryamontri, Andrew G. Winter, Livia Perfetto, Leonardo Briganti, Luana Licata, Marta Iannuccelli, Luisa Castagnoli, Gianni Cesareni, Mike Tyers, Gerold Schneider, Fabio Rinaldi, Robert Leaman, Graciela Gonzalez, Sérgio Matos, Sun Kim, W. John Wilbur, Luis M. Rocha, Hagit Shatkay, Ashish V. Tendulkar, Shashank Agarwal, Feifan Liu, Xinglong Wang, Rafal Rak, Keith Noto, Charles Elkan, Zhiyong LuBMC Bioinformatics 2011 · DOI

TL;DR

The BioCreative III challenge focused on two crucial tasks for understanding protein-protein interactions (PPIs) from scientific literature: classifying articles relevant to PPIs and linking these interactions to specific experimental methods described in the text. This work highlights the ongoing challenges in automated biological text mining, particularly with complex relationships and the need for practical, user-oriented tools.

What Problem Does This Paper Address?

Understanding protein-protein interactions (PPIs) is fundamental to biology, driving research in areas from experimental biology to systems biology. Manually curating PPI information from the ever-growing scientific literature is a time-consuming bottleneck. This paper addresses the need for automated systems to assist in this process by tackling two key challenges:

  1. Article Classification (ACT): Identifying which scientific abstracts describe PPIs. This is the first step in manual curation, and efficient systems can significantly reduce the time researchers spend sifting through literature.
  2. Interaction Method Linking (IMT): Connecting specific PPIs described in full-text articles to the experimental methods used to detect them. This is crucial for assessing the reliability and context of reported interactions, but is more complex due to the variability in how methods are described.

The paper aims to advance text-mining tools that can go beyond simple keyword matching, providing ranked results and textual evidence to aid human annotators, ultimately improving the efficiency and accuracy of biological database curation.

Methods

The BioCreative III challenge involved two main tasks, each with specific methodologies:

  • Article Classification Task (ACT):
    • Machine Learning: Most participating teams employed machine learning techniques, including Support Vector Machines (SVMs), Naïve Bayes, and Nearest Neighbor algorithms.
  • Feature Engineering: Features extracted from articles included word tokens (unigrams, bigrams), lexical resources (MeSH, PSI-MI), textual patterns, character strings, syntactic features (dependency parsing), Part-of-Speech tags, and article metadata.
  • Ranking: Systems were evaluated on their ability to rank relevant articles, aiming to reduce manual curation time.
  • Interaction Method Task (IMT):
    • Ontology-based Classification: This task involved linking experimental method terms from the PSI-MI ontology to full-text articles.
  • Machine Learning & Pattern Matching: Teams used a combination of machine learning (e.g., logistic regression, SVMs) and pattern matching strategies to identify mentions of interaction detection methods.
  • Lexical Enrichment: Strategies included expanding the PSI-MI ontology with synonyms and related terms from other resources like MeSH and UMLS.
  • Full-Text Analysis: Unlike ACT, IMT required processing full-text articles, posing challenges related to PDF conversion and the richer textual context.

Datasets

  • Article Classification Task (ACT):
    • A large, manually curated corpus of over 12,000 PubMed abstracts was provided.
  • This corpus included training, development, and test sets, reflecting a realistic class imbalance of PPI-relevant vs. non-relevant abstracts.
  • Annotation guidelines were refined iteratively by domain experts.
  • Interaction Method Task (IMT):
    • Data collections consisted of full-text articles.
  • Training and development sets were compiled from articles annotated with PSI-MI compliant data.
  • A test set of 223 unseen publications was used for evaluation.
  • The PSI-MI ontology, containing 115 interaction detection method terms, was central to this task.

Key Insights and Ideas

  • Ranking is Key for Efficiency: For article classification, ranked lists of relevant articles significantly reduce the time needed for manual curation compared to unranked results. Systems that provide a good ranking can help curators find half of the relevant articles in a fraction of the time.
  • Full-Text Analysis is Challenging but Necessary: Linking interaction methods to full text is more difficult than abstract-based tasks due to variability in method descriptions, PDF conversion issues, and the heterogeneity of method terms.
  • Ontology Expansion is Crucial: The PSI-MI ontology, while valuable, benefits from expansion with synonyms and related terms to improve the detection of interaction methods.
  • Curator Experience Matters: Manual classification times vary significantly based on curator experience and the complexity of the article, with positive (PPI-relevant) articles generally taking longer to classify than negative ones.
  • Consensus Systems Outperform Individual Runs: For ACT, a consensus prediction generated by majority voting among multiple runs achieved better performance than the best single run.
  • Ambiguity and Generalization are Hurdles: Many method terms are general and can appear in contexts unrelated to PPIs, making accurate classification difficult.

Key Results

  • ACT:
    • The highest Matthew's Correlation Coefficient (MCC) score was 0.55, with an accuracy of 89% and an AUC iP/R of 68%.
  • A consensus system achieved an MCC of 0.57 and a better AUC iP/R (68.98%) and F-Score (63.16%) than the best single run.
  • Ranked results showed that curators could find half of the relevant articles by examining only the top 7% of the ranked list.
  • IMT:
    • The highest AUC iP/R achieved was 53%, with a best MCC score of 0.55.
  • Macro-averaged precision ranged between 50% and 80% for competitive systems with acceptable recall.
  • Some teams achieved full recall (100%) but with lower precision, highlighting the trade-off between recall and precision.
  • Performance varied significantly across different interaction method terms, with some being much harder to detect than others.

Limitations and Future Work

  • ACT Limitations: The classification of large, unbalanced article collections remains challenging.
  • IMT Limitations: Variability in method term mentions, errors from PDF pre-processing, and the heterogeneity of method concepts in the ontology pose significant challenges.
  • Future Work:
    • Developing more sophisticated text-mining modules that can provide evidence strings for human interpretation.
  • Creating practical modules for biological annotation workflows.
  • Further research into handling ambiguous and general method terms.
  • Providing more complete lexical resources for interaction methods.
  • Exploring strategies for grouping equivalent methods.
  • Future settings could distinguish between closed and open predictions (using only provided data vs. external resources).
  • The development of customizable alert systems and tools for uploading PDF full-text articles with ranked interaction detection methods and evidence passages.

Summary updated

BibTeX
@article{krallinger2011protein,
  author = {Martin Krallinger and Miguel Vazquez and Florian Leitner and David Salgado and Andrew Chatr-aryamontri and Andrew G. Winter and Livia Perfetto and Leonardo Briganti and Luana Licata and Marta Iannuccelli and Luisa Castagnoli and Gianni Cesareni and Mike Tyers and Gerold Schneider and Fabio Rinaldi and Robert Leaman and Graciela Gonzalez and Sérgio Matos and Sun Kim and W. John Wilbur and Luis M. Rocha and Hagit Shatkay and Ashish V. Tendulkar and Shashank Agarwal and Feifan Liu and Xinglong Wang and Rafal Rak and Keith Noto and Charles Elkan and Zhiyong Lu},
  title = {{The Protein-Protein Interaction tasks of BioCreative III: classification/ranking of articles and linking bio-ontology concepts to full text}},
  journal = {BMC Bioinformatics},
  year = {2011},
  doi = {10.1186/1471-2105-12-s8-s3},
}
Figure