Unsupervised Identification of Study Descriptors in Toxicology Research: An Experimental Study

Drahomira Herrmannova, Steven Young, Robert Patton, Christopher Stahl, Nicole Kleinstreuer, Mary WolfeEMNLP 2018

TL;DR

This paper introduces an unsupervised method to automatically identify key study descriptors within toxicology research publications. By leveraging word embeddings and semantic similarity, the approach can pinpoint relevant text segments without requiring manually annotated data, significantly streamlining the process of extracting crucial information for systematic reviews and database construction.

What Problem Does This Paper Address?

Extracting specific data elements, such as species, route of administration, or dosing regimens, from full-text scientific publications is a critical but incredibly time-consuming and labor-intensive task. This is particularly true in fields like toxicology, where the sheer volume of published literature (over a million new papers annually) makes manual review infeasible for comprehensive analyses. Existing methods often rely on supervised learning, which necessitates large amounts of manually labeled data – a process that is itself the bottleneck the automation aims to solve. This paper addresses the challenge of automating this extraction process in an unsupervised manner, meaning it doesn't require pre-labeled examples to learn from.

Methods

The core of the proposed method lies in using word embeddings (specifically Word2Vec) to represent words and phrases as vectors in a semantic space. The authors treat the criteria descriptions (e.g., "MC 1: Animal model") as "questions" and the text segments within a document as potential "answers."

Here's a breakdown of their approach:

  1. Segment Extraction: Documents are first broken down into sentences.
  2. Segment/Description Representation: Both the sentences and the criteria descriptions are converted into vector representations using Word2Vec embeddings trained on the corpus of toxicology publications.
  3. Word-to-Word Similarities: Cosine similarity is used to calculate the relatedness between individual words in a sentence and words in the criteria description, generating a similarity matrix.
  4. Segment-to-Description Similarities: A single similarity score is computed for each sentence relative to the criteria description. This is achieved by taking the maximum similarity value for each word in the sentence and then averaging these maximums.
  5. Candidate Segments: The top-k sentences with the highest similarity scores are identified as candidate segments.
  6. N-gram Refinement (Optional): For finer granularity, the top sentences are further analyzed by breaking them down into continuous n-grams (e.g., 5-grams) to pinpoint the most relevant phrase within the sentence.

The key innovation is the unsupervised nature of this similarity-based retrieval, allowing it to work with just the document and the criteria description, without needing pre-annotated text.

Datasets

The primary dataset used for both training and evaluation is a curated database of rodent uterotrophic bioassay publications. This database contains 670 publications, each labeled with 0/1 values for six minimum criteria (MC) and an overall guideline-like (GL) label, indicating whether the study adhered to OECD guidelines. The authors note that this database, while containing publication-level labels, lacks fine-grained sentence-level annotations, making it an ideal testbed for their unsupervised approach. They successfully converted 624 full-text documents from PDFs into structured text for their experiments.

Key Insights and Ideas

  • Semantic Similarity as a Proxy for Relevance: The fundamental idea that semantic relatedness between a criteria description and text segments can effectively identify relevant information is a powerful insight. This bypasses the need for explicit keyword matching or complex rule-based systems.
  • Unsupervised Learning for Information Extraction: Demonstrating that unsupervised methods can achieve competitive results in a domain typically reliant on supervised learning is a significant contribution. This opens doors for more scalable information extraction.
  • Word Embeddings Capture Nuance: The use of Word2Vec embeddings allows the model to understand variations in language. For instance, "killed" and "euthanized" can be recognized as semantically similar in the context of animal handling, even if the exact terms aren't in the criteria description.
  • Sentence-level vs. Full-text Classification: The evaluation shows that a classifier trained on top-ranked sentences identified by their method often performs better than one trained on the entire document, suggesting that focusing on semantically relevant segments is more efficient.

Key Results

The paper's evaluation methodology involved training binary classifiers for each criterion. They compared their "top-k sentences" model against several baselines:

  1. Baseline 1 (Most Frequent Label): A classifier that always predicts the majority class for each criterion.
  2. Baseline 2 (All Full Text): A classifier trained on the entire text of each document.
  3. Bottom-k Sentences: A classifier trained on sentences least similar to the criteria.
  4. Random-k Sentences: A classifier trained on randomly selected sentences.

Key findings include:

  • For four out of six criteria (MC 1, MC 4, MC 5, and MC 6), the "top-k sentences" model significantly outperformed the "bottom-k" and "random-k" models, and often baseline 1.
  • Crucially, for three of these criteria, the "top-k sentences" model also outperformed the "all full text" baseline, despite being given less information. This strongly supports the hypothesis that semantic relatedness helps identify relevant segments.
  • Qualitative examples (Figures 3-8) demonstrate the method's ability to highlight sentences and specific phrases that accurately capture the criteria.

Limitations and Future Work

  • Class Imbalance: For criteria with highly imbalanced class distributions (e.g., MC 2, where very few studies met the criterion), the top-k sentences model sometimes performed worse. The authors hypothesize that the selected sentences might not contain enough negative examples to learn effectively, and that other models might inadvertently pick up on linguistic patterns associated with the criteria due to this imbalance.
  • Criteria Descriptions: The current method uses criteria descriptions as input. The authors suggest that replacing these with example sentences from documents that do meet the criteria could be an improvement.
  • Future Directions: They plan to test their method on other annotated datasets (e.g., PICO element datasets) and explore different ways of calculating sentence-to-description similarity, such as using Doc2Vec or other embedding approaches. They also aim to investigate how to handle criteria with class imbalance more effectively.

Summary updated

BibTeX
@inproceedings{herrmannova2018unsupervised,
  author = {Drahomira Herrmannova and Steven Young and Robert Patton and Christopher Stahl and Nicole Kleinstreuer and Mary Wolfe},
  title = {{Unsupervised Identification of Study Descriptors in Toxicology Research: An Experimental Study}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure