Into the Single Cell Multiverse: an End-to-End Dataset for Procedural Knowledge Extraction in Biomedical Texts

Ruth Dannenfelser, Jeffrey Zhong, Ran Zhang, Vicky YaoNeurIPS 2023

TL;DR

Researchers have introduced FlaMBé, the first end-to-end expert-curated dataset designed to extract procedural knowledge—the "how-to" of scientific experiments—from biomedical literature. By focusing on the complex "multiverse" of single-cell RNA sequencing workflows, the dataset enables NLP models to move beyond simple fact extraction toward reconstructing entire computational pipelines.

What Problem Does This Paper Address?

Most Natural Language Processing (NLP) in the biomedical domain focuses on declarative knowledge—extracting static facts like "gene X is associated with disease Y." However, science is fundamentally about procedural knowledge: the specific sequence of steps, tools, and decisions that lead to a discovery.

Extracting these workflows is notoriously difficult. Scientific "Methods" sections are often unstructured, non-linear, and ambiguous. A single tool (like PCA) might be used for three different purposes in one paper, and different labs might use entirely different "tool chains" to answer the same biological question—a phenomenon the authors call the "multiverse." Without structured data to train models, we cannot automate the mining of these workflows, which is a major bottleneck for research reproducibility and meta-analysis.

Methods

The authors developed a systematic framework to distill unstructured text into a structured representation of procedural knowledge. Their approach breaks the problem down into three manageable layers:

  1. Targets: Identifying the biological entities involved (tissues and cell types).
  2. Tools & Contexts: Identifying the software used and, crucially, the analysis task (context) it was performing (e.g., using the tool "Seurat" for "clustering" vs. "normalization").
  3. Workflows: Mapping the flow between these (tool, context) pairs to reconstruct the pipeline.
Example of an extracted single-cell analysis workflow mapping tools to their specific contexts.

The curation was performed by expert computational biologists using the Prodigy annotation tool. To ensure high quality, they mapped entities to standardized ontologies like the NCI Thesaurus for Named Entity Disambiguation (NED).

Datasets

The paper introduces FlaMBé (Flow annotations for Multiverse Biological entities), which consists of:

  • Full-text Corpus: 55 papers (420,000 tokens) fully annotated for entities, tool contexts, and workflow relations.
  • Abstract Corpus: 1,195 abstracts (290,000 tokens) annotated for tissue and cell types.
  • Benchmarks: The authors compare FlaMBé against Softcite, a previous software-mention dataset, and evaluate several transformer models (BERT, RoBERTa, PubMedBERT, etc.) on the new data.

Key Insights and Ideas

  • The Contextual Tool Gap: The authors found that most existing datasets only identify that a tool was used. FlaMBé highlights that the why is just as important. For example, the popular tool "Seurat" can be associated with up to 10 different analysis categories.
  • Declarative vs. Procedural: By framing procedural extraction as a series of linked declarative tasks (NER for tools + classification for context + relation extraction for flow), the authors provide a roadmap for solving complex "know-how" extraction using existing NLP architectures.
  • Reproducibility Mining: Automating the extraction of these "multiverses" allows researchers to see how sensitive scientific conclusions are to specific software choices, potentially flagging results that only appear when using a very specific, idiosyncratic pipeline.

Key Results

  • NER Performance: PubMedBERT emerged as the top performer for identifying cell types and tissues, achieving an F1-score of approximately 0.79 for cell types and 0.84 for tissues.
Performance of various transformer models on extracting Cell Type and Tissue entities.
  • Superior Tool Recognition: When trained on FlaMBé, PubMedBERT significantly outperformed models trained on the Softcite dataset for tool identification (F1 of 0.83 vs. 0.47). This is likely because FlaMBé captures the long, multi-word names common in bioinformatics.
  • Context Prediction: The models were surprisingly good at predicting the purpose of a tool mention (e.g., distinguishing if a tool was used for "Alignment" vs. "Clustering"), with AUC scores ranging from 0.81 to 0.95.

Limitations and Future Work

The primary limitation is the scale of workflow annotations. While entity-level tags (like cell types) are abundant, the more complex "workflow" links are fewer in number because they require intense expert labor.

The authors suggest that future work should focus on "human-in-the-loop" curation, where models perform the initial NER/NED and experts focus solely on verifying the complex logic of the workflow. They also point toward Reinforcement Learning (RL) as a promising avenue, as RL is naturally suited for learning "scripts" and sequential decision-making processes.

Summary updated

BibTeX
@inproceedings{dannenfelser2023into,
  author = {Ruth Dannenfelser and Jeffrey Zhong and Ran Zhang and Vicky Yao},
  title = {{Into the Single Cell Multiverse: an End-to-End Dataset for Procedural Knowledge Extraction in Biomedical Texts}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
}
Figure