NERO: a biomedical named-entity (recognition) ontology with a large, annotated corpus reveals meaningful associations through text embedding

Kanix Wang, Robert Stevens, Halima Alachram, Yu Li, Larisa Soldatova, Ross King, Sophia Ananiadou, Annika M. Schoene, Maolin Li, Fenia Christopoulou, José Luis Ambite, Joel Matthew, Sahil Garg, Ulf Hermjakob, Daniel Marcu, Emily Sheng, Tim Beißbarth, Edgar Wingender, Aram Galstyan, Xin Gao, Brendan Chambers, Weidi Pan, Bohdan B. Khomtchouk, James A. Evans, Andrey RzhetskyNpj Systems Biology and Applications 2021 · OpenReview

Abstract

Machine reading (MR) is essential for unlocking valuable knowledge contained in millions of existing biomedical documents. Over the last two decades, the most dramatic advances in MR have followed in the wake of critical corpus development. This study contributes six components to an advanced, named entity analysis tool for biomedicine: (a) a new, Named Entity Recognition Ontology (NERO) developed specifically for describing textual entities in biomedical texts, which accounts for diverse levels of ambiguity, bridging the scientific sublanguages of molecular biology, genetics, biochemistry, and medicine; (b) detailed guidelines for human experts annotating hundreds of named entity classes; (c) pictographs for all named entities, to simplify the burden of annotation for curators; (d) an original, annotated corpus comprising 35,865 sentences, which encapsulate 190,679 named entities and 43,438 events connecting two or more entities; (e) validated, off-the-shelf, named entity recognition (NER) automated extraction, and; (f) embedding models that demonstrate the promise of biomedical associations embedded within this corpus.

The Challenge of Biomedical Ambiguity

Biomedical language is notoriously complex and context-dependent. A term like "mutant hemoglobin $\alpha_2$" might refer to a gene in one instance and the resulting protein in another. Existing Named Entity Recognition (NER) systems typically force annotators to choose a single specific category, introducing noise into training data when the text is inherently ambiguous. This "forced choice" paradigm fails to capture the semantic gray areas that even human experts struggle to resolve, leading to inconsistencies in automated knowledge extraction.

The NERO Framework and Corpus

To address this, researchers developed NERO (Named Entity Recognition Ontology), an extension of the Information Artifact Ontology (IAO). NERO’s defining feature is "variable-level meaning resolution," which formalizes uncertainty. For example, it includes parent classes like GeneOrProtein for cases where the specific entity type cannot be determined from context.

The NERO ontology hierarchy, featuring pictographs for entity classes and frequency counts within the annotated corpus.

The project produced a massive benchmark dataset: the NERO Corpus. This expert-annotated resource contains 35,865 sentences, encapsulating 190,679 named entities and 43,438 events. To streamline the annotation process for experts, the team developed unique pictographs for each of the hundreds of entity classes, significantly reducing the cognitive load on human curators and improving consistency across the dataset.

Machine Learning Validation and the "Digital Double"

The corpus was validated using multiple architectures, including NERsuite (CRF-based), spaCy, and a specialized nested NER model designed to capture overlapping entities. The nested model achieved an F1 score of 53.9%, a notable result given the multi-layered complexity of the biomedical domain.

Beyond recognition, the study explored the concept of the "digital double" through word embeddings. By projecting entities into a high-dimensional space using word2vec, the researchers found that the context of words in the text reflects real-world properties.

Visualization of entity embeddings showing meaningful clustering of drugs and diseases, reflecting real-world clinical and biological associations.

These embeddings showed significant correlations with external ground truth data, such as drug prices (0.42 correlation), toxicity (0.32 correlation with LD50), and gender-coded disease prevalence (0.436 correlation). This suggests that the corpus captures latent knowledge about entities even when not explicitly stated in a single document, effectively acting as a digital representation of biological reality.

Discussion and Future Directions

A significant challenge identified is the "long tail" of entity distribution. While common classes like GeneOrProtein and Process are well-represented, many specific classes appear rarely, leading to difficulties for standard machine learning models. Future work will focus on semi-supervised learning, using the high-quality NERO labels to seed broader knowledge extraction across millions of unannotated documents, bridging the gap between molecular biology and clinical practice.

Summary updated

Abstract

Machine reading (MR) is essential for unlocking valuable knowledge contained in millions of existing biomedical documents. Over the last two decades1,2, the most dramatic advances in MR have followed in the wake of critical corpus development3. Large, well-annotated corpora have been associated with punctuated advances in MR methodology and automated knowledge extraction systems in the same way that ImageNet4 was fundamental for developing machine vision techniques. This study contributes six components to an advanced, named entity analysis tool for biomedicine: (a) a new, Named Entity Recognition Ontology (NERO) developed specifically for describing textual entities in biomedical texts, which accounts for diverse levels of ambiguity, bridging the scientific sublanguages of molecular biology, genetics, biochemistry, and medicine; (b) detailed guidelines for human experts annotating hundreds of named entity classes; (c) pictographs for all named entities, to simplify the burden of annotation for curators; (d) an original, annotated corpus comprising 35,865 sentences, which encapsulate 190,679 named entities and 43,438 events connecting two or more entities; (e) validated, off-the-shelf, named entity recognition (NER) automated extraction, and; (f) embedding models that demonstrate the promise of biomedical associations embedded within this corpus.
BibTeX
@inproceedings{wang2021nero,
  author = {Kanix Wang and Robert Stevens and Halima Alachram and Yu Li and Larisa Soldatova and Ross King and Sophia Ananiadou and Annika M. Schoene and Maolin Li and Fenia Christopoulou and José Luis Ambite and Joel Matthew and Sahil Garg and Ulf Hermjakob and Daniel Marcu and Emily Sheng and Tim Beißbarth and Edgar Wingender and Aram Galstyan and Xin Gao and Brendan Chambers and Weidi Pan and Bohdan B. Khomtchouk and James A. Evans and Andrey Rzhetsky},
  title = {{NERO: a biomedical named-entity (recognition) ontology with a large, annotated corpus reveals meaningful associations through text embedding}},
  booktitle = {Npj Systems Biology and Applications},
  year = {2021},
  url = {https://openreview.net/forum?id=ZlPCuLTrCB},
}
Figure