Hierarchy exploitation to detect missing annotations on hierarchical multi-label classification

Miguel Romero, Felipe Kenji Nakano, Jorge Finke, Camilo Rocha, Celine Vens

TL;DR

The REASSIGN method improves gene function prediction by exploiting the inherent hierarchical structure of biological data. By aggregating prediction probabilities along paths from specific leaf nodes to the root, it successfully identifies missing annotations in genomic datasets, outperforming standard classification techniques that ignore these structural relationships.

What Problem Does This Paper Address?

In the world of genomics, knowing a gene's sequence is only half the battle; we need to know what it does. This is called functional annotation. Because biological functions are complex, they are organized into hierarchies (like the Gene Ontology), where specific functions (e.g., "regulation of DNA replication") are children of more general ones (e.g., "biological process").

The challenge is twofold:

  1. Incompleteness: Our current knowledge is full of "missing links"—annotations that exist in nature but haven't been experimentally verified yet.
  2. Structural Constraints: In a hierarchy, if a gene has a specific function, it must also have all its parent functions (the "true-path rule"). Many machine learning models struggle to maintain this consistency, especially at deeper levels of the hierarchy where data is sparse and probabilities are naturally lower.

Methods

The authors introduce REASSIGN (hieRarchical multi-labEl clAsSification to diScover mIssinG aNnotations). The workflow consists of three main steps:

  1. Base Prediction: Use a global Hierarchical Multi-label Classification (HMC) model—specifically a Random Forest of Predictive Clustering Trees—to generate initial probability scores for every gene-function pair.
  2. Path Aggregation: Instead of looking at labels in isolation, REASSIGN looks at entire paths from a "leaf" (the most specific function) up to the "root" (the most general). It calculates an aggregated probability for the path using one of three functions:
  3. Min: The probability of the "weakest link" in the path.
  4. Sum: Favors longer, more informative paths.
  5. Average: Balances the high-probability general classes with the low-probability specific ones.
  6. Selection: The top-ranked paths are suggested as missing annotations, ensuring that if a specific function is suggested, all its ancestors are included to satisfy the hierarchy constraint.

Datasets

The study focuses on Oryza sativa Japonica (a common variety of rice).

  • Training Data: Functional annotations from 2018.
  • Evaluation Data: Annotations from 2021, which serve as the "ground truth" to see if the model correctly predicted annotations that were discovered/added in the intervening three years.
  • Features: The model uses structural properties of Gene Co-expression Networks (GCNs) and low-dimensional node embeddings to represent gene interactions.
  • Benchmarks: The paper introduces eight specific HMC sub-hierarchy datasets based on different biological processes in rice.

Key Insights and Ideas

  • Hierarchy as a Filter: The core "aha moment" is that the hierarchy isn't just a constraint to follow; it's a source of information. A low-probability specific function becomes much more "believable" if its parent functions also show strong predictive signals.
  • The Power of Averaging: The REASSIGN (average) variant proved to be the most robust. By averaging probabilities along a path, the model avoids being over-confident about general terms (which are common and easy to predict) or being too discouraged by the naturally lower scores of specific, "deep" terms.
  • Addressing Sparsity: Standard models often fail at the deeper levels of a tree because there are fewer examples to learn from. REASSIGN effectively "pulls" signal from the more populated upper levels of the tree to support predictions at the bottom.

Key Results

  • Superior Precision: REASSIGN (avg) outperformed competitor methods (like "Noise detect" and "No aggr") in most sub-hierarchies. In one specific dataset (GO:0008152), it achieved 5% higher precision than the closest competitor across all tested thresholds.
  • Statistical Significance: Using Friedman-Nemenyi tests, the authors confirmed that REASSIGN is significantly better than random selection and standard noise-detection algorithms for this task.
  • Biological Relevance: The method was particularly successful at identifying annotations in the middle-to-deep levels of the hierarchy, which are the most useful for biologists looking to design in vivo experiments.

Limitations and Future Work

  • The "Deepest" Frontier: While REASSIGN improved predictions in deeper levels, the very deepest tips of the hierarchy remain a challenge for all methods due to the extreme lack of training data.
  • Fixed Aggregation: The current model uses static functions (min, sum, avg). Future work could involve learning the best way to aggregate path probabilities using more complex neural architectures.
  • Transfer Learning: The authors suggest that incorporating data from better-studied "model organisms" (like Arabidopsis thaliana) could further improve the accuracy of predictions for crops like rice.
  • Wrong vs. Missing: This paper focuses on missing labels. A natural next step is identifying incorrect existing labels (false positives) in genomic databases.

Summary updated

Abstract

The availability of genomic data has grown exponentially in the last decade, mainly due to the development of new sequencing technologies. Based on the interactions between genes (and gene products) extracted from the increasing genomic data, numerous studies have focused on the identification of associations between genes and functions. While these studies have shown great promise, the problem of annotating genes with functions remains an open challenge. In this work, we present a method to detect missing annotations in hierarchical multi-label classification datasets. We propose a method that exploits the class hierarchy by computing aggregated probabilities to the paths of classes from the leaves to the root for each instance. The proposed method is presented in the context of predicting missing gene function annotations, where these aggregated probabilities are further used to select a set of annotations to be verified through in vivo experiments. The experiments on Oriza sativa Japonica, a variety of rice, showcase that incorporating the hierarchy of classes into the method often improves the predictive performance and our proposed method yields superior results when compared to competitor methods from the literature.
BibTeX
@misc{romero2022hierarchy,
  author = {Miguel Romero and Felipe Kenji Nakano and Jorge Finke and Camilo Rocha and Celine Vens},
  title = {{Hierarchy exploitation to detect missing annotations on hierarchical multi-label classification}},
  year = {2022},
  eprint = {2207.06237},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.06237},
}
Figure