AUC-maximized Deep Convolutional Neural Fields for Sequence Labeling

Sheng Wang, Siqi Sun, Jinbo XuarXiv 2015 · OpenReview

TL;DR

This paper introduces Deep Convolutional Neural Fields (DeepCNF), a novel deep learning architecture for sequence labeling tasks, particularly effective for imbalanced datasets. They propose a new training method, "maximum-AUC," which directly optimizes the Area Under the ROC Curve (AUC), outperforming traditional methods on challenging protein sequence labeling problems.

What Problem Does This Paper Address?

Many real-world sequence labeling tasks, especially in bioinformatics, suffer from highly imbalanced label distributions. This means some labels appear far more frequently than others. Standard training methods like maximum-likelihood and maximum labelwise accuracy, which perform well on balanced datasets, struggle to achieve good performance on these imbalanced datasets. This leads to models that might be accurate overall but perform poorly on the rare, yet often critical, labels. The authors specifically target protein sequence labeling tasks like predicting solvent accessibility, secondary structure, and disorder, where such imbalances are common.

Methods

The core of the proposed solution is DeepCNF, which combines Deep Convolutional Neural Networks (DCNNs) with Conditional Random Fields (CRFs). DCNNs are adept at capturing hierarchical patterns in sequential data, while CRFs are excellent for modeling dependencies between adjacent labels in a sequence.

The key innovation lies in the maximum-AUC training algorithm. Instead of optimizing for overall accuracy or label-specific accuracy, this method directly maximizes the empirical Area Under the ROC Curve (AUC). AUC is a robust metric for imbalanced data as it measures the model's ability to distinguish between positive and negative classes, irrespective of their prevalence. To make AUC optimizable, the authors:

  1. Formulate AUC within a pairwise ranking framework.
  2. Approximate the AUC function using a polynomial Chebyshev approximation.
  3. Employ a gradient-based optimization procedure (specifically L-BFGS) to train the DeepCNF model.

This approach allows the model to learn to prioritize correct predictions for rare classes, which is crucial for imbalanced datasets.

Datasets

The authors utilize publicly available datasets for their experiments:

  • JPRED dataset: Used for training and 7-fold cross-validation. This dataset contains 1338 training and 149 test proteins, with each protein belonging to a different superfamily to minimize bias.
  • CASP10 and CASP11 datasets: Merged and used as a CASP dataset for testing.
  • CAMEO dataset: Used for testing.

For CASP and CAMEO datasets, they filtered out targets sharing >25% sequence identity with the JPRED training set to ensure independent evaluation.

Key Insights and Ideas

  • AUC as a superior metric for imbalanced sequence labeling: The paper strongly advocates for AUC as a more appropriate objective function than traditional metrics when dealing with imbalanced sequence data, demonstrating its effectiveness in improving performance on rare classes.
  • DeepCNF effectively captures sequential dependencies: The combination of DCNNs and CRFs proves powerful for protein sequence labeling, leveraging both local feature extraction and global sequence context.
  • Maximum-AUC training is crucial for imbalanced data: The experimental results clearly show that the maximum-AUC training method significantly outperforms maximum-likelihood and maximum labelwise accuracy when label distributions are highly imbalanced, particularly for disorder and 8-state secondary structure prediction.
  • Architecture matters, but less than training method for imbalance: While DCNN architecture (number of layers, neurons, window size) influences performance, the choice of training method (especially maximum-AUC) has a more pronounced impact on imbalanced datasets.

Key Results

The AUC-maximized DeepCNF model achieved state-of-the-art performance on all three protein sequence labeling tasks (solvent accessibility, disorder prediction, and 8-state secondary structure prediction) when compared to existing popular predictors.

  • Disorder Prediction (DISO): On imbalanced DISO data, maximum-AUC achieved a mean Mcc of 0.51 and AUC of 0.89, significantly outperforming other methods.
  • 8-State Secondary Structure Prediction (SS8): For SS8, which has several rare labels, maximum-AUC achieved a mean Mcc of 0.44 and AUC of 0.86, surpassing other methods.
  • Solvent Accessibility Prediction (ACC): For ACC, where labels are more balanced, maximum-AUC performed comparably to other methods, achieving a Q3 accuracy of 0.69, mean Mcc of 0.45, and mean AUC of 0.82.
  • Comparison with state-of-the-art: In evaluations on CASP and CAMEO datasets, the AUC-trained DeepCNF model consistently outperformed other leading predictors in terms of accuracy, Mcc, and AUC, especially for the highly imbalanced DISO and SS8 tasks. For instance, in DISO prediction, it achieved 0.53 Mcc and 0.88 AUC, significantly better than DNdisorder (0.37 Mcc, 0.81 AUC) and DisoPred3_ab (0.47 Mcc, 0.84 AUC).

Limitations and Future Work

  • Computational Complexity: The paper notes that while the space complexity of the training methods is similar, the time complexity for maximum-AUC is higher (O(d² · |Σ|³ · L)) compared to maximum-likelihood and labelwise accuracy (O(|Σ|² · L)), where 'd' is the degree of the polynomial approximation.
  • Suboptimal Solutions: Due to the complexity of optimizing AUC, the authors use L-BFGS to find a suboptimal solution rather than a global optimum.
  • Future Work: The authors suggest that their maximum-AUC training algorithm can be applied to a wide range of sequence labeling problems with imbalanced label distributions, such as post-translation modification (PTM) site prediction. They also mention that instead of linear-chain CRFs, Markov Random Fields (MRFs) could be explored for modeling protein interactions.

Summary updated

Abstract

Deep Convolutional Neural Networks (DCNN) has shown excellent performance in a variety of machine learning tasks. This manuscript presents Deep Convolutional Neural Fields (DeepCNF), a combination of DCNN with Conditional Random Field (CRF), for sequence labeling with highly imbalanced label distribution. The widely-used training methods, such as maximum-likelihood and maximum labelwise accuracy, do not work well on highly imbalanced data. To handle this, we present a new training algorithm called maximum-AUC for DeepCNF. That is, we train DeepCNF by directly maximizing the empirical Area Under the ROC Curve (AUC), which is an unbiased measurement for imbalanced data. To fulfill this, we formulate AUC in a pairwise ranking framework, approximate it by a polynomial function and then apply a gradient-based procedure to optimize it. We then test our AUC-maximized DeepCNF on three very different protein sequence labeling tasks: solvent accessibility prediction, 8-state secondary structure prediction, and disorder prediction. Our experimental results confirm that maximum-AUC greatly outperforms the other two training methods on 8-state secondary structure prediction and disorder prediction since their label distributions are highly imbalanced and also have similar performance as the other two training methods on the solvent accessibility prediction problem which has three equally-distributed labels. Furthermore, our experimental results also show that our AUC-trained DeepCNF models greatly outperform existing popular predictors of these three tasks.
BibTeX
@misc{wang2015auc,
  author = {Sheng Wang and Siqi Sun and Jinbo Xu},
  title = {{AUC-maximized Deep Convolutional Neural Fields for Sequence Labeling}},
  year = {2015},
  eprint = {1511.05265},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1511.05265},
}
Figure