Using the Future to Sort Out the Present: Rankprop and Multitask Learning for Medical Risk Evaluation

Rich Caruana, Shumeet Baluja, Tom M. MitchellNeurIPS 1995

Introduction

In medical triage, specifically for conditions like pneumonia, the primary goal is to accurately identify which patients are at high risk (requiring hospitalization) and which are at low risk (safe for home treatment). This paper introduces two novel neural network techniques—Rankprop and Multitask Learning (MTL)—designed to significantly improve pneumonia risk assessment. By learning to rank patients instead of just predicting binary mortality, and by using "future" lab results as auxiliary training signals, the authors achieved a 10-50% improvement in identifying low-risk patients.

The Problem with Standard Approaches

Standard machine learning approaches typically treat medical risk assessment as a binary classification problem: will the patient live (0) or die (1)? However, this presents two major challenges:

  1. The "Noisy" Target Problem: Two patients with nearly identical symptoms might have different outcomes. Training a network on these strict 0/1 targets forces it to learn a highly non-linear, "jagged" function that often generalizes poorly to new patients.
  2. The "Future Data" Paradox: The most predictive data (like specific blood gas levels) often comes from lab tests performed after a patient is admitted. At the moment of triage, this data is unavailable, yet it contains vital information about the disease's underlying severity.

Proposed Methods

The authors propose two distinct but complementary methods to overcome these hurdles.

1. Rankprop: Learning to Rank

Instead of forcing the network to predict an absolute 0 or 1, Rankprop teaches the network to rank patients by risk. It is fundamentally easier for a model to learn a smooth, monotonic function that determines "Patient A is riskier than Patient B" than it is to predict the exact probability of death.

Adjusted targets for ranking patients

Rankprop alternates between a "ranking pass" and a "backprop pass." During the ranking pass, the model uses its current predictions to break ties among patients who both lived or both died. This creates a more refined, continuous target for the next round of training, effectively "smoothing" the learning landscape and avoiding the pitfalls of noisy binary targets.

2. Multitask Learning (MTL): Learning from the Future

MTL allows the model to leverage future data during the training phase. While lab results aren't available during real-world triage, they are available in historical training databases.

Neural network architecture for Multitask Learning

The network is designed with extra output nodes for these "future" lab results. During training, the model must predict both the mortality risk and these subsequent lab values. This forces the shared hidden layers to develop internal representations that capture the fundamental physiology of the disease. By acting as a form of intelligent regularization, MTL prevents the model from focusing on spurious correlations, ultimately making the primary risk prediction more accurate.

Experimental Setup and Key Results

The study utilized the Medis Pneumonia Database, a large-scale dataset containing 14,199 pneumonia cases from 78 hospitals. Each patient record included 30 "pre-hospital" features (age, sex, pulse) and 35 "post-hospital" lab results, with an overall mortality rate of 10.9%. The researchers used small training subsets (2,000 patterns) to simulate data-scarce environments.

The combination of Rankprop and MTL provided a massive performance boost over standard Sum of Squares Error (SSE) training:

  • Error Reduction: Rankprop alone reduced error rates by 8% to 40% depending on the fraction of the population being evaluated.
  • Combined Performance: When Rankprop was combined with MTL, the error rate for the lowest-risk 10% of patients dropped from 1.4% to 0.74%—a nearly 47% improvement.
  • Competitive Edge: These methods outperformed other popular techniques of the era, including Bayes Nets and K-Nearest Neighbors, which experts had previously applied to the same dataset.

Limitations and Future Work

While highly effective, the authors acknowledge a few limitations. First, they do not yet have a mathematical proof for the conditions under which Rankprop’s "parallel search" is guaranteed to converge. Second, MTL strictly requires a training set that includes "future" features, which may not be available in every domain.

Despite these constraints, the authors suggest these methods have broad applicability beyond medicine. Techniques like Rankprop and MTL could revolutionize other fields, such as investment analysis (ranking stocks) and autonomous vehicle navigation (using high-resolution sensors available only during training to bias the model).

Summary updated

BibTeX
@inproceedings{caruana1995using,
  author = {Rich Caruana and Shumeet Baluja and Tom M. Mitchell},
  title = {{Using the Future to Sort Out the Present: Rankprop and Multitask Learning for Medical Risk Evaluation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1995},
}
Figure