Noise Pollution in Hospital Readmission Prediction: Long Document Classification with Reinforcement Learning

Liyan Xu, Julien Hogan, Rachel E. Patzer, Jinho D. Choi

TL;DR

Predicting hospital readmission from clinical notes is often a "needle in a haystack" problem due to massive amounts of irrelevant text. This paper demonstrates that a Reinforcement Learning (RL) approach can successfully prune about 25% of noisy content from long clinical documents, improving prediction accuracy. Surprisingly, the researchers found that traditional Bag-of-Words models outperformed modern deep learning architectures like ClinicalBERT on this specific, small-scale medical dataset.

What Problem Does This Paper Address?

Hospital readmission after major surgery, such as a kidney transplant, is a significant burden on healthcare systems. While Electronic Health Records (EHRs) contain rich clinical notes that could predict these events, these documents are notoriously "noisy." A single patient's record might contain tens of thousands of tokens, much of which consists of repetitive boilerplate, tabular lab results, and task-irrelevant descriptions.

The authors tackle two main hurdles:

  1. The "Long Document" Problem: Standard NLP models struggle with sequences of 10,000+ tokens.
  2. The "Small Data" Problem: Medical datasets for specific procedures are often small (under 2,000 patients), leading complex deep learning models to overfit rapidly.

Methods

The researchers took a two-step approach. First, they benchmarked four document encoders to find the best representation for these long, messy texts:

  • Bag-of-Words (BoW): Simple TF-IDF scores.
  • Averaged Word Embedding (AWE): Using fastText embeddings.
  • ClinicalBERT: A version of BERT pre-trained on clinical corpora.
  • Weight-dropped LSTM: A recurrent network designed to handle long sequences with heavy regularization.

Second, they implemented a Reinforcement Learning (RL) framework to act as an automated editor. They modeled noise extraction as a sequential decision process:

  • Policy Network: For each segment of a document, a policy network decides whether to Keep or Prune it.
  • Environment: The "selected" text is passed to a downstream classifier.
  • Reward: The policy receives a reward based on the classifier's accuracy, plus a small bonus for pruning text to encourage efficiency.
Figure 1: Overview of our reinforcement learning approach. Rewards are calculated and sent back to the policy network after all actions a1:T have been sampled for the given episode.

Datasets

The study uses the Emory Kidney Transplant Dataset (EKTD), which includes records for 2,060 patients. The dataset contains 8 distinct types of clinical notes, ranging from surgical reports (Operative) to social worker encounters. These notes are highly unstructured and often contain HTML fragments, duplicates, and complex tabular data.

Key Insights and Ideas

  • Simplicity Wins on Small Data: In a surprising twist, the "old-school" Bag-of-Words (BoW) encoder consistently outperformed ClinicalBERT and LSTMs. The deep learning models suffered from severe overfitting because the feature space was too large relative to the number of training samples.
  • RL as a Task-Specific Filter: Unlike static filters (like stop-word lists), the RL agent learned to identify noise specific to the task. It successfully pruned segments containing "lab fishbones" (formatted lab results) and repetitive headers that didn't contribute to predicting readmission.
  • Entropy is Essential: The authors found that strong entropy regularization was critical. Without it, the RL agent would fall into a "lazy" local optimum where it simply kept all text to avoid the risk of losing information.
Figure 3: Retaining ratios on the development set of SC while training the reinforcement learning model. En- tropy regularization encourages more exploration.

Key Results

  • Performance Boost: The RL-augmented BoW model achieved approximately a 1% improvement in AUC (Area Under the Curve) across several note types compared to the best baseline.
  • Significant Pruning: The model achieved these gains while discarding up to 26% of the input text as noise.
  • Noise Identification: Quantitative analysis showed that the RL policy successfully identified both low-frequency "junk" tokens and high-frequency but uninformative tokens (like "chloride" or "creatinine" appearing in noisy tables).
Figure 4: Log scale distribution on document frequency of tokens with top negative feature importance.

Limitations and Future Work

The primary limitation is the dataset size; kidney transplants are relatively rare, limiting the training data to roughly 2,000 patients. This likely explains why ClinicalBERT struggled to generalize.

Future work could explore:

  • Hybrid Encoders: Combining the robustness of BoW with the semantic understanding of transformer-based models.
  • Multimodal Integration: While this paper focused purely on unstructured text to test the limits of NLP, real-world systems would benefit from integrating this with structured lab data.
  • Transfer Learning: Investigating if a policy network trained to denoise one type of medical note (e.g., Discharge Summaries) can be effectively transferred to another.

Summary updated

Abstract

This paper presents a reinforcement learning approach to extract noise in long clinical documents for the task of readmission prediction after kidney transplant. We face the challenges of developing robust models on a small dataset where each document may consist of over 10K tokens with full of noise including tabular text and task-irrelevant sentences. We first experiment four types of encoders to empirically decide the best document representation, and then apply reinforcement learning to remove noisy text from the long documents, which models the noise extraction process as a sequential decision problem. Our results show that the old bag-of-words encoder outperforms deep learning-based encoders on this task, and reinforcement learning is able to improve upon baseline while pruning out 25% text segments. Our analysis depicts that reinforcement learning is able to identify both typical noisy tokens and task-specific noisy text.
BibTeX
@inproceedings{xu2020noise,
  author = {Liyan Xu and Julien Hogan and Rachel E. Patzer and Jinho D. Choi},
  title = {{Noise Pollution in Hospital Readmission Prediction: Long Document Classification with Reinforcement Learning}},
  booktitle = {BioNLP},
  year = {2020},
  url = {https://openreview.net/forum?id=hjmd9N3gQY},
}
Figure