Artificial neural networks condensation: A strategy to facilitate adaption of machine learning in medical settings by reducing computational burden

Dianbo Liu, Néstor Sepúlveda, Ming ZhengarXiv 2018 · OpenReview

TL;DR

This paper tackles the computational burden of machine learning models in healthcare, particularly for resource-limited settings. They demonstrate that techniques like pruning and quantization can significantly reduce model size and inference time for both recurrent and dense neural networks without sacrificing, and sometimes even improving, prediction accuracy for in-hospital mortality.

What Problem Does This Paper Address?

Machine learning holds immense promise for improving healthcare, from diagnostics to drug development. However, a major hurdle is the significant computational resources required to train and deploy these models, especially when dealing with large and sparse medical datasets. This poses a challenge for widespread adoption, particularly in developing nations where access to high-performance computing is limited. The authors aim to bridge this gap by exploring methods to make ML models more computationally efficient, specifically focusing on neural networks used for predicting in-hospital mortality.

Methods

The researchers investigated three primary methods for "condensing" neural networks:

  • Pruning: This involves removing "unused" neurons (and their connections) from trained models. They applied this at a channel level, removing neurons whose average weights across inputs were below a certain quantile (50% in this study).
  • Hidden-layer LSTM (hLSTM): For recurrent neural networks (RNNs), they modified the standard Long Short-Term Memory (LSTM) cell by inserting an additional dense layer with a ReLU activation function within its internal gates. This aims to increase the representational power of each LSTM unit, potentially allowing for simpler overall architectures.
  • Quantization: This technique reduces the precision of model parameters. They converted 32-bit floating-point weights to 8-bit integers after training.

These methods were applied to two common neural network architectures: Recurrent Neural Networks (RNNs) and Dense Neural Networks (DNNs).

Datasets

The study utilized the MIMIC-III critical care database. This dataset contains de-identified health-related data associated with over 53,000 intensive care unit admissions. The researchers focused on adult patients, excluding neonatal and pediatric cases, and used data from the first 48 hours of ICU stay for their in-hospital mortality prediction task.

Key Insights and Ideas

  • Efficiency without Accuracy Loss is Achievable: The core finding is that significant computational gains can be realized without compromising, and sometimes even enhancing, predictive accuracy. This is crucial for real-world deployment.
  • Pruning is a Powerful Tool: Channel-level pruning proved effective in reducing model size and increasing inference speed for both RNNs and DNNs, maintaining competitive accuracy.
  • hLSTM Offers a Structural Advantage: Modifying the LSTM cell with an internal dense layer (hLSTM) allowed for a simpler RNN architecture (one layer instead of two) that achieved better performance and faster training. This suggests that architectural innovation within the core building blocks can be highly effective.
  • Quantization is Highly Effective for DNNs: For DNNs, 8-bit quantization dramatically reduced model size (by 5 times) with minimal impact on accuracy and a notable speedup in inference.
  • Bridging the Digital Divide: The research directly addresses the disparity in computational resources, offering a pathway for advanced ML applications to benefit a wider population.

Key Results

  • RNNs:
    • The baseline RNN achieved an AUROC of 0.836.
  • Pruned LSTM (PLSTM): Achieved an AUROC of 0.853, with a 50% reduction in parameters, a 44% reduction in file size, and doubled inference speed compared to the baseline.
  • Hidden-layer LSTM (hLSTM): Achieved the highest AUROC of 0.860, with a 13% reduction in parameters and a 48% faster inference speed compared to the baseline.
  • DNNs:
    • The baseline DNN achieved an AUROC of 0.82.
  • Pruned DNN (pDNN): Maintained a similar AUROC (0.81) while reducing parameters by 55% and doubling inference speed.
  • Quantized DNN (qDNN): Achieved an AUROC of 0.82, reducing file size by 92% compared to the baseline DNN.

Overall, the condensed models demonstrated significant improvements in file size and inference speed, with some achieving higher accuracy than their baseline counterparts.

Limitations and Future Work

  • Limitations:
    • The study focused on a specific task (in-hospital mortality prediction) and dataset (MIMIC-III). The generalizability of these methods to other medical tasks and datasets needs further investigation.
  • While the authors modified Keras code for hLSTM, it's noted that this specific architecture isn't readily available in standard libraries, requiring custom implementation.
  • Future Work:
    • The authors suggest exploring the combined effects of pruning and quantization on the same models to potentially achieve even greater efficiency gains.
  • They also propose investigating these condensation strategies for other medical applications and potentially non-medical domains.
  • Further research into optimizing the "lottery ticket" hypothesis (finding pruned networks that can be retrained quickly) could also be beneficial.

Summary updated

Abstract

Machine Learning (ML) applications on healthcare can have a great impact on people's lives helping deliver better and timely treatment to those in need. At the same time, medical data is usually big and sparse requiring important computational resources. Although it might not be a problem for wide-adoption of ML tools in developed nations, availability of computational resource can very well be limited in third-world nations. This can prevent the less favored people from benefiting of the advancement in ML applications for healthcare. In this project we explored methods to increase computational efficiency of ML algorithms, in particular Artificial Neural Nets (NN), while not compromising the accuracy of the predicted results. We used in-hospital mortality prediction as our case analysis based on the MIMIC III publicly available dataset. We explored three methods on two different NN architectures. We reduced the size of recurrent neural net (RNN) and dense neural net (DNN) by applying pruning of "unused" neurons. Additionally, we modified the RNN structure by adding a hidden-layer to the LSTM cell allowing to use less recurrent layers for the model. Finally, we implemented quantization on DNN forcing the weights to be 8-bits instead of 32-bits. We found that all our methods increased computational efficiency without compromising accuracy and some of them even achieved higher accuracy than the pre-condensed baseline models.
BibTeX
@misc{liu2018artificial,
  author = {Dianbo Liu and Néstor Sepúlveda and Ming Zheng},
  title = {{Artificial neural networks condensation: A strategy to facilitate adaption of machine learning in medical settings by reducing computational burden}},
  year = {2018},
  eprint = {1812.09659},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.09659},
}
Figure