Adaptive Inference for Medical Vision Transformers: Token Reduction or Early Exit?
Lists
Only you see this
Abstract
Vision Transformers (ViTs) have demonstrated exceptional performance in medical image analysis, yet their computational demands hinder clinical deployment, particularly in time-sensitive applications. Medical imaging requires sample-adaptive optimization due to dataset heterogeneity across modalities and sample complexity; uniform strategies do not well balance efficiency and accuracy. We propose a unified adaptive inference framework that combines Token Reduction (TR) and Early Exiting (EE) through dataset-specific profiling. Our approach quantifies spatial redundancy via Jensen-Shannon Divergence (JSD) and prediction confidence at intermediate layers to train a lightweight predictor that dynamically selects inference strategies at test time. Across five medical datasets, including a real-world cataract dataset (INSIGHT), our framework achieves 71.4% average floating-point operations per second (FLOPs) reduction with only 0.1pp accuracy loss, substantially outperforming individual strategies (EE-only: 55.9%, TR-only: 57.7%). On PathMNIST, our adaptive inference framework simultaneously improves accuracy by 1.3pp while reducing computation by 77.2%. On INSIGHT, we maintain baseline accuracy with 69.8% FLOPs reduction, demonstrating robust real-world clinical applicability.
BibTeX
@inproceedings{byun2026adaptive,
author = {Ji Young Byun and Hyun Seo Lee and Jordan M Shuff and Nakul S. Shekhawat and Kunal S Parikh and Rama Chellappa},
title = {{Adaptive Inference for Medical Vision Transformers: Token Reduction or Early Exit?}},
booktitle = {MIDL},
year = {2026},
url = {https://openreview.net/forum?id=JyJsaVDzOl},
}Loading related content...