SPEED: Selective Prediction for Early Exit DNNs

Divya Jyoti Bajpai, Bundeliya Harsh Jitendrakumar, Manjesh Kumar HanawalOpenReview 2025 · Submission to ICLR 2025 · Withdrawn · OpenReview

Its authors withdrew this paper, so there is no full text to read or summarize.

Abstract

Inference latency and trustworthiness of Deep Neural Networks (DNNs) are the bottlenecks in deploying them in critical applications like autonomous driving. Early Exit (EE) DDNs overcome the latency issues by allowing samples to exit from intermediary layers if they attain high confidence scores on the predicted class. However, the DNNs are known to exhibit overconfidence, which can lead to many samples exiting early and render EE strategies untrustworthy. We use Selective Prediction (SP) to overcome this issue by checking the hardness of the samples rather than just relying on the confidence score alone. We propose SPEED, a novel approach that uses Deferral Classifiers (DCs) at each layer to check the hardness of samples before performing EEs. The DCs at each layer identify if a sample is hard and either differ its inference to the next layer or directly send it to an expert. Early detection of hard samples and using an expert for inference prevents the wastage of computational resources and improves trust. We also investigate the generalization capability of DCs trained on one domain when applied to other domains where target domain data is not readily available. We observe that EE aided with SP improves both accuracy and latency. Our method minimizes the risk by 50% with a speedup of $2.05\times$ as compared to the final layer. The anonymized source code is available at https://anonymous.4open.science/r/SPEED-35DC/README.md.
BibTeX
@misc{bajpai2025speed,
  author = {Divya Jyoti Bajpai and Bundeliya Harsh Jitendrakumar and Manjesh Kumar Hanawal},
  title = {{SPEED: Selective Prediction for Early Exit DNNs}},
  year = {2025},
  url = {https://openreview.net/forum?id=sceqRsa0oo},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2025},
}
Figure