Targeted Deep Learning: Framework, Methods, and Applications

Shih-Ting Huang, Johannes LedererarXiv 2021 · OpenReview

Abstract

Standard deep learning systems are designed for broad generalization, aiming to perform well across a wide range of unseen test inputs. However, in many high-stakes applications like precision medicine, the specific test inputs (e.g., a patient's genetic profile) are known beforehand, even if their labels are not. We propose Targeted Deep Learning (TDL), a framework that optimizes model performance for these specific, known-but-unlabeled test inputs. By reweighting training data based on its similarity to these "target" inputs, TDL achieves faster convergence and higher accuracy for the cases that matter most. We demonstrate the effectiveness of this approach across six diverse real-world datasets.

Introduction

In the traditional deep learning paradigm, the primary objective is generalization: the ability of a model to perform well on any unseen data drawn from the same distribution. This is essential for applications like autonomous driving or fraud detection, where the next input is unpredictable. However, the authors argue that in many fields, we actually do know our test inputs beforehand. For example, a clinician might have the genetic profile (covariates) of a specific patient but lack the diagnosis (label). Training a model to be universally competent is inefficient if the goal is to be "perfect for this specific person."

The Targeted Deep Learning Framework

Targeted Deep Learning (TDL) shifts the focus from broad generalization to optimizing model performance for specific, pre-specified test inputs. While "Personalized Deep Learning" often requires significant labeled data for a specific entity, TDL fills the gap where you have a small number of target inputs (even just one) that are known but completely unlabeled.

Conceptual diagram of the Targeted Deep Learning framework showing the integration of unlabeled test data into the training process.

As illustrated in Figure 1, the TDL framework incorporates the unlabeled test data directly into the training pipeline, allowing the model to specialize on the relevant regions of the input space before inference.

Methodology

The core of TDL is a similarity-weighted training strategy that requires minimal changes to existing pipelines, making it highly accessible for practitioners:

  1. Similarity Measure ($s$): The framework uses a similarity metric based on cosine similarity (or correlation). For a training sample $x$ and a set of target vectors $w$, the similarity is calculated as the maximum cosine of the angle between $x$ and any target $w_i$.
  2. Weighted Stochastic Gradient Descent (SGD): Instead of drawing training samples uniformly, TDL draws them with probabilities proportional to their similarity to the target vectors.
  3. Data Preprocessing Implementation: To make this "plug-and-play" for frameworks like PyTorch or TensorFlow, the authors suggest a preprocessing step where the dataset is oversampled based on similarity to targets. This allows the use of standard training code while the model naturally focuses on the most relevant data.

Experimental Evaluation

The authors evaluated TDL across six diverse real-world applications to prove its versatility, including regression tasks (Parkinson’s telemonitoring, concrete strength) and classification tasks (diabetic retinopathy, fetal heart-rate patterns, and image classification using KMNIST and Fashion-MNIST).

Key Results

Across all six applications, TDL consistently outperformed standard deep learning in two primary ways:

  • Convergence Speed: TDL converged much faster than standard deep learning. In many cases, TDL reached a lower loss in 25 epochs than standard DL did in 200.
  • Accuracy and Stability: TDL achieved higher final accuracy and lower final loss on the target inputs. The testing accuracy curves for TDL were notably higher and more stable than the baselines.
Training and testing loss curves for the Parkinson’s telemonitoring application, comparing TDL with standard deep learning for target group sizes g=1 and g=5.

As shown in Figure 2, the performance advantage of TDL is evident in both training and testing phases, maintaining an edge even as the target group size ($g$) increases.

Conclusion and Limitations

Targeted Deep Learning demonstrates that when the goal is specific, broad generalization can act as a source of "noise." By focusing the model's attention on training data that resembles the target, TDL significantly boosts performance. The primary trade-off is that a TDL-optimized model will likely perform worse on the general population because it has specialized for the provided targets. Future work may explore more complex, deep-learning-based similarity measures to further refine how the model selects its relevant training data.

Summary updated

Abstract

Deep learning systems are typically designed to perform for a wide range of test inputs. For example, deep learning systems in autonomous cars are supposed to deal with traffic situations for which they were not specifically trained. In general, the ability to cope with a broad spectrum of unseen test inputs is called generalization. Generalization is definitely important in applications where the possible test inputs are known but plentiful or simply unknown, but there are also cases where the possible inputs are few and unlabeled but known beforehand. For example, medicine is currently interested in targeting treatments to individual patients; the number of patients at any given time is usually small (typically one), their diagnoses/responses/... are still unknown, but their general characteristics (such as genome information, protein levels in the blood, and so forth) are known before the treatment. We propose to call deep learning in such applications targeted deep learning. In this paper, we introduce a framework for targeted deep learning, and we devise and test an approach for adapting standard pipelines to the requirements of targeted deep learning. The approach is very general yet easy to use: it can be implemented as a simple data-preprocessing step. We demonstrate on a variety of real-world data that our approach can indeed render standard deep learning faster and more accurate when the test inputs are known beforehand.
BibTeX
@misc{huang2021targeted,
  author = {Shih-Ting Huang and Johannes Lederer},
  title = {{Targeted Deep Learning: Framework, Methods, and Applications}},
  year = {2021},
  eprint = {2105.14052},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.14052},
}
Figure