Comparing Sample-Wise Learnability across Deep Neural Network Models

Seung-Geon Lee, Jaedeok Kim, Hyun-Joo Jung, Yoonsuck ChoeAAAI 2019 · DOI

Introduction

In deep learning, "learnability" is typically discussed at the dataset level. However, practitioners intuitively understand that individual data points vary in difficulty. A clear, centered image of a horse is inherently "easier" for a model to classify than a blurry, occluded one. This paper formalizes this intuition by introducing sample-wise learnability, a metric that quantifies the ease with which a deep neural network (DNN) learns a specific sample. Crucially, the authors investigate whether this "hardness" is a model-specific artifact or an intrinsic property of the data itself.

Defining Sample-Wise Learnability

The authors propose a metric, $L_f$, based on the model's confidence during the training process. For a given sample $i$, the learnability is defined as the expected value of the predicted probability of the correct label, averaged over all training steps $T$:

$$L_{f,i} = \frac{1}{T} \sum_{t=1}^{T} P(y_i | x_i; \theta_t)$$

A high score indicates a sample that the model identifies correctly early in training and maintains high confidence in. Conversely, a low score indicates a "hard" sample that the model struggles with or only learns late in the process. To account for variations in absolute probability scales across different architectures, the authors also utilize a Learnability Rank ($R_{f,i}$), which orders samples from easiest (1) to hardest ($N$).

Cross-Architecture Consistency

To test if learnability is architecture-dependent, the researchers evaluated the metric on the CIFAR-10 dataset using three distinct architectures: ResNet-20, VGG-16, and MobileNet.

The results reveal a striking degree of consistency. Despite the fundamental differences in how these models process information (e.g., residual connections in ResNet vs. standard convolutions in VGG), they tend to agree on which samples are easy and which are hard.

Learnability Rank correlation between VGG-16 and ResNet-20 showing a high correlation coefficient of 0.87.

As shown in the correlation analysis, the learnability ranks between VGG-16 and ResNet-20 exhibit a linear correlation coefficient of 0.867. Similar high correlations were found across all model pairs, suggesting that learnability is largely a property of the data itself rather than the specific model architecture.

Visual Characteristics of Learnability

Qualitative analysis of the ranked samples provides further validation. "Easy" samples typically feature objects that are centered, well-lit, and possess prototypical features of their class. "Hard" samples, on the other hand, often contain atypical viewpoints, ambiguous backgrounds, or objects that are very small or partially obscured.

Examples of "Easy" and "Hard" images from CIFAR-10 with their corresponding learnability ranks for VGG-16 and ResNet-20.

This visual evidence supports the "data-centric" view of difficulty. If a human observer finds an image ambiguous or difficult to categorize quickly, it is highly likely that a DNN will also assign it a low learnability score.

Implications and Future Work

The high correlation between models has significant practical implications. It suggests that a small, computationally efficient "proxy" model (like MobileNet) can be used to score the difficulty of a large dataset. These scores can then be used to design curriculum learning strategies for training much larger, more expensive models, potentially reducing total training time.

However, the current methodology requires a full training run to calculate the scores, presenting a "chicken and egg" problem for initial training speedups. Future research could explore methods to estimate sample-wise learnability without exhaustive training or investigate if these findings generalize to more complex domains like natural language processing or large-scale, noisy real-world datasets.

Summary updated

BibTeX
@inproceedings{lee2019comparing,
  author = {Seung-Geon Lee and Jaedeok Kim and Hyun-Joo Jung and Yoonsuck Choe},
  title = {{Comparing Sample-Wise Learnability across Deep Neural Network Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33019961},
}
Figure