Transferability and Hardness of Supervised Classification Tasks

Anh Tuan Tran, Cuong V. Nguyen, Tal Hassner

Abstract

We propose a novel approach for estimating the difficulty and transferability of supervised classification tasks. Unlike previous work, our approach is solution agnostic and does not require or assume trained models. Instead, we estimate these values using an information theoretic approach: treating training labels as random variables and exploring their statistics. When transferring from a source to a target task, we consider the conditional entropy between two such variables (i.e., label assignments of the two tasks). We show analytically and empirically that this value is related to the loss of the transferred model. We further show how to use this value to estimate task hardness. We test our claims extensively on three large scale data sets - CelebA (40 tasks), Animals with Attributes 2 (85 tasks), and Caltech-UCSD Birds 200 (312 tasks) - together representing 437 classification tasks. We provide results showing that our hardness and transferability estimates are strongly correlated with empirical hardness and transferability.

Introduction: Moving Beyond Model Zoos

In modern deep learning, transfer learning is ubiquitous. We frequently take representations learned from a source task (e.g., face recognition) and apply them to a target task (e.g., detecting facial attributes). However, determining which source task will yield the best performance for a specific target has traditionally been a computationally expensive "black art." Frameworks like Taskonomy or Task2Vec require training hundreds of "probe" models to map task relationships, often consuming tens of thousands of GPU hours.

Furthermore, these existing methods are solution-dependent. They measure how a specific architecture (like a ResNet) transfers, which may conflate architectural biases with the inherent relationship between the tasks. This paper introduces a solution-agnostic alternative that predicts transferability and hardness by looking only at label statistics, before any training occurs.

A Solution-Agnostic Framework

The core innovation lies in treating supervised classification as a relationship between random variables. Instead of training models, the authors calculate the Conditional Entropy (CE) between label sets.

Conceptual illustration of task relationships using label statistics.

To estimate if Task A transfers to Task B, they compute $H(Y|Z)$, where $Y$ represents target labels and $Z$ represents source labels. If knowing the source label makes the target label highly predictable, the CE is low, indicating high transferability. Task hardness is similarly defined as the transferability from a "trivial" task (where all labels are identical). If a task remains unpredictable even with this baseline, it is inherently hard.

Theoretical Foundation: Theorem 1

The researchers bridge information theory and practical performance through Theorem 1. This theorem proves that the log-likelihood of a transferred model is lower-bounded by the source task's performance minus the conditional entropy of the labels. This provides a rigorous mathematical guarantee that label statistics are a valid proxy for empirical transfer success, assuming an optimal representation of the source task.

Empirical Validation

The theory was tested across 437 classification tasks using three major datasets: CelebA (40 facial attributes), Animals with Attributes 2 (85 attributes), and Caltech-UCSD Birds 200 (312 attributes).

Correlation between conditional entropy and empirical error across various CelebA attributes.

As shown in the results, there is a remarkably strong correlation—often exceeding 0.9 Pearson coefficient—between the theoretical CE estimates and the actual test errors observed after training. This consistency across diverse attributes (from "Heavy Makeup" to "Paws") confirms that label sequences alone carry sufficient signal to predict transferability.

Efficiency and Impact

The practical implications are significant. While previous benchmarks like Taskonomy required over 47,000 GPU hours to map task relationships, calculating conditional entropy takes less than a second on a standard CPU.

In a case study transferring a face recognition model to CelebA attributes, the authors achieved state-of-the-art (SOTA) accuracy on highly transferable tasks using only a simple linear SVM. This suggests that for many problems, the right representation is more important than a complex model architecture.

Conclusion and Future Directions

By shifting the focus from model weights to label statistics, this work provides a mathematically grounded shortcut for designing multi-task systems. While currently limited to tasks sharing the same input domain and discrete labels, it sets the stage for more efficient, data-centric approaches to transfer learning. Future work may extend these information-theoretic tools to continuous tasks like regression or cross-domain transfer.

Summary updated

Abstract

We propose a novel approach for estimating the difficulty and transferability of supervised classification tasks. Unlike previous work, our approach is solution agnostic and does not require or assume trained models. Instead, we estimate these values using an information theoretic approach: treating training labels as random variables and exploring their statistics. When transferring from a source to a target task, we consider the conditional entropy between two such variables (i.e., label assignments of the two tasks). We show analytically and empirically that this value is related to the loss of the transferred model. We further show how to use this value to estimate task hardness. We test our claims extensively on three large scale data sets - CelebA (40 tasks), Animals with Attributes 2 (85 tasks), and Caltech-UCSD Birds 200 (312 tasks) - together representing 437 classification tasks. We provide results showing that our hardness and transferability estimates are strongly correlated with empirical hardness and transferability. As a case study, we transfer a learned face recognition model to CelebA attribute classification tasks, showing state of the art accuracy for tasks estimated to be highly transferable.
BibTeX
@misc{tran2019transferability,
  author = {Anh Tuan Tran and Cuong V. Nguyen and Tal Hassner},
  title = {{Transferability and Hardness of Supervised Classification Tasks}},
  year = {2019},
  eprint = {1908.08142},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.08142},
}
Figure