Transferability and Hardness of Supervised Classification Tasks

Anh T. Tran, Cuong V. Nguyen, Tal Hassner

TL;DR

This paper introduces a solution-agnostic, information-theoretic method to estimate the hardness and transferability of supervised classification tasks using only the statistics of their training labels. By bypassing the need to train deep networks or maintain expensive "model zoos," this approach reduces the computational cost of mapping task relationships from thousands of GPU hours to less than a second on a standard CPU.

What Problem Does This Paper Address?

When building universal perception systems, engineers frequently reuse representations learned from a source task (like face recognition) and apply them to a target task (like facial attribute classification). However, knowing beforehand which tasks will transfer well or how hard a target task is typically requires brute-force experimentation.

Prior frameworks like Taskonomy or Task2Vec attempt to map these relationships by training massive networks or maintaining extensive "model zoos." This creates two major bottlenecks:

  1. High Computational Cost: Measuring empirical transferability across dozens of tasks can consume tens of thousands of GPU hours.
  2. Solution Dependency: These methods measure relationships between specific deep learning models (proxies) rather than the underlying tasks themselves.

The authors tackle this gap by asking: Can we determine task hardness and transferability directly from the data itself, before a single weight is updated?

Methods

The authors propose a completely solution-agnostic approach rooted in information theory. They consider supervised classification tasks that share the same input domain (e.g., the same set of images) but have different label sets.

Task Transferability via Conditional Entropy

Let $Z$ be the label sequence for a source task and $Y$ be the label sequence for a target task. By treating these labels as random variables, the authors compute the Conditional Entropy (CE), denoted as $H(Y|Z)$, from the empirical joint distribution of the training labels.

Mathematically, they prove (Theorem 1) that if you train an optimal representation on the source task and transfer it to the target task, the target task's log-likelihood is lower-bounded by the source task's log-likelihood minus $H(Y|Z)$. When the source task is fixed, transferability depends entirely on this conditional entropy.

Task Hardness

To estimate how inherently difficult a task is, the authors calculate its transferability from a "trivial task"—a hypothetical task where every input is assigned the exact same constant label. This simplifies the hardness metric of a task $T^Z$ down to the conditional entropy $H(Z|C)$, where $C$ is the constant label sequence.

Datasets

The approach was rigorously validated across 437 classification tasks using three massive image benchmarks:

  • CelebA: 202,599 face images annotated with subject identities and 40 binary attributes (40 tasks).
  • Animals with Attributes 2 (AwA2): 37,322 images across 50 animal classes, annotated with 85 attributes (85 tasks).
  • Caltech-UCSD Birds 200 (CUB): 11,788 images of 200 bird species, filtered to 312 reliable attribute tasks.

Key Insights and Ideas

  • Labels Carry the Signal: The most surprising finding is that you don't need to look at the pixel data or the model architecture to predict transferability. The joint distribution of the training labels alone provides a highly accurate proxy for feature relevance.
  • Massive Efficiency Gains: While empirical methods like Taskonomy require over 47,000 GPU hours to map relationships between 26 tasks, computing conditional entropy takes less than a second on a standard CPU, even for complex target sets.
  • Asymmetric Relationships: The framework naturally handles the reality that transferability is a one-way street; transferring from a fine-grained task to a coarse task yields different entropy values than the reverse.

Key Results

The authors validated their theoretical bounds by comparing their a-priori CE estimates against actual empirical test errors obtained by training ResNet-18 models.

  • Strong Empirical Correlation: Across all three datasets, the conditional entropy showed an incredibly strong positive correlation with actual test errors (Pearson correlation coefficient $r \geq 0.92$ for AwA2 and CUB, and $r \geq 0.73$ for CelebA, all with statistical significance $p < 0.001$).
Attribute prediction error vs Conditional Entropy
  • Predicting Hardness: The hardness metric $H(Z|C)$ strongly correlated with the test errors of dedicated networks trained from scratch, proving that label distributions alone act as strong predictors of task difficulty.
Estimated task hardness vs empirical errors
  • Case Study (Identity to Facial Attributes): The authors transferred a deep face recognition model (trained on MS-Celeb-1M and VGGFace2) to CelebA attributes using a simple linear SVM on frozen features. For attributes predicted to be highly transferable (like 'Male' or 'Bald'), this basic setup achieved state-of-the-art accuracy, matching or beating dedicated networks while using only a fraction of the training data.

Limitations and Future Work

  • Shared Input Domain Constraint: The current formulation assumes that the source and target tasks share the exact same input instances (e.g., the same images with different labels). It cannot yet predict transferability between entirely different domains, such as transferring features from face images to animal images.
  • Discrete Labels Only: The math relies on discrete conditional entropy, limiting its current application to classification tasks. The authors note that extending this to continuous distributions could open the door for regression-heavy tasks like 3D reconstruction, pose estimation, and semantic segmentation.

Summary updated

Abstract

We propose a novel approach for estimating the difficulty and transferability of supervised classification tasks. Unlike previous work, our approach is solution agnostic and does not require or assume trained models. Instead, we estimate these values using an information theoretic approach: treating training labels as random variables and exploring their statistics. When transferring from a source to a target task, we consider the conditional entropy between two such variables (i.e., label assignments of the two tasks). We show analytically and empirically that this value is related to the loss of the transferred model. We further show how to use this value to estimate task hardness. We test our claims extensively on three large scale data sets -- CelebA (40 tasks), Animals with Attributes 2 (85 tasks), and Caltech-UCSD Birds 200 (312 tasks) -- together representing 437 classification tasks. We provide results showing that our hardness and transferability estimates are strongly correlated with empirical hardness and transferability. As a case study, we transfer a learned face recognition model to CelebA attribute classification tasks, showing state of the art accuracy for tasks estimated to be highly transferable.
BibTeX
@misc{tran2019transferability,
  author = {Anh T. Tran and Cuong V. Nguyen and Tal Hassner},
  title = {{Transferability and Hardness of Supervised Classification Tasks}},
  year = {2019},
  eprint = {1908.08142},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.08142},
}

Loading related content...

Figure