Learning Purified Feature Representations from Task-irrelevant Labels

Yinghui Li, Ruiyang Liu, Chen Wang, Yangning Li, Ning Ding, Haitao Zheng

TL;DR

This paper introduces "Purified Learning," a novel framework that leverages task-irrelevant labels from auxiliary datasets to improve deep learning model generalization, especially on small, noisy datasets. By minimizing the divergence between feature representations from task-irrelevant labels and the general dataset, it purifies features, enhancing performance on tasks like facial expression recognition and digit classification.

What Problem Does This Paper Address?

Deep neural networks excel with abundant, well-annotated data. However, acquiring such datasets is often costly and time-consuming, leading to scenarios with limited or noisy training samples. In these situations, deep models tend to overfit the training data, resulting in poor generalization to unseen data. Traditional approaches like data augmentation or transfer learning have limitations: data augmentation might not be effective on small datasets, and transfer learning can suffer from "negative transfer" if the auxiliary data's labels are not relevant to the target task. This paper addresses the challenge of improving deep model generalization on small datasets by exploring a less-utilized source of information: task-irrelevant labels.

Methods

The core of Purified Learning is to extract useful information from task-irrelevant labels to mitigate the negative impact of spurious correlations in small training sets. The framework consists of three main components: a feature extractor, a discriminator, and a classifier.

The key idea is to minimize the Wasserstein distance between the distribution of feature representations extracted from samples with a specific task-irrelevant label and the distribution of representations from the entire auxiliary dataset. This process "purifies" the feature representations by reducing the influence of task-irrelevant features.

The optimization is achieved through two main goals:

  1. Empirical Risk Minimization (Goal 1): This is the standard objective of minimizing classification loss on the target training data.
  2. Wasserstein Distance Minimization (Goal 2): This goal uses task-irrelevant samples to minimize the divergence between feature distributions. This is implemented using a discriminator (inspired by WGAN) that tries to distinguish between feature representations from a specific task-irrelevant label group and the general task-irrelevant dataset. The feature extractor is trained to fool this discriminator, thereby reducing the influence of task-irrelevant features.

The overall loss function combines these two goals with weighting factors. The framework is model-agnostic, allowing various existing deep neural network architectures to be used as feature extractors.

Datasets

The paper evaluates Purified Learning on two main tasks:

  • Facial Expression Recognition (FER):
    • Target Task Datasets: ck+, oulu, mmi (small-scale datasets).
  • Task-Irrelevant Label Dataset: CelebA (used for hair color labels).
  • Digit Recognition:
    • Target Task Datasets: MNIST (small-scale training set).
  • Task-Irrelevant Feature: Background color of digits.
  • Test Datasets: SVHN, MNIST-M.

Key Insights and Ideas

  • Task-Irrelevant Labels Can Be Beneficial: Contrary to the common understanding that irrelevant information leads to negative transfer, this paper demonstrates that task-irrelevant labels, when used appropriately, can actually improve model generalization.
  • Purification Through Divergence Minimization: The core innovation lies in using Wasserstein distance to measure and minimize the divergence between feature distributions derived from specific task-irrelevant labels and the broader dataset. This effectively "purifies" the learned representations.
  • Addressing Negative Transfer: Purified Learning offers a way to overcome negative transfer issues encountered in traditional transfer learning by actively leveraging the irrelevance of certain labels.
  • Model Agnosticism: The framework's flexibility allows it to be combined with any existing deep neural network architecture, making it broadly applicable.
  • PCA Analysis Supports Feature Purification: Principal Component Analysis (PCA) results show that Purified Learning leads to representations where task-relevant information has a higher proportion, supporting the claim of feature purification.

Key Results

  • Facial Expression Recognition: Purified Learning achieved better results than baseline methods (Goal 1 Only, Transfer Learning, Multi-task Learning, Adversarial Multi-task Learning) in 14 out of 18 cross-dataset tests. It significantly improved accuracy over "Goal 1 Only" and successfully resolved negative transfer issues.
  • Digit Recognition: Purified Learning outperformed DANN (a state-of-the-art domain adaptation method) and "Goal 1 Only" by a significant margin, improving DANN's accuracy by over 10% on the challenging MNIST to SVHN transfer task.
  • Impact of Irrelevance: Experiments showed that using a more irrelevant label (hair color for FER) yielded better results than a label with some relevance (smiling for FER), reinforcing the idea that Purified Learning benefits from true irrelevance.

Limitations and Future Work

  • Dataset Size for Task-Irrelevant Labels: While the target training dataset can be small, the paper implies that the auxiliary dataset with task-irrelevant labels should be large-scale to provide a reliable distribution for comparison.
  • Defining "Irrelevance": The paper notes that the practical irrelevance of a label is related to performance loss in transfer learning, not just human judgment. Further exploration into quantifying and selecting the "optimal" level of irrelevance could be a future direction.
  • Hyperparameter Tuning: The performance of Purified Learning depends on hyperparameters like the weighting factors ($\lambda_1, \lambda_2$) for the two loss goals.
  • Future Work: The authors believe their exploration of task-irrelevant labels offers a valuable insight for future research, suggesting further investigation into how to best exploit this type of information.

Summary updated

Abstract

Learning an empirically effective model with generalization using limited data is a challenging task for deep neural networks. In this paper, we propose a novel learning framework called PurifiedLearning to exploit task-irrelevant features extracted from task-irrelevant labels when training models on small-scale datasets. Particularly, we purify feature representations by using the expression of task-irrelevant information, thus facilitating the learning process of classification. Our work is built on solid theoretical analysis and extensive experiments, which demonstrate the effectiveness of PurifiedLearning. According to the theory we proved, PurifiedLearning is model-agnostic and doesn't have any restrictions on the model needed, so it can be combined with any existing deep neural networks with ease to achieve better performance. The source code of this paper will be available in the future for reproducibility.
BibTeX
@inproceedings{li2021learning,
  author = {Yinghui Li and Ruiyang Liu and Chen Wang and Yangning Li and Ning Ding and Haitao Zheng},
  title = {{Learning Purified Feature Representations from Task-irrelevant Labels}},
  booktitle = {IJCNN},
  year = {2021},
  url = {https://openreview.net/forum?id=MmOxdN2QEv},
}
Figure