Exploiting Class Activation Value for Partial-Label Learning

Fei Zhang, Lei Feng, Bo Han, Tongliang Liu, Gang Niu, Tao Qin, Masashi SugiyamaICLR 2022 · OpenReview

TL;DR

Researchers have developed CAV Learning (CAVL), a novel approach to Partial-Label Learning that identifies true labels by looking at a model's internal "Class Activation Value" rather than relying on rigid statistical assumptions about data noise. By generalizing the concept of Class Activation Maps (CAM) to any neural architecture, CAVL achieves state-of-the-art performance across both image and non-image datasets.

What Problem Does This Paper Address?

In traditional supervised learning, every training instance comes with one exact label (e.g., "Dog"). In Partial-Label Learning (PLL), the annotator provides a set of candidate labels (e.g., {"Dog", "Cat", "Fox"}), only one of which is correct. This is much cheaper and faster for humans to produce, but significantly harder for machines to learn from.

Most existing deep learning methods for PLL rely on specific assumptions—for example, assuming that the "noise" labels are distributed uniformly. If the real-world data doesn't fit these mathematical assumptions, the model's performance tanks. The authors of this paper wanted to find a way to identify the true label using the model's own "gut feeling" (its intrinsic representations) rather than external assumptions.

Methods

The researchers started with a "pilot experiment" using Class Activation Maps (CAM). CAM is a standard computer vision technique that highlights which pixels a Convolutional Neural Network (CNN) is "looking at" to make a prediction. They discovered two fascinating properties:

  1. Power: The class with the most "activated" regions in the CAM is highly likely to be the true label.
  2. Dynamic: The accuracy of CAM-based label selection improves in lockstep with the model's overall training progress.
Pilot experiment demonstrating the Power and Dynamic properties of CAM.

However, CAM only works for CNNs and image data. To make this concept universal, the authors proposed the Class Activation Value (CAV).

The CAVL Algorithm

  1. Generalization: They derived a mathematical value ($v^j = |f^j(x) - 1|f^j(x)$) that captures the "importance" of a class label based on the model's logits (outputs before the final softmax). This works for any architecture, including simple Linear models and Multi-Layer Perceptrons (MLPs).
  2. Warm-up: In the first epoch, the model treats all candidate labels equally (an "average-based" strategy).
  3. Label Selection: From the second epoch onward, the model calculates the CAV for every candidate label. It picks the label with the maximum CAV as the "true" label for that training step.
  4. Iterative Refinement: As the model gets smarter, its CAVs become more accurate, which in turn provides cleaner labels for the next round of training.

Datasets

The authors evaluated CAVL on two types of data:

  • Benchmark-Simulated: They took standard datasets—MNIST, Fashion-MNIST, Kuzushiji-MNIST (KMNIST), and CIFAR-10—and manually added "candidate" labels using two different strategies (Uniform Sampling and Flipping Probability) to simulate real-world ambiguity.
  • Real-World PLL Datasets: They tested on five datasets where labels are naturally ambiguous: Lost, MSRCv2, BirdSong, Soccer Player, and Yahoo!News. These include tasks like automatic face naming in images and bird species classification from audio.

Key Insights and Ideas

  • The Model "Knows" More Than It Shows: Even when a model is confused by multiple candidate labels, its internal activation patterns (the CAV) often point toward the ground truth long before the final output layer reaches high confidence.
  • Moving Beyond Assumptions: By using CAV, the model doesn't care how the candidate labels were generated. It simply exploits its own learned representations to disambiguate the data.
  • Versatility is Key: Unlike CAM, which is restricted to spatial image data, CAV allows this "internal look" to be applied to feature vectors (like the BirdSong dataset) and non-convolutional models.

Key Results

  • Superior Accuracy: CAVL outperformed state-of-the-art methods like PRODEN and RC/CC across almost all benchmarks. For example, on CIFAR-10 with a DenseNet backbone, CAVL achieved 81.58% accuracy compared to PRODEN's 79.05%.
Performance comparison of CAVL against state-of-the-art methods across various datasets and ambiguity levels.
  • Robustness to Noise: In high-ambiguity scenarios (where many false labels are included in the candidate set), CAVL maintained a significant lead over other methods.
  • Real-World Success: On the "Lost" dataset (face naming), CAVL achieved 82.07% accuracy, significantly higher than the next best method (RC at 80.45%).

Limitations and Future Work

  • The "Starting Epoch" Sensitivity: The authors found that if you wait too long to start using CAVL (the "warm-up" period), performance drops. This is because deep networks eventually "memorize" the noisy candidate labels if they aren't corrected early enough.
  • Initial Training Quality: CAVL relies on the model learning something useful in the first epoch. If the initial "average-based" training is too poor, the CAVs might initially point to the wrong labels, leading to a "garbage in, garbage out" cycle.
  • Future Direction: Integrating CAVL with other weakly supervised techniques (like semi-supervised learning) could further boost performance when candidate labels are extremely sparse or noisy.

Summary updated

Abstract

Partial-label learning (PLL) solves the multi-class classification problem, where each training instance is assigned a set of candidate labels that include the true label. Recent advances showed that PLL can be compatible with deep neural networks, which achieved state-of-the-art performance. However, most of the existing deep PLL methods focus on designing proper training objectives under various assumptions on the collected data, which may limit their performance when the collected data cannot satisfy the adopted assumptions. In this paper, we propose to exploit the learned intrinsic representation of the model to identify the true label in the training process, which does not rely on any assumptions on the collected data. We make two key contributions. As the first contribution, we empirically show that the class activation map (CAM), a simple technique for discriminating the learning patterns of each class in images, could surprisingly be utilized to make accurate predictions on selecting the true label from candidate labels. Unfortunately, as CAM is confined to image inputs with convolutional neural networks, we are yet unable to directly leverage CAM to address the PLL problem with general inputs and models. Thus, as the second contribution, we propose the class activation value (CAV), which owns similar properties of CAM, while CAV is versatile in various types of inputs and models. Building upon CAV, we propose a novel method named CAV Learning (CAVL) that selects the true label by the class with the maximum CAV for model training. Extensive experiments on various datasets demonstrate that our proposed CAVL method achieves state-of-the-art performance.
BibTeX
@inproceedings{zhang2022exploiting,
  author = {Fei Zhang and Lei Feng and Bo Han and Tongliang Liu and Gang Niu and Tao Qin and Masashi Sugiyama},
  title = {{Exploiting Class Activation Value for Partial-Label Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=fAzQkShlPlc},
}
Figure