Exploiting Unlabeled Data in CNNs by Self-Supervised Learning to Rank.

Xialei Liu, Joost van de Weijer, Andrew D. Bagdanov

Introduction

Deep learning models are notoriously data-hungry, yet for many regression tasks, obtaining high-quality labels is a significant bottleneck. In Image Quality Assessment (IQA), a single reliable score often requires the consensus of dozens of human raters. Similarly, in Crowd Counting, researchers must manually annotate every individual in images containing thousands of people. This "labeling tax" limits dataset sizes and hinders model generalization.

This paper introduces a self-supervised framework that bypasses this bottleneck by using "learning to rank" as an auxiliary task. By automatically generating ranked image sets from unlabeled data—leveraging synthetic distortions or geometric properties—the authors enable models to learn the underlying logic of a task without human intervention.

Methodology: Multi-task Siamese Learning

The core of the proposed approach is a multi-task Siamese network designed to learn from both labeled and unlabeled data simultaneously.

The general architecture of the multi-task Siamese network, showing shared backbone layers and separate branches for regression and ranking.

The architecture consists of three main components:

  1. Shared Backbone: A CNN (e.g., VGG16) extracts features from both labeled and ranked image sets.
  2. Regression Branch: Trained on labeled data to predict absolute values (e.g., quality scores or person counts).
  3. Ranking Branch: Trained on unlabeled, automatically ranked pairs using a pairwise ranking loss ($L_{rank}$).

To address the computational overhead of Siamese networks, the authors implemented an efficient backpropagation layer. Instead of passing image pairs through the backbone multiple times, each image in a mini-batch is processed once. All possible pairs are then constructed within the loss layer, resulting in a speedup proportional to the mini-batch size.

Self-Supervised Ranking Strategies

The effectiveness of the framework relies on "knowing" the relative rank of images without ground-truth labels. The authors propose two domain-specific strategies:

Image Quality Assessment (IQA)

For IQA, the authors utilize synthetic distortions. While the absolute quality of a natural image is unknown, applying increasing levels of Gaussian blur or JPEG compression creates a deterministic quality hierarchy. The model is trained to recognize that an image with higher distortion intensity has lower quality.

The IQA-specific framework demonstrating how synthetic degradation (e.g., increasing blur) creates a ranked sequence for self-supervision.

Crowd Counting

In crowd counting, the authors employ a "containment" rule. If one image patch is a sub-crop of another, the larger patch must contain a number of people greater than or equal to the smaller one. This geometric guarantee allows for the generation of infinite training pairs from any unlabeled crowd scene.

Key Results and Insights

The multi-task approach yields significant improvements across multiple benchmarks:

  • Performance Gains: On the TID2013 IQA dataset, the Spearman correlation (SROCC) improved from a baseline of 0.612 to 0.806. In crowd counting on the UCF-QNRF dataset, the Mean Absolute Error (MAE) was reduced from 137 to 124.
  • Labeling Efficiency: By using the model's ranking uncertainty as a proxy for active learning, the authors reached target accuracy levels using 50-75% fewer labeled images compared to random sampling.
  • Feature Robustness: Training on ranking tasks forces the network to learn monotonic, robust features that directly enhance the primary regression performance, proving that relative order is a powerful proxy for absolute value.

Conclusion

This research demonstrates that self-supervised ranking is a universal and effective auxiliary task for regression. By exploiting the "free" information available in the physical or geometric properties of unlabeled data, the framework significantly reduces the reliance on expensive human labels. Future work may extend this logic to other domains like age estimation or depth perception, where natural rankings are inherently present.

Summary updated

Abstract

For many applications the collection of labeled data is expensive laborious. Exploitation of unlabeled data during training is thus a long pursued objective of machine learning. Self-supervised learning addresses this by positing an auxiliary task (different, but related to the supervised task) for which data is abundantly available. In this paper, we show how ranking can be used as a proxy task for some regression problems. As another contribution, we propose an efficient backpropagation technique for Siamese networks which prevents the redundant computation introduced by the multi-branch network architecture. We apply our framework to two regression problems: Image Quality Assessment (IQA) and Crowd Counting. For both we show how to automatically generate ranked image sets from unlabeled data. Our results show that networks trained to regress to the ground truth targets for labeled data and to simultaneously learn to rank unlabeled data obtain significantly better, state-of-the-art results for both IQA and crowd counting. In addition, we show that measuring network uncertainty on the self-supervised proxy task is a good measure of informativeness of unlabeled data. This can be used to drive an algorithm for active learning and we show that this reduces labeling effort by up to 50 percent.
BibTeX
@article{liu2019exploiting,
  author = {Xialei Liu and Joost van de Weijer and Andrew D. Bagdanov},
  title = {{Exploiting Unlabeled Data in CNNs by Self-Supervised Learning to Rank.}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2019},
  url = {https://openreview.net/forum?id=8etNlv5gLKQ},
}
Figure