Label super-resolution networks

Kolya Malkin, Caleb Robinson, Le Hou, Rachel Soobitsky, Jacob Czawlytko, Dimitris Samaras, Joel H. Saltz, Lucas Joppa, Nebojsa JojicICLR 2019 · OpenReview

Introduction

High-resolution (HR) labeled data is the "gold standard" for training semantic segmentation models, but it is incredibly expensive and time-consuming to produce. For instance, creating a 1-meter resolution land cover map for the Chesapeake Bay area cost over $1 million. Conversely, low-resolution (LR) labels—like the 30-meter National Land Cover Database (NLCD)—are abundant and cover vast geographic areas.

This paper introduces Label Super-Resolution (SR) Networks, a deep learning framework that trains high-resolution semantic segmentation models using only these coarse, low-resolution labels. By matching the predicted pixel-level statistics to the known composition of low-resolution classes, the authors achieve performance comparable to fully supervised models in land cover mapping and medical imaging.

The Challenge of Weak Labels

The primary challenge with LR labels is that they are often "weak." A single 30m pixel might be labeled "Developed, Low Intensity," but in reality, it contains a complex mixture of houses, roads, and trees. Most existing weakly supervised methods require the LR classes to match the HR classes exactly.

Super-resolving low-resolution labels to high-resolution maps

This paper tackles a more general and difficult problem: How do we train a model when the coarse labels don't match the fine labels, and we only know the statistical relationship between them?

Methodology: Label Super-Resolution

The authors propose a novel training pipeline that converts a standard segmentation network (like a U-Net) into a Label Super-Resolution network through several key innovations:

  1. Label Counting Layer: Instead of comparing every pixel to a ground-truth label, the network predicts HR labels for a block of pixels. A "label counting layer" then calculates the mean and variance of these predictions within that block.
  2. Statistics Matching Loss: The core innovation is a loss function that compares the predicted distribution of classes within a block to the expected distribution provided by the LR label.
  3. Joint Distribution Modeling: The method utilizes a joint distribution $P(Y, Z)$ between "accessory" (LR) classes and "application" (HR) classes. For example, if an LR pixel is "Suburban," the model knows it should expect roughly 40% impervious surfaces and 60% vegetation.
  4. Differentiability: The entire process is formulated to be fully differentiable. This allows the network to learn the visual cues (texture, color, shapes) that correspond to specific HR labels, even though it never sees a pixel-level ground truth during SR training.

Key Insights and Experimental Results

The paper evaluates the method across distinct domains, including land cover mapping (using 1-meter NAIP aerial imagery paired with 30-meter NLCD labels), medical imaging (segmenting tumor-infiltrating lymphocytes), and pedestrian segmentation.

Class Mismatch as a Feature

Unlike previous work, this method doesn't need a 1-to-1 mapping between labels. It treats LR labels as "statistical descriptors." This allows the model to learn that "Urban" (LR) is a mixture of "Building" and "Pavement" (HR). By treating the counts of predicted labels as approximately Gaussian, the authors created a tractable loss function that is stable during training and avoids the computational overhead of more complex models.

The Power of "HR + SR" Hybrid Training

One of the most striking findings is that adding a tiny amount of HR data to a large pool of LR data (the HR+SR approach) allows the model to generalize much better to new geographic regions than a model trained on HR data alone. The LR data acts as a powerful regularizer.

In a "Chesapeake 2014" test set (evaluating a different year and geography), the hybrid HR+SR model achieved 82.4% accuracy, significantly outperforming the HR-only model (67.2%). This proves that weak labels help models adapt to new environments. Even an SR-only model (trained without any pixel-level labels) achieved 82.6% accuracy on standard tests, coming remarkably close to the 91.1% achieved by a fully supervised HR model.

Medical Imaging and Change Detection

The framework's utility extends beyond satellite imagery. For lymphocyte segmentation, the SR model achieved an AUC of 0.7833, which is statistically equivalent to the best-performing supervised baseline (0.7856). Furthermore, the authors demonstrated that their super-resolved maps are accurate enough to detect fine-grained land cover changes (like new buildings) over time, a task notoriously difficult to perform by comparing raw satellite images directly.

Limitations and Future Work

While powerful, the method relies on knowing the joint distribution $P(Y, Z)$ beforehand. While the authors show this can be estimated from small samples or text descriptions, the model's accuracy is fundamentally tied to the accuracy of this prior. Additionally, the current U-Net architecture applies filters uniformly, meaning spatial invariance can be a disadvantage for tasks where pixel location within a block matters (like bounding boxes).

Future directions suggest extending this framework beyond computer vision to fields like public health or natural language processing, where summary statistics for groups are often available even when individual data is private or missing.

Summary updated

Abstract

We present a deep learning-based method for super-resolving coarse (low-resolution) labels assigned to groups of image pixels into pixel-level (high-resolution) labels, given the joint distribution between those low- and high-resolution labels. This method involves a novel loss function that minimizes the distance between a distribution determined by a set of model outputs and the corresponding distribution given by low-resolution labels over the same set of outputs. This setup does not require that the high-resolution classes match the low-resolution classes and can be used in high-resolution semantic segmentation tasks where high-resolution labeled data is not available. Furthermore, our proposed method is able to utilize both data with low-resolution labels and any available high-resolution labels, which we show improves performance compared to a network trained only with the same amount of high-resolution data. We test our proposed algorithm in a challenging land cover mapping task to super-resolve labels at a 30m resolution to a separate set of labels at a 1m resolution. We compare our algorithm with models that are trained on high-resolution data and show that 1) we can achieve similar performance using only low-resolution data; and 2) we can achieve better performance when we incorporate a small amount of high-resolution data in our training. We also test our approach on a medical imaging problem, resolving low-resolution probability maps into high-resolution segmentation of lymphocytes with accuracy equal to that of fully supervised models.
BibTeX
@inproceedings{malkin2019label,
  author = {Kolya Malkin and Caleb Robinson and Le Hou and Rachel Soobitsky and Jacob Czawlytko and Dimitris Samaras and Joel H. Saltz and Lucas Joppa and Nebojsa Jojic},
  title = {{Label super-resolution networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=koSxeDmmQ8},
}
Figure