Deep Consensus Learning

Wei Sun, Tianfu WuarXiv 2021 · arXiv

Introduction

Deep Consensus Learning (DCL) introduces a three-player framework that jointly tackles layout-to-image synthesis and weakly-supervised semantic segmentation. By enforcing "consensus" between a generator and an inference network—ensuring synthesized masks match inferred ones and reconstructed images match real ones—the model learns to segment images using only bounding box labels while maintaining high-quality image generation.

In computer vision, generative learning (creating images from labels) and discriminative learning (predicting labels from images) are typically treated as separate domains. For structured tasks like layout-to-image synthesis and semantic segmentation, researchers usually develop independent models. However, this separation leads to two major hurdles: label scarcity, where pixel-level annotations are expensive to produce, and disconnected learning, where generative models may hallucinate details that lack geometric consistency. DCL aims to bridge this gap by making these two processes cooperate, using the generative process to provide "free" supervision for the discriminative one.

Methodology

The paper builds upon the LostGAN architecture, expanding the standard two-player GAN (Generator vs. Discriminator) into a three-player game by adding an Inference Network ($I$). The core of the framework is the dual "Consensus Mappings":

  1. Latent Consensus (Mask-to-Mask): The Generator ($G$) creates a synthetic mask from a layout and then synthesizes an image. The Inference Network ($I$) then attempts to predict the mask from that synthetic image. DCL enforces that the generated mask and the inferred mask must match, allowing $G$ to act as a teacher for $I$.
  2. Data Consensus (Image-to-Image): For a real image, the Inference Network ($I$) predicts a semantic mask. The Generator ($G$) then takes this predicted mask and attempts to reconstruct the original real image. Minimizing the difference between the real image and its reconstruction ensures that the masks predicted by $I$ are meaningful and contain sufficient scene information.
Overview of the DCL framework showing the layout-to-image synthesis and real image reconstruction paths.

The framework is trained end-to-end using a combination of reconstruction loss, perceptual loss (VGG-based), and a hinge-version adversarial loss. A key insight is that the generative model provides an infinite stream of synthetic training data for the segmenter. By forcing the generator to reconstruct a real image from the inference network's predicted mask, the authors prevent the inference network from collapsing into a state where it predicts "garbage" masks.

Experiments and Results

The authors evaluated DCL on the COCO-Stuff dataset (171 categories). For the segmentation task, the model only uses bounding box annotations (weak supervision) rather than ground-truth pixel maps.

Image Synthesis: DCL achieved an Inception Score (IS) of 15.40 and a Frechet Inception Distance (FID) of 50.45 on COCO-Stuff. While slightly behind the state-of-the-art LostGAN-V2 (trained for 100 epochs), DCL reached these competitive results in only 25 epochs, suggesting the consensus mechanism accelerates learning.

Weakly-Supervised Segmentation: Despite the lack of pixel-level ground truth during training, the inference network achieved a 9.59% Mean IOU and 28.12% Frequency Weighted (FW) IOU. This is remarkably close to the fully-supervised baseline (12.59% mIOU / 29.71% FW IOU) trained for four times as long. Qualitative results show the model successfully disentangles objects in complex scenes using only bounding box guidance.

Discussion

While promising, DCL faces optimization instability common in multi-player GAN frameworks. The authors noted training was stopped at 25 epochs partly due to these stability issues. Future improvements could involve using more modern, pre-trained backbones (like HRNet) for the inference network or exploring separate feature normalization branches for real vs. synthesized images to account for the domain gap.

Summary updated

Abstract

Both generative learning and discriminative learning have recently witnessed remarkable progress using Deep Neural Networks (DNNs). For structured input synthesis and structured output prediction problems (e.g., layout-to-image synthesis and image semantic segmentation respectively), they often are studied separately. This paper proposes deep consensus learning (DCL) for joint layout-to-image synthesis and weakly-supervised image semantic segmentation. The former is realized by a recently proposed LostGAN approach, and the latter by introducing an inference network as the third player joining the two-player game of LostGAN. Two deep consensus mappings are exploited to facilitate training the three networks end-to-end: Given an input layout (a list of object bounding boxes), the generator generates a mask (label map) and then use it to help synthesize an image. The inference network infers the mask for the synthesized image. Then, the latent consensus is measured between the mask generated by the generator and the one inferred by the inference network. For the real image corresponding to the input layout, its mask also is computed by the inference network, and then used by the generator to reconstruct the real image. Then, the data consensus is measured between the real image and its reconstructed image. The discriminator still plays the role of an adversary by computing the realness scores for a real image, its reconstructed image and a synthesized image. In experiments, our DCL is tested in the COCO-Stuff dataset. It obtains compelling layout-to-image synthesis results and weakly-supervised image semantic segmentation results.
BibTeX
@misc{sun2021deep,
  author = {Wei Sun and Tianfu Wu},
  title = {{Deep Consensus Learning}},
  year = {2021},
  eprint = {2103.08475},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.08475},
}
Figure