Introduction
Deep Consensus Learning (DCL) introduces a three-player framework that jointly tackles layout-to-image synthesis and weakly-supervised semantic segmentation. By enforcing "consensus" between a generator and an inference network—ensuring synthesized masks match inferred ones and reconstructed images match real ones—the model learns to segment images using only bounding box labels while maintaining high-quality image generation.
In computer vision, generative learning (creating images from labels) and discriminative learning (predicting labels from images) are typically treated as separate domains. For structured tasks like layout-to-image synthesis and semantic segmentation, researchers usually develop independent models. However, this separation leads to two major hurdles: label scarcity, where pixel-level annotations are expensive to produce, and disconnected learning, where generative models may hallucinate details that lack geometric consistency. DCL aims to bridge this gap by making these two processes cooperate, using the generative process to provide "free" supervision for the discriminative one.
Methodology
The paper builds upon the LostGAN architecture, expanding the standard two-player GAN (Generator vs. Discriminator) into a three-player game by adding an Inference Network ($I$). The core of the framework is the dual "Consensus Mappings":
- Latent Consensus (Mask-to-Mask): The Generator ($G$) creates a synthetic mask from a layout and then synthesizes an image. The Inference Network ($I$) then attempts to predict the mask from that synthetic image. DCL enforces that the generated mask and the inferred mask must match, allowing $G$ to act as a teacher for $I$.
- Data Consensus (Image-to-Image): For a real image, the Inference Network ($I$) predicts a semantic mask. The Generator ($G$) then takes this predicted mask and attempts to reconstruct the original real image. Minimizing the difference between the real image and its reconstruction ensures that the masks predicted by $I$ are meaningful and contain sufficient scene information.
The framework is trained end-to-end using a combination of reconstruction loss, perceptual loss (VGG-based), and a hinge-version adversarial loss. A key insight is that the generative model provides an infinite stream of synthetic training data for the segmenter. By forcing the generator to reconstruct a real image from the inference network's predicted mask, the authors prevent the inference network from collapsing into a state where it predicts "garbage" masks.
Experiments and Results
The authors evaluated DCL on the COCO-Stuff dataset (171 categories). For the segmentation task, the model only uses bounding box annotations (weak supervision) rather than ground-truth pixel maps.
Image Synthesis: DCL achieved an Inception Score (IS) of 15.40 and a Frechet Inception Distance (FID) of 50.45 on COCO-Stuff. While slightly behind the state-of-the-art LostGAN-V2 (trained for 100 epochs), DCL reached these competitive results in only 25 epochs, suggesting the consensus mechanism accelerates learning.
Weakly-Supervised Segmentation: Despite the lack of pixel-level ground truth during training, the inference network achieved a 9.59% Mean IOU and 28.12% Frequency Weighted (FW) IOU. This is remarkably close to the fully-supervised baseline (12.59% mIOU / 29.71% FW IOU) trained for four times as long. Qualitative results show the model successfully disentangles objects in complex scenes using only bounding box guidance.
Discussion
While promising, DCL faces optimization instability common in multi-player GAN frameworks. The authors noted training was stopped at 25 epochs partly due to these stability issues. Future improvements could involve using more modern, pre-trained backbones (like HRNet) for the inference network or exploring separate feature normalization branches for real vs. synthesized images to account for the domain gap.
Summary updated