DejaVu: Conditional Regenerative Learning to Enhance Dense Prediction

Shubhankar Borse, Debasmit Das, Hyojin Park, Hong Cai, Risheek Garrepalli, Fatih Porikli

Introduction

Dense prediction tasks, such as semantic segmentation and depth estimation, are fundamental to autonomous systems and robotics. These tasks require models to assign precise values to every pixel in an image. However, standard training paradigms often rely on discriminative losses (e.g., cross-entropy) that encourage the network to discard information deemed "non-essential" for classification. This often results in the loss of fine structural details, leading to blurred object boundaries and poor global spatial consistency.

To address this, the authors propose DejaVu, a novel training framework that introduces an auxiliary "Conditional Regeneration" task. By forcing the model to reconstruct a redacted version of the input image using its own dense predictions, DejaVu ensures that the network preserves and utilizes critical structural information.

Methodology: The DejaVu Framework

The core innovation of DejaVu is a training-time loop that tasks the network with "re-earning" the details of the input image. This process involves three main components: image redaction, dense prediction, and conditional regeneration.

The DejaVu training architecture, illustrating the flow from input image to dense prediction and the subsequent conditional regeneration loop used for auxiliary supervision.

Image Redaction Strategies

Before being processed by the auxiliary module, the input image is intentionally degraded to remove structural cues. The authors explore two primary redaction strategies:

  • Spatial Redaction: Pixels or blocks are randomly masked (e.g., using a checkerboard or random block pattern). This forces the model to learn spatial relationships to fill in the gaps.
  • Spectral Redaction: Using the Discrete Cosine Transform (DCT), specific frequency bands are removed. For instance, removing high frequencies blurs edges, requiring the model to provide sharp structural gradients in its predictions to reconstruct the original image.
Examples of spatial and spectral redaction techniques used to challenge the model during the regeneration task.

Conditional Regeneration Module (CRM)

The CRM is a small auxiliary network that takes the redacted image and the base network's dense prediction as inputs. Its objective is to produce a "Regenerated Input" that matches the original, un-redacted image. Because the redacted input lacks sufficient detail, the CRM must rely on the structural information embedded within the dense prediction. This creates a "DejaVu Loss" that backpropagates through the base network, incentivizing it to produce feature-rich, structurally consistent outputs. For applications where inference complexity is less constrained, the authors also introduce DejaVu-SA, which utilizes a shared attention mechanism to integrate this logic directly into the architecture.

Experiments and Results

The DejaVu framework was evaluated across multiple benchmarks, including Cityscapes and ADE20K for segmentation, COCO for panoptic segmentation, and KITTI and NYUD-v2 for depth and surface normal estimation.

Key Findings

  • Performance Gains: DejaVu consistently improved baseline models. For example, it boosted HRNet18 by 1.3 mIoU points on Cityscapes and helped HRNet48 achieve a state-of-the-art score of 87.1 mIoU. In panoptic segmentation, it improved Mask2Former (Swin-T) from 53.2 to 54.3 PQ.
  • Structural Fidelity: Qualitative results indicate that models trained with DejaVu produce significantly sharper object silhouettes and fewer "bleeding" artifacts at boundaries compared to standard training methods.
  • Zero Inference Cost: Since the CRM and the regeneration loop are only used during training, the final model maintains the same inference speed and memory footprint as the original baseline.

Conclusion and Future Work

DejaVu provides an elegant solution to the loss of structural detail in dense prediction by rethinking auxiliary supervision. By making the reconstruction of the input image conditional on the model's predictions, it forces the learning of more robust and detailed features. Future research may extend this concept to video data through "Temporal Redaction" to improve consistency across frames, or further optimize the complexity of the regeneration module to avoid potential bypasses in the conditioning logic.

Summary updated

Abstract

We present DejaVu, a novel framework which leverages conditional image regeneration as additional supervision during training to improve deep networks for dense prediction tasks such as segmentation, depth estimation, and surface normal prediction. First, we apply redaction to the input image, which removes certain structural information by sparse sampling or selective frequency removal. Next, we use a conditional regenerator, which takes the redacted image and the dense predictions as inputs, and reconstructs the original image by filling in the missing structural information. In the redacted image, structural attributes like boundaries are broken while semantic context is largely preserved. In order to make the regeneration feasible, the conditional generator will then require the structure information from the other input source, i.e., the dense predictions. As such, by including this conditional regeneration objective during training, DejaVu encourages the base network to learn to embed accurate scene structure in its dense prediction. This leads to more accurate predictions with clearer boundaries and better spatial consistency. When it is feasible to leverage additional computation, DejaVu can be extended to incorporate an attention-based regeneration module within the dense prediction network, which further improves accuracy. Through extensive experiments on multiple dense prediction benchmarks such as Cityscapes, COCO, ADE20K, NYUD-v2, and KITTI, we demonstrate the efficacy of employing DejaVu during training, as it outperforms SOTA methods at no added computation cost.
BibTeX
@inproceedings{borse2023dejavu,
  author = {Shubhankar Borse and Debasmit Das and Hyojin Park and Hong Cai and Risheek Garrepalli and Fatih Porikli},
  title = {{DejaVu: Conditional Regenerative Learning to Enhance Dense Prediction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure