Conditional Diffusion Process for Inverse Halftoning

Hao Jiang, Yadong MuNeurIPS 2022 · OpenReview

Overview

This paper introduces the Conditional Diffusion model for image Halftoning (CDH), the first generative approach to inverse halftoning. By treating halftone pixels as moving particles in a diffusion process and incorporating a novel "meta-halftone" guidance system, the authors achieve state-of-the-art restoration quality. Crucially, CDH demonstrates significantly better generalization to unseen printing patterns compared to traditional U-Net architectures.

The Challenge: The Dithering Pattern Gap

Halftoning is the process of representing a continuous-tone image (like a photograph) using only black and white dots—a technique ubiquitous in newspapers and vintage books. Inverse halftoning is the task of reversing this process to recover the original smooth gradients from the dotted print.

Baseline methods often struggle with unseen dithering patterns, resulting in moiré-like artifacts and ghosting.

The primary challenge in modern inverse halftoning is the "dithering pattern gap." Most deep learning models, such as U-Nets, are trained on specific mathematical rules (dithering algorithms) for placing dots. However, there are dozens of different algorithms (e.g., Floyd-Steinberg, Stucki), and they evolve over time. When a model trained on one pattern encounters a different one during testing, it fails to generalize, producing ugly, unintended visual patterns and artifacts.

Proposed Method: A Generative Approach

To address this gap, the authors propose a two-stage framework based on Diffusion Models, shifting from a discriminative regression approach to a generative one.

1. Halftone Dithering Diffusion

Instead of a simple regression, the model treats black pixels as "physically moving particles." It trains a diffusion model to move these particles from a random distribution into a valid halftone pattern. This process is conditioned on the image's gray-scale level and a Laplacian prior, which helps preserve important structural edges.

2. Meta-halftone Guided Network

To ensure the generated dots have "blue noise" properties—a high-quality distribution that avoids visual clusters—the authors introduce a guidance network. This network extracts features from a "Meta-halftone Set" (a collection of reference halftones) and uses an affine learning layer to inject these high-quality patterns into the diffusion process. This acts as a "style reference" to keep the dots looking realistic.

3. Inverse Halftoning Diffusion

Finally, a second diffusion model is trained to map these diverse halftone distributions back to continuous-tone images. Because the first stage can generate a wide variety of halftone patterns, the inverse model learns a "universal" mapping rather than overfitting to one specific dot pattern, making it much more robust to different printing styles.

Experimental Setup and Key Results

The model was trained and validated on the UTKFace dataset and tested on the VOC2012 dataset. To rigorously test generalization, the authors collected 9 classical dithering algorithms. They trained the model on 5 of these and tested it on the remaining 4 "unseen" patterns.

The proposed CDH method successfully recovers smooth, continuous-tone images without the artifacts seen in previous approaches.

The proposed CDH method outperformed several strong baselines, including specialized inverse halftoning networks and general-purpose diffusion models:

  • Quantitative Superiority: CDH achieved a PSNR of 24.24 and an SSIM of 0.727, outperforming the next best baseline which scored 23.35 and 0.693, respectively.
  • Qualitative Improvements: Visual results demonstrate that CDH effectively removes the "ghosting" and moiré-like artifacts that plague U-Net-based methods when they encounter unfamiliar dithering patterns. The restored images look significantly smoother and more faithful to the ground truth.

Limitations and Future Work

While highly effective, CDH shares a common limitation with most diffusion models: inference speed. It requires hundreds of steps to "denoise" an image, making it computationally expensive and significantly slower than a standard U-Net. This currently limits its viability for mobile devices or real-time applications.

For future work, the authors suggest exploring parallel diffusion methods to restore low-frequency and high-frequency components separately, which could potentially accelerate the sampling process. They also acknowledge the broader impact concern that such powerful generative models could be misused to create "fake" vintage documents.

Summary updated

Abstract

Inverse halftoning is a technique used to recover realistic images from ancient prints (\textit{e.g.}, photographs, newspapers, books). The rise of deep learning has led to the gradual incorporation of neural network designs into inverse halftoning methods. Most of existing inverse halftoning approaches adopt the U-net architecture, which uses an encoder to encode halftone prints, followed by a decoder for image reconstruction. However, the mainstream supervised learning paradigm with element-wise regression commonly adopted in U-net based methods has poor generalization ability in practical applications. Specifically, when there is a large gap between the dithering patterns of the training and test halftones, the reconstructed continuous-tone images have obvious artifacts. This is an important issue in practical applications, since the algorithms for generating halftones are ever-evolving. Even for the same algorithm, different parameter choices will result in different halftone dithering patterns. In this paper, we propose the first generative halftoning method in the literature, which regards the black pixels in halftones as physically moving particles, and makes the randomly distributed particles move under some certain guidance through reverse diffusion process, so as to obtain desired halftone patterns. In particular, we propose a Conditional Diffusion model for image Halftoning (CDH), which consists of a halftone dithering process and an inverse halftoning process. By changing the initial state of the diffusion model, our method can generate visually plausible halftones with different dithering patterns under the condition of image gray level and Laplacian prior. To avoid introducing redundant patterns and undesired artifacts, we propose a meta-halftone guided network to incorporate blue noise guidance in the diffusion process. In this way, halftone images subject to more diverse distributions are fed into the inverse halftoning model, which helps the model to learn a more robust mapping from halftone distributions to continuous-tone distributions, thereby improving the generalization ability to unseen samples. Quantitative and qualitative experimental results demonstrate that the proposed method achieves state-of-the-art results.
BibTeX
@inproceedings{jiang2022conditional,
  author = {Hao Jiang and Yadong MU},
  title = {{Conditional Diffusion Process for Inverse Halftoning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=EEcFW47sktI},
}
Figure