Bayesian Tensor Inference for Sketch-Based Facial Photo Hallucination

Wei Liu, Xiaoou Tang, Jianzhuang Liu

Abstract

This paper develops a statistical inference approach, Bayesian Tensor Inference, for style transformation between photo images and sketch images of human faces. Motivated by the rationale that image appearance is determined by two cooperative factors: image content and image style, we first model the interaction between these factors through learning a patch-based tensor model. Second, by introducing a common variation space, we capture the inherent connection between photo patch space and sketch patch space, thus building bidirectional mapping/inferring between the two spaces. Subsequently, we formulate a Bayesian approach accounting for the statistical inference from sketches to their corresponding photos in terms of the learned tensor model. Comparative experiments are conducted to contrast the proposed method with state-of-the-art algorithms for facial sketch synthesis in a novel face hallucination scenario: sketch-based facial photo hallucination. The encouraging results obtained convincingly validate the effectiveness of our method.

Introduction

While the computer vision community has seen considerable success in image sketching (generating a sketch from a photo), the reverse process—hallucinating a photorealistic image from a sketch—is significantly more challenging. Sketches and photos exist in vastly different visual domains: sketches consist of sparse, high-contrast strokes, while photos contain dense, continuous textures and lighting variations.

Sketching vs. Hallucinating: The bidirectional relationship between photos and sketches.

Previous attempts to bridge these domains often relied on global linear models, which tend to produce blurry, washed-out results. This research introduces a system that reliably infers missing photorealistic textures from sparse sketch lines, a technology with significant potential in law enforcement and forensic identification.

Methodology

The proposed framework, Bayesian Tensor Inference, operates in two primary phases to disentangle content from style and bridge the modality gap.

TensorPatches Model

Images are divided into overlapping patches. To bridge the gap between sketch lines and photo textures, the model extracts image derivatives (gradients) rather than raw pixel intensities. A 4-order tensor is constructed encompassing four dimensions: people, patch positions, styles (photo vs. sketch), and features. Using N-mode Singular Value Decomposition (SVD), this tensor is decomposed to isolate a "common variation space"—representing the underlying identity and structure shared by both modalities—from a "special variation space" representing the specific style.

Bayesian Inference and Gradient Correction

To perform the sketch-to-photo transformation, the authors formulate a Bayesian Maximum A Posteriori (MAP) optimization problem. The learned tensor relations are used to infer local photo patches from sketch patches. To ensure global coherence, this local inference is constrained by a global PCA-based prior of the photo space. Finally, a gradient correction step is applied to render the final pixel intensities.

The proposed framework for sketch-based facial photo hallucination.

Experimental Setup and Results

The method was evaluated on a proprietary database of over 600 individuals, using 500 for training and the remainder for testing. Each entry consists of a normalized frontal face photo and a corresponding "plain style" sketch drawn by an artist.

Qualitative and Quantitative Performance

Visually, the Bayesian Tensor Inference method produces significantly sharper and more realistic faces compared to baseline methods like Eigentransform (global) and Local Geometry Preserving (LGP, local). While the proposed method may show a slightly higher Root-Mean-Square Error (RMSE) than LGP (e.g., 44.90 vs 44.24), its visual perceptual quality is vastly superior. This highlights that pixel-wise error metrics often penalize sharp, realistic textures if they are slightly misaligned, favoring blurry averages instead.

Conclusion

By combining a patch-based multilinear tensor model with Bayesian inference, this framework successfully maps distinct visual domains into a shared latent space. While currently restricted to "plain style" sketches due to the requirement for strict spatial correspondence, future work may incorporate spatial deformation models to handle the non-linear exaggerations common in stylized or forensic sketches.

Summary updated

Abstract

This paper develops a statistical inference approach, Bayesian Tensor Inference, for style transformation between photo images and sketch images of human faces. Motivated by the rationale that image appearance is determined by two cooperative factors: image content and image style, we first model the interaction between these factors through learning a patch-based tensor model. Second, by introducing a common variation space, we capture the inherent connection between photo patch space and sketch patch space, thus building bidirectional mapping/inferring between the two spaces. Subsequently, we formulate a Bayesian approach accounting for the statistical inference from sketches to their corresponding photos in terms of the learned tensor model. Comparative experiments are conducted to contrast the proposed method with state-of-the-art algorithms for facial sketch synthesis in a novel face hallucination scenario: sketch-based facial photo hallucination. The encouraging results obtained convincingly validate the effectiveness of our method.
BibTeX
@inproceedings{liu2007bayesian,
  author = {Wei Liu and Xiaoou Tang and Jianzhuang Liu},
  title = {{Bayesian Tensor Inference for Sketch-Based Facial Photo Hallucination}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2007},
  url = {https://openreview.net/forum?id=SJWgsrfd-B},
}
Figure