Multi-modal dictionary learning for image separation with application in art investigation

Nikos Deligiannis, João F. C. Mota, Bruno Cornelis, Miguel R. D. Rodrigues, Ingrid Daubechies

Abstract

In support of art investigation, researchers have proposed a novel source separation method to unmix single X-ray scans acquired from double-sided paintings. Traditional methods struggle when sources have similar morphological characteristics, such as overlapping brushstrokes. This approach utilizes high-resolution photographs of the front and back of a panel as side information to drive a coupled dictionary learning framework. By capturing both common features across modalities and modality-specific innovations, the method enables accurate separation of complex X-ray mixtures, significantly outperforming state-of-the-art techniques.

The Challenge of Double-Sided Masterpieces

In art conservation, X-ray radiography is a vital tool for seeing beneath the surface of a painting. However, many masterpieces—like the panels of the famous Ghent Altarpiece (1432)—are painted on both sides of a single wooden support. Because X-rays penetrate the entire structure, the resulting image is a confusing jumble of the front painting, the back painting, and the internal wood grain.

The input data for the Ghent Altarpiece: Front photographs (left), back photographs (center), and the mixed X-ray radiography (right).

Standard "blind" source separation techniques, such as Morphological Component Analysis (MCA), usually rely on the different layers having distinct textures (e.g., smooth vs. grainy). In this case, the two sources are both paintings with similar brushstroke characteristics, making them nearly impossible to distinguish without extra help. This research addresses how to use visible-light photographs—which only capture one side at a time—to guide the separation of the complex X-ray mixture.

Methodology: Coupled Dictionary Learning

The proposed framework relies on Coupled Dictionary Learning. The core idea is to represent images as a combination of "common" and "innovation" components:

  • Common Component: Captures features shared between the visible photograph and the X-ray (like the shapes and structures of the figures).
  • Innovation Component: Captures features unique to the X-ray modality (like the wood grain of the panel or internal support structures).

To solve the sparse coding problem across these dictionaries, the authors developed a variation of the Orthogonal Matching Pursuit algorithm (mOMP). Additionally, the algorithm uses binary masks to identify "craquelure" (the fine network of cracks on old paintings). By assigning a weight of zero to these pixels during training, the dictionaries avoid being contaminated by high-frequency noise, leading to cleaner reconstructions.

The Multi-Scale Advantage

A critical finding of the study was that single-scale dictionary learning often leaves "ghosts" of the other side in the separated images because it cannot handle the DC (average brightness) values of patches correctly. The authors implemented a recursive pyramid decomposition to address this.

The multi-scale pyramid decomposition allows the model to separate low-frequency content at coarse scales and fine details at higher resolutions.

This multi-scale approach effectively "drains" the low-frequency content down through the pyramid, allowing the model to capture global structures at coarse scales and fine details at high resolutions. This was found to be essential for achieving high-quality visual separation.

Key Results and Insights

Experimentation on both synthetic data and the 2TB digital acquisition of the Ghent Altarpiece confirmed the superiority of this method:

  • Quantitative Gains: On simulated mixtures, the proposed method outperformed state-of-the-art MCA by up to 1.41 dB in PSNR and significantly improved the Structural Similarity Index (SSIM).
  • Qualitative Improvement: Visual results on the Adam and Eve panels show a dramatic improvement. Where previous methods left faces visible on both separated sides, this method successfully isolated the face to one side and the fabric textures to the other. It also successfully isolated the vertical wood grain into the "innovation" layer.

Limitations and Future Work

The method relies heavily on the photographs and X-rays being perfectly registered (aligned pixel-for-pixel). Any misalignment can lead to artifacts in the separation. Furthermore, dictionary learning and multi-scale sparse coding are computationally expensive, especially for ultra-high-resolution art scans.

Future directions include extending this framework to other multi-modal problems, such as medical imaging (combining MRI and CT scans) or remote sensing, and automating the "innovation vs. common" selection process to prevent brushstrokes from being misclassified as wood grain.

Summary updated

Abstract

In support of art investigation, we propose a new source separation method that unmixes a single X-ray scan acquired from double-sided paintings. In this problem, the X-ray signals to be separated have similar morphological characteristics, which brings previous source separation methods to their limits. Our solution is to use photographs taken from the front and back-side of the panel to drive the separation process. The crux of our approach relies on the coupling of the two imaging modalities (photographs and X-rays) using a novel coupled dictionary learning framework able to capture both common and disparate features across the modalities using parsimonious representations; the common component models features shared by the multi-modal images, whereas the innovation component captures modality-specific information. As such, our model enables the formulation of appropriately regularized convex optimization procedures that lead to the accurate separation of the X-rays. Our dictionary learning framework can be tailored both to a single- and a multi-scale framework, with the latter leading to a significant performance improvement. Moreover, to improve further on the visual quality of the separated images, we propose to train coupled dictionaries that ignore certain parts of the painting corresponding to craquelure. Experimentation on synthetic and real data - taken from digital acquisition of the Ghent Altarpiece (1432) - confirms the superiority of our method against the state-of-the-art morphological component analysis technique that uses either fixed or trained dictionaries to perform image separation.
BibTeX
@article{deligiannis2017multi,
  author = {Nikos Deligiannis and João F. C. Mota and Bruno Cornelis and Miguel R. D. Rodrigues and Ingrid Daubechies},
  title = {{Multi-modal dictionary learning for image separation with application in art investigation}},
  journal = {IEEE Trans. Image Process.},
  year = {2017},
  url = {https://openreview.net/forum?id=hTPqq-7VHXmV},
}
Figure