Wavelet-Based Dual-Stream Network for Inpainting of Dunhuang Murals

Jiacheng Li, Yuqing Shi, Jing Lian, Shiqiang Du, Yang ZhangResearch Square 2024 · DOI

Abstract

The preservation and restoration of the Dunhuang murals, a paramount element of China's ancient artistic heritage, are confronted with significant challenges due to their extensive degradation over centuries. Traditional inpainting methods, which are mainly focused on the spatial domain, frequently fall short in authentically restoring these murals, especially considering their distinct artistic style. In response to these challenges, our study introduces a novel wavelet-based dual-stream network (WDSN). This innovative approach bifurcates the restoration process into distinct structure and texture pathways, employing wavelet decomposition to dissect the damaged image into low-frequency (structural) and high-frequency (textural) subbands. A key feature of our method is the incorporation of an Efficient Transformer Block (ETB) within the structure branch, alongside a Residual in Residual Dense Block (RRDB) in the texture branch. These elements synergistically enhance the fidelity of global structural reconstruction and local texture refinement. Furthermore, the integration of the CIEDE2000 color difference formula plays a pivotal role in ensuring the color authenticity of the restored murals, aligning more closely with human visual perception. Our comprehensive tests indicate that this methodology substantially outperforms existing algorithms in mural inpainting, offering a groundbreaking solution for the restoration and preservation of these invaluable cultural artifacts.

Introduction

The Dunhuang murals of China’s Mogao Grottoes represent a world-renowned cultural treasure. However, centuries of environmental exposure have left them riddled with pigment peeling, flaking, and color degradation. Digital image inpainting—the process of "filling in" missing or damaged parts of an image—is a vital tool for restoration.

Standard inpainting methods often struggle with these murals for two primary reasons. First, most models operate purely in the spatial domain (pixels), which often leads to blurred textures or structural inconsistencies when dealing with complex artistic styles. Second, standard loss functions like $\ell_1$ or Mean Squared Error treat color as a simple numerical distance in RGB space, failing to account for how the human eye actually perceives color. This often results in "restored" sections that look jarringly different from the original ancient pigments.

The WDSN Architecture

To address these limitations, we propose the Wavelet-Based Dual-Stream Network (WDSN). The core innovation lies in how the network decomposes the image into different frequency components to handle structure and texture separately.

Overview of the Wavelet-Based Dual-Stream Network (WDSN) architecture, featuring structure and texture branches.

Wavelet Decomposition

Instead of processing raw pixels, the model uses a Discrete Wavelet Transform (DWT) to split the damaged image into a low-frequency subband (representing global structure) and three high-frequency subbands (representing fine texture). This allows the network to apply specialized processing to each component.

Dual-Stream Processing

  • Structure Branch: This branch utilizes an Efficient Transformer Block (ETB). Unlike standard convolutions with limited receptive fields, the ETB uses specialized attention mechanisms to capture global dependencies, ensuring that large-scale shapes and lines remain coherent across the mural.
  • Texture Branch: This branch employs Residual in Residual Dense Blocks (RRDB). These blocks are optimized for extracting and reconstructing the fine-grained, local details found in the high-frequency subbands, preserving the unique brushwork of the original artists.

Efficient Transformer Blocks and Color Fidelity

A significant technical hurdle in using Transformers for high-resolution images is computational overhead. The WDSN overcomes this by using stripe-window and axial self-attention within the ETB.

Comparison of stripe-window self-attention and axial attention mechanisms used in the ETB.

By splitting attention into horizontal and vertical stripes (axial attention) or localized windows, the model gains the benefits of global context without the massive memory requirements of standard self-attention.

Furthermore, to solve the color-shifting problem, we integrated the CIEDE2000 color difference formula into the loss function. This formula is designed to mimic human visual perception, ensuring that the restored colors feel authentic to the original mural's palette by prioritizing visual accuracy over simple numerical RGB distance.

Results and Evaluation

The WDSN was evaluated on a specialized Dunhuang Mural Dataset (expanded to 13,291 images) and the standard Paris StreetView benchmark.

Quantitative Performance

On the Dunhuang dataset, WDSN achieved a PSNR (Peak Signal-to-Noise Ratio) of 23.42 and an SSIM (Structural Similarity Index) of 0.8460, consistently outperforming leading models such as PConv, RFR, and CTSDG.

Qualitative Performance

Visually, the WDSN was significantly better at reconstructing sharp edges and delicate facial features. While other models produced "muddy" or "ghostly" artifacts in heavily damaged areas, WDSN maintained structural integrity and color consistency, producing results that align more closely with the original artistic intent.

Conclusion

The Wavelet-Based Dual-Stream Network offers a robust solution for the digital restoration of ancient art. By leveraging wavelet decomposition and perceptual color loss, it bridges the gap between mathematical image processing and human artistic perception. Future work will explore integrating semantic maps and additional wavelet-based priors to handle extreme cases of degradation where original structural information is almost entirely lost.

Summary updated

Abstract

<title>Abstract</title> The preservation and restoration of the Dunhuang murals, a paramount element of China's ancient artistic heritage, are confronted with significant challenges due to their extensive degradation over centuries. Traditional inpainting methods, which are mainly focused on the spatial domain, frequently fall short in authentically restoring these murals, especially considering their distinct artistic style. In response to these challenges, our study introduces a novel wavelet-based dual-stream network (WDSN). This innovative approach bifurcates the restoration process into distinct structure and texture pathways, employing wavelet decomposition to dissect the damaged image into low-frequency (structural) and high-frequency (textural) subbands. A key feature of our method is the incorporation of an Efficient Transformer Block (ETB) within the structure branch, alongside a Residual in Residual Dense Block (RRDB) in the texture branch. These elements synergistically enhance the fidelity of global structural reconstruction and local texture refinement. Furthermore, the integration of the CIEDE2000 color difference formula plays a pivotal role in ensuring the color authenticity of the restored murals, aligning more closely with human visual perception. Our comprehensive tests indicate that this methodology substantially outperforms existing algorithms in mural inpainting, offering a groundbreaking solution for the restoration and preservation of these invaluable cultural artifacts, thereby contributing significantly to the conservation of global art heritage.
BibTeX
@inproceedings{li2024wavelet,
  author = {Jiacheng Li and Yuqing Shi and Jing Lian and Shiqiang Du and Yang Zhang},
  title = {{Wavelet-Based Dual-Stream Network for Inpainting of Dunhuang Murals}},
  booktitle = {Research Square},
  year = {2024},
  doi = {10.21203/rs.3.rs-4346893/v1},
}
Figure