SEIT: Structural Enhancement for Unsupervised Image Translation in Frequency Domain

Zhifeng Zhu, Yaochen Li, Yifan Li, Jinhuo Yang, Peijun Chen, Yuehu LiuAAAI 2024 · DOI

Introduction

Unsupervised image-to-image (I2I) translation aims to map images between two domains (e.g., day to night, sunny to cloudy) without paired training data. While Generative Adversarial Networks (GANs) have become the standard for this task, they face a persistent challenge: the "style vs. structure" trade-off. Models often struggle to apply significant stylistic changes (color, lighting, texture) while faithfully preserving the underlying structural integrity (roads, buildings, fine details). Processing images primarily in the spatial domain often leads to structural distortions or "hallucinated" artifacts when style shifts are drastic.

Methodology

To address these limitations, the proposed SEIT (Structural Enhancement in Frequency Domain) framework shifts the translation process from the spatial domain to the frequency domain. By decoupling content and style using Fourier and Wavelet transforms, the model achieves superior style transfer without compromising structural fidelity.

The SEIT architecture utilizes a multi-scale generator integrated with FDA and WSE modules to decouple and enhance features across domains.

Frequency Dynamic Adaptive (FDA) Module

The FDA module leverages the Discrete Fourier Transform (DFT). In the Fourier domain, an image is decomposed into its Amplitude and Phase components. The Amplitude typically carries style and texture information, while the Phase encapsulates structural and content information. The FDA module extracts the target domain's style from its amplitude and applies it to the source image while keeping the source's phase information intact, ensuring the "what" of the image remains unchanged while the "how it looks" is transformed.

Wavelet-based Structure Enhancement (WSE) Module

To prevent the blurring of sharp edges often associated with global Fourier transforms, the Wavelet-based Structure Enhancement (WSE) module is employed. Using the Discrete Wavelet Transform (DWT), the image is decomposed into low-frequency and high-frequency sub-bands. The WSE module specifically reinjects high-frequency details from the original source image back into the translated output, preserving fine structural elements like power lines and distant architectural features.

Experimental Results

SEIT was evaluated on three benchmark datasets: SYNTHIA (Day → Night), BDD100k (Sunny → Cloudy), and GTA5 → Cityscapes (Synthetic-to-Real). The model consistently outperformed state-of-the-art baselines such as CUT, MUNIT, and VSAIT.

Performance comparison showing SEIT achieving the highest SSIM and lowest FID on the SYNTHIA Day to Night task.

Quantitative Performance

On the SYNTHIA Day → Night task, SEIT achieved a Structural Similarity (SSIM) of 0.87, a significant improvement over the previous state-of-the-art (LPTN at 0.83). Furthermore, SEIT reached the lowest Fréchet Inception Distance (FID) across tasks, such as 82.55 for GTA5 → Cityscapes, indicating that its generated images are statistically the most similar to real target-domain data.

Conclusion

SEIT demonstrates that frequency-domain decoupling is a highly effective strategy for unsupervised I2I translation. By treating the Fourier Phase as a structural anchor and using Wavelets for edge recovery, the model allows for aggressive style changes while maintaining rigid structural consistency. Future work will explore extending this framework to multimodal translation tasks and improving performance on non-rigid transformations.

Summary updated

BibTeX
@inproceedings{zhu2024seit,
  author = {Zhifeng Zhu and Yaochen Li and Yifan Li and Jinhuo Yang and Peijun Chen and Yuehu Liu},
  title = {{SEIT: Structural Enhancement for Unsupervised Image Translation in Frequency Domain}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i7.28617},
}
Figure