CSHNet: A Novel Information Asymmetric Image Translation Method

Xi Yang, Haoyuan Shi, Zihan Wang, Nannan Wang, Xinbo Gao

TL;DR

CSHNet is a novel hybrid architecture that combines the local detail-capturing power of CNNs with the global structural awareness of Swin Transformers to solve "information asymmetric" image translation tasks. By introducing a specialized bottleneck (SCB), an interactive guidance module (IGC), and a dynamic edge-aware loss (AEPL), it significantly outperforms existing methods in transforming sparse data (like radar or sketches) into rich, detailed images.

What Problem Does This Paper Address?

Image-to-image translation has seen massive success in tasks like style transfer. However, a specific subset of these tasks—information asymmetric translation—remains a major hurdle. These tasks involve moving from a "sparse" domain (limited information) to a "rich" domain (detailed content). Examples include:

  • SAR-to-Optical (S2O): Converting grainy Synthetic Aperture Radar (radar) images into clear optical satellite photos.
  • Sketch-to-Instance: Turning simple line drawings into fully rendered anime characters.

The authors identify a "Goldilocks" problem with current architectures:

  1. CNNs are great at fine details but have a "local bias." They often lose the big picture, leading to "region blending" where distinct objects (like a river and a forest) bleed into each other.
  2. Transformers excel at global structure but often miss the "intricate details," such as the specific highlights in a character's eyes or sharp textural edges.

Methods

The authors propose the CNN-Swin Hybrid Network (CSHNet), which uses three core innovations to balance local precision with global reasoning:

  1. SEC-CES-Bottleneck (SCB): Instead of just stacking layers, the bottleneck uses a cross-combination of two modules. The Swin Embedded CNN (SEC) uses a CNN backbone to extract details while using a Swin Transformer to provide structural context. Conversely, the CNN Embedded Swin (CES) uses the Transformer as a foundation and embeds CNN modules to recover lost textural details.
  2. Interactive Guided Connection (IGC): As networks get deeper, they often "forget" the raw, low-level details from the input. IGC extracts these early features and uses a learnable linear transformation ($ax + b$) to modulate and "guide" the high-level features in the bottleneck, ensuring fine details aren't lost to abstraction.
  3. Adaptive Edge Perception Loss (AEPL): Standard edge-detection losses (like Canny) use fixed thresholds that don't work for every image. AEPL uses information entropy to dynamically calculate the optimal threshold for each image, forcing the network to maintain sharp, accurate boundaries between different regions.

Datasets

The model was evaluated on two primary benchmarks representing scene-level and instance-level challenges:

  • SEN12: A large-scale dataset of SAR-Optical image pairs acquired by Sentinel-1 and Sentinel-2 satellites.
  • Sketch2Anime: A dataset of anime character images paired with their corresponding sketches.

For both datasets, the authors used a subset of 3,000 pairs for training and 800 for testing to ensure experimental efficiency while maintaining a representative distribution.

Key Insights and Ideas

  • Synergy over Stacking: The "aha moment" here is that simply putting a Transformer after a CNN isn't enough. By embedding one within the other (the SEC and CES modules), the network can leverage the inductive bias of both simultaneously at every step of the bottleneck.
  • Dynamic Constraints: The use of entropy-based thresholding for loss calculation acknowledges that "what constitutes an edge" changes depending on the image content. This adaptivity is crucial for remote sensing data where contrast varies wildly.
  • Efficiency Matters: While pure Transformer models like SwinG or CMT can be massive (up to 2078M parameters), CSHNet achieves better results with only 30.6M parameters, making it much more practical for real-world deployment.

Key Results

CSHNet consistently outperformed state-of-the-art (SOTA) models like Pix2pix, CycleGAN, and SwinG across multiple metrics:

  • Quantitative: On the SEN12 dataset, CSHNet achieved a PSNR of 16.37 and an SSIM of 0.2895, significantly higher than the CNN-based Pix2pix (15.84 / 0.2568). It also achieved the lowest RMSE (45.1), indicating superior pixel-level accuracy.
  • Qualitative: Visual results show that CSHNet successfully avoids the "blending" seen in CNNs (e.g., clearer river boundaries in satellite images) and captures the "intricate details" missed by Transformers (e.g., accurate eye reflections in anime sketches).

Limitations and Future Work

While CSHNet shows robust performance, the authors note a few areas for growth:

  • Multi-modal Integration: Future work could explore how to better integrate semantic and multi-modal translation mechanisms to handle even more complex "asymmetric" scenarios.
  • Pre-trained Models: The authors suggest that leveraging large-scale pre-trained models (Foundation Models) could further improve the generalizability of the network in dynamic settings.
  • Complexity: While more efficient than pure Transformers, it still has slightly more parameters than basic CNN-based generators, which might be a consideration for edge-device applications.

Summary updated

Abstract

Despite the considerable advancements in cross-domain image translation, a significant challenge remains in addressing information asymmetric translation tasks such as SAR-to-Optical and Sketch-to-Instance conversions. These tasks involve transforming data from a domain with limited information into one with more detailed and richer content. Traditional CNN-based methods, while effective at capturing intricate details, often struggle to grasp the overall structural composition of the image, leading to unintended blending or merging of distinct regions within the generated images. In light of these limitations, research has increasingly turned toward Transformers. Though Transformers excel at capturing global structures, they often lack the ability to preserve fine-grained details. Recognizing the importance of both detailed features and structural relationships in information asymmetric translation tasks, we introduce the CNN-Swin Hybrid Network (CSHNet). This network employs a novel bottleneck architecture featuring two key modules: Swin Embedded CNN (SEC) and CNN Embedded Swin (CES), which together form the SEC-CES-Bottleneck (SCB). Within this structure, SEC capitalizes on CNN’s capability for detailed feature extraction while incorporating the Swin Transformer’s inherent structural bias. In contrast, CES preserves the Swin Transformer’s strength in maintaining global structural integrity, while compensating for CNN’s tendency to emphasize detail. In addition to the SCB architecture, CSHNet integrates two essential components designed to improve cross-domain information retention and ensure structural consistency. The Interactive Guided Connection (IGC) fosters dynamic information exchange between SEC and CES, encouraging a deeper understanding of image details. At the same time, Adaptive Edge Perception Loss (AEPL) is implemented to preserve well-defined structural boundaries throughout the translation process. Experimental evaluations demonstrate that CSHNet surpasses current state-of-the-art methods, achieving superior results in both visualization and performance metrics across scene-level and instance-level datasets. Our code is available at: https://github.com/XduShi/CSHNet
BibTeX
@article{yang2026cshnet,
  author = {Xi Yang and Haoyuan Shi and Zihan Wang and Nannan Wang and Xinbo Gao},
  title = {{CSHNet: A Novel Information Asymmetric Image Translation Method}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2026},
  url = {https://openreview.net/forum?id=i1KDvWm1JQ},
}
Figure