PI-Trans: Parallel-Convmlp and Implicit-Transformation Based Gan for Cross-View Image Translation

Bin Ren, Hao Tang, Yiming Wang, Xia Li, Wei Wang, Nicu Sebe

TL;DR

PI-Trans is a novel Generative Adversarial Network (GAN) designed to synthesize realistic ground-level images from aerial photos. By combining convolutional layers with a unique parallel MLP structure and an "implicit transformation" module, it effectively maps long-range pixel dependencies while preserving fine spatial details, outperforming current state-of-the-art models on major benchmarks.

What Problem Does This Paper Address?

Cross-view image translation—specifically generating a street-level view from a satellite or aerial image—is a notoriously difficult task in computer vision. Unlike standard image-to-image translation (like turning a sketch into a photo), cross-view tasks involve drastic geometric shifts and very little pixel-to-pixel overlap.

The authors identify three main gaps in existing methods:

  1. The Locality Trap: Traditional Convolutional Neural Networks (CNNs) use small kernels that struggle to model "long-range dependencies" (e.g., understanding how a building's roof in an aerial view relates to its facade on the ground).
  2. The Detail Loss: Recent attempts to use Multi-Layer Perceptrons (MLPs) to solve the long-range problem often sacrifice fine-grained spatial details, leading to blurry or distorted results.
  3. Ignored Cues: Most models ignore "direct translation" information—the raw mapping from image to image without semantic guidance—which actually contains crucial color and appearance data.

Methods

The authors introduce PI-Trans, which utilizes two primary innovations:

1. Parallel-ConvMLP Module

Instead of stacking convolutions and MLPs sequentially, PI-Trans uses a parallel approach. The input feature map is split into two chunks:

  • Channel-wise MLP: Focuses on "what" the objects are.
  • Spatial-wise MLP: Focuses on "where" they should be located. By processing these in parallel alongside convolutional layers, the model maintains sharp spatial boundaries while reasoning about the global layout of the scene.

2. Implicit Transformation Module

This module acts as a sophisticated fusion center. It takes three inputs: the target semantic map (the "blueprint"), the features from the aerial image, and a "direct translation" branch.

  • The "Direct" Branch: The authors found that even a "bad" translation (one made without a semantic map) contains the correct color palette.
  • Attention Mechanism: The module uses an attention-based mechanism where the semantic map "selects" the most relevant color and texture information from the direct translation branch to populate the final image.

Datasets

The model was evaluated on two standard, challenging datasets for cross-view synthesis:

  • Dayton: A benchmark containing matched aerial and ground-level pairs from urban environments.
  • CVUSA: A large-scale dataset of ground-level panoramas and corresponding satellite images across the United States.

Key Insights and Ideas

  • The Power of Parallelism: Decoupling spatial and channel reasoning into parallel paths prevents the "vanishing detail" problem common in deep MLP architectures. This allows the model to be "global" without becoming "blurry."
  • Don't Throw Away the "Messy" Data: One of the most interesting insights is the reuse of the "direct translation" branch. While this branch produces images that look structurally incorrect to humans, it serves as a vital "color bank" that the rest of the network can draw from.
  • Implicit vs. Explicit: Rather than trying to explicitly warp pixels from the sky to the ground, the "Implicit Transformation" allows the network to learn a latent mapping, which is much more robust to the drastic perspective changes between a drone and a car.

Key Results

PI-Trans achieved new state-of-the-art (SOTA) results across both datasets:

  • Accuracy: On the Dayton dataset, it achieved a Top-1 accuracy of 49.48%, significantly higher than the previous SOTA, CrossMLP (47.65%).
  • Image Quality: It showed marked improvements in Inception Score (IS) and LPIPS (a measure of perceptual similarity), indicating that the generated images look more realistic and sharper to the human eye.
  • Qualitative Superiority: Visual comparisons show that PI-Trans avoids the "ghosting" artifacts and structural distortions (marked by red boxes in the paper's figures) that plague earlier models.

Limitations and Future Work

  • Computational Complexity: The paper utilizes a dual-branch encoder-decoder structure with multiple fusion levels. While effective, the authors do not deeply discuss the computational overhead or inference speed compared to simpler CNN models.
  • Semantic Dependency: The model relies heavily on having an accurate target semantic map as guidance. In real-world scenarios where a semantic map might be noisy or unavailable, the performance might degrade.
  • Future Directions: The authors suggest that this parallel MLP-Conv approach could be extended to other difficult synthesis tasks, such as indoor scene generation or 3D-aware image synthesis.

Summary updated

Abstract

For semantic-guided cross-view image translation, it is crucial to learn where to sample pixels from the source view image and where to reallocate them guided by the target view semantic map, especially when there is little overlap or drastic view difference between the source and target images. Hence, one not only needs to encode the long- range dependencies among pixels in both the source view image and target view semantic map but also needs to translate these learned dependencies. To this end, we propose a novel generative adversarial network, PI-Trans, which mainly consists of a novel Parallel-ConvMLP module and an Implicit Transformation module at multiple semantic levels. Extensive experimental results show that PI-Trans achieves the best qualitative and quantitative performance by a large margin compared to the state-of-the-art methods on two challenging datasets. The source code is available at https://github.com/Amazingren/PI-Trans.
BibTeX
@inproceedings{ren2023pi,
  author = {Bin Ren and Hao Tang and Yiming Wang and Xia Li and Wei Wang and Nicu Sebe},
  title = {{PI-Trans: Parallel-Convmlp and Implicit-Transformation Based Gan for Cross-View Image Translation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095449},
}

Loading related content...

Figure