Guided Image-to-Image Translation with Bi-Directional Feature Transformation

Badour Albahar, Jia-Bin HuangICCV 2019 · DOI

Introduction

Guided image-to-image translation aims to generate images that adhere to the structural constraints of an input (e.g., a sketch or low-resolution map) while adopting the stylistic or semantic attributes of a guidance image (e.g., a texture patch or a target pose). Traditional conditioning methods, such as Feature-wise Linear Modulation (FiLM) or simple concatenation, often suffer from two primary limitations: uni-directionality and spatial invariance.

In uni-directional models, information flows only from the guidance to the input, preventing the model from reconciling conflicting constraints between the two sources. Furthermore, many methods apply transformations globally across feature maps, failing to capture the localized, fine-grained details required for complex tasks like pose synthesis or high-fidelity depth upsampling. This paper introduces Bi-directional Feature Transformation (bFT) to address these gaps through mutual modulation and spatially varying tensors.

Methodology: The bFT Layer

The core innovation is the bFT layer, designed as a drop-in replacement for standard normalization layers in generative architectures like U-Nets or ResNets.

Spatially Varying Transformation

Unlike FiLM, which uses vectors for scaling ($\gamma$) and shifting ($\beta$), bFT generates 3D tensors. This allows the model to apply unique transformations to different spatial locations, preserving local structures and handling dense guidance (like RGB images) more effectively.

Bi-directional Flow

The architecture employs a "mutual modulation" scheme. In each layer, the guidance features scale and shift the input features, while the input features simultaneously scale and shift the guidance features. This creates a "dialogue" between the two branches, allowing them to iteratively refine one another as information passes through the network.

Qualitative results across three tasks: Pose Transfer (DeepFashion), Texture Transfer (Handbags/Shoes), and Depth Upsampling (NYU v2).

Applications and Evaluation

The authors evaluated bFT across three distinct domains:

  1. Texture Transfer: Using sketch-to-photo datasets, bFT demonstrated superior ability to map specific textures onto object geometries. On the Handbag dataset, it achieved an LPIPS score of 0.161, significantly outperforming standard pix2pix (0.234).
  2. Pose Transfer: Using the DeepFashion dataset, the model synthesized humans in new poses defined by guidance keypoints. Qualitative results showed sharper images with fewer artifacts than previous state-of-the-art methods.
  3. Depth Upsampling: On the NYU v2 dataset, bFT generated high-resolution depth maps from low-resolution inputs using RGB guides. It achieved a Root Mean Square Error (RMSE) of 3.35 at 4x scale, outperforming specialized depth-sensing models.

Key Insights and Results

The transition from channel-wise vectors to 3D tensors provides the flexibility needed to handle both sparse guidance (like pose keypoints) and dense guidance (like RGB images) within a single framework. Quantitative metrics across all tasks confirm that bi-directional communication helps the model better understand how guidance constraints should apply to specific input geometries.

While highly effective, the model faces challenges when there is a significant semantic mismatch—for instance, attempting to apply a highly complex texture to an overly simplified sketch. In such cases, the model may struggle to hallucinate the missing details correctly.

Conclusion

Bi-directional Feature Transformation represents a significant step forward in guided image synthesis. By enabling mutual modulation and spatial variance, bFT allows for more nuanced integration of guidance information, leading to state-of-the-art performance in texture transfer, pose synthesis, and depth upsampling.

Summary updated

BibTeX
@inproceedings{albahar2019guided,
  author = {Badour Albahar and Jia-Bin Huang},
  title = {{Guided Image-to-Image Translation with Bi-Directional Feature Transformation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00911},
}
Figure