TL;DR
This paper investigates the limitations of the FUNIT unsupervised image-to-image translation framework when applied to complex, content-rich scenes. The authors propose and evaluate methods to improve FUNIT's performance by leveraging object detection, demonstrating that incorporating object-level information and increasing dataset diversity significantly enhances translation quality.
What Problem Does This Paper Address?
Unsupervised image-to-image (I2I) translation aims to transform an image from a source domain to a target domain without paired examples. While recent methods like FUNIT have shown promise, they often struggle with complex scenes containing multiple objects and can produce unusable results on unrealistic styles (e.g., cartoons). Furthermore, their performance degrades when applied to target classes not seen during training. This paper specifically addresses the challenge of extending I2I translation capabilities to more diverse and content-rich images, moving beyond simpler, single-object scenarios. The authors also explore how to make these methods more robust to unseen target classes and improve their handling of complex scenes by incorporating object detection techniques.
Methods
The core of this work involves adapting and evaluating the FUNIT (Few-Shot Unsupervised Image-to-Image Translation) framework. FUNIT uses a generator with separate encoders for content and style, and a decoder that combines these representations. The authors' approach involves several key methodological steps:
- Dataset Extension with Object Detection: To address the limitations of existing datasets and improve the model's ability to handle complex scenes, the authors employ RetinaNet, a single-shot object detector. This detector is used to identify objects within images, and the resulting object detections are used to create new, more specific classes (e.g., "sunny - car" instead of just "sunny"). This effectively expands the dataset and provides finer-grained information.
- Adapting FUNIT: The paper explores ways to integrate object detection insights into the FUNIT framework. This includes proposing a future architecture where an object detector preprocesses images before they are fed into FUNIT, and another where object information is integrated directly into FUNIT's latent space.
- Training and Evaluation: The authors conduct experiments with different configurations, including varying dataset sizes, class distributions (balanced vs. unbalanced), and the use of RetinaNet-generated classes. They train and evaluate models using standard I2I loss functions, including GAN loss, content reconstruction loss, and feature matching loss.
Datasets
The primary dataset used in this paper is the INIT dataset. This dataset consists of street images from Tokyo, Japan, captured under various weather and lighting conditions, leading to four initial classes: sunny, cloudy, night, and rainy. The original images are high-resolution (3000x4000 pixels), but the authors downsampled them to 1280x1920 pixels for computational feasibility. They also reduced the dataset size by removing some images to balance class distributions and manage computational constraints.
A significant contribution is the creation of a new, extended dataset by applying the RetinaNet object detector to the INIT dataset. This process generated a much larger number of classes (initially 178, refined to 97 after manual checks) by creating class-object combinations (e.g., "sunny - car").
Key Insights and Ideas
- Object Detection is Crucial for Complex Scenes: The most significant insight is that incorporating object-level information is vital for improving I2I translation on complex, content-rich images. Simply applying a global style transfer to an entire scene often fails when multiple objects are present.
- More Data, More Classes, Better Performance: The experiments consistently show that increasing the amount of training data and, crucially, the number of distinct classes (achieved through object detection) leads to substantially better translation results. This counters the initial expectation that fewer, more distinct classes might be sufficient.
- Dataset Balance Matters, But Not Exclusively: While balancing class distributions can be beneficial, the overall quantity of data and the diversity of classes (even if unbalanced) play a more dominant role in performance improvement.
- FUNIT's Limitations are Evident on Diverse Data: The paper clearly demonstrates that FUNIT, while effective on simpler datasets, struggles significantly when faced with the complexity and diversity of the INIT dataset and its object-augmented versions.
- RetinaNet-generated Classes Offer Promising Results: Training with classes derived from object detection (e.g., "night - car") yields the most promising qualitative and quantitative results, suggesting this is a viable path for enhancing I2I translation.
Key Results
- Model 1 ("Balanced"): With a balanced dataset and night as the test class, this model produced "terrible results," indicating that class balance alone was insufficient.
- Model 2 ("Unbalanced"): Using the whole, unbalanced dataset with night as the test class, this model showed "much more interesting results," with generated outputs looking closer to real images. This highlights the benefit of more data and the impact of unbalanced class distributions.
- Model 3 ("RetinaNet"): This model, trained with classes generated by RetinaNet and night as the test class, demonstrated the most promising results, both qualitatively and quantitatively. The generated images were significantly closer to actual night scenes.
- Quantitative Metrics: Tables 3 and 4 show LPIPS (lower is better) and Inception Score (IS, higher is better) for different models. Model 3 ("RetinaNet") consistently achieved better IS scores and competitive LPIPS scores compared to Model 2, especially with 2 target style images. For instance, the average IS for Model 3 with 2 target images was 0.801, significantly higher than Model 2's 0.666.
- Qualitative Improvements: Figure 4 shows example translations. The Baseline and Model 1 produce images that are less convincing. Model 2 shows improvement, but Model 3, using RetinaNet classes, generates the most realistic and coherent night scenes.
Limitations and Future Work
Limitations: * Computational Constraints: The authors faced significant limitations due to computational power and GPU memory, forcing them to use small batch sizes and a subset of the data for some experiments. This restricted their ability to fully explore larger batch sizes or longer training durations. * Dataset Quality: The object detector (RetinaNet) produced some noisy classes (e.g., "boomerang," "giraffe") that required manual removal, indicating that automated object detection is not perfect. * Unbalanced Data Bias: While unbalanced data improved results, it also led to biased translations, with classes that had more images translating more realistically. * Target Style Variability: The "night" target style images represented different moments (sunset to sunrise), leading to variations in the translated outputs.
Future Work: * New Model Architectures: The paper proposes two new architectures: 1. An architecture that uses an object detector before FUNIT, feeding both individual object images and the global image into FUNIT, and then merging the results. 2. An architecture inspired by DUNIT, where object information is merged directly within FUNIT's latent space. * Larger Datasets and Longer Training: The authors suggest that using a larger portion of the dataset (with RetinaNet-extracted objects) and training for longer durations with bigger batches would further enhance results. * Exploring More Classes: The success with RetinaNet-generated classes suggests that increasing the number of distinct classes is a key avenue for improvement.
Summary updated