TL;DR
This paper introduces a novel hierarchical attention-based Image-to-Image (I2I) translation model designed to correct multiple types of image defects across diverse product categories on e-commerce platforms. By leveraging both high-level defect groups and specific defect types, the model efficiently focuses on problematic regions, and its semi-paired training approach makes it robust to the low-quality paired data common in real-world shopping datasets.
What Problem Does This Paper Address?
Online shopping heavily relies on product images to attract customers. However, these images often suffer from various defects like non-white backgrounds or watermarks, which can negatively impact customer experience and purchasing decisions. While detecting these defects is well-studied, automatically correcting them at scale is a significant challenge. The sheer diversity of products and their unique defects makes training a separate model for each defect impractical. Existing I2I methods struggle to handle multiple defect types simultaneously and often require expensive, manually labeled data (like masks) to pinpoint defects, which is not feasible for large-scale e-commerce scenarios. Furthermore, real-world e-commerce datasets often contain imperfectly paired images, where the "corrected" version isn't truly defect-free, posing a challenge for standard I2I training.
Methods
The core innovation is a hierarchical attention-guided conditional generator. This generator incorporates defect information at two levels:
- High-level defect groups: For example, classifying defects as "background-related" or "angle-related."
- Specific defect types: Within a group, distinguishing between "non-white background" and "watermark."
This hierarchical information is injected into the network via attention modules, guiding the generator to focus its efforts on the relevant image regions. The model is also designed to be semi-paired. This means it can utilize both perfectly paired images (using L1 reconstruction loss) and unpaired images (using cycle-consistent loss). This semi-paired approach is crucial for robustness, as it allows the model to learn effectively even when the available paired data is of low quality, a common issue with synthesized or user-uploaded images on shopping sites. The overall framework uses a standard GAN architecture with two generators (for X to Y and Y to X transformations) and discriminators, augmented with adversarial, reconstruction, cycle, and identity losses.
Datasets
The authors evaluate their model on a combination of public and custom datasets:
- Public Paired Datasets: Maps, Facades, Edges2Shoes, Edges2Handbags.
- Public Unpaired Datasets: Horse2Zebra, Apple2Orange, Monet2Photo, Vangogh2Photo.
- Custom Image Defects Dataset: Collected from a shopping website, containing images with three specific defects: non-white background (non-Wbg), watermark (WM), and incorrect sunglasses angle (in-SA). For non-Wbg and WM, they synthesized paired images by cleaning backgrounds. For in-SA, they used main and secondary images of sunglasses products.
Key Insights and Ideas
- Hierarchical Attention is Key: The paper demonstrates that explicitly guiding the I2I model with both broad defect categories and specific defect types significantly improves its ability to focus on and correct localized defects, especially across diverse product types.
- Semi-Paired Learning for Real-World Robustness: The inclusion of cycle loss for unpaired data makes the model resilient to the imperfectly paired data often encountered in practical e-commerce scenarios, where synthesized pairs might not be perfectly clean.
- Scalability Through Unification: By creating a single model capable of handling multiple defect types and product categories, the authors address the critical scalability issue faced by e-commerce platforms.
Key Results
- On eight public datasets, the proposed model achieved an average 24.6% reduction in Frechet Inception Distance (FID) compared to MoNCE, a state-of-the-art unpaired I2I method.
- On their custom shopping website dataset, correcting three image defects, the model achieved an average 63.2% reduction in FID compared to WS-I2I, a state-of-the-art semi-paired I2I method.
- Ablation studies confirmed that both the hierarchical defect grouping (g) and specific defect type (η) guidance, as well as the use of unpaired data, significantly contributed to performance improvements.
Limitations and Future Work
- Limitations: The paper acknowledges that while their model is robust to low-quality paired data, the synthesized pairs for non-Wbg and WM might still have limitations, especially with semi-transparent objects or complex backgrounds. The effectiveness of the model relies on the quality of the defect group and type labels provided.
- Future Work: The authors suggest that their framework is scalable to multiple image defects in various domains and can be further improved. Potential future directions could involve exploring more sophisticated ways to automatically infer defect types or groups, or extending the model to handle a wider range of image manipulations beyond defect correction.
Summary updated