GAN inversion and shifting: recommending product modifications to sellers for better user preference

Satyadwyoom Kumar, Abhijith Sharma, Apurva NarayanPeerJ Comput. Sci 2025 · OpenReview

TL;DR

This paper introduces a novel framework that leverages GANs to generate modified product images, offering sellers actionable aesthetic feedback to improve user preference. By bridging the gap between user-centric recommendations and seller-oriented insights, the method generates distinct, realistic images that boost user preference scores by 16.7% compared to existing approaches.

What Problem Does This Paper Address?

Traditional recommendation systems excel at suggesting new items to users based on their past behavior. However, they largely overlook a crucial stakeholder: the seller. Sellers lack direct insights into how to aesthetically refine their existing products to appeal to a broader customer base and adapt to evolving user preferences. This paper addresses this gap by proposing a system that provides sellers with concrete visual guidance for product modifications, aiming to enhance item saleability and customer coverage. The challenge lies in generating modifications that are not only aesthetically pleasing and realistic but also cater to the dynamic preferences of multiple users simultaneously, without requiring drastic changes to the original product.

Methods

The core of the proposed method lies in a two-stage process involving GAN inversion and GAN shifting.

  1. GAN Inversion: This stage takes an existing item image and finds its corresponding latent vector within a pre-trained Generative Adversarial Network (GAN). The goal is to optimize this latent vector to produce an image that closely resembles the original item, using a combination of perceptual loss (LPIPS) and mean-squared error (MSE). This ensures the generated image maintains the item's core identity.
  2. GAN Shifting: Once the base latent vector is established, this stage explores the latent space around it. It introduces "shifting vectors" that, when added to the base latent vector, generate new, aesthetically diverse images. These shifted images are optimized to maximize user preference scores, while also minimizing diversity loss (ensuring distinctness among generated images) and shift prediction loss. A ResNet18 feature extractor and linear layers are used to calculate these losses. The method also incorporates a "shift predictor" to help disentangle the latent space and promote distinctiveness among generated images.

The framework is designed to be model-agnostic, meaning it can work with various pre-trained GANs and recommender systems.

Datasets

The primary dataset used for training and evaluation is the Amazon Fashion Dataset. This dataset is described as a comprehensive real-world dataset containing a large number of user-item interactions across 6 different item classes (shirts, jackets, trousers, shoes, sandals) for both genders. The authors also mention using subsets of this dataset (shirts, footwear, trousers) for specific experiments.

Key Insights and Ideas

  • Seller-Centric Feedback: The most significant insight is shifting the focus from purely user recommendations to providing actionable, visual feedback for sellers. This opens up a new avenue for AI in e-commerce, directly impacting product development.
  • Leveraging GANs for Aesthetic Refinement: GANs are shown to be powerful tools not just for generating novel items but for suggesting subtle, aesthetic modifications to existing ones, making them more appealing.
  • Bridging User Preference and Seller Needs: The method effectively connects what users like (preference scores) with what sellers can do (aesthetic modifications), creating a mutually beneficial loop.
  • Generalizability: The framework's ability to work with various GANs and recommender systems, and its performance across different item categories, highlights its potential for broad applicability.
  • Diversity and Realism: The combination of GAN inversion and shifting successfully generates images that are both realistic and aesthetically diverse, addressing a key limitation of previous generative approaches.

Key Results

  • Improved User Preference: The proposed method achieved a mean preference score of 8.97, which is a 16.7% improvement over the personalized variant of the DVBPR baseline (Kang et al., 2017), which scored 7.68.
  • Enhanced Diversity: The method achieved a mean diversity score (opposite SSIM) of 0.63, outperforming the non-personalized variant of Kang et al. (2017) by 12.5% and the personalized variant by 18.8%.
  • Overall Performance: The method outperformed the personalized method of Kang et al. (2017) by 7.4% in terms of overall quality.
  • Item-Specific Generation: For item-specific generation, the method produced unique, realistic images with a mean incremental preference score shift of 7.31 for unpreferred items, indicating potential for improved ranking.
  • Scalability: Experiments with varying numbers of users (n=1 to 10,000) showed a consistent mean incremental preference score shift of 3.2 for user counts greater than 10, without significantly impacting image quality metrics like FID and Inception Score.

Limitations and Future Work

  • Search Radius Limitation: In item-specific generation, a few generated images were noted to be unrelated to the ground truth image, attributed to the lack of constraints beyond the search radius. This suggests that exploring more sophisticated constraints could be beneficial.
  • Future Directions:
    • Latent Diffusion Models: Investigating latent diffusion models for more controllable modifications and insights into how changes are influenced by preferences for other product categories.
  • Refined GAN Inversion: Improving the accuracy of GAN inversion by constraining outputs to more closely resemble the original item and filter out irrelevant object associations.
  • Privacy-Preserving Techniques: Incorporating differential privacy or selective masking of shifted latent codes to safeguard proprietary patterns while preserving insights, addressing ethical concerns and intellectual property issues.

Summary updated

Abstract

In efforts to better accommodate users, numerous researchers have endeavored to model customer behavior, seeking to comprehend how they interact with diverse items within online platforms. This exploration has given rise to recommendation systems, which utilize customer similarity with other customers or customer-item interactions to suggest new items based on the existing item catalog. Since these systems primarily focus on enhancing customer experiences, they overlook providing insights to sellers that could help refine the aesthetics of their items and increase their customer coverage. In this study, we go beyond customer recommendations to propose a novel approach: suggesting aesthetic feedback to sellers in the form of refined item images informed by customer-item interactions learned by a recommender system from multiple consumers. These images could serve as guidance for sellers to adapt existing items to meet the dynamic preferences of multiple users simultaneously. To evaluate the effectiveness of our method, we design experiments showcasing how changing the number of consumers and the class of item image used affect the change in preference score. Through these experiments, we found that our methodology outperforms previous approaches by generating distinct, realistic images with user preference higher by 16.7%, thus bridging the gap between customer-centric recommendations and seller-oriented feedback.
BibTeX
@inproceedings{kumar2025gan,
  author = {Satyadwyoom Kumar and Abhijith Sharma and Apurva Narayan},
  title = {{GAN inversion and shifting: recommending product modifications to sellers for better user preference}},
  booktitle = {PeerJ Comput. Sci},
  year = {2025},
  url = {https://openreview.net/forum?id=2w2p1jK2jk},
}
Figure