Text-Guided Image Editing Based on Post Score for Gaining Attention on Social Media

Yuto Watanabe, Ren Togo, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama

TL;DR

This paper introduces a novel text-guided image editing method that aims to create images optimized for social media engagement. By generating multiple edited versions of an image based on varied text prompts and then predicting which version will achieve the highest "post score" (a measure of likes and comments), the system selects the most attention-grabbing output. This approach addresses the challenge of creating visually appealing and engaging content for social media platforms.

What Problem Does This Paper Address?

In the age of social media, images are a primary form of communication and currency. While text-guided image editing tools can modify images based on textual descriptions, they often produce results that may not resonate with a social media audience. The problem is that existing methods focus on accurately reflecting the text prompt but don't consider the crucial aspect of social media engagement. Users often struggle to select the "best" edited image from multiple options, especially if they lack expertise in social media marketing. This paper tackles this by bridging the gap between accurate text-guided editing and the prediction of social media virality.

Methods

The proposed method involves a two-stage process. First, it leverages a pre-trained text-guided image editing model (Instruct-Pix2Pix) to generate multiple edited images. To achieve this, a large language model (GPT-3) is used to create several paraphrased versions of the original text prompt. This diversification of prompts increases the likelihood of generating a range of edited images with varying appeal.

The second and most novel stage is the introduction of a "post score" prediction model. This model takes an edited image and its associated text prompt as input and predicts an engagement rate. The prediction is based on three key features:

  1. Multimodal Feature: A combined representation of the image and text, extracted using contrastive language-image pre-training (CLIP).
  2. Aesthetics Feature: An assessment of the image's visual appeal, derived from the NIMA model.
  3. Category Feature: The predicted category of the post (e.g., fashion, travel), also leveraging CLIP for classification without manual labeling.

These features are concatenated and fed into a fully connected layer to predict a probability distribution over engagement rate classes. The final "post score" is calculated from this distribution. The system then selects the edited image that receives the highest predicted post score.

Datasets

The study utilizes the Instagram influencer dataset [18] for training and evaluating the post score prediction model. This dataset contains over 10 million Instagram posts with associated engagement rates. For the classification task of engagement rates, the dataset was divided into four classes. The text-guided image editing experiments also used images from this dataset.

Key Insights and Ideas

  • Bridging Editing and Engagement: The core insight is that simply editing an image to match text isn't enough for social media success. The system's ability to predict and optimize for engagement is crucial.
  • Leveraging LLMs for Prompt Diversity: Using LLMs to generate multiple text prompts is a clever way to explore a wider latent space of edited images, increasing the chances of finding a highly engaging one.
  • Multifaceted Post Score Prediction: The post score model's reliance on multimodal, aesthetic, and categorical features provides a more holistic understanding of what makes a post engaging, going beyond just the text or image in isolation.
  • Zero-Shot Category Prediction: The use of CLIP to predict post categories without explicit manual labeling is an efficient and powerful approach.

Key Results

  • Improved Engagement Prediction: The proposed post score prediction model achieved an accuracy of 0.409 and an F-measure of 0.409 in predicting engagement rate classes, outperforming models that excluded aesthetic or category features.
  • Subjective Evaluation Success: In a user study, the proposed method's edited images were rated highest across all three perspectives: Editing (4.17 ± 1.08), Response (3.56 ± 1.18), and Aesthetics (3.83 ± 1.13). This indicates that the method not only edits accurately but also produces images that users are more likely to "like" or comment on and find aesthetically pleasing.
  • Qualitative Improvements: Visual examples demonstrate that the proposed method generates edited images that are more visually appealing and better aligned with the goal of gaining social media attention compared to existing methods like CLIPstyler, DiffEdit, and InstructPix2Pix.

Limitations and Future Work

  • Editing Accuracy Dependency: The method's effectiveness relies on the underlying text-guided image editing model producing sufficiently accurate edits. In cases where all generated edits are poor, the system might still select the "best" of the bad options.
  • Post Score Prediction Accuracy: The current accuracy of the post score prediction model is around 40%. Future work aims to improve this by exploring new integration techniques for image and text features.
  • Ensuring Editing Quality: To further enhance results, future work suggests combining post score selection with evaluation metrics for the quality of the text-guided image editing itself.
  • Real-World Deployment: While subjective experiments were conducted, a demonstration experiment in an actual social media environment is planned, along with verifying the robustness of the method by operating social media accounts.

Summary updated

Abstract

Text-guided image editing has been highlighted in the fields of computer vision and natural language processing in recent years. The approach takes an image and text prompt as input and aims to edit the image in accordance with the text prompt while preserving text-unrelated regions. The results of text-guided image editing differ depending on the way the text prompt is represented, even if it has the same meaning. It is up to the user to decide which result best matches the intended use of the edited image. This paper assumes a situation in which edited images are posted to social media and proposes a novel text-guided image editing method to help the edited images gain attention from a greater audience. In the proposed method, we apply the pre-trained text-guided image editing method and obtain multiple edited images from the multiple text prompts generated from a large language model. The proposed method leverages the novel model that predicts post scores representing engagement rates and selects one image that will gain the most attention from the audience on social media among these edited images. Subject experiments on a dataset of real Instagram posts demonstrate that the edited images of the proposed method accurately reflect the content of the text prompts and provide a positive impression to the audience on social media compared to those of previous text-guided image editing methods.
BibTeX
@inproceedings{watanabe2024text,
  author = {Yuto Watanabe and Ren Togo and Keisuke Maeda and Takahiro Ogawa and Miki Haseyama},
  title = {{Text-Guided Image Editing Based on Post Score for Gaining Attention on Social Media}},
  booktitle = {Sensors},
  year = {2024},
  url = {https://openreview.net/forum?id=t4v1BLd0Ig},
}
Figure