Finetuning LLMs for Automatic Concept to TTI Prompt Generation (Student Abstract)

Jeremy Rutter, Maneesh Reddy Chamakura, Justin Delgado, Gene Louis KimAAAI 2024 · DOI

Introduction

Text-to-image (TTI) generative models, such as Stable Diffusion, have revolutionized digital art creation. However, these models are notoriously sensitive to input nuances; a minor change in phrasing or punctuation can lead to vastly different visual outcomes. This sensitivity has necessitated the rise of "prompt engineering," a specialized skill where users must learn specific keywords and structural patterns to elicit high-quality results.

This research explores the automation of this process by finetuning a Large Language Model (LLM) to act as a bridge between a user’s simple conceptual input and the complex, descriptive prompts that TTI models require. By treating prompt generation as a translation task, the authors aim to lower the barrier to entry for high-quality image generation and enable automated visualization for other NLP systems.

Methodology

The researchers utilized GPT-2 XL as the base model for their prompt generator. The core hypothesis was that an LLM trained on the same dataset as the TTI model would learn to generate prompts that align with the TTI model's "comfort zone"—its specific training distribution.

Finetuning Strategy

To adapt the LLM effectively while maintaining its general linguistic capabilities, the team employed a selective finetuning approach. They froze the first 42 layers of GPT-2 XL and trained only the final 6 layers. This prevented the model from losing its foundational understanding of language while allowing it to specialize in the specific style and vocabulary of image captions found in the LAION-400M dataset.

The Generation Pipeline

The workflow transforms a simple subject into a detailed prompt. For instance, a user input of "London" is processed by the finetuned GPT-2 model to produce a descriptive string like "London London Skyline Canvas Print by Michael Tompsett-Moore." This expanded prompt is then fed into Stable Diffusion 2 to generate the final image.

The automated prompt generation pipeline, illustrating the transformation of a simple subject into a descriptive prompt for Stable Diffusion.

To ensure the model produced complete and coherent captions rather than truncated fragments, the researchers experimented with adding start (<bos>) and end (<eos>) tags to the training data.

Results and Evaluation

The performance of the model was evaluated through a human survey using a Likert scale (1 to 4) focusing on two primary metrics: Quality (Q), representing the aesthetic appeal of the image, and Faithfulness (F), representing how well the image matched the user's original concept.

Quantitative comparison of model variants across Quality (Q) and Faithfulness (F), including standard deviations.

The Quality-Faithfulness Seesaw

The results revealed a fundamental trade-off in generative AI. While the finetuned model (with tags) achieved a measurable boost in aesthetic quality (+0.15 points over the baseline), it suffered a significant drop in faithfulness (-0.30 points). As the LLM becomes more "creative" by adding artistic styles or specific artist names (e.g., "Michael Tompsett-Moore"), it risks drifting away from the user's core intent.

Interestingly, the inclusion of start/end tags proved critical. The "FT w/ Tags" model outperformed the "FT No Tags" version in both quality and faithfulness, suggesting that structural markers help the model produce more coherent and relevant descriptions.

Discussion and Future Directions

The study highlights the concept of distributional alignment: by training the LLM on LAION data, it learns the specific "dialect" of prompts that Stable Diffusion understands best. However, the primary limitation remains the loss of faithfulness. When the LLM hallucinates details to improve aesthetics, it may ignore the user's original subject.

Future research directions include:

  1. Explicit Topic Tagging: Forcing the model to prioritize the user's subject by explicitly tagging the main topic in training data.
  2. Scaling Up: This study utilized only 0.1% of the LAION-400M dataset. Training on a larger portion of the data could significantly improve robustness.
  3. Refining Evaluation: Given the subjective nature of image quality (noted by a moderate inter-annotator agreement of κ = 0.386), more robust human or automated metrics are required to accurately measure success in this domain.

Summary updated

BibTeX
@inproceedings{rutter2024finetuning,
  author = {Jeremy Rutter and Maneesh Reddy Chamakura and Justin Delgado and Gene Louis Kim},
  title = {{Finetuning LLMs for Automatic Concept to TTI Prompt Generation (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30505},
}
Figure