Abstract
"Creative" remains an inherently abstract concept for both humans and diffusion models. While text-to-image (T2I) diffusion models can easily generate out-of-distribution concepts like "a blue banana", they struggle with generating combinatorial objects such as "a creative mixture that resembles a lettuce and a mantis", due to difficulties in understanding the semantic depth of "creative". Current methods rely heavily on synthesizing reference prompts or images to achieve a creative effect, typically requiring retraining for each unique creative output—a process that is computationally intensive and limits practical applications. To address this, we introduce CreTok, which brings meta-creativity to diffusion models by redefining "creative" as a new token, <CreTok>, thus enhancing models' semantic understanding for combinatorial creativity. CreTok achieves such redefinition by iteratively sampling diverse text pairs from our proposed CangJie dataset to form adaptive prompts and restrictive prompts, and then optimizing the similarity between their respective text embeddings. Extensive experiments demonstrate that <CreTok> enables the universal and direct generation of combinatorial creativity across diverse concepts without additional training, achieving state-of-the-art performance with improved text-image alignment and higher human preference ratings.
Introduction
Current text-to-image (T2I) models like Stable Diffusion 3 or DALL-E 3 are excellent at generating "out-of-distribution" but concrete concepts—think of a "blue banana." However, they stumble when faced with abstract instructions like "a creative mixture of a lettuce and a mantis." Instead of a seamless fusion, these models often just render a lettuce and a mantis sitting next to each other.
The core issue is a lack of semantic depth. To the model, "creative" is just another word, not a functional instruction on how to combine concepts. Existing solutions to this problem are computationally expensive, often requiring the model to be retrained for every new pair of objects or relying on a massive bank of reference images to "hallucinate" a blend. CreTok addresses this by teaching the model the logic of hybridization through a specialized token.
Methodology: Redefining Creativity
The authors propose a framework called CreTok (Creative Token) that treats "creative" as a universal adjective rather than a static label. The method focuses on redefining the semantic space of the text encoder rather than fine-tuning the image generation weights.
Token Redefinition and Image-Free Training
CreTok introduces a new specialized token, <CreTok>, into the model's vocabulary. The training process is uniquely image-free, optimizing the token entirely within the text embedding space. The framework utilizes two types of prompts:
- Restrictive Prompts: A literal combination (e.g., "A Lettuce Mantis").
- Adaptive Prompts: Using the new token in a natural template (e.g., "A photo of a
<CreTok>mixture").
The model is trained to minimize the cosine similarity loss ($L_{mix}$) between the embeddings of these two prompts. By doing this iteratively across many different pairs from the CangJie dataset (200 diverse text pairs), the <CreTok> token learns the generalized concept of "fusion."
Loss Thresholding and Positional Bias
To prevent the model from overfitting to just one of the two concepts (e.g., making a bug that looks 100% like a mantis and 0% like lettuce), the researchers implement a loss threshold ($\theta$). This ensures the model finds a "sweet spot" where both parent concepts are recognizable in the hybrid. Additionally, CreTok mitigates positional bias (the tendency of models to favor the first word in a prompt) by alternating the order of concepts during the training phase.
Key Results and Evaluation
CreTok achieves state-of-the-art (SOTA) performance in both efficiency and quality. It generates images in roughly 4 seconds, compared to 40 seconds for BASS (10x speedup) and 120 seconds for ConceptLab (30x speedup).
As shown in the comparative results, standard models like SD 3.5 and Midjourney often struggle with the "mixture" instruction, frequently rendering the two objects separately. CreTok, however, successfully synthesizes integrated hybrids that preserve the features of both source concepts. In a user study of 50 participants, CreTok was ranked as the most "creative" and "integrated" compared to SD3.5, Kandinsky 3, and BASS. Furthermore, because <CreTok> acts like a standard adjective, it maintains stylistic flexibility, allowing users to combine it with various artistic styles (e.g., "a pop art painting of a <CreTok> hybrid") without further tuning.
Conclusion
CreTok represents a significant step toward "meta-creativity" in diffusion models. By shifting the focus from instance-specific training to semantic redefinition, it enables zero-shot, high-quality combinatorial generation. While the quality remains tethered to the base model's knowledge, the ability to teach a model the logic of hybridization opens new doors for more intuitive and powerful creative AI tools. Future work will explore expanding this logic to multi-concept fusion and dynamic thresholding for complex scenes.
Summary updated