Introduction
Professional-grade visual AI tools often require complex workflows involving multiple specialized models for tasks like inpainting, style transfer, or object-specific generation. These "Frankenstein" systems frequently struggle with maintaining instance identity—the ability to keep a specific object recognizable across different scenes—and often lack diversity in their outputs. WeGen addresses these challenges by introducing a unified "design copilot" that merges Multimodal Large Language Models (MLLMs) with diffusion models to enable seamless, conversational image generation and editing.
Methodology: Bridging Reasoning and Generation
WeGen’s architecture integrates an EVA-CLIP visual encoder, a LLaMA-2-7B language backbone, and an SDXL diffusion decoder. This combination allows the model to not only generate images but also to reason about the user's instructions in a conversational context.
Two primary innovations drive WeGen's performance:
- Dynamic Instance Identity Consistency (DIIC): Traditional models often fail to capture how an object's appearance changes naturally with lighting or perspective. WeGen utilizes a video-based data pipeline to learn these dynamics. By tracking objects across video frames using foundation models like InternVL2 and SAM2, the model learns to preserve the "essence" of an object while allowing for realistic environmental variations.
- Prompt Self-Rewriting (PSR): To overcome the limited diversity of standard diffusion models, WeGen employs a self-rewriting mechanism. The MLLM expands brief user prompts into detailed textual descriptions. This process leverages the discrete token space of the LLM to sample diverse interpretations, which are then used to guide the diffusion process.
Data and Training
The model was trained on the DIIC Dataset, a large-scale collection of interleaved text and visual sequences extracted from internet videos. This dataset provides the necessary temporal context for learning robust object identity. For evaluation, the researchers utilized standard benchmarks like COCO2014 for general text-to-image quality and DreamBench for subject-driven generation consistency.
Key Results and Insights
WeGen demonstrates state-of-the-art performance across several metrics:
- Identity Preservation: On DreamBench, WeGen achieved a DINO score of 0.823 and a CLIP-I score of 0.882, significantly outperforming models like Emu2 and Kosmos-G.
- Visual Diversity: The PSR mechanism led to an LPIPS_d score of 0.6286, nearly triple that of its closest competitors, indicating a much broader range of creative outputs for identical prompts.
- Efficiency: Despite its multi-modal capabilities, WeGen maintains a competitive FID score of 9.39 on COCO2014, achieving high quality with lower computational overhead than many pure diffusion-based systems.
Conclusion
WeGen represents a significant step toward intuitive, interactive AI design tools. By treating video as a primary source for learning identity and leveraging the creative reasoning of LLMs, it provides a unified framework for complex visual tasks. Future work will likely focus on reducing inference latency and extending the model's capabilities to consistent video generation.