TL;DR
TexTailor is a training-free, inference-time framework that significantly improves text-image alignment in Multimodal Diffusion Transformers (MMDiTs) like FLUX and Stable Diffusion 3.5. By systematically probing how different transformer blocks handle text-visual interactions, TexTailor strategically amplifies textual guidance in key blocks to boost prompt adherence, enable precise image editing, and accelerate inference.
What Problem Does This Paper Address?
State-of-the-art diffusion models have shifted from traditional UNet architectures to Multimodal Diffusion Transformers (MMDiTs), which concatenate text and visual tokens into a single sequence for joint self-attention. While powerful, these models frequently struggle with complex compositional prompts—such as correctly binding colors to specific objects, counting, or handling spatial relationships.
When these text-visual interactions fail, the generated images do not match the prompt. The authors address a fundamental gap: we lack a comprehensive understanding of how individual MMDiT blocks contribute to semantic understanding versus detail rendering, making it difficult to fix these alignment failures without undergoing expensive model retraining.
Methods
The authors first build an automatic probing pipeline to profile MMDiT blocks using three operations: removing blocks, disabling text conditions (muting them with zero tensors), and enhancing text conditions (scaling hidden states).
Based on the probing results, they introduce TexTailor, a training-free framework that modifies text-visual interactions at inference time. It operates via two main strategies:
- Full-Sentence Tailoring: Scaling the entire text condition vector by a factor ($\lambda$) in specific "sensitive" blocks (typically earlier layers).
- Token-Level Tailoring: Using an attention-map-derived binary mask to amplify only specific target phrases (e.g., "brown horse") within the prompt, ensuring critical attributes receive greater emphasis.
TexTailor is applied to three downstream tasks: improving standard text-to-image generation, enabling precise text-based editing (by injecting source features into a parallel generation stream while tailoring target text embeddings), and accelerating inference by skipping blocks identified as non-critical for core semantics.
Datasets
- Probing Dataset: A newly constructed dataset of 633 diverse prompts covering color, shape, and spatial relationships, sourced from HPDv2 and COCO.
- Editing Benchmark: A new benchmark consisting of 1,000 images and corresponding editing prompts covering object addition, background changes, quantity variations, and more.
- Evaluation Benchmarks: Standard benchmarks including T2I-CompBench / T2I-CompBench++ (8,000 compositional prompts) and GenEval (553 structured prompts for object-focused evaluation).
Key Insights and Ideas
- Early Semantics vs. Late Details: Probing reveals that core semantic information (like layout and object identity) is established in the earlier blocks, while later blocks focus on fine-grained detail rendering. Disabling text conditions in late blocks barely hurts the overall CLIP score.
- Text Guidance is Crucial: Disabling text conditions in early blocks is far more disruptive to image synthesis than completely removing the blocks themselves, showing that MMDiTs heavily rely on continuous conditional guidance.
- Unified Attribute Behavior: Surprisingly, different semantic attributes (color, shape, position) exhibit highly similar block-wise behaviors. This means a unified block-selection strategy can enhance overall prompt adherence without needing attribute-specific tuning.
Key Results
- Better Text Alignment: On SD3.5, TexTailor improves T2I-Combench scores from 56.92% to 63.00% and GenEval scores from 66.42% to 71.63%.
- Attribute-Level Gains: Token-level tailoring achieved substantial improvements on specific challenging attributes in T2I-CompBench, including a 12% boost on Shape, 10% on Texture, and 8% on Color.
- Inference Speedup: By skipping less critical blocks, TexTailor reduces FLUX generation time on an NVIDIA 4090 from 36.79 seconds to 33.34 seconds standalone, and down to 24.53 seconds when combined with TeaCache, with negligible impact on image quality.
Limitations and Future Work
- Dependence on Probing: The method relies on a preliminary, automated block-wise analysis to identify which blocks to tailor for a given model.
- Complex Prompt Bottlenecks: For extremely complex or highly detailed prompts, TexTailor can still fail if the underlying base model's pretraining data distribution lacks representation for those rare object combinations.
- Numerical Weakness: While token-level tailoring improves counting, performance gains remain bounded by the intrinsic limitations of current diffusion models in numerical reasoning.
- Future Directions: The authors suggest exploring trainable modules and token-level dynamic routing to optimize text-visual interactions dynamically.
Summary updated