When we interact with generative models, we rarely have just one type of instruction. A fashion designer might have a rough sketch of a dress silhouette, a specific fabric texture in mind, and a target human pose. An industrial engineer might have a textual requirement for a mechanical part alongside a precise geometric boundary constraint.
Traditional conditional generative models are typically specialists, optimized to handle a single modality of guidance—such as text-to-image or sketch-to-image. When users attempt to chain these single-modality models together or force multiple inputs into a single model, the system encounters a fundamental architectural friction: heterogeneous control signals often fight each other, leading to information overwriting, mode collapse, or a loss of user control.
To solve this, a major research direction has emerged focused on unifying multi-conditional inputs within a single coherent generative framework. The goal is to gracefully blend high-level semantic intent (like text prompts or style references) with low-level structural constraints (like sketches, depth maps, or poses) across text, images, video, and 3D geometry.
The Core Friction: Why Simple Conditioning Fails
To understand why multi-conditional frameworks are necessary, consider what happens inside a standard diffusion model or Generative Adversarial Network (GAN) when you inject two different constraints, such as a text prompt ("a blue silk dress") and a structural sketch.
If you simply concatenate these inputs or inject them sequentially through standard cross-attention layers, the dense spatial information of the sketch often clashes with the global semantic embeddings of the text. As noted by Alberto Baldrati et al. in their work on the Multimodal Garment Designer, standard models fail to maintain the distinct identities and spatial boundaries of these inputs simultaneously. The model either ignores the sketch to satisfy the text, or produces a literal interpretation of the sketch while discarding the requested texture and style.
Furthermore, real-world workflows require flexibility; a user might provide only a text prompt, or a sketch and an image reference, or all three. Forcing a model trained on a fixed set of modalities to accept missing or partial inputs typically breaks its internal representations. Shin-I Cheng et al. highlight this in their development of DiSS, demonstrating that independent control over shape (sketches), color (strokes), and realism requires explicit architectural decoupling so that one attribute can be adjusted without degrading the others.
Researchers have converged on three primary architectural paradigms to resolve this friction: Probabilistic Fusion, Disentangled Attention Routing, and Unified In-Context Tokenization.
Paradigm 1: Probabilistic Fusion via Product of Experts
One elegant way to handle an arbitrary combination of missing or present inputs is to treat each conditioning modality as an independent "expert" that guides the generation process.
Instead of training a massive joint encoder to model the combined probability distribution $P(\text{Image} \mid \text{Text}, \text{Sketch}, \text{Style})$, the Product-of-Experts (PoE) formulation multiplies the individual conditional distributions together:
$$P(\text{Image} \mid \text{Text}, \text{Sketch}, \text{Style}) \propto P(\text{Image} \mid \text{Text}) \times P(\text{Image} \mid \text{Sketch}) \times P(\text{Image} \mid \text{Style})$$
In PoE-GAN, Xun Huang et al. implemented this by assigning a separate encoder to each modality. Each encoder predicts the parameters (mean and variance) of a latent distribution. By mathematically combining these predictions using a product-of-Gaussians formula, the model finds the mathematical intersection of all provided constraints.
[Text Input] --> [Text Expert] ---\
[Sketch Input] --> [Sketch Expert] ----> [PoE Fusion Layer] --> [Unified Latent] --> [Generator]
[Style Input] --> [Style Expert] ---/
Why does this work so well? If a user leaves out a modality (e.g., provides no sketch), that expert simply outputs a non-informative, uniform distribution. The fusion layer handles this missing data gracefully without requiring retraining or specialized sub-networks.
A similar philosophy of non-destructive guidance blending can be seen in the training-free framework Semantic Diffusion Guidance. Here, the authors leverage a self-supervised, noised version of CLIP to compute separate text and image guidance gradients, combining them dynamically during the diffusion denoising steps to steer the generation toward both semantic and visual targets simultaneously.
Paradigm 2: Disentangled Attention Routing
When generating complex layouts—such as graphic designs containing products, backgrounds, and text slogans—different constraints must govern different spatial regions. If you pass all constraints into a global attention mechanism, the text attributes will "bleed" into the background, and the product features will distort.
To prevent this cross-contamination, Hui Zhang et al. introduced a multimodal attention mask mechanism in CreatiDesign. Instead of allowing every pixel token to attend to every conditioning token, the attention matrix is explicitly masked based on spatial layouts. Visual features of a product are routed strictly to the product's bounding box, while textual slogans are isolated to their designated regions, allowing a single Diffusion Transformer to harmonize disparate elements without losing their individual identities.
Standard Attention: [Pixel Tokens] <---> All Conditioning Tokens (Bleeding)
Masked Attention: [Pixel Region A] <---> [Spatial Constraint A] (Isolated)
[Pixel Region B] <---> [Semantic Constraint B] (Isolated)
This concept of routing information through specialized pathways also applies to separating content from style. In ControlStyle, Jingwen Chen upgrades pre-trained diffusion models by introducing a trainable modulation network. This architecture isolates the text prompt (which dictates what to draw) from the reference image (which dictates how to draw it), preventing the geometric structure of the reference image from overriding the text's semantic instructions.
Paradigm 3: Unified In-Context Tokenization
With the rise of Large Multimodal Models (LMMs), another powerful approach has emerged: treating every control signal—whether it is a word, a pixel patch, or a spatial coordinate—as a discrete token within a single autoregressive sequence.
By projecting heterogeneous inputs into a shared embedding space, the model can learn cross-modal dependencies natively through standard self-attention blocks. For example, Zhu Zhang et al. developed UFC-BERT to unify text prompts, visual style references, and inpainting masks into a single tokenized stream, eliminating the need to chain multiple disparate models together.
This unified sequence approach allows for "in-context" generation. As demonstrated by Xichen Pan et al. in Kosmos-G, users can interleave multiple images and text descriptions naturally (e.g., "A photo of [Image_Ref_1] next to [Image_Ref_2] on a beach"). The model treats the visual tokens as context clues, generating a new image that respects the identities and spatial relationships of all references without requiring tedious per-subject fine-tuning.
This paradigm scales beyond 2D images. In C3LLM, Zixuan Wang et al. map video frames, text prompts, and discrete acoustic vocabularies into a single sequence, enabling precise multi-conditional audio synthesis. Similarly, Junhao Chen et al. developed Idea23D, which coordinates collaborative LMM agents to ingest interleaved text, images, and existing 3D assets to generate unified 3D models.
Breaking the Linguistic Bottleneck in Spatial Domains
The necessity of blending heterogeneous constraints becomes starkest in highly technical domains like CAD engineering, video composition, and LiDAR simulation. In these fields, language alone acts as an expressive bottleneck; complex spatial forms and temporal trajectories are incredibly difficult to verbalize accurately.
[Pure Text Prompt] --------> "Add a curved bracket here" ------> High Ambiguity / Hallucination
[Text + Sketch Blend] ----> "Add this [Sketch] here" ---------> High Precision Geometry
In computer-aided design (CAD), relying solely on text instructions often causes models to hallucinate invalid geometry. To circumvent this, Qiangya Guo et al. proposed Plan then Act (PTA), a bi-level framework that uses an LLM to generate a high-level operational plan before a specialized actioner executes precise command sequences. However, as noted by Zhanwei Zhang et al. in GeoCAD and Yu-Zhe Shi et al. in their work on domain-specific interfaces, true precision requires bridging the gap between abstract descriptions and precise spatial forms by allowing hand-drawn sketches to complement text commands directly.
We see an identical pattern in 3D scene generation and video synthesis:
- LiDAR Generation: In T2LDM, Xiaoshui Huang et al. show that synthesizing realistic 3D LiDAR environments requires a "self-conditioned" guidance mechanism that reconciles high-level linguistic descriptions of a street scene with precise, low-level semantic maps and directional encodings to prevent spatial distortion.
- Video Synthesis: In VideoComposer, Xiang Wang et al. demonstrate that preventing temporal flickering and unpredictable movement requires a compositional framework. By simultaneously injecting textual intent, structural depth maps, and explicit motion vectors into the latent space, the model prevents high-level semantic instructions from overriding the low-level physical constraints of the scene.
By moving away from isolated, single-modality inputs and embracing architectures designed to balance, mask, and fuse heterogeneous constraints, this research direction is transforming generative models from unpredictable text-to-image engines into highly controllable, multi-modal design tools.
Key Foundational Work
Multimodal Conditional Image Synthesis with Product-of-Experts GANs (2022) — contributed 1 concept elements to this direction.
UFC-BERT: Unifying Multi-Modal Controls for Conditional Image Synthesis (2021) — contributed 1 concept elements to this direction.