When we ask a generative model to create an image from a rough sketch, reconstruct a 3D floor plan from a few photos, or generate a text sequence under strict grammatical rules, we are forcing it to navigate a fundamental conflict.
On one hand, the model must generate an output that looks realistic—meaning it must conform to a learned data distribution (the "manifold" of real-world images, objects, or text). On one hand, it must conform to the user's constraints. The core challenge is that user constraints are frequently sparse, noisy, or physically impossible. A wobbly, hand-drawn sketch of a face does not intersect with the manifold of real human faces.
If a model prioritizes conformance (adhering strictly to every line of the sketch), it is forced off the realistic data manifold, resulting in flat textures, uncanny distortions, or severe visual artifacts. If it prioritizes realism, it may ignore the user's structural intent entirely, generating a beautiful but irrelevant output.
Strict Conformance (Off-Manifold)
[ Distorted / Uncanny ]
▲
│ ▲ (User Constraint Force)
│ │
─────────────────┼──┴───────────────── Data Manifold (Realism)
│
▼
Strict Realism (Ignores User)
Recent research across image synthesis, structural reconstruction, and neuro-symbolic AI focuses on how to gracefully navigate this trade-off.
The Geometry of Constraint Guidance
To understand why this trade-off occurs, it helps to look at how conditional generative models operate. In models like conditional diffusion or flow matching, the generation process is guided by an external force—often a gradient—that pulls the trajectory toward the user's condition.
However, as Minsi Ren et al. observe, simply turning up the strength of this guidance force to ensure compliance often backfires. Strong guidance introduces estimation biases and cross-condition interference, which rapidly degrades the overall data fidelity. To mitigate this, the authors propose an additional stochastic backward step using a Monte-Carlo sampler, which stabilizes the guidance path and prevents the model from collapsing into unrealistic states.
In flow matching, a similar geometric tension exists. Zhengyan Huan et al. demonstrate that when strong constraint pressure is applied deterministically, sample quality degrades. They introduce a randomized exploration strategy that allows the generation path to stochastically explore the boundaries of the constraint space. By treating constraints probabilistically rather than rigidly, the model can satisfy external conditions while remaining firmly within the underlying data distribution.
This tension extends beyond continuous image spaces into discrete domains. For instance, Virasone Manibod et al. address sequence generation by combining neural sequence models with constraint programming. They show that forcing a language or token model to satisfy complex combinatorial structures requires a careful algorithmic balance to prevent the output from drifting away from the learned dataset distribution.
Forgiving Human Imperfection in Sketch-to-Image
The realism-conformance trade-off is most visible in sketch-based image synthesis. Human drawings are rarely anatomically or geometrically perfect.
A foundational mechanism for resolving this is "hijacking" the generative process via noise, as pioneered by Chenlin Meng et al. in their SDEdit framework. Instead of generating an image from scratch based on a sketch, the authors add a controlled amount of noise to the user's rough guide, effectively dissolving the wobbly, high-frequency details of the drawing. They then run the diffusion model's denoising process. Because the low-frequency layout is preserved under the noise, the model's natural prior can reconstruct realistic high-frequency textures and details, successfully "forgiving" the user's lack of drawing skill.
Rough Sketch ──► Add Noise (Erase Wobbly Lines) ──► Denoise via Prior ──► Realistic Image
Other approaches modify how the model interprets the user's input:
- Semantic Alignment: Ahmed Bourouis et al. argue that models should prioritize semantic meaning over pixel-perfect alignment. By focusing on the "vibe" and intent of a human drawing rather than its exact geometry, their framework avoids the visual artifacts that come from trying to reconstruct a distorted sketch too literally.
- Anatomical Compensation: Yuhang Li et al. utilize a Spatial Attention Pooling module and a dual-generator strategy to explicitly compensate for anatomical errors in freehand sketches, prioritizing structural symmetry and plausible textures over strict line adherence.
- Coarse-to-Fine Refinement: Shuai Yang et al. simulate the multi-stage drawing process of real artists. They introduce a style-based control parameter that lets users explicitly decide how much the AI should "fix" their drawing versus following it strictly.
Explicit Control and Latent Mapping
Rather than relying on fixed heuristics, modern frameworks attempt to turn the realism-conformance trade-off into an explicit user-controlled dial.
Shin-I Cheng et al. frame realism as a controllable distance from the user's input. By adjusting the weights of classifier-free guidance on low-frequency structural constraints, their framework allows users to independently tune how closely an image follows a sketch's shape, a stroke's color, and the overall level of photographic realism.
[ Shape Adherence: ──■── ] [ Color Fidelity: ──■── ] [ Photorealism: ──■── ]
When working with frozen, pre-trained models, maintaining this balance requires lightweight adapter networks. Michael Blumenstein et al. show that over-relying on sparse sketch inputs with pre-trained latent diffusion models leads to flat, unrealistic outputs. They use a lightweight mapping network to bridge sketch features and image latents without retraining the base model, preserving the model's rich texture generation capabilities while respecting structural boundaries.
Similarly, Dina Bashkirova et al. exploit the internal self-attention maps of pre-trained transformers to guide structure without any paired training data, finding that attention maps naturally offer a softer, more flexible constraint that preserves visual plausibility.
Robustness vs. Intentionality
An inherent risk of optimizing for realism is that the model might become too robust. If a generative system is designed to automatically correct every error, it may override a user's intentional, unconventional design choices, smoothing them out into generic, common shapes.
Juyeon Ko et al. highlight this tension in semantic image synthesis. If a user provides a noisy or highly unusual semantic label map, a hyper-robust model will ignore the layout to ensure a realistic image, while a hyper-faithful model will replicate every error. The authors propose stochastically diffusing the input labels during training, which teaches the model to bridge the gap between messy inputs and clean data distributions without erasing the user's underlying intent.
A parallel problem occurs in image restoration and physical reconstruction:
- Image Restoration: Hwayoon Lee et al. note that enforcing strict structural fidelity to a degraded input image often forces a model to sacrifice sharpness. Because the system prioritizes consistency with the coarse input, it limits the generation of high-frequency synthetic details.
- Physical Reconstruction: In macro-level tasks like floor plan generation, Arnaud Gueze et al. show that unconstrained generative models easily produce plausible-looking room layouts that are physically inaccurate based on the actual sparse views available. They resolve this by embedding explicit geometric constraint satisfaction directly within a diffusion framework.
Ultimately, the goal of this research direction is not to eliminate the tension between realism and conformance, but to map its geometry. Whether through stochastic exploration, attention manipulation, or multi-stage refinement, controlling this trade-off is what transforms generative models from unpredictable text-to-image engines into precise, interactive design tools.
Key Foundational Work
DeepFacePencil: Creating Face Images from Freehand Sketches (2020) — contributed 2 concept elements to this direction.
Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches (2020) — contributed 1 concept elements to this direction.
Constrained Generative Adversarial Networks for Interactive Image Generation (2019) — contributed 1 concept elements to this direction.