Procedural Scene Synthesis for Diagnostic Vision Research

33 concepts33 papers1 proposal

In computer vision, real-world datasets are inherently "entangled." If a model fails to identify a pedestrian at dusk, it is often impossible to determine whether the failure stemmed from the low light, the specific camera angle, or the person's unusual pose. Because these variables co-occur in natural images, researchers cannot easily isolate the root cause of a model's error.

To solve this, a growing body of research utilizes procedural scene synthesis—the use of 3D rendering engines to programmatically generate training and evaluation data. By treating a visual scene as a set of controllable parameters, researchers can "dissect" model performance with a level of precision that static, web-scraped datasets cannot provide.

Disentangling Visual Variables

The primary intuition behind controllable synthesis is the ability to hold all scene elements constant while varying exactly one "nuisance variable." This ceteris paribus approach allows researchers to map the boundaries of a model's robustness.

For instance, Kaneva et al. demonstrated that high-fidelity 3D renders can serve as a valid proxy for real-world evaluation, allowing for the systematic isolation of environmental factors like sun angle or complex viewpoint shifts. Similarly, Sun et al. introduced PersonX, a 3D engine that allows researchers to rotate a pedestrian model degree-by-degree. This granularity revealed that side-view queries are more effective for person re-identification than front or back views—a nuance that would be lost in a dataset where viewpoint is randomly distributed.

This control extends beyond simple rotation. He et al. showed that by extracting environmental distributions (like lighting and camera height) from a specific target scene and applying them to a 3D rendering pipeline, one can generate synthetic data that specifically bridges the "domain gap" for that environment.

Breaking Statistical Shortcuts

Deep learning models are notorious for finding "shortcuts"—exploiting accidental correlations in data rather than learning underlying concepts. If every "cow" in a dataset appears on green grass, a model might learn to recognize "grass" instead of "cow."

Procedural generation breaks these correlations by enforcing compositional diversity. Johnson et al. pioneered this with the CLEVR dataset, using a ray-tracer to place simple geometric shapes in configurations that are statistically balanced. This ensures that a model cannot "cheat" by relying on linguistic biases (e.g., "most cubes are red").

Building on this, Eulig et al. developed DiagViB-6, which independently controls shape, hue, and texture. This allows researchers to measure compositional generalization: the ability to recognize a familiar attribute (like a specific texture) when it appears on a brand-new shape. By programmatically decoupling these factors, the authors can quantify exactly when a model begins to rely on "lazy" features like color over robust features like shape.

Isolating Logic from Perception

In complex tasks like Visual Question Answering (VQA) or Vision-Language alignment, failures are often ambiguous. Does a model fail to count objects because it can't see them, or because it doesn't understand the concept of "three"?

Controllable synthesis provides "perfect ground truth" that eliminates perceptual ambiguity. Alonso 0001 et al. utilized 3D environments to show that state-of-the-art models struggle with "cross-modal entity linking"—mapping specific objects to structured text—even when the visual attributes are perfectly clear. Because the researchers controlled the spatial coordinates and geometry, they could definitively attribute failures to alignment errors rather than recognition errors.

Furthermore, Cascante-Bonilla et al. argued that the controllability of a simulation is often more important than its visual realism. They found that being able to precisely place objects to create diverse reasoning challenges (like complex counting scenarios) taught models logic that transferred effectively to real-world images, even if the synthetic training images looked "artificial."

Measuring Internal Representations

Finally, this research direction allows us to look "under the hood" of how models represent the world. Issa et al. used controlled synthetic datasets to test for factorization—the degree to which a model stores different scene parameters (like lighting vs. identity) in independent neural subspaces. By systematically varying one parameter at a time, the authors could measure how "brain-like" a model’s internal organization truly is.

Similarly, Abbasi et al. used the ComCO dataset to manipulate object size and quantity, revealing that models like CLIP have structural biases (e.g., favoring the first object mentioned in a caption) that are learned from the skewed distributions of natural language.

By moving from static image collections to parametric 3D engines, the field is shifting from asking what a model sees to why it sees it. As Newell et al. suggest, this parametric approach is essential for understanding how specific visual complexities impact the learning and transferability of features in the modern AI era.

Key Foundational Work

Keyframe Selection for Camera Motion and Structure Estimation from Multiple Views (2004) — contributed 1 concept elements to this direction.
Simulation of video sequences for an accurate evaluation of tracking algorithms on complex scenes (2008) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_controllable_synthetic_data_generation_2e5b0ba2,
  title = {controllable synthetic data generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/controllable-synthetic-data-generation/rdr_2e5b0ba2437b44b7a2165bf09996a266},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_controllable_synthetic_data_generation_2e5b0ba2_v1,
  title = {controllable synthetic data generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/controllable-synthetic-data-generation/rdr_2e5b0ba2437b44b7a2165bf09996a266/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake