Spatial Diagrams with an Explicit Frame of Reference for Multilingual Image Generation

When someone says "the ball is in front of the car," the image they picture depends on a convention their language carries. English speakers usually mirror their own front-back axis onto the car, so "in front" means between the car and the viewer. Speakers of Hausa or Tamil often use rotated or translated conventions instead, and "in front" can then mean the far side. Vision-language models ignore this. The COMFORT benchmark (Do Vision-Language Models Represent Space and How?) shows that GPT-4o and LLaVA apply the English convention across 109 languages. The same paper finds their left/right answers are often no more consistent than chance. Text-to-image generators inherit the same problem, because nothing in the pipeline ever writes down which convention is in force. The text prompt leaves it implicit, the bounding boxes have no slot for it, and the attention layers have no place to put it.

We propose to make the convention a visible, checkable object. A short instruction in any language is compiled into a small spatial diagram. The diagram records each entity's region, its depth order, its facing direction, and a typed frame-of-reference marker: whose viewpoint, and which axis transformation. A handful of sound diagram rules, borrowed in spirit from diagrammatic theorem provers, then check the diagram for contradictions. Only a consistent diagram is rendered, through per-entity attention streams that are kept separate. The same diagram language is used in reverse to audit generated images. The claim is that most of today's perspective and convention errors are interface errors. Fixing the interface should beat both prompting tricks and post-hoc editing, and the experiments are designed to show whether the diagram itself, rather than the renderer, is responsible.

Core Claim

A "frame of reference" is the coordinate convention that gives "left," "front," and "behind" their meaning: the camera's view, the related object's own view, or some third party's. It also fixes whether the axis is mirrored, rotated, or translated. Our claim is that representing this frame explicitly, as one slot in an intermediate spatial diagram, and checking the diagram with a few sound rules before generation, will produce images that match the intended convention more often than either of two alternatives. The first is an LLM emitting bounding boxes directly. The second is a rule-based correction loop applied after generation. We expect the advantage to be largest for non-English conventions and for scenes where entities overlap.

The diagram contributes two things a box list lacks. First, a contradiction becomes a geometric fact rather than a judgment call. Placing an entity in the "left-of-X" region excludes the "right-of-X" region by construction, so opposite-consistency failures cannot reach the generator. Second, the diagram is readable by a checker that does not depend on a large multimodal model's judgment. That matters because such judges carry the same Western defaults we are trying to correct (Culture in Action).

Research Program

Diagram language. The representation draws on three diagrammatic traditions. From Concept Diagrams and the iCon prover (iCon; Reasoning with Concept Diagrams about Antipatterns) we take regions for entities, arrows for directed relations, and shaded regions for places an entity is forbidden to be. iCon proves inconsistency by forcing an individual into a shaded region; we use the same move to reject an unsatisfiable layout before any pixels are produced. From inter-diagrammatic reasoning (Towards Diagram Processing) we take the planar tessellation and its unary and binary operators, which make relational checks direct operations on a grid. From Chandrasekaran's cognitive architecture (Multimodal Cognitive Architecture) we take the split into "action routines" that edit the diagram and "perceptual routines" that query it.

Our addition is a frame-of-reference marker attached to every directional relation. The marker has two fields. The first is a viewpoint: camera, relatum, or addressee. The second is a transformation: reflected, rotated, translated, or absolute (cardinal). The inference rules stay deliberately few. They cover opposite exclusion, transitivity of left/right and front/behind within a single frame, frame conversion between viewpoints, and acyclic depth order. These rules are what make the diagram checkable rather than just another data format.

Compiling language into diagrams. An LLM produces only the spatial primitives, a step FoREST showed helps (FoREST). Those primitives are the located object, the reference object, the relation, distance, topology, and frame. The frame transformation is not left to the LLM's defaults. When the instruction does not state it, the system fills it from a per-language convention prior taken from the typological preferences COMFORT documents. When a speaker states the frame explicitly, that statement overrides the prior.

This design differs from FoR-SALE (FoR-SALE) in two ways. FoR-SALE converts intrinsic and allocentric phrases to camera coordinates through 32 fixed, English-centric rules. It also applies its corrections after an image already exists. Here, the conversion is one parameterized rule applied to a diagram, and checking happens before generation. COMFORT omits absolute frames, so we include cardinal-direction frames only for a small, human-annotated subset and treat them as exploratory.

Rendering. Decoupled cross-attention gives each conditioning source its own key and value projections against a shared query. Separate streams stop one source from swamping another in a single softmax. We reuse the training-free version from AnyMS (AnyMS), which crops latent features to each subject's box, attends separately, and merges. We change one thing. Each entity's stream is masked by its diagram region rather than a rectangle, and where regions overlap, the diagram's depth order decides which stream owns the occluded pixels. AnyMS identifies overlap as its failure mode. Facing direction enters through the entity's text stream, for example "car, facing left," and is verified afterwards. We do not claim the renderer solves 3D orientation. If facing errors dominate, we will graft in FoR-SALE's facing-direction latent edit as a bounded repair step.

Auditing. Each generated image is turned back into a diagram. An open-vocabulary detector supplies regions, a monocular depth estimator supplies depth order, and an orientation estimator supplies facing. The rule checker then reports which specified relations hold under the specified frame. This automated audit is calibrated against native-speaker judgments before it is trusted. The point of this calibration is to address the evaluator-bias problem raised in Culture in Action rather than to inherit it.

Baselines

The naive reference point is the base text-to-image model given the raw instruction in its original language, with no layout and no correction. A second naive point is the same model given an English translation of the instruction. Translation alone is the cheapest fix, and COMFORT and CVLUE (CVLUE) both suggest it fails, but we have to show this on our prompts.

The strongest existing method for frame-dependent generation is FoR-SALE. It reports gains of up to 5.3% over prior text-to-image systems after one correction round and 9.9% after three on FoR-LMD. On intrinsic-frame prompts it reports up to 5% after one round and 15% after three, over SLD and GraPE. We run its pipeline on the same base diffusion model as ours and allow three correction rounds. We also cap our own method at the same total number of denoising passes. Beating it means a higher convention-correct rate on FoR-LMD at equal sampling budget, plus a larger margin on the multilingual split. On that split FoR-SALE's English rules should misfire, and no reported result found on Lacuna for multilingual frame-conditioned generation.

The strongest layout-faithfulness competitor is AnyMS, which reports 49.75 mIoU layout accuracy and 35.82 CLIP-T on its multi-subject benchmark. We feed it the boxes implied by our diagram, so that it differs from our system only in rectangular versus depth-ordered region masks. Beating it means higher mIoU and audit pass rate on the overlapping-subject subset, without losing more than a point of CLIP-T on the non-overlapping subset. For layout pipelines using spatial-primitive prompting, FoREST reports a GLIGEN VISOR score of 83.75%, up from 81.43%. We run that pipeline as an additional box-based baseline.

The key ablation, the one this proposal stands or falls on, is "LLM boxes into our renderer." The same LLM receives the same primitive-extraction prompt and the same per-language convention hint, but it outputs boxes, depth order, and facing directly. There is no diagram, no frame marker, and no rule checking. Its output then goes to our identical renderer. A second ablation keeps the diagram but disables the rule checker. A third replaces the per-language prior with the English default. All variants share the LLM, the prompt-tuning budget (the same number of development-set iterations), the seeds, and the base model.

Why not just…?

Why not just ask a strong LLM, with chain-of-thought, to produce the boxes? This is the most serious threat. If the LLM is told the language's convention, it might apply it correctly without any diagram. The evidence against it is real but indirect. FoREST finds relatum-perspective accuracy 20 to 30% below camera-perspective accuracy, with systematic left/right reversals. COMFORT finds GPT-4o reverting to the egocentric frame even when told the viewpoint. ViewSpatial-Bench (ViewSpatial-Bench) finds GPT-4o at 34.98% on allocentric localization and documents mid-answer perspective switching. None of these papers tested an LLM given an explicit convention hint and asked for coordinates, which is exactly our first ablation. If that ablation comes within noise of the full system, we will report that the diagram is unnecessary. The contribution then shrinks to the convention prior and the audit, and we will say so.

Why not just fine-tune the generator on spatially and culturally diverse images? Kamath et al. (What's "up" with vision-language models?) found that spatial fine-tuning raised in-distribution scores but lowered performance on their controlled What'sUp benchmark, which suggests the models learned shortcuts. Said et al. (Deconstructing Bias) show that optimizing for photorealism suppresses cultural nuance. More fundamentally, the same English sentence can need different images depending on the speaker's convention. A model fine-tuned on images alone has no input that says which convention applies, so the convention has to be an explicit input. We will still include a small fine-tuned control on synthetic frame-labelled scenes. The test is whether it transfers to the held-out multilingual split.

Why not just run FoR-SALE with a multilingual rule table? This is a fair middle path, and we test it by giving FoR-SALE's interpreter the same per-language convention prior. Our hypothesis is that post-hoc editing pays repeatedly for errors that a pre-generation consistency check prevents, and that its 2D latent edits struggle with overlap. If prior-augmented FoR-SALE matches us at equal sampling budget, the diagram's value lies in auditing rather than generation, and the paper's claim will narrow to that.

Why not just judge and iteratively fix images with a multimodal model? Culture in Action reports that MLLM judges reward stereotypical objects and that its descriptor-based FAITH metric achieves 27% higher correlation with human judgments than an MLLM judge does. ViewSpatial-Bench shows frontier models near chance on perspective. We still include a GPT-4o critique-and-reprompt loop as a baseline at equal call budget, scored by humans rather than by itself.

Proposed Experiments

The first experiment isolates compilation. On COMFORT scenes, we measure how often the compiled diagram's frame matches the native-speaker-preferred convention for each language, compared with an LLM answering directly. Opposite consistency holds in the diagram by construction, so the informative number is compilation accuracy. We also measure how often the checker flags a contradictory instruction instead of silently resolving it.

The second experiment is generation. On FoR-LMD we compare against FoR-SALE, SLD-style correction, and GLIGEN with spatial-primitive prompting. We also build FoR-LMD-ML. It translates FoR-LMD prompts into a subset of COMFORT languages with documented non-reflected conventions plus English controls. Native speakers validate each translation and annotate the intended layout. The primary metric is the human-judged convention-correct rate. The audit pass rate is reported only after its agreement with those human labels is established on a held-out set.

The third experiment covers overlap. On AnyMS's multi-subject benchmark, augmented with occluding pairs, we report mIoU, CLIP-T, and audit pass rate against AnyMS.

The fourth experiment is a small, explicitly exploratory test on CULTIVate activities where spatial arrangement is part of the cultural specification, such as seating or greeting position. We report FAITH and native-rater judgments of the layout dimension only. We will not claim gains on attire or artifacts, which the diagram does not touch.

The decisive check is the three-way comparison on FoR-LMD-ML among the full system, the "LLM boxes into our renderer" ablation, and prior-augmented FoR-SALE, all at equal sampling and tuning budgets. The proposal's central claim survives only if the full system beats both on non-English conventions by a margin larger than seed variance, and the no-checker ablation falls between them. A second check guards against a self-flattering evaluator. The audit must rank systems in the same order as native-speaker raters on a held-out set. If it does not, every audit-based number is withdrawn and only the human scores are reported.