Rotate Outside the Model: Frame-Normalized Spatial Questions for Noisy, Domain-Specific Settings

Vision-language models (systems that answer questions about images) are fairly good at saying whether one object is left of another from the camera's point of view. They are much worse when the same question is asked from someone else's point of view, such as "to the car's left," "on the patient's right," or "on the ship's port side." Current fixes ask the model to reason through the change of viewpoint in words. They add structured prompts, chains of reasoning steps, or consistency checks. This proposal tests a different idea. The model should never perform the viewpoint rotation itself. It should answer two short questions it is good at: which way the reference object faces, and where the target sits in the camera's view. A few lines of deterministic geometry then combine the two answers. Domain conventions, such as radiology images being displayed mirrored so the patient's left appears on the viewer's right, become entries in a lookup table that this geometry applies. They are not text the model must absorb and act on.

The motivation is a conflict in the literature. Resolving which coordinate system a sentence uses (its "frame of reference") seems to require extra prompt scaffolding. Extra context, however, is known to dilute attention and induce confident errors. Reasoning chains also make things worse when the frame is wrong from the start. If the rotation moves outside the model, the prompt stays short, the reasoning chain never builds on a misread viewpoint, and ambiguity can be detected before any answer is committed. These three properties together are what we would test.

Core Claim

Our hypothesis is that most failures in non-camera spatial questions come from the model having to carry out the rotation in language, not from poor perception. The rest come from frame ambiguity that the model resolves silently and wrongly. We call the method frame-normalized querying. A natural-language spatial question is rewritten into two minimal queries. The first is an orientation query: "which way is the relatum facing: toward camera, away, left, right?" The second is a camera-frame relation query. A fixed transform maps the camera-frame answer into the requested frame, applying any domain convention from a table.

When the language does not pin down the frame, the same procedure is run under every candidate frame (camera, addressee, object-centric). If the answers disagree, the system reports the ambiguity instead of answering. We predict three things. First, the relatum-perspective penalty documented by FoREST will largely close. Second, accuracy will hold as distractor text and clutter are added, because the prompts stay short. Third, domain conventions supplied to the external transform will help, while the same conventions pasted into the prompt will not. That third prediction mirrors TheoremQA's finding that knowing a definition is not the same as using it.

Grounding in Prior Work

Several papers motivate the gap. FoREST builds Unity scenes with ambiguous and frame-specified splits. It finds that Llama3-70B is 20–30% less accurate on relatum-perspective questions than on camera-perspective ones, and that text models reverse left and right when adopting an object's view. It also finds an "extraction bias": models pull the right entities from the prompt but echo relations instead of transforming them. COMFORT rotates fronted objects through 360° in 10° steps. It shows that GPT-4o and LLaVA revert to the camera frame even when told to take the object's view, collapse at the 0°/180° boundaries, and fail left/not-right consistency at near chance. Both results suggest the model can perceive but cannot rotate, which is exactly the step we remove.

Other papers constrain the design. When More Is Less finds that injecting one good spatial cue helps by up to 9.6% on the VSR benchmark, while five or more cues fall below the no-context baseline. It also finds that reasoning chains help on unambiguous scenes but cost up to 2.0% on VSR, where frames are ambiguous. SpatiaLab reports that in cluttered real photos, models act as "salience detectors," and that reasoning chains elaborate on false perceptual premises. Distraction-aware answer selection shows in text QA that irrelevant documents produce confidently wrong answers, and that an explicit escape option ("unanswerable") recovers much of the loss. Together these rule out long scaffolds and argue for an explicit abstention path, which our ambiguity check supplies.

Some work supplies the method or sits close to it. REP and top-view perspective transformation also rotate the scene geometrically. However, they rebuild the visual scene through depth estimation and bird's-eye projection, for referring-expression grounding. We rotate only the answer and use the model itself to estimate orientation. That makes the method a training-free wrapper around any model, and it handles linguistic frame ambiguity and domain conventions, which those systems do not target.

Constraint-aware prompting is the closest hallucination-side work. It enforces converse and transitive relations inside a reasoning chain and raises average accuracy on ARO, GQA and MMRel from 66.37% to 80.05%. But it stays in the camera frame, uses yes/no questions, and costs 2–3x the tokens. We address frame transformation directly, and our consistency check runs across frames rather than across converse relations.

Finally, TheoremQA frames the domain angle. Prepending theorem definitions gave no significant gain there, because models fail to put rules to use. Spatial domain conventions are an unusually clean case where putting a rule to use is a fixed function. They therefore let us test whether the domain-knowledge gap shrinks when application is taken out of the model.

Research Program

The system has three parts. A parser maps the question to a locatum (the target object), a relatum (the reference object), a relation term, and either an explicit frame or "unspecified." The parser can be the same model with a short extraction prompt, which FoREST suggests models do well. The orientation query asks the vision-language model for the relatum's facing direction as one of a small set of bins. The relation query asks for the camera-frame relation. A deterministic transform composes the two and applies the convention table. The table covers radiological display mirroring, port and starboard as vessel-intrinsic left and right, stage left as performer-intrinsic, and anatomical anterior and posterior as patient-intrinsic front and back.

For unspecified frames, the transform is applied under each candidate frame. Disagreement among the candidates triggers an "ambiguous" output. A variant ranks candidates by language-specific defaults, which matters given COMFORT's cross-lingual findings.

We will analyze where errors originate by giving the transform oracle orientations from scene metadata. This separates perception error from composition error. It also tells us whether the 0°/180° and lateral-facing failures in COMFORT and FoREST are orientation-estimation failures. If they are, the method inherits them, and the honest conclusion becomes that the bottleneck is facing-direction perception.

To study domain conventions, we build a small diagnostic split by rewording FoREST and COMFORT-CAR scenes into domain lexicons. The geometry is unchanged; only the frame and convention words vary. For example, the car becomes a vessel ("port of the boat") or a patient figure ("the patient's left"). The split includes cases where the display convention flips the image. To study noise, we add two kinds of distractors. Text distractors are irrelevant narrative sentences and unrelated spatial statements about other objects. Visual distractors are additional salient objects placed in FoREST scenes. We will also use SpatiaLab's naturally cluttered orientation and relative-positioning items.

Baselines

The naive reference point is the base vision-language model asked the original question directly, with no scaffolding. We use the same models throughout: an open model in the LLaVA-NeXT family and Qwen2-VL, plus GPT-4o where budget allows. Every result is reported as a delta from this baseline.

The strongest prompting competitor is Spatial-Guided prompting from FoREST. It decomposes the query into locatum, relatum, topology, distance and frame before reasoning, and is reported to improve Llama3-70B C-split accuracy by 10–25% over standard reasoning chains. That paper gives no absolute accuracies we can quote. It is the right competitor because it attacks the same frame problem with in-prompt structure. The comparison isolates whether rotating outside the model beats asking the model to reason about the rotation carefully.

The strongest hallucination-side competitor is constraint-aware prompting, at a reported 80.05% average on ARO/GQA/MMRel and 92.70% on MMRel. We will extend it to frame-specified questions by adding the converse check in the requested frame.

The strongest trained competitor is fine-tuning on RoboSpatial. Its data covers ego-, world- and object-centric frames, and it lifted LLaVA-NeXT from 30.3% to 60.5% on its validation set. We use the released fine-tuned model if available, or otherwise fine-tune the same base model on the RoboSpatial data with a matched compute budget.

For fairness, all prompting methods share the same base models and the same decoding settings. All get the same small development set for prompt tuning, about 200 FoREST items that are excluded from the test split. All are scored on identical items. We also report tokens per question, so the cost of each method is visible.

Two ablations isolate the key ingredient. The first, "in-prompt rotation," asks the model for orientation and camera relation in the same two queries but then has it compose them in text. This tests whether the external transform is what matters. The second, "convention in prompt," pastes the convention table into the prompt instead of applying it externally.

Beating a baseline means four things. On FoREST C-split relatum-perspective items, we must exceed Spatial-Guided prompting, and the gap between relatum and camera accuracy must shrink by at least half. Accuracy loss under added distractors must be smaller than for every scaffolded baseline. On ambiguous items, the confident-error rate must be lower than every baseline at matched coverage.

Why not just…?

Why not just state the frame explicitly in the prompt? FoREST's C-split already does this, and the 20–30% relatum-perspective penalty is measured in exactly that setting. COMFORT likewise reports models reverting to the camera frame under explicit instructions like "from the car's viewpoint." Stating the frame is necessary but evidently not sufficient. Our C-split comparison measures how much of the remaining gap external rotation removes.

Why not just fine-tune on multi-frame 3D data like RoboSpatial? This is the real threat. RoboSpatial's doubling of accuracy shows that training can teach frame transformation. Two pieces of evidence give us reason to doubt it is the end of the story. SpatiaLab reports that spatially fine-tuned models fall to 10–13% on open-ended in-the-wild questions. And fine-tuning cannot cover domain conventions absent from household scans. We will run the RoboSpatial-tuned model on our domain-lexicon split, on SpatiaLab open-ended items, and on COMFORT boundary angles. If it matches frame-normalized querying everywhere, the conclusion is that training suffices and our contribution narrows to the cheap, training-free case and the abstention mechanism. The two approaches can also be combined, by using the tuned model as the orientation estimator.

Why not just paste the domain convention definitions into the prompt? TheoremQA found that definition augmentation gave no significant gain, and FoREST's extraction bias suggests models read such text but do not act on it. Rather than assume the same here, the "convention in prompt" ablation tests it directly. If in-prompt conventions do as well as external application, our account of the domain gap is wrong, and we will report that.

Why not just add consistency checks or reasoning chains to catch errors? Constraint-aware prompting works in the camera frame but triples token use. When More Is Less and SpatiaLab show that reasoning chains amplify errors when the frame is misread. Our distractor sweep measures whether these scaffolds degrade faster than two short queries as noise grows.

Proposed Experiments and Decisive Checks

The first experiment is the main comparison on FoREST A- and C-splits and COMFORT-CAR. We report accuracy by frame class, the relatum-versus-camera gap, accuracy by angle (with the 0°/180° and lateral bins separated out), and left/not-right consistency. There is no reported numeric COMFORT accuracy in our sources, so we establish our own base-model numbers. Because the transform is deterministic, consistency should rise to near the consistency of the orientation estimates. If it does not, the parser or orientation step is the culprit, and the oracle-orientation run will show which.

The second experiment is the noise sweep. We add zero to eight distractor sentences and zero to four salient distractor objects, and plot the slope of accuracy loss for each method. The prediction, drawn from the one-cue optimum in When More Is Less, is a flatter slope for frame-normalized querying. If our slope matches Spatial-Guided prompting, the noise argument fails.

The third experiment is ambiguity handling on FoREST's A-split. We score coverage against the error rate on answered items, and compare our cross-frame disagreement check to the "return unanswerable" escape instruction borrowed from distraction-aware answer selection.

The fourth experiment is the domain-lexicon split, with conventions applied externally versus pasted into the prompt. This is the decisive test of whether the spatial side of the domain-knowledge gap is a failure to put conventions to use.

The fifth experiment checks transfer to real images. We use SpatiaLab orientation and relative-positioning items in open-ended format, where GPT-5-mini's reported best open-ended score is 40.93% against 64.93% for humans. We also use RoboSpatial-Home object-centric configuration questions. These show whether orientation estimation survives clutter.

The project fails cleanly if the oracle-orientation upper bound is high but model-estimated orientation is so poor that end-to-end accuracy does not beat Spatial-Guided prompting. That outcome would itself be informative. It would place the frame-of-reference bottleneck in facing-direction perception rather than in reasoning, and redirect effort accordingly.