Vision-language models, which answer questions about images and video, are reasonably good at naming what is in a scene and poor at saying where things are relative to a viewer. The failure is worst when the two objects in a question were never on screen together. On egocentric video, meaning footage from a head-worn or vehicle-mounted camera, accuracy falls from 60% to 30% as the time between sightings grows, and the best model scores 65.60 against a human 93.96 (Out of Sight, Not Out of Context?). That same study contains a puzzle. Giving the model a top-down map, a bird's-eye view (BEV), helped by only about 2.6%. Giving it full ground-truth 3D coordinates helped by about 18%. We think the map failed for a specific reason: it was drawn north-up in world coordinates, while the questions are asked from the camera's position at some moment in the video. The model still had to do the mental rotation, which is the step it cannot do.
We propose to remove that step. From the video itself we estimate the camera's motion across the floor using planar motion solvers, a classical family of geometric methods that assume the platform moves on a flat surface and so need only a heading angle and a 2D translation. We place each detected object on the floor plane. Then, for every question, we redraw a simple symbolic floor map centered on the camera's pose at the time the question refers to, with "forward" pointing up. The model then answers by reading the picture, with no rotation required of it. The core claim is that viewpoint alignment of the map is the missing ingredient behind the 2.6% versus 18% gap. A second claim is that the planar restriction is what makes this cheap and robust: the answer depends mostly on heading accuracy, and heading is what planar solvers estimate well.
Core Claim and Mechanism
Planar motion constraints shrink camera motion from six degrees of freedom to three: yaw $\psi$ and floor translation $t \in \mathbb{R}^2$. Closed-form solvers exist for this reduced problem. Two point matches give a single quadratic (Zhu et al.). One affine correspondence plus histogram voting runs in 16 ms, against 588–749 ms for 3- and 5-point RANSAC on Malaga (Barath and Hajder). A globally optimal least-squares refinement runs in 0.4 ms (Hajder and Barath). Chaining these frame-to-frame estimates gives a pose $(\psi_k, t_k)$ for every frame $k$. The same floor plane fixes object placement without a depth network. An object's ground-contact pixel, mapped through the floor homography of the frame where it was seen, gives a floor position, and the frame's pose then moves that position into a common floor coordinate system $p$.
For a question anchored at frame $q$, each object is re-expressed in that frame's egocentric coordinates:
$$p^{(q)} = R(-\psi_q)\,(p - t_q),$$
where $R(\cdot)$ is a 2D rotation. The rendered map shows the camera as a fixed upward-pointing marker and the objects as labeled dots, following the abstraction that let Keep it SymPL lift allocentric accuracy from about 50% to 97.33% on "closer" queries. Like SymPL, we can also shade decision regions such as left and right half-planes.
This construction lets us predict how pose error propagates to answers. A left/right or front/behind answer flips only when the heading error $\delta\psi$ exceeds the object's angular distance $\gamma$ from the relevant decision line:
$$\Pr[\text{flip}] = \Pr[|\delta\psi| > \gamma].$$
Accuracy should therefore be nearly flat under heading noise until the noise approaches the typical margin $\gamma$, and then fall off. These directional relations, and "which is closer" comparisons, are unchanged by a global scale factor. The scale ambiguity of a single camera therefore matters only for questions about absolute distance, and for those we recover scale from the known camera height above the floor plane.
Relation to Prior Work
Two static systems motivate the representation and supply its rendering style. Keep it SymPL projects one image onto a top-down plane using GroundingDINO, Depth Pro and OrientAnything. TEP draws a human-centric top-view grid from GLIP and MiDaS, reaching 66.73% on InterRef at about 13 s per image. Both work on a single frame, so neither can relate an object seen a minute ago to where the camera stands now. We keep their symbolic map and replace per-frame depth and orientation networks with accumulated planar odometry.
Direction-aware 3D LMMs is the closest in spirit. It rotates whole point clouds into a recovered camera frame and gains 30% mIoU on ScanRefer. It differs from our proposal in three ways. It works in full 3D (SE(3)), it takes camera poses from offline sensor logs, and its input is a point-cloud encoder rather than a rendered map. Its finding that pose information placed in the text prompt did not cure directional confusion, while transforming the coordinates did, is direct support for our core claim.
Out of Sight, Not Out of Context? motivates the project. It supplies the benchmark and the 2.6%/18% gap, and it used simulator-generated, world-aligned maps. On the geometry side, Zhu et al. constrains our design. Their planar estimate was precise enough for trajectories but not for dense reconstruction, because real roads are not flat. Hajder and Barath show that the constraint tolerates 1–3° of tilt. We only need trajectory-level accuracy in heading, which is the regime where these solvers are strong. Linear Memory SE(2) Invariant Attention builds rotation and translation invariance into trajectory forecasting. We do not alter the model's architecture. We put the floor-plane geometry into the input, which works with closed models as well as open ones.
Research Program
The pipeline has four parts: gravity alignment, planar odometry, object placement, and rendering.
Head-worn cameras pitch and roll, so we first rotate each frame to an upright, gravity-aligned view. We use the device's inertial sensor when it is available, and otherwise a floor normal estimated from early frames. What remains is approximately planar motion of the head. Frame-to-frame motion comes from the 1AC solver with voting, followed by least-squares refinement on the inliers, and the results are chained into a trajectory. Objects are detected with an open-vocabulary detector and placed on the floor through their contact pixels. When an object has no visible floor contact, such as a wall shelf, we fall back to monocular depth and record which source was used.
At question time we choose the anchor frame and render the aligned map. The map goes to the vision-language model together with the original frames. The same pipeline runs on driving video for STSBench, where nuScenes provides gravity-aligned forward cameras and the vehicle's motion is close to the circular arcs that Ackermann steering produces (Lee et al., Häne et al.). STSBench also lets us test layouts that include other moving agents.
Baselines
The naive reference point is the vision-language model on raw video. On DISJOINT-3DQA the strongest reported version of this is GPT-4o with visual markers at 65.60 normalized LLM-Match (Out of Sight, Not Out of Context?). We rerun it with the same frame budget and prompt template we use for our method. The same paper provides two reference points that we reproduce under its protocol: sparse world-aligned BEV cues, reported at about +2.6%, and full 3D oracle metadata, reported at about +18%. Beating the method means exceeding the sparse-BEV condition by a clear margin and recovering most of the oracle gain, using only estimated poses.
The strongest structured competitor is our layout built from a full six-degree-of-freedom visual SLAM trajectory projected onto the floor. It uses the same detector, the same object placement and the same renderer, and only the pose source changes. A second fair competitor is a SymPL-style single-frame layout, placed using monocular depth and an object-orientation network. This isolates whether accumulating odometry over time matters.
Two ablations isolate the key ingredient. The first renders the identical map, from ground-truth poses, either north-up or aligned to the anchor frame. This difference is the test of the core claim. The second replaces estimated poses with ground truth, which bounds the cost of odometry error.
On STSBench, the strongest reported numbers are text-only GPT-4o given ground-truth 3D trajectories at 57.08% and InternVL 2.5 8B on video at 46.07% (STSBench). We run InternVL 2.5 8B with and without our camera-only layout. We also run GPT-4o with the same ground-truth trajectories either as text or rendered as an aligned map, so that representation is separated from information. Beating InternVL on video is the minimum bar. Approaching 57.08% without ground-truth trajectories is the meaningful bar.
Any fine-tuned baseline gets the same training data, compute and tuning budget as any fine-tuned version of our model.
Why not just…?
Why not just give the model the camera trajectory or coordinates as text? The evidence is mixed, and we treat this as a real competitor. Textual trajectory cues helped little on DISJOINT-3DQA (Out of Sight, Not Out of Context?). Prompting with pose did not fix direction errors in Direction-aware 3D LMMs. TEP beat GPT-4V given orientation metadata by over 18 points (TEP). On the other hand, text-only GPT-4o with ground-truth trajectories is the best reported STSBench system. The text-versus-rendered-map comparison on identical ground-truth trajectories settles which presentation carries the gain. If text matches the aligned map, our contribution narrows to supplying the trajectory from camera alone.
Why not just use full 6-DoF SLAM? This may well match planar odometry in answer accuracy, and we run it head-to-head rather than assume otherwise. Our expectation that planar odometry holds up rests on three things: its speed (16 ms versus hundreds of milliseconds), its stability under forward motion, and its higher rate of translation estimates within 20° on Malaga (90% versus 75%) (Hajder and Barath). Heading is the quantity answers depend on. If SLAM and planar odometry tie, we report the representation as the result and planar odometry as the cheaper way to get it, with no accuracy claim.
Why not just fine-tune on perspective data? A 3B model trained for perspective alignment (MVSM) reaches 82.09% on ViewSpatial-Bench (ViewSpatial-Bench). That benchmark is single-frame, however, and there is no reported result found on Lacuna for MVSM on DISJOINT-3DQA. On m2sv, supervised fine-tuning followed by reinforcement learning lifted Qwen3-VL-8B only from 34.3% to 43.9%, and the gain did not transfer to other benchmarks (m2sv). We fine-tune the same open model on DISJOINT-3DQA training questions, with and without layouts, under an equal budget. This shows whether training alone closes the gap that persists across time.
Decisive Checks
The first check is the north-up versus aligned ablation with ground-truth poses on DISJOINT-3DQA. If the aligned map does not clearly beat the north-up map, the central claim is wrong and the rest of the pipeline is moot.
The second check injects controlled yaw noise into ground-truth poses and plots accuracy against noise. The flip formula predicts the shape of this curve from the measured distribution of decision margins $\gamma$. Matching that prediction would confirm that heading is the quantity that matters.
The third check measures the gap from ground-truth to estimated poses, broken down by temporal separation between sightings, against the reported 60%→30% decline. Success means accuracy stays flat as the time gap grows. We also report odometry heading error on Malaga as a sanity check, and the share of objects placed by floor contact versus depth fallback, along with per-source accuracy.
References
- Out of Sight, Not Out of Context? Egocentric Spatial Reasoning in VLMs Across Disjoint Frames
- Monocular Visual Odometry and Dense 3D Reconstruction for On-Road Vehicles
- Relative planar motion for vehicle-mounted cameras from a single affine correspondence
- Least-squares Optimal Relative Planar Motion for Vehicle-mounted Cameras
- Keep it SymPL: Symbolic Projective Layout for Allocentric Spatial Reasoning in Vision-Language Models
- Walk in Others' Shoes with a Single Glance: Human-Centric Visual Grounding with Top-View Perspective Transformation
- Direction-aware 3D Large Multimodal Models
- Linear Memory SE(2) Invariant Attention
- Motion Estimation for Self-Driving Cars with a Generalized Camera
- 3D visual perception for self-driving cars using a multi-camera system: Calibration, mapping, localization, and obstacle detection
- STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving
- ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models
- m2sv: A Scalable Benchmark for Map-to-Street-View Spatial Reasoning