Gravity as a Check on Monocular Depth in Pose-Free Sparse-View Reconstruction

Building a 3D model from a handful of unposed photos usually starts with a single-image depth network, a model that guesses how far each pixel is from one photo. Those guesses are then used to estimate where each camera was, and the camera estimates are used to fit the 3D scene. Errors made at the first step flow down the chain. Single-image depth is wrong by an unknown scale and an unknown offset, so the camera estimates inherit that error and the scene fit cannot fully undo it. We propose to break this chain with a signal nearly every phone already records: the direction of gravity, read from the accelerometer when each photo is taken.

Gravity is usually treated as a pose shortcut. It fixes two of the three rotation angles, which leaves only the heading (the rotation about the vertical axis) to estimate. Our claim is that gravity does a second, overlooked job. It can test and correct the depth guesses before they reach the pose step. Surfaces the image network believes are horizontal, such as floors, tabletops and ground, must come out flat and perpendicular to gravity when they are lifted into 3D. A wrong depth offset or a wrong focal length bends and tilts them in a measurable way. The same gravity reading can therefore calibrate the depth prior, shrink the pose search, and anchor the joint 3D optimization. We would test whether this stops errors from compounding in sparse, unordered captures, where continuous visual-inertial tracking is not available.

Core Claim

In pose-free reconstruction from 2 to 8 unordered views, where each view carries a noisy per-frame gravity reading, three steps together should give more accurate camera poses and renderings than any single component alone. The first step calibrates each frame's depth using gravity. The second estimates relative pose with the heading as the only unknown rotation, using the calibrated depth. The third runs a Gaussian-splatting optimization in which gravity is a soft prior rather than a hard lock. The new ingredient is the first step, together with the insistence that gravity be weighted by how much it can be trusted. We expect the gain to come mainly from correcting the depth offset. If gravity only helps through the smaller rotation search, the proposal reduces to a known trick, and our ablations are built to show which of the two is doing the work.

What Prior Work Establishes

Gravity-constrained geometry is well developed for sparse point correspondences. Globally Optimal Relative Pose Estimation With Gravity Prior reduces relative rotation to a single heading angle. gDLS* adds gravity and scale penalties to a pose-and-scale solver for multi-camera rigs, with manually fixed weights. Gravity-Aligned Rotation Averaging with Circular Regression turns multi-view rotation consistency into a one-dimensional problem. These works supply our pose parameterization. None of them uses dense depth priors or 3D scene fitting, and gDLS* leaves the question of how to weight gravity by its reliability explicitly open, which we take up.

On the depth side, MadPose shows that even strong depth models such as Depth Anything and Marigold carry a scale error and a shift error. It builds minimal solvers that recover relative pose together with these affine parameters, raising ScanNet-1500 pose AUC at 20° from 55.60% to 61.08%. MadPose motivates our focus on the shift and supplies the solver template, but it solves all three rotation angles and never uses gravity. Metric3D v2 predicts metric depth and surface normals zero-shot, but needs the focal length from photo metadata. We reuse its normals to find candidate horizontal surfaces, and we test whether gravity can partly stand in for the missing focal length. Monocular Depth Estimation for Tilted Images via Gravity Rectifier predicts gravity from pixels and rotates tilted images upright before estimating depth. It cuts relative depth error by 29% on its tilted dataset, but fails where vertical and horizontal cues are missing. We use a measured gravity vector instead, and we use it to correct depth, not just to rotate the image.

Pose-free scene fitting is the application setting. NoPe-NeRF jointly optimizes a radiance field, camera poses, and a per-frame depth scale and shift. It reports 0.030 m trajectory error and 31.86 dB on ScanNet from dense video. Look Gauss, No Pose puts pose gradients inside the Gaussian-splatting renderer and reports 31.24 dB on Tanks and Temples versus NoPe-NeRF's 26.34 dB. It optimizes an unconstrained six-degree-of-freedom pose and seeds points from depth without modeling the shift. Both methods rely on dense, overlapping video. DepthSplat fuses monocular depth with multi-view matching for fast sparse-view Gaussians, but requires known poses. Our method is meant to supply those poses.

The error-cascade literature constrains our design. Invariant Teacher and Equivariant Student shows sharp drops in 3D lifting when clean 2D inputs are replaced with real detections. View Invariant 3D Human Pose Estimation notes that a lifting stage that sees only coordinates cannot recover what the earlier stage lost. We treat the depth-to-pose-to-scene pipeline as the same kind of cascade. That argues for correcting depth before it is lifted, and for carrying uncertainty forward rather than passing on hard values. The file's own warning about corrupted gravity, from IMU bias and hand acceleration, adds a further requirement: a hard gravity lock would itself start a new cascade of roll and pitch bias.

Research Program

Stage 1: gravity-anchored depth calibration. For each frame, predicted surface normals within a set angle of the measured gravity direction mark candidate horizontal pixels. Let $\hat D(\mathbf{x})$ be the predicted depth at pixel $\mathbf{x}$, and let $\mathbf{r}(\mathbf{x}; f)$ be the unit viewing ray for focal length $f$. Lifting a pixel with corrected depth gives the 3D point $\mathbf{P} = (\hat D(\mathbf{x})/\alpha - \beta)\,\mathbf{r}(\mathbf{x}; f)$, where $\alpha$ is the unknown scale and $\beta$ the unknown shift. For horizontal points, the height along the unit gravity direction $\mathbf{g}$ should be constant within each surface. We therefore choose the shift, and optionally the focal length, to minimize

$$\sum_{k}\sum_{\mathbf{x}\in S_k} \rho\big(\mathbf{g}^\top \mathbf{P}(\mathbf{x};\beta,f) - h_k\big),$$

where $S_k$ is the $k$-th horizontal surface, $h_k$ is its height, and $\rho$ is a robust loss that limits the influence of outliers. Scale drops out of this flatness test, since scaling every point leaves flat surfaces flat. The test is sensitive to shift and focal length because both bend or tilt planes when points are lifted. Scale is left to the pose stage. If a frame has no horizontal surface, it falls back to uncalibrated depth and is flagged. This keeps the method honest about its main failure mode.

Stage 2: heading-only affine pose solver. With roll and pitch taken from gravity and the shift already estimated, the relative pose between two views has five unknowns: heading, three translation components, and relative scale. We derive a minimal solver in the style of MadPose, using its cross-view 3D-distance constraints. We expect it to need fewer correspondences than MadPose's calibrated 3-point solver. It runs inside the same robust sampling loop. The file's sampling-cost formula implies the number of required iterations falls steeply as the sample size shrinks, which matters under the high outlier rates of sparse views.

Stage 3: soft-gravity joint optimization. The poses and depths initialize a Look-Gauss-style Gaussian-splatting optimization. The key change is that each camera's rotation is split into a gravity-aligned part and a small tilt correction $\delta_i$, penalized by

$$\lambda_i\,\|\delta_i\|^2,\qquad \lambda_i = 1/\sigma_i^2,$$

where $\sigma_i$ is the estimated angular uncertainty of frame $i$'s gravity reading. That uncertainty is computed from how far the accelerometer magnitude departs from 9.81 m/s² and from its short-window variance. This answers the weighting question gDLS* leaves open. Per-frame shift and scale remain free, but start from Stage 1.

Analysis. We will measure how the three input errors propagate to the final result: depth shift error, gravity angle error, and focal length error. The outputs are pose error and rendering quality. We inject each error in a controlled way and report the slope of final error against injected error, both with and without each stage.

Baselines

The naive reference is Look Gauss, No Pose run on the sparse unordered views, starting from identity or depth-bootstrapped poses with no gravity input. It shows what pose-free Gaussian splatting does in this regime without any help.

The strongest existing pose-initialization competitor is MadPose's calibrated affine solver. On ScanNet-1500 it reports pose AUC at 20° of 61.08%, against 55.60% for the PoseLib 5-point baseline (MadPose). We run it with the same depth model, the same matcher, the same iteration budget and the same local refinement. The only difference is that our solver additionally receives gravity. Beating it means exceeding its AUC at 20° on ScanNet-1500 under matched gravity-noise conditions. Because our method sees extra information, we also require that the gain survive realistic gravity noise of 2–3°, not only perfect gravity.

For full reconstruction, the strongest pose-free systems are NoPe-NeRF and Look Gauss, No Pose. The file reports NoPe-NeRF at 0.030 m trajectory error on ScanNet, and Look Gauss at 31.24 dB on Tanks and Temples. Both numbers come from dense video. We found no reported result on Lacuna for either method on 2–8-view unordered subsets, so we will rerun both on our subsets with equal optimization time and per-scene tuning budget. We also include a posed upper bound: DepthSplat fed ground-truth poses. On ScanNet it bounds how much of the gap to perfect poses we close.

A gravity-given-poses baseline, visual-inertial or ARKit poses, is discussed under the alternatives below.

Three ablations isolate the key ingredient. The first uses gravity only for the heading-only rotation parameterization, without Stage 1 calibration. The second uses Stage 1 calibration but a full-rotation solver. The third uses a hard gravity lock instead of the soft uncertainty-weighted prior. The core claim holds only if the first ablation falls clearly short of the full system.

Why not just…?

Why not just use the phone's visual-inertial or ARKit poses? If a continuous camera-and-IMU stream exists, visual-inertial tracking is a serious competitor. The TUM VI Benchmark reports indoor error under 2 cm for systems such as BASALT. That is why we target a different regime: a few photos taken at separate moments, or collected from different people or devices, where each photo carries only a gravity reading and there is no connecting motion stream. We will still report ARKit poses on ARKitScenes sequences as an upper reference. We say plainly that on continuous video the proposal is not expected to win, and that its value depends on how common sparse, gravity-tagged captures are.

Why not just constrain the rotation with gravity and keep everything else as is? This is the most likely threat, since it amounts to a one-line change to Look Gauss or MadPose. The first ablation is exactly this change. If it matches the full system, the depth-calibration claim fails, and we would report the result as a strong, simple finding rather than argue around it. Our reason to expect a gap is MadPose's own evidence that shift errors cause large 3D misalignment independently of rotation. A rotation constraint does nothing for such errors.

Why not just use a metric depth model such as Metric3D v2? Its accuracy (0.047 relative error on NYUv2 and 0.044 on KITTI) depends on a known focal length. MadPose finds residual affine error even in strong foundation models. We run Metric3D v2 as the depth source in every variant, with and without metadata. If Stage 1 adds nothing on top of Metric3D v2 when the focal length is known, the contribution narrows to the missing-metadata case, which we report separately.

Why not just run COLMAP? NoPe-NeRF documents that COLMAP fails on low-texture and rotation-dominant captures, and sparse views make this worse. We report COLMAP's success rate and its accuracy when it does succeed, on the same subsets.

Proposed Experiments

We use ScanNet and ScanNet-1500 for two-view pose. Gravity is derived from the ground-truth poses, then corrupted with a noise sweep from 0° to 5° and with bias. We use ARKitScenes for sparse 3–8-view reconstruction, since its phone captures are gravity-aligned and include laser-scan geometry for depth evaluation. We use TUM VI for real accelerometer readings, including dynamic motion. There we subsample frames to simulate unordered sparse captures and test the uncertainty weighting on real data. To test tilt robustness, we apply synthetic roll and pitch rotations up to the ranges used by the gravity-rectifier paper.

The metrics are pose AUC at 5°, 10° and 20°, median rotation and translation error, trajectory error, depth relative error before and after calibration, and PSNR on held-out views.

The decisive checks are as follows. First, Stage 1 must measurably reduce the depth shift error and the tilt of lifted planes relative to scale-only alignment. Second, our full system must beat the rotation-only ablation, and it must beat MadPose on ScanNet-1500 AUC at 20° at 2–3° gravity noise. Third, the soft prior must degrade gracefully while the hard lock collapses as injected gravity bias grows; TUM VI's dynamic segments give the real-world version of this test. Fourth, the error-propagation slopes should fall when each stage is added.

A negative outcome on the second check would be informative on its own terms. It would say that for sparse reconstruction, gravity is only worth its rotation shortcut.