Modern feed-forward 3D reconstructors are networks that turn a handful of photos into a full 3D scene in a single pass, with no per-scene optimization. They now produce geometry that is internally consistent: angles, ratios and shapes are right. The overall size of the scene, however, is often wrong or simply undefined. This is unavoidable for a camera alone, because a toy car up close and a real car far away project to the same pixels. Current systems fix the size in one of two ways. Some regress it with an opaque learned head, such as a predicted median log-depth or a global "scale token." Others borrow it from a per-image metric depth network that needs the camera's focal length. Neither approach reports when it is wrong. Neither uses the most direct evidence a person would use: a reconstructed door is about two meters tall, a chair seat sits about 45 cm off the floor, and a car is about 4.5 m long.
We propose to treat metric scale as a single unknown number that the reconstruction lets us measure, not guess. Once a multi-view reconstruction is consistent up to one global factor, every recognized object gives an independent noisy reading of that factor. The reading is the object's known physical size, taken from a language-conditioned size prior, divided by its extent in the reconstruction. Combining many such readings with other cues as an explicit posterior over the scale gives a metric 3D Gaussian scene with calibrated uncertainty. We expect it to stay accurate where learned scale heads fail: unseen domains, cropped or zoomed images, and unknown focal lengths.
Core Claim
The research file shows a consistent ordering in prior work. Semantics are lifted into 3D after geometry, and play no part in fixing its scale. SegSplat attaches clustered CLIP indices to Gaussians from a frozen DepthSplat backbone, using camera-pose coordinates. EmbodiedSplat streams compressed CLIP features into Gaussians but assumes known poses and good input depth. Meanwhile, AMB3R adds a metric head on a frozen VGGT front-end, and MapAnything predicts a single learned scale token from early-fused inputs. Both take semantic information only implicitly, through the image encoder.
On the monocular side, ScaleDepth regresses one image-level scale from CLIP scene-text similarity. Single View Physical Distance Estimation using Human Pose uses a fixed human-height prior inside a RANSAC solve, and the file notes that a fixed prior breaks for children versus adults. Our claim is that the right place to fuse semantics with geometry, for the purpose of scale, is the 3D object instance. Two observations support this.
First, a single global scale is exactly the correct unknown for a multi-view reconstruction that is consistent up to a similarity transform. This removes the "global versus local scale" objection that the file raises against ScaleDepth and MapAnything in monocular settings.
Second, per-object readings can be combined as independent evidence, which yields both accuracy and an honest error bar. Concretely, let $s$ be the unknown factor converting reconstruction units to meters. Let $e_i$ be the measured extent of instance $i$ along a chosen dimension. Let $p_\theta(\log h \mid z_i)$ be a learned distribution over that object's true metric size $h$, conditioned on its fused image-and-text embedding $z_i$. Let $q_c(\log s)$ be optional likelihoods from other cues $c$, such as per-view monocular metric depth or sparse depth points. The scale posterior is
$$ \log p(\log s \mid \cdot) = \sum_i \log\Big[(1-\epsilon)\, p_\theta(\log s + \log e_i \mid z_i) + \epsilon\, u\Big] + \sum_c \log q_c(\log s) + \text{const}, $$
where $\epsilon$ is a learned outlier rate covering misdetections and broken instances, and $u$ is a flat density. Conditioning on the instance's own crop embedding, rather than only its category name, is what lets the prior tell a child from an adult or a toy from a car. Once at least two cue types are present, the posterior width becomes a usable confidence signal.
Research Program
The pipeline is built from parts the file already supplies.
A frozen relative reconstructor produces pointmaps, cameras and Gaussians. The front-end is VGGT-style, as in AMB3R, with Gaussian heads in the manner of DepthSplat. A SegSplat-style lift-and-index step groups Gaussians into open-vocabulary instances across views. For each instance we measure robust extents: percentile-based oriented-box dimensions and height above an estimated support plane.
The new learned module is the size prior $p_\theta$. It is a small network that maps a CLIP crop embedding, fused through cross-attention with the instance's coarse 3D shape tokens (in the spirit of SGFormer's geometry-queries-semantics attention), to a mixture over log-dimensions. It is trained on metric object annotations from ScanNet, ScanNet++ and nuScenes boxes. The shape tokens are needed because the prior must know which dimension $e_i$ refers to, and whether the instance is truncated by the field of view.
The posterior is a one-dimensional search over a log-scale grid. It is differentiable and cheap, and its mode rescales the Gaussians. During training, the posterior also replaces the free scale in pointmap regularization. Revisiting Depth Representations for Feed-Forward 3D Gaussian Splatting aligns Gaussians to foundation pointmaps with a full similarity fit, which discards metric scale. Pinning that fit's scale to our posterior keeps the regularizer from washing units out.
We will analyze three things: how scale error falls as the number of informative instances grows, which object categories carry most of the information, and whether the posterior width predicts actual error.
Baselines
The naive reference is a do-nothing scale: output the relative reconstruction multiplied by the training-set median scale for its normalization convention. Every method must clearly beat this, or the task is trivially easy on that dataset.
The strongest existing competitor is a learned metric head on the same front-end, as in AMB3R. It is the most direct answer to "feed-forward metric reconstruction," and it shares our backbone, so any difference isolates how scale is obtained. AMB3R reports TUM RGB-D ATE RMSE dropping from 7.1 cm to 3.2 cm, and 98.2% RRA@5 on ETH3D. The file gives no direct scale-error number for it, so we will retrain its metric head on our exact training mix and evaluate it ourselves. Beating it means lower median absolute log-scale error in-domain with no loss in ATE on TUM, and a clearly lower error and tail rate under the cross-domain and zoom shifts described below. MapAnything is run from its released form as an RGB-only global scale token; no reported result found on Lacuna for its scale error on these benchmarks.
The second strong competitor is per-view monocular metric depth followed by a scale-only median fit. We use Metric3D v2, which reports KITTI AbsRel 0.044 and NYUv2 AbsRel 0.047 with ViT-Large. It is fed the reconstructor's estimated focal lengths, not EXIF, so that it receives no information our method lacks. ScaleDepth reports AbsRel 0.048 on KITTI and 0.074 on NYU. Its holistic CLIP-scene scale head is re-implemented on our backbone as the "semantics, but not at the object level" control.
All learned heads receive the same training data, the same frozen front-end, and the same hyperparameter search budget.
Three ablations isolate the key ingredient. The first removes the object term, leaving only the monocular-depth likelihood. The second replaces the crop-embedding prior with a category-name-only prior, which tests the child/adult argument. The third feeds the same per-object readings into a learned regression head instead of the explicit product, which tests whether the posterior structure matters or only the cues do.
Why not just…?
Why not just trust a learned scale head like AMB3R's or MapAnything's? This is the main threat. Their image encoders already see objects, so they may already exploit object sizes implicitly. The difference we expect is not in-domain accuracy but behavior under shift. A head that regresses scale from appearance can lock onto scene-type or field-of-view cues that change when the camera changes. Metric3D v2 identifies focal length as the dominant driver of monocular scale ambiguity. Our zoom-crop test is designed to expose this. If the learned head matches us on the cross-domain and zoom tests, the object-measurement idea adds nothing, and we will report it that way.
Why not just align the reconstruction to Metric3D v2 depth? It is simple and in-domain it may be nearly as good. However, it inherits Metric3D v2's need for correct intrinsics, which the file lists as its main open limitation. It also gives one reading per pixel that is highly correlated within an image, so it offers no independent cross-check. Our method includes it as one likelihood term. The test that settles the question is whether adding the object term reduces error and tail failures when the focal estimate is perturbed. If it does not, the simple alignment wins.
Why not just add sparse depth points, as LiDARTouch or MapAnything do? When such points exist, they should dominate. LiDARTouch reports 2.504 m RMSE on KITTI with only four beams and eliminates catastrophic distance errors. In our formulation they are simply another sharp likelihood term. We include this setting to confirm that the posterior defers to them. The proposal targets the common case of RGB-only phones, drones and archival video, where no such sensor exists.
Why not just use a holistic CLIP scene scale, as ScaleDepth does? Its scene-category text anchors capture "kitchens are about this big," which is a weak prior with wide spread. It cannot exploit the fact that a multi-view reconstruction makes object extents measurable. The holistic-CLIP baseline on the same backbone directly tests whether object-level evidence beats scene-level evidence.
Proposed Experiments
We train on ScanNet, ScanNet++ and nuScenes training splits. We evaluate in-domain on ScanNet++ and nuScenes, and on TUM RGB-D and ETH3D, which are not used for training. We test cross-domain by training on indoor data only and evaluating on KITTI, and the reverse.
The primary metric is absolute log-scale error between the predicted and true scale. The relative geometry is first aligned with rotation and translation only, so that scale error is not hidden. We report its median and the rate of errors above 10% and 25%. Secondary metrics are AbsRel of rendered depth with no per-image rescaling, TUM ATE in metric units, and novel-view PSNR on nuScenes. The PSNR check confirms that rescaling does not hurt rendering; VGGD reports 24.85 PSNR in its single-frame surround-view protocol, which serves as a reference point for that setting. Calibration is measured as the empirical coverage of the posterior's 68% and 95% intervals.
The decisive checks are three.
The first is the zoom-crop test. Center-cropping and resizing the inputs changes the apparent focal length while leaving true metric geometry unchanged. A method that truly measures should be invariant to this; methods that infer scale from appearance should drift.
The second is the instance-count curve. If per-object errors were independent, scale error should fall roughly as one over the square root of the number of informative instances, because averaging independent log-readings shrinks variance that way. The error should then flatten at a floor set by shared prior bias. Showing both the decline and the floor tells us where the method's value ends.
The third is the calibration test. The posterior width should rank scenes by actual error, which no current feed-forward metric reconstructor offers.
The idea fails if, across these tests, the object term does not improve the strongest baseline or its combination with Metric3D v2 alignment.
References
- SegSplat: Feed-forward Gaussian Splatting and Open-Set Semantic Segmentation
- EmbodiedSplat: Online Feed-Forward Semantic 3DGS for Open-Vocabulary 3D Scene Understanding
- AMB3R: Accurate Feed-forward Metric-scale 3D Reconstruction with Backend
- MapAnything: Universal Feed-Forward Metric 3D Reconstruction
- ScaleDepth: Decomposing Metric Depth Estimation into Semantic-aware Scale Prediction and Adaptive Relative Depth Estimation
- Single View Physical Distance Estimation using Human Pose
- SGFormer: Semantic-Geometry Fusion Transformer for Multi-modal 3D Panoptic Segmentation
- Revisiting Depth Representations for Feed-Forward 3D Gaussian Splatting
- Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation
- LiDARTouch: Monocular metric depth estimation with a few-beam LiDAR