When people rate the quality of a compressed 3D model, they typically watch it rotate on screen. The SJTU-TMQA study, for example, collected its human scores on 16-second turntable videos. Automatic metrics work differently. They compare the reference and the compressed model either pixel by pixel in those rendered frames or directly on the 3D geometry. Both approaches break down in the same case, when geometry and texture are compressed together. On that subset of SJTU-TMQA, no tested metric reaches a rank correlation with human scores above 0.60 (SJTU-TMQA).
This note argues that the failure has a simple cause and a simple fix. When geometry is quantized, every surface point lands a few pixels away from where it appeared in the reference rendering. A pixel-aligned metric counts that shift as a large error. Human viewers largely ignore such shifts on busy textures and notice them on smooth surfaces and silhouettes. Because we render the frames ourselves, we know exactly which surface point produced each pixel. We can therefore compare reference and distorted frames at the same surface point. This splits each frame's error into two parts: an appearance change (texture blur, color bleeding, shading) and a screen-space displacement. The displacement can then be discounted where texture masks it.
The second ingredient borrows a lesson from image-to-video transfer: image-trained models can handle video without any video training if correspondence across frames is supplied. Here correspondence is free from the renderer. A frozen, image-trained perceptual model can therefore score the whole camera orbit. A small external semantic prior, marking regions like faces where viewers are unforgiving, is tested as a secondary, falsifiable addition.
Core Claim
Our central hypothesis is about where today's metrics compare things. Perceptual-geometric misalignment on textured meshes comes mostly from comparing in screen coordinates. The fix is to compare in surface coordinates, using a decomposition into appearance error plus texture-masked displacement, and then pool over a full camera orbit. On the hybrid-compression subset of SJTU-TMQA this should beat every pixel-aligned metric, including the same deep image features computed without correspondence. It should also do so with at most a handful of fitted parameters, so the result cannot be explained by fitting the small human-score datasets.
The prior work plays three roles here. Some of it motivates the claim. Human studies show that viewers care more about color and texture than about small coordinate shifts (Wu et al.; Su et al.). Sharp textures hide aggressive geometric quantization (SJTU-TMQA). Small depth errors become visible artifacts once shaded (Voynov et al.). Point-distance metrics weight all points equally and so miss what matters to viewers (Patil et al.).
Other work constrains the design. TGE, a point-based metric that never renders, argues against rendered-view metrics because their scores jitter with camera angle and cost about 93 GFLOPs versus TGE's 14.7. Our response is that the jitter comes from comparing unaligned pixels. Surface-anchored pooling over many views should remove it, and we measure this directly. HybridMQA models the masking effect implicitly. It fuses six fixed rendered views with features from a graph network over the mesh, using learned cross-attention. We instead make masking an explicit, nearly parameter-free term and cover the full orbit rather than six fixed views.
Finally, some work supplies the method. MinVIS segments video objects using a model trained only on images, tracking objects by matching their per-frame descriptions across frames. ClipBERT shows that a few sparsely sampled frames through an image model beat dense video models. We adopt both principles but replace learned matching with exact correspondence from the renderer. This sidesteps the identity-swap failures MinVIS reports when frames are degraded.
Research Program
The pipeline renders the reference and distorted meshes along the same orbit. Each frame comes with two auxiliary images: a texture-coordinate image (which texture-map location each pixel shows) and a surface-normal image. For each distorted-frame pixel we find the reference pixel showing the nearest texture coordinate. This works because the compressed meshes in these benchmarks keep an approximately shared texture parameterization. If quantization of texture coordinates breaks correspondence on some content, we fall back to a nearest-surface-point match in 3D.
That correspondence gives a per-pixel displacement field. It also lets us warp the reference frame into alignment with the distorted frame. We compute deep features from a frozen image-trained full-reference metric (LPIPS-style) on the aligned pair, which yields an appearance-error map with the displacement removed. We then score displacement separately:
$$ q_t = \text{mean}_p\big[a_t(p) + \lambda\, w_t(p)\, \|d_t(p)\|\big] $$
The symbols are as follows:
- $q_t$ is the distortion score of frame $t$.
- $a_t(p)$ is the aligned appearance error at pixel $p$.
- $d_t(p)$ is the screen-space displacement of that pixel's surface point.
- $w_t(p)$ is a masking weight. It decreases with local texture contrast in the reference frame and increases near silhouettes and at grazing view angles.
- $\lambda$ is a single scalar fitted on training content.
Frame scores are then mapped back onto the mesh surface and pooled across the orbit. Pooling has two parts. First, each surface point keeps a high percentile of its error over the views in which it is visible. This targets artifacts that appear only from oblique angles and that six fixed views miss. Second, a temporal term measures frame-to-frame change in displacement. This captures surface wobble that exists only in motion.
The external-knowledge arm adds a per-region weight. A frozen open-vocabulary segmenter labels regions such as faces, hands, eyes, and printed text, and errors in those regions are up-weighted. The motivation is SJTU-TMQA's observation that the same distortion on a human face is judged far worse than on a plant. Prior retrieval work supports separating world knowledge from small training sets. InfoSeek found that correct entity identification, not reasoning, was the bottleneck: oracle entity linking raised accuracy from 18.2% to 45.6%. EntRAG and EnFoRe show that grounding at the level of specific entities, rather than whole images, is what helps. We are honest that this evidence comes from question answering, not quality assessment. The semantic weight is therefore an add-on whose value the experiments must establish, not part of the core claim.
Baselines
The naive reference point is YUV_PSNR on rendered frames, the best of 13 metrics benchmarked in SJTU-TMQA, at an overall SRCC of 0.65. VMAF applied to the turntable video is the off-the-shelf video answer. The file reports an SRCC of 0.85 for VMAF on texture downsampling, but no metric exceeds 0.60 on the hybrid geometry-plus-texture subset (GTC). All of these use the paper's own renders, so no tuning is involved.
The strongest model-based competitors are HybridMQA and GeodesicPSIM. GeodesicPSIM compares curvature and color statistics on small surface patches and reports PLCC 0.81 and SRCC 0.81 on TSMD. That number is the bar on TSMD. HybridMQA is the strongest learned fusion method, but the file gives only a relative gain of 6.5–7.7% PLCC over 3D-PSSIM. No absolute HybridMQA number was found on Lacuna for SJTU-TMQA GTC, so we will retrain it with its released configuration on our exact content-disjoint splits.
The most dangerous baseline is per-frame deep features without correspondence. This applies the same frozen LPIPS-style model to unaligned reference and distorted frames over the same orbit, with the same pooling. It differs from our method only in the alignment step. We pair it with a no-reference per-frame model from the HYPERIQA/MANIQA family, which exceeds SRCC 0.93 on single 3D Gaussian-splatting renders in 3DGS-IEval-15K. No result for these models on SJTU-TMQA was found on Lacuna.
Two ablations isolate the key ingredients. The first sets $\lambda$ to zero and removes masking, giving appearance error only. The second keeps alignment but drops orbit pooling, using six fixed views instead. All methods are evaluated on identical renders, splits, and frame counts. Any fitted parameters are tuned under the same cross-validation budget.
Beating the baselines means three things. Overall, the method must exceed the best competitor's SRCC on SJTU-TMQA. On GTC, it must clear 0.60 by a margin outside the bootstrap confidence interval over content. On TSMD, it must match or exceed GeodesicPSIM's 0.81 when $\lambda$ is fitted on SJTU-TMQA only.
Why not just…?
Why not just run a strong deep image metric on every frame and average? This is the real threat. Single-view deep models already correlate above 0.93 with human scores on Gaussian-splatting renders. If our diagnosis is wrong and pixel misalignment is not the main problem, the unaligned per-frame baseline will match us on GTC. The paired comparison above isolates exactly the alignment step, so it settles the question. A controlled test sharpens it further: we apply rigid sub-pixel to few-pixel vertex shifts to textured and smooth regions. Our method should stay nearly flat on textured regions, while the unaligned baseline should climb with shift size.
Why not just give HybridMQA more views or a video branch? A learned network with many views could in principle discover the masking rule itself. However, the human-score datasets contain tens to hundreds of distorted meshes. TSMD, for instance, has 210 distorted meshes over 42 references, so learning a viewpoint- and texture-dependent rule from that data risks overfitting to the training content. We test this directly with a dense-view HybridMQA and compare cross-dataset transfer from SJTU-TMQA to TSMD. A near-parameter-free rule should transfer better if the rule is right.
Why not just ask a vision-language model, or train a video network end to end on the turntables? SRAM shows a frozen 3D encoder feeding an LLM can regress shape realism, reaching SROCC 0.70 on RealismGrading (SRAM). But it handles only untextured shapes and has no notion of a reference. ClipBERT and MinVIS argue that dense spatio-temporal training is costly and unnecessary when an image model has the needed discriminative power. Small human-score datasets make the cost of end-to-end training worse, not better. We include a closed-book vision-language model prompted on sampled frames as an extra no-reference point of comparison. We expect it to fall below full-reference methods on compression artifacts, and will report it if it does not.
Proposed Experiments
The primary experiment evaluates on SJTU-TMQA with splits that keep each reference model's content entirely in train or test. We report SRCC, PLCC, and RMSE overall and for each distortion type, with GTC as the headline. Secondary evaluations cover TSMD and the colored-mesh datasets used by HybridMQA (CMDM and Nehmé et al.). For these we fit nothing on the test set, and parameters come only from SJTU-TMQA. Score stability is measured as the variance of the score under random rotations of the orbit's starting angle, directly answering TGE's viewpoint-jitter objection. Cost is reported in GFLOPs alongside TGE's figures.
Three outcomes are decisive. First, the aligned-minus-unaligned gap on GTC should be positive and larger than on texture-only distortions; if it is not, the central claim fails. Second, the fitted $\lambda$ should be stable across folds, and the learned masking weight should decrease with texture contrast. Third, orbit pooling should beat six fixed views mainly on content with oblique or self-occluded detail.
The semantic arm counts as successful only if region weighting improves face-bearing content without hurting the rest. We will report it as negative if not, which would itself be evidence that region-level priors from outside data are not where the remaining human-metric gap lies.
References
- SJTU-TMQA: A Quality Assessment Database for Static Mesh with Texture Map
- Subjective Quality Database and Objective Study of Compressed Point Clouds With 6DoF Head-Mounted Display
- Perceptual Quality Assessment of Colored 3D Point Clouds
- Perceptual Deep Depth Super-Resolution
- DR-KFS: A Differentiable Visual Similarity Metric for 3D Shape Reconstruction
- Textured Geometry Evaluation: Perceptual 3D Textured Shape Metric via 3D Latent-Geometry Network
- HybridMQA: Exploring Geometry-Texture Interactions for Colored Mesh Quality Assessment
- MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training
- Less Is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
- Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
- EntRAG: Entity-Centric Retrieval-Augmented Generation for Knowledge-based Visual Question Answering
- Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering
- GeodesicPSIM: Predicting the Quality of Static Mesh with Texture Map via Geodesic Patch Similarity
- 3DGS-IEval-15K: A Large-scale Image Quality Evaluation Database for 3D Gaussian-Splatting
- SRAM: Shape-Realism Alignment Metric for No Reference 3D Shape Evaluation