When people rate the quality of a 3D shape, they often disagree. A chair with a missing leg may look fine to most raters and broken to a few. A mesh whose sharp texture hides heavily quantized geometry fools some viewers and not others. Current learned quality metrics, such as a 3D-native network that scores textured point clouds or a language-model realism scorer, predict only the average rating, the mean opinion score. That average hides exactly the cases where coordinate metrics and human perception come apart. We propose to predict the whole distribution of human ratings for each asset. The model outputs any requested percentile of "what a random rater would say," with a built-in guarantee that lower percentiles never exceed higher ones. On top of that we add a calibrated lower bound suited to automated accept/reject decisions for generated or compressed 3D content.
The technical core is a known failure of quantile regression, the family of models that predict a chosen percentile $\tau$ of an outcome instead of its mean. When several percentiles are predicted separately, they "cross": the model claims the 10th-percentile rating is higher than the 50th, which is impossible. Crossing is worst where data are sparse, and per-asset rating distributions are sparse by construction, with roughly 20 to 70 raters per stimulus. We carry the crossing-free architecture from distributional reinforcement learning over to perceptual 3D quality assessment. We also extend it from a fixed grid of percentiles to arbitrary, continuously queried percentiles, which that architecture explicitly cannot handle.
Core Claim
A quality model trained with the pinball loss directly on individual rater scores, rather than on their average, can predict the per-asset opinion distribution. The pinball loss is the asymmetric absolute error whose minimizer is the $\tau$-th percentile. The model uses a head that is monotone in $\tau$ for every $\tau$. We claim it will match mean-regression metrics on rank correlation with the mean opinion score. We also claim it will beat Gaussian, histogram, and sort-after-the-fact alternatives on held-out-rater likelihood-free scores and on the efficiency of calibrated lower bounds.
We further claim that the predicted spread is informative about perceptual-geometric misalignment. Assets where geometric metrics such as Chamfer distance rank quality very differently from humans should be exactly the assets with high predicted rater disagreement and a heavy lower tail. If that holds, disagreement is the observable symptom of misalignment, and a lower-percentile score is the right gatekeeping signal.
Relation to Prior Work
Two lines of work motivate the proposal by documenting the gap.
The first shows that coordinate metrics misjudge 3D quality. Pix3D builds an image–shape benchmark and measures how well standard metrics track human judgments: 0.32 for IoU, 0.49 for Chamfer, and 0.43 for Earth Mover's distance. DR-KFS shows Chamfer barely penalizes a missing chair leg and proposes a rendered keypoint-descriptor loss. Perceptual Deep Depth Super-Resolution shows that a 46 mm RMSE depth map can look worse than a 99 mm one under lighting. The SJTU-TMQA study (Cui et al.) shows texture masking geometric degradation, with no metric above SRCC 0.6 on combined geometry–texture compression. All of these treat the human side as a single number per stimulus. We keep the human side as a distribution.
The second line supplies the backbones we reuse. TGE scores colored point sets with a PointNet++ backbone and geometry-to-color cross-attention. IW-SSIMp weights structural similarity over 12 projected views. SRAM regresses no-reference realism through an LLM. Each outputs a scalar. We change only the output head and the training target, so any gain is attributable to the distributional formulation rather than to a better feature extractor.
The method comes from the crossing literature. NC-QR-DQN guarantees ordered quantiles through a softmax, a cumulative sum, and a positive scale plus shift, and reports that monotonicity acts as a useful regularizer. It is, however, restricted to a fixed quantile grid and to scalar returns. Deep Lattice Networks for CDFs enforce monotonicity functionally in forecasting. Conformalized Quantile Regression wraps quantile estimators to obtain finite-sample coverage and notes that crossed base quantiles force heuristic patching.
The file cites two further sources, Tang et al. and Decke et al., as locating crossing in data-sparse regions. That constrains our design: an unconstrained multi-head model is expected to fail precisely in the sparse extreme-quality ranges that matter for gatekeeping.
To our knowledge from this file, no work predicts per-rater opinion distributions for 3D quality, and none applies crossing-free quantile heads outside scalar reinforcement-learning or forecasting targets.
Research Program
The head. For an asset $x$, the head predicts $$q(x,\tau) = \beta(x) + \alpha(x)\,\sum_{k=1}^{K} w_k(x)\,\sigma\!\left(\frac{\tau - \mu_k(x)}{s_k(x)}\right),$$ where the symbols are as follows:
- $\beta(x)$ is the predicted lowest rating.
- $\alpha(x) \ge 0$ is the predicted range.
- $w_k(x)$ are softmax weights.
- $\sigma$ is the logistic function.
- $\mu_k(x)$ and $s_k(x) > 0$ are the location and width of $K$ monotone components over $\tau$.
The sum, normalized to span $[0,1]$ on $\tau \in [0,1]$, is increasing in $\tau$ for every input. The head therefore cannot cross at any percentile, including percentiles sampled at random during training in the style of implicit quantile networks. It reduces to the NC-QR-DQN construction when the logistic components are replaced by steps on a fixed grid. We bound $\beta$ and $\alpha$ so that the output stays inside the rating scale.
The training target. We train on individual ratings. Each (asset, rater score) pair is one sample of $Y \mid X$, and the loss is the pinball loss averaged over randomly drawn $\tau$. This avoids estimating noisy empirical percentiles from 20–70 scores. It also handles ties on discrete scales such as SJTU-TMQA's 11 levels, because the pinball minimizer on discrete data is still a valid percentile.
Calibration. For deployment we apply split conformal calibration, following CQR, to the lower percentile $q(x,\tau_0)$. We hold out assets and calibrate so that a new rater's score falls below the adjusted bound at most at the nominal rate. Because $\tau_0$ can be chosen after training, one model serves any gatekeeping threshold.
The misalignment analysis. On each dataset we compute the rank disagreement between Chamfer (or PSNR) and the mean opinion score per stimulus. We then test whether predicted spread, $q(x,0.9)-q(x,0.1)$, and lower-tail heaviness predict that disagreement better than the empirical spread does on held-out raters. We also break results down by distortion type, with SJTU-TMQA's combined geometry–texture compression as the key slice.
Baselines
Naive reference. The do-nothing reference assigns every asset the pooled marginal rating distribution of its training set. A second naive variant shifts that distribution to the asset's predicted mean. A method that cannot beat the shifted marginal on held-out-rater pinball loss has learned nothing about per-asset disagreement.
Mean-regression backbone. This baseline is TGE, or IW-SSIMp features with a learned regressor, trained on the mean opinion score. It anchors mean accuracy. The reported numbers we must not fall below are TGE's PLCC of 0.7887 on the Colored Shape Fidelity dataset (TGE), IW-SSIMp's PLCC of 0.8504 and SRCC of 0.8481 on WPC (Su et al.), and the best reported SRCC of 0.65 (YUV_PSNR) on SJTU-TMQA (Cui et al.). We retrain the backbone ourselves on our splits so that the comparison uses identical data. Beating it means the median or mean of our predicted distribution matches its SRCC within 0.01 while additionally supplying calibrated spread.
Gaussian head. The same backbone predicts a mean and a softplus standard deviation, fit by Gaussian likelihood on individual ratings. This cannot cross.
Categorical head. The same backbone predicts a softmax histogram over rating levels, binned where the scale is continuous. It is also crossing-free and non-parametric.
Independent heads with sorting. Separate quantile heads on a fixed grid, sorted after inference, then passed through CQR. This is the post-hoc fix the literature warns about.
Fairness protocol. All heads share the backbone, the data splits (split by source asset, so no content leaks), the optimizer, and an equal hyperparameter-search budget.
Ablations. One ablation isolates the constraint: the identical continuous head with the non-negativity of $\alpha$ and the ordering removed, plus a crossing penalty instead. A second isolates continuity: a fixed-grid NC-QR-DQN head.
What counts as winning. Beating a baseline means lower held-out-rater pinball loss averaged over $\tau$, equivalently the continuous ranked probability score, under a paired bootstrap across assets. It also means narrower conformal lower-bound margins at equal empirical coverage within one percentage point.
For distributional prediction on any of these 3D quality datasets, there is no reported result found on Lacuna. We therefore establish all distributional baselines ourselves.
Why not just…?
Why not just predict a mean and standard deviation? This is the most natural competitor, and it is a real threat. The argument against it is that rating distributions for 3D content are skewed and sometimes bimodal. The file reports bimodal responses to facial distortions of digital humans (Zhou et al.), and parametric forms fail on heavy tails in forecasting (Erdmann et al.). But if per-asset rater distributions on SJTU-TMQA and WPC turn out close to Gaussian, the Gaussian head will tie us at lower cost. We will therefore first report goodness-of-fit of per-asset empirical distributions, and then compare lower-tail pinball loss specifically, since gatekeeping lives there. A tie would be reported as a finding: disagreement is real but symmetric.
Why not just predict a histogram over the rating levels? On a discrete 11-level scale, a softmax histogram is also a crossing-free CDF, and it may do as well as our head. Its weaknesses are twofold. It ignores the ordering of levels during training, so neighbouring levels do not share statistical strength with 20–70 samples per asset. It also needs arbitrary binning on continuous scales. The categorical baseline tests both points directly. If it matches us on SJTU-TMQA but loses on continuous-scale data, the honest conclusion is that our head's value is on continuous scales and arbitrary-$\tau$ queries, not on discrete ones.
Why not just sort independent quantiles and conformalize? Sorting yields valid intervals at inference. However, NC-QR-DQN reports that enforcing monotonicity during training improved mean human-normalized Atari score from 864% to 1598% and reached baseline performance 30% faster. That suggests the constraint regularizes learning rather than merely tidying outputs. Whether this transfers to small perceptual datasets is exactly what the sorting baseline and the constraint ablation measure. We will also report the pre-sort crossing rate stratified by quality range, to test whether crossing concentrates in the sparse extremes.
Why not just conformalize a mean-score regressor? Such an interval covers the average opinion, not a new rater's opinion. It therefore cannot express that an asset is fine for most viewers but broken for a meaningful minority. We compare coverage of held-out individual ratings for both approaches. We expect the mean-based interval to be forced much wider on high-disagreement assets.
Proposed Experiments and Decisive Checks
We use SJTU-TMQA (textured meshes, 73 subjects, 11-level scale) and WPC (colored point clouds, 22,800 scores). 3DGS-IEval-15K (228,000 ratings) serves as a larger 2D-rendered check where HyperIQA and MANIQA reach SRCC above 0.93 (Xing et al.). RealismGrading is an optional no-reference setting.
The first check is a feasibility gate. Individual per-rater scores must be obtainable for at least two of these datasets. If only means and standard deviations are released, the project reduces to fitting published moments, and we would say so rather than proceed.
The decisive experiments are four:
- Distributional accuracy. On held-out raters and held-out assets, the continuous non-crossing head should beat the Gaussian, categorical, and sorted baselines in averaged pinball loss, with the largest margin at $\tau \le 0.2$.
- Calibrated gatekeeping. We define a defective asset as one where at least 20% of raters score it below the scale midpoint. At a fixed false-alarm rate, the conformalized lower percentile should flag defective assets with higher recall than the conformalized mean.
- Misalignment link. Predicted spread should correlate positively with Chamfer-versus-human rank disagreement, most strongly on SJTU-TMQA's combined geometry–texture compression slice.
- No mean penalty. Mean-score SRCC must stay within 0.01 of the retrained scalar backbone.
Failure of the third check would mean rater disagreement is noise rather than a signature of misalignment, and would sink the scientific claim even if the first two succeed.
References
- Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling
- DR-KFS: A Differentiable Visual Similarity Metric for 3D Shape Reconstruction
- Perceptual Deep Depth Super-Resolution
- SJTU-TMQA: A Quality Assessment Database for Static Mesh with Texture Map
- Textured Geometry Evaluation: Perceptual 3D Textured Shape Metric via 3D Latent-Geometry Network
- Perceptual Quality Assessment of Colored 3D Point Clouds
- SRAM: Shape-Realism Alignment Metric for No Reference 3D Shape Evaluation
- Non-Crossing Quantile Regression for Distributional Reinforcement Learning
- Multi-Horizon Time Series Forecasting of Non-Parametric CDFs with Deep Lattice Networks
- Conformalized Quantile Regression
- fastkqr: A Fast Algorithm for Kernel Quantile Regression
- An Efficient Multi Quantile Regression Network with Ad Hoc Prevention of Quantile Crossing
- DDH-QA: A Dynamic Digital Humans Quality Assessment Database
- 3DGS-IEval-15K: A Large-scale Image Quality Evaluation Database for 3D Gaussian-Splatting