Many pose estimators, the systems that recover an object's 3D rotation and position from an image, learn best when they must also solve a helper task derived from the object's CAD model (its computer-aided design mesh). A common helper task asks the network to predict, for every object pixel, which point on the CAD surface it shows. That extra supervision is only as good as the mesh. If the mesh is the wrong trim level of a car, a generic chair, or a scan with a missing handle, the helper task trains the network on labels that contradict the image. We propose to treat this "CAD fidelity dependency" as a multi-task learning problem. Multi-task learning trains one shared network on several objectives at once, and its known failure mode is negative transfer, where one objective damages another. Our claim is that a flawed CAD model makes its helper task a source of negative transfer that is concentrated in space. It hurts through specific image regions, such as a reshaped bumper, rather than uniformly. The remedy is a small head that learns, from the image and the rendered mesh, which pixels to trust. Per-task loss weights cannot do this.
In practice the question is how much of the benefit of dense CAD supervision survives when the CAD model is only approximately right, and whether a model can learn which parts of an approximate mesh to believe. Answering it would make these accurate pipelines usable in settings such as robot grasping, vehicle docking and augmented reality, where exact meshes are the exception.
Core Claim
We make three testable claims. First, the gain from a dense CAD-coordinate helper task decays as mesh fidelity drops, and past some fidelity level it turns into a loss relative to training on pose alone. Second, the fidelity level at which this happens can be predicted from gradient conflict: the cosine similarity between the pose-loss gradient and the helper-loss gradient on the shared backbone should fall toward zero or below as fidelity drops, in the sense measured by RotoGrad. Third, a per-pixel gate that sees both the image and the substitute mesh recovers most of the exact-mesh gain across fidelity levels. It does this better than global remedies (tuned loss weights, gradient homogenization) and better than image-blind robust losses. The gate's output is also a map of where the CAD model disagrees with the object, which we evaluate against ground truth.
Relation to Prior Work
Two papers motivate the premise. The dense "3D location field" head of A Refined 3D Pose Dataset for Fine-Grained Object Categories maps each pixel to coordinates on an exact ShapeNet car. Adding it to a Mask R-CNN pose head improved StanfordCars 3D from 60.04% to 66.07% projection accuracy and cut median rotation error from 6.68° to 5.68°. That paper assumes an exact mesh for every sub-category, and it never asks what happens without one. MRC-Net replaces parallel multi-task pose learning with a render-then-correlate pipeline, and its authors acknowledge that the pipeline depends on accurate CAD. We do not claim to rescue parallel coarse-to-fine classification. We accept MRC-Net's critique on that point and target a different coupling: pose regression paired with CAD-derived dense targets.
Two papers constrain the design by showing mismatch failures in classical refinement. Combining 3D Model Contour Energy and Keypoints for Object Tracking shows contour energies settling in wrong minima when the geometry differs. Geometric Correspondence Fields addresses the synthetic-to-real gap in render-and-compare. Both are failures at test time. We study the complementary failure at training time, where flawed targets reshape the shared features.
The closest multi-task pose work is SSP-Pose. It trains pose regression jointly with shape-prior deformation, point matching and symmetry branches on RGB-D input. Shape Prior Deformation deforms a category mean and solves pose afterwards with a geometric solver. Both handle intra-class variation by deforming a category template using depth. Neither measures how auxiliary benefit depends on reference-mesh fidelity, and neither asks whether an auxiliary task should be switched off locally. We work from RGB alone with a given, imperfect instance mesh and treat its errors as label noise to be localized.
The method side reuses ideas from general multi-task learning. RotoGrad balances gradient magnitudes and rotates the shared features for each task. DeMT uses task-specific deformable sampling, and it achieved a 3.36% average per-task gain over single-task models on NYUD-v2, compared with 0.31% for MQTransformer. Both act per task or per feature. Neither can express "trust the helper labels on the door but not on the bumper", and that spatial granularity is our contribution.
Research Program
We first build a controlled fidelity ladder. StanfordCars 3D and CompCars 3D pair each image with its exact ShapeNet car and a 7-parameter pose annotation. Keeping the pose labels fixed, we replace the mesh used to generate helper targets with progressively worse substitutes. The rungs are the exact mesh, the nearest different car model by chamfer distance, a random car of the same body type, and a category mean shape. We add synthetic degradations of the exact mesh: decimation, local bumps and dents, and removal of parts such as mirrors and spoilers. Every rung has a measured fidelity score (chamfer distance to the exact mesh) and a ground-truth per-pixel mismatch map, obtained by rendering both meshes at the true pose. The mismatch maps are used only for evaluation, never for training.
The model is the location-field architecture of the Refined 3D Pose paper with one addition: a gate head that outputs a per-pixel weight $w_p \in (0,1]$. The gate sees backbone features together with a rendering of the substitute mesh's coordinates at the training pose, so it can compare what the mesh claims against what the image shows. The training objective is
$$\mathcal{L} = \mathcal{L}_{pose} + \lambda \sum_{p} \left( w_p \, \lVert \hat{y}_p - y_p \rVert_1 - \beta \log w_p \right),$$
where $\hat{y}_p$ is the predicted surface coordinate at pixel $p$ and $y_p$ is the substitute-mesh target. The $-\beta \log w_p$ term stops the gate from switching supervision off everywhere. For a fixed residual $e$, the optimal gate is $w=\min(1,\beta/e)$. To keep the gate from simply hiding difficult pixels, we detach the helper residual when it is fed to the gate. Where instance masks are available, as on Pix3D, disagreement between the mask and the rendered silhouette serves as a weak extra signal on the object boundary.
For each fidelity rung we also log the cosine between $\nabla_\theta \mathcal{L}_{pose}$ and $\nabla_\theta \mathcal{L}_{aux}$ on the shared parameters $\theta$. This tests whether gradient conflict predicts the crossover point before any validation run. If it does, practitioners get a cheap check for whether their CAD library is good enough to be worth using as supervision.
Baselines
The naive reference point is the same backbone trained on pose alone, with no helper task. It is immune to CAD errors by construction, and any method that falls below it on a given rung has suffered negative transfer. On exact meshes its expected level is the 6.68° median rotation error and 60.04% projection accuracy reported for the direct pose head in A Refined 3D Pose Dataset for Fine-Grained Object Categories.
The strongest existing method is that paper's location-field head, trained with a plain summed loss. It reports 5.68° median rotation error and 66.07% projection accuracy on StanfordCars 3D with exact meshes. We found no reported result on Lacuna for this method with substitute or degraded meshes. Producing those numbers is part of our contribution. We reimplement it with identical backbone, data splits, schedule and augmentation, and we first confirm that our reimplementation roughly matches the reported exact-mesh figures.
Two global multi-task remedies isolate the question of spatial granularity. The first is the summed loss with $\lambda$ tuned separately on each rung by grid search on held-out validation data. This gives the global approach an oracle advantage. The second is RotoGrad applied to the pose and location-field tasks. Every method gets the same number of tuning trials.
Two local but image-blind remedies isolate the gate's use of image evidence. The first replaces the gate with a truncated $L_1$ loss on the helper residual. The second is an ablated gate that sees only the residual magnitude, not the image or the rendered mesh. We also train with mesh-perturbation augmentation, meaning random degradations of the exact mesh applied during training. The oracle upper bound applies the true mismatch mask to the helper loss.
We count the proposal as beating the field if the image-conditioned gate, on the random-car and category-mean rungs, beats the pose-only model, the per-rung-tuned summed loss and the truncated loss. The margin must exceed the spread across three seeds, and the gate must close at least half the gap between the vanilla location-field head and the oracle-mask bound.
Why not just…?
Why not just drop the helper task whenever the CAD model is poor? That requires knowing in advance that the mesh is poor, and it throws away the parts of an approximate mesh that are correct. A car of the wrong make still has correct wheels, doors and roofline. Our fidelity ladder directly measures whether partial meshes carry value beyond pose-only training. If the gated model never beats pose-only training below the exact-mesh rung, this alternative wins and we will report that.
Why not just tune the loss weight or use RotoGrad? Both apply one scalar or one rotation to the whole task. They can shrink the helper task's influence but cannot keep its good pixels while rejecting its bad ones. The DeMT result above supports the view that where features are shared matters in dense geometric prediction. The per-rung-tuned $\lambda$ baseline tests this with an unfair advantage given to the global approach. If tuned $\lambda$ matches the gate, our spatial claim fails.
Why not just use a robust loss? This is the most serious threat, because a truncated loss is already a per-pixel gate. The difference we expect is that CAD mismatch often produces small, spatially coherent errors. A slightly different bumper curve biases many neighbouring coordinates by a few centimetres, below any truncation threshold, and such a bias shifts pose systematically. Only a gate that compares the rendered mesh with the image can flag it. The residual-only gate ablation separates the two explanations. If it performs as well as the full gate, the finding reduces to "use robust helper losses under CAD mismatch". That is still useful, but it is a smaller claim, and we would state it as such.
Why not just augment with perturbed meshes, or avoid CAD entirely? Augmentation teaches tolerance to random noise, but real substitute meshes differ in structured ways, and the random-car rungs test whether that tolerance transfers. CAD-free alternatives cost accuracy. OSOP reaches only 43.6% ADD on LineMOD from RGB. Pose from Shape reports that conditioning on shape raises unseen-category accuracy on ObjectNet3D from 50% to 62%. Geometry therefore helps even when it is imperfect.
Proposed Experiments
The main experiment is the full fidelity ladder on StanfordCars 3D, repeated for transfer on CompCars 3D. We report median rotation error and projection accuracy at the 0.1 threshold, always computed against the exact mesh and true pose, so that evaluation never inherits the substitute's errors. We plot each method's accuracy against measured chamfer fidelity. For the gate we additionally report area under the ROC curve for detecting true mismatch pixels, which tests whether the gate genuinely localizes CAD errors.
We test the diagnostic claim by checking whether the fidelity level at which mean gradient cosine crosses zero predicts the fidelity level at which the vanilla helper task falls below pose-only training.
For in-the-wild transfer we use Pix3D. Its annotated meshes serve as the exact rung and retrieved nearest neighbours as the substitute rung, and we measure pose accuracy under strict thresholds. The reference point there is the 55% versus 19% strict-threshold result of Geometric Correspondence Fields with exact meshes. Finally, as an RGB check on textureless industrial parts, we degrade the T-LESS meshes and measure BOP Average Recall (BOP).
The decisive outcomes are as follows. The project's premise fails if no crossover appears, meaning degraded helper tasks never hurt. The spatial claim fails if the per-rung-tuned summed loss or the truncated loss matches the image-conditioned gate. The gate's value as a fidelity verifier fails if its mismatch-detection AUROC is near chance. A positive result on all three would show that fidelity dependency can be managed pixel by pixel, and that the same head can report where the CAD model is wrong.
References
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- A Refined 3D Pose Dataset for Fine-Grained Object Categories
- MRC-Net: 6-DoF Pose Estimation with MultiScale Residual Correlation
- Combining 3D Model Contour Energy and Keypoints for Object Tracking
- Geometric Correspondence Fields: Learned Differentiable Rendering for 3D Pose Refinement in the Wild
- SSP-Pose: Symmetry-Aware Shape Prior Deformation for Direct Category-Level Object Pose Estimation
- Shape Prior Deformation for Categorical 6D Object Pose and Size Estimation
- DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Prediction
- OSOP: A Multi-Stage One Shot Object Pose Estimation Framework
- Pose from Shape: Deep Pose Estimation for Arbitrary 3D Objects
- BOP: Benchmark for 6D Object Pose Estimation