When a model completes a 3D shape from a partial depth scan and a photo, it has to decide, region by region, which source to believe. It can believe the scan where the object was actually seen. It can believe a full-shape guess inferred from the image. Or it can believe what it has learned about the categories it was trained on. Recent systems make this decision with cross-modal attention, a layer in which features from one input choose, by learned similarity, which features from another input to borrow. We argue that these attention layers are where limited object diversity actually hurts. The image encoders and image-to-3D generators feeding them have seen many kinds of objects. The small fusion module on top, however, is trained on a dozen furniture and vehicle categories, and it learns category-specific trust rules, such as "for cabinets, follow the image draft." On an unseen category those rules misfire. The fusion can defer to a wrong draft, or it can throw away a correct observation.
The proposal is to make the fusion decision depend on evidence that means the same thing for every category: how well the modalities agree locally, and whether a region was observed. It should not depend on content features that encode category identity. We also train the fusion on deliberately manufactured disagreements, so that it learns to arbitrate rather than memorize. The main claim is testable. With the encoders held fixed, agreement-conditioned fusion should shrink the gap between seen-category and unseen-category completion error. We also predict that a probe will be able to read category identity out of the attention weights of standard fusion, and not out of ours.
Where this sits in prior work
Two papers bracket the idea, and neither does it. 3D Shape Completion on Unseen Categories: A Weakly-supervised Approach completes shapes from 8 unseen ShapeNet categories. It does so by letting partial scans cross-attend to a bank of complete shapes from seen categories and borrow shared local primitives, reaching mean IoU 0.474 and Chamfer Distance 3.02 at $32^3$ voxels. Its cross-attention is purely 3D-to-3D, however, so it never faces the question of whom to trust across modalities. Rethinking Multimodal Point Cloud Completion: A Completion-by-Correction Perspective (PGNet) does face that question. It takes a complete draft from a pretrained image-to-3D model and aligns it with the partial point cloud using cross-attentive "grounding" transformers, lowering Chamfer Distance on ShapeNetViPC from 1.211 to 0.926. However, it evaluates only within the 13 training categories. Its authors also note that when the draft has wrong topology on an out-of-distribution object, the grounding stage struggles to correct it. That admission motivates this proposal directly. We reuse PGNet's completion-by-correction structure and change only what the fusion is allowed to condition on.
Other work supplies the mechanism or constrains the design. An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention shows that a learned gate mixing cross-attended context with original features, plus sparse attention, counters one modality dominating the other. We borrow the gating form. Its gates, though, are computed from content features, which is exactly the channel through which category shortcuts can leak. SDFusion injects image and text conditions through cross-attention and exposes per-modality guidance weights. Those weights are global and set by hand, with no automatic way to stop the image from overriding weak geometry. That is the region-level decision we want to learn. The OOD-CV benchmarks constrain the design: OOD-CV-v2 finds no significant robustness difference between CNNs and transformers, while explicit 3D models are markedly more robust to geometric shifts. We read this as evidence that attention alone will not fix category shift. What matters is which signal the model is built to rely on, and this is why we change the fusion's inputs rather than its architecture.
Core Claim
Consider a query point $x$ on the object's surface. Let $h_x$ be the content features at $x$, and let $c_x^{(m)}$ be the context that cross-attention retrieves from modality $m$. The modalities are the partial scan, the image-to-3D draft, and the seen-category shape bank. Standard fusion computes mixing weights from $h_x$. We instead compute them only from a category-agnostic evidence vector $e_x$:
$$\alpha_x = \mathrm{softmax}\big(f_\theta(e_x)\big), \qquad z_x = \sum_m \alpha_x^{(m)} c_x^{(m)}.$$
Here $\alpha_x$ is the vector of per-modality trust weights, $f_\theta$ is a small network, and $z_x$ is the fused feature passed to the decoder. The evidence vector $e_x$ contains local geometric discrepancy between the draft and the observed points, measured as distance and normal disagreement over a neighborhood. It also contains a visibility flag derived from the camera ray, the draft generator's own sample variance, and the distance to the nearest shape-bank match. None of these quantities names a category, so a rule learned on chairs transfers to jellyfish-shaped lamps or backpacks.
The second ingredient is disagreement training. During training we corrupt one source in controlled ways. We swap the draft for a draft of a different object, warp it locally, drop observed regions, or add sensor-like noise to the scan. Supervision still comes from the true complete shape, so the fusion is forced to learn which evidence predicts which source is right. Without such corruptions the training distribution rarely contains confident-but-wrong drafts, and those are exactly what unseen categories produce.
Research Program
We first establish the phenomenon diagnostically. We will reimplement PGNet-style fusion, since the file lists no public PGNet code. We will train it on category-held-out splits of ShapeNetViPC and record where its attention mass goes on seen and unseen categories. We then train a linear probe to predict object category from the per-point fusion weights. High probe accuracy on seen categories, together with errors that concentrate in regions where the draft disagrees with observation on unseen categories, would confirm that the fusion has learned category-conditional trust. If the probe fails and the errors are uniform, the premise is wrong, and the fault lies in the draft generator rather than the fusion. We would report that finding and stop.
We then build agreement-conditioned fusion on the same encoders, draft generator, decoder and refinement blocks, and add the disagreement curriculum. Finally, we test transfer to a different dataset. We render partial depth scans for Pix3D's real images and aligned CAD models (Pix3D), holding out Pix3D categories absent from training. We also run the ScanNet evaluation used by the unseen-category completion paper, which provides real sensor partials. The same fusion rule could later sit inside visuo-tactile policies such as the one in Grasping Deformable Objects via Reinforcement Learning with Cross-Modal Attention to Visuo-Tactile Inputs. That paper reports that its cross-attention module overfits under environment randomization, a symptom consistent with our hypothesis. We treat this as a follow-up and do not test it here.
Baselines
The naive reference points are the two unfused sources. The first is the image-to-3D draft alone, rigidly registered to the partial scan. The second is a strong single-modality completion network given only the partial scan. If fusion of any kind cannot beat the better of these two on unseen categories, the problem is not a fusion problem.
The strongest existing multimodal competitor is PGNet (Rethinking Multimodal Point Cloud Completion), which reports an average Chamfer Distance of 0.926 on the standard in-domain ShapeNetViPC protocol, against 1.211 for EGIInet. No reported result was found on Lacuna for PGNet on category-held-out splits, so we will produce that number ourselves. We will first reproduce the in-domain figure to validate our reimplementation. We then retrain on each held-out fold with the same draft generator, encoders, epochs and a matched hyperparameter search budget. Beating it means lower unseen-category Chamfer Distance and higher F-Score across folds, and specifically a smaller seen-to-unseen gap, without losing more than a small margin in-domain.
The strongest unseen-category method is CS-SC from 3D Shape Completion on Unseen Categories, with mean IoU 0.474 and Chamfer Distance 3.02 on its 8 held-out ShapeNet categories. That paper uses a different protocol and a $32^3$ voxel output. For a fair comparison, we will run our model on its exact split and voxelize our outputs to its resolution. We will also give CS-SC access to the partial scan only, as it was designed. If our model cannot beat CS-SC, then adding an image modality does not help on unseen categories under that protocol, which is itself informative.
A content-gated baseline adds a gate in the style of Gated Cross Attention to PGNet, computed from features, with the same parameter count as our fusion head. It separates "gating helps" from "what the gate conditions on helps." Two ablations isolate our ingredients. The first gives our fusion head the content features $h_x$ in addition to $e_x$, which should reintroduce the shortcut. The second removes disagreement training while keeping $e_x$.
Why not just…?
Why not just train the fusion on more categories? This is the most serious threat. If category diversity in the fusion's training data fixes the problem cheaply, our method is unnecessary. We will measure it directly. We vary the number of training categories from few to all but the held-out fold, and plot the unseen-category error of content-based fusion against ours. If content fusion's curve converges to ours within the categories available, we will report that the remedy is data rather than architecture. We expect otherwise because the Pix3D and ScanNet transfer tests hold out geometry types absent from all of ShapeNetViPC.
Why not just trust the image-to-3D draft, since its generator already saw diverse objects? PGNet's authors note that the draft can have wrong topology out of distribution, and the draft-only baseline measures how often that happens. We will stratify unseen objects by draft error. Our method should help most where the draft is wrong and match the draft where it is right.
Why not a hand rule: trust the scan where observed, the draft elsewhere? This rule is category-agnostic, so it is a real competitor. It breaks when the scan itself is noisy, or when the unobserved region of the draft is wrong with no observation to contradict it. We include it as a baseline with the same registration step. If it matches our method, the learned part adds nothing, and we will say so.
Why not just add a gate, as in drug-target interaction work? A gate computed from content features can still encode "this looks like a cabinet, so trust the draft." The content-gated baseline and the category probe on its gate values test this directly.
Proposed Experiments
The main experiment uses leave-categories-out folds of ShapeNetViPC. We report Chamfer Distance and F-Score on seen and unseen categories, the seen-to-unseen gap, and a breakdown by draft correctness. It is supplemented by the CS-SC unseen-category protocol and by rendered-partial Pix3D and ScanNet transfer. Three decisive checks follow. First, our model must shrink the unseen-category gap relative to matched PGNet, not merely lower in-domain error. Second, a linear probe should decode category from content-fusion weights well above chance and from our weights near chance. Third, removing disagreement training should hurt specifically on objects with confident-but-wrong drafts. If the first check fails while the probe succeeds, category shortcuts exist but do not drive error, and the proposal's causal claim is refuted.
References
- 3D Shape Completion on Unseen Categories: A Weakly-Supervised Approach
- Rethinking Multimodal Point Cloud Completion: A Completion-by-Correction Perspective
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation
- OOD-CV-v2: An extended Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images
- Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling
- Grasping Deformable Objects via Reinforcement Learning with Cross-Modal Attention to Visuo-Tactile Inputs