Best-Buddy Reranking for Zero-Shot Recognition of Real 3D Scans

Large 3D–language models such as Uni3D and OpenShape recognize objects they were never explicitly trained on. They do this by pooling a point cloud into one vector and comparing it with text embeddings of class names. On clean synthetic CAD models this works well. On real sensor scans it degrades, because the single pooled vector absorbs everything in the scan: floor, neighboring furniture, occlusion holes and scan-line artifacts. We propose a training-free fix that compares parts rather than wholes, and only counts parts that agree in both directions.

For each candidate class, the method uses the text embedding of the class name to retrieve a few clean synthetic shapes of that class from the 3D–language model's own pretraining library. It then breaks the real scan and each retrieved shape into local patches, each represented by a feature vector from the model. The class score counts "best buddies": pairs of patches that are each other's nearest neighbor. Floor and clutter patches in the scan have no counterpart in a clean CAD chair, so they form no mutual pairs and drop out. Parts of the chair that the scan missed also form no pairs and simply go uncounted. The result is a per-instance foreground mask and an occlusion mask in one operation, with no training, no labeled real scans, and no access to the rest of the test set.

Core Claim

Mutual nearest neighbor filtering has three established uses. It handles occlusion in 3D registration (DeepBBS, Best Buddies Registration). It rejects background in 2D few-shot classification (DMN4). It finds correspondences between deep features across very different domains (Neural Best-Buddies). It has never been used for open-vocabulary 3D recognition. The obstacle is that zero-shot recognition has no visual reference set to match against.

Our claim is that the semantic label embedding can supply that reference set. Text-to-shape retrieval inside a frozen 3D–language model turns a class name into a small set of clean exemplars. Patch-level bilateral matching against those exemplars then corrects the global score on cluttered, occluded real scans. The prediction is falsifiable. Gains should be large on ScanObjectNN variants that keep background points, smaller on object-only crops, and near zero on clean ModelNet40. They should also disappear when reciprocity is removed and matching becomes one-directional.

We deliberately do not match 3D patches directly to text tokens. In Uni3D and OpenShape, only the pooled embedding is trained to align with text, so individual patch tokens have no reason to sit near any word embedding. Patch-to-text matching would mostly measure noise. Patch-to-patch matching within one 3D feature space avoids this cross-modal mismatch. Language is used only where these models are reliable, at the global level.

Research Program

Encoder and exemplar library. The encoder is a frozen Uni3D model (Uni3D). It groups points into $k$-NN patches with a small PointNet tokenizer and processes them with a Vision Transformer. We also run OpenShape's PointBERT (OpenShape) to check that results hold across encoders. The exemplar library is the Objaverse/ShapeNet pool these models were trained on.

Exemplar retrieval. For class $c$ we embed several prompts. We retrieve the $R$ library shapes whose global 3D embedding is closest to the averaged text embedding, plus the $R$ shapes whose own text metadata is closest. This reduces the prompt sensitivity noted for language-embedding methods.

Scoring. Let $S$ be the set of patch features of the test scan and $E$ those of one exemplar. The Best-Buddies Similarity is

$$\mathrm{BBS}(S,E)=\frac{1}{\min(|S|,|E|)}\sum_{s\in S,\,e\in E}\mathbb{1}\big[e\in \mathrm{NN}_K(s)\ \wedge\ s\in \mathrm{NN}_K(e)\big],$$

where $\mathrm{NN}_K(x)$ is the set of the $K$ nearest patches to $x$ in the other set under cosine distance. Setting $K=1$ gives strict best buddies. Larger $K$ gives the mutual $K$-NN relaxation used for deformations and shared model units in Rosetta Neurons.

The final class score is

$$s_c=\cos(z,t_c)+\lambda\,\frac{1}{R}\sum_{r=1}^{R}\mathrm{BBS}(S,E_{c,r}),$$

where $z$ is the scan's global embedding, $t_c$ the class text embedding, and $\lambda$ a single weight tuned on synthetic validation data only.

Computational cost. Each comparison is between roughly 512 and 512 patches. Cost per class is therefore small, and the memory problem of point-level soft matching does not arise. For the 1,156 Objaverse-LVIS classes we rerank only the global model's top-20 shortlist.

Two research questions beyond "does it help". The first is which transformer layer supplies patch features. Deep layers mix information across all patches through attention, so a background patch may already carry object context and become a spurious buddy. Shallow tokenizer features are local but less semantic. We will measure the trade-off layer by layer.

The second is the density-mismatch failure mode named in the matching literature. Real scans are dense near the sensor and sparse far away, while CAD samples are uniform. We test whether a fixed patch count with farthest-point sampling, together with a moderate $K$, removes this sensitivity. The alternative would be density-normalized patch weighting.

Reported diagnostics. For every scan we report the fraction of patches that find a buddy. We check it against ground-truth background masks, which ScanObjectNN provides, to verify that the method actually acts as a foreground mask rather than only raising accuracy.

Baselines

Naive reference. The do-nothing reference is the frozen encoder's own zero-shot prediction, cosine between the global embedding and the class text embedding. We use the same prompts, the same point sampling and no tuning. For context, Uni3D reports 88.2% zero-shot accuracy on ModelNet40 and OpenShape reports 85.3% on ModelNet40 and 46.8% on Objaverse-LVIS. These numbers come from Uni3D and OpenShape. We found no reported result on Lacuna for Uni3D or OpenShape zero-shot accuracy on ScanObjectNN broken down by split, and no exact Uni3D scalar on Objaverse-LVIS. We will rerun both models ourselves under identical preprocessing. Beating this baseline means a gain on the ScanObjectNN background-retaining splits that exceeds seed and prompt variance. It also means no more than a small drop on ModelNet40.

Exemplar retrieval without patch matching. This is the strongest competitor and is built for this study. It uses the same retrieved exemplars, but scores the class by the cosine between the scan's global embedding and the exemplars' global embeddings. It is a retrieval-augmented zero-shot classifier with exactly the same extra information as our method. If it matches our gains, the benefit comes from retrieval, not from bilateral matching, and the core claim fails.

One-directional patch matching. This is the ablation isolating reciprocity, built in the style of DN4 as discussed in DMN4. It scores the average similarity of each scan patch to its nearest exemplar patch, plus a symmetric Chamfer-style variant. The motivation comes from the 2D few-shot result: on CUB, mutual matching reached 78.36% 1-shot accuracy against 73.42% for greedy one-way matching. That gap is the effect we expect to reappear in 3D.

Inductive hubness correction. This is cross-domain local scaling applied to the global scores, which lowers the score of classes that are close to everything. It is the natural inductive answer to the hubness problem documented in Zero-Shot Learning on 3D Point Cloud Objects and Beyond.

Few-shot reference. We include PointTFA at 1, 4 and 16 shots. It uses labeled real scans, which we do not, so it tells us how much labeled target data our training-free method is worth.

Fairness. All methods share the frozen encoder, the prompts, the exemplar pool, and a hyperparameter budget tuned only on held-out synthetic data with synthetic clutter.

Why not just…?

Why not just remove the background with a plane fit or a segmenter before classifying? On ScanObjectNN, much of the clutter is floor and walls, so plane removal is a real threat. We include it as a preprocessing step applied to the global baseline and to our method alike. The remaining clutter is neighboring objects and occlusion holes, which no plane fit handles. If plane removal recovers most of the gain on the background splits, we will report that honestly and narrow the claim to occlusion and non-planar clutter. We test that narrower claim with the controlled corruption study below.

Why not just retrieve CAD exemplars and compare global embeddings? This baseline isolates whether patch-level reciprocity matters at all. Our expectation that it falls short rests on registration evidence. In DeepBBS, global and one-way matching (PRNet) gave 1.712° rotation error on partial ModelNet40 pairs, against 0.008° for best-buddy matching. That shows reciprocity, not just exemplars, is what tolerates missing geometry. But registration is not recognition, and the experiment decides.

Why not just fix hubness transductively or with local scaling? The transductive hubness loss of Zero-Shot Learning on 3D Point Cloud Objects and Beyond needs the whole unlabeled test set. Even so, it reached only 45.63% generalized zero-shot accuracy on ScanObjectNN. Local scaling is inductive and cheap, so we run it. It corrects class-level bias but cannot remove corrupted patches inside one scan. We expect the two corrections to be complementary and will report their combination.

Why not just use few-shot adaptation like PointTFA? PointTFA needs labeled target scans. Its query reconstruction is a one-directional weighted sum, so scan regions without a counterpart still pull on the adapted feature. Our setting has zero labeled scans. The comparison reports at how many shots PointTFA overtakes us.

Proposed Experiments

Main experiment. The main experiment is zero-shot classification on the three standard ScanObjectNN variants (object-only, object with background, and the hardest perturbed split). We use two encoders and five prompt seeds. The decisive pattern is a gain that ranks the splits in order of clutter, together with a failure of the global-retrieval and one-way ablations to reproduce it. The mask diagnostic must also show buddy-matched patches overlapping the ground-truth object mask well above chance.

Controlled dose-response study. On ModelNet40 we add synthetic clutter from other ModelNet objects and a floor plane at increasing fractions. We also cut out occlusion wedges and resample points with a range-dependent density falloff. The method should leave clean accuracy near the 88.2% Uni3D reference and widen its margin as corruption grows. For this study we exclude any library shape that also appears in ModelNet/ShapeNet, to avoid retrieving near-duplicates of test items.

Scale test. On Objaverse-LVIS we test whether shortlist reranking still helps with 1,156 fine-grained classes, where exemplar quality is lower.

Sweeps. We sweep the patch-feature layer, $K$ from 1 to 8, and the number of exemplars $R$, and measure latency per scan. The method is not worth adopting if the background-split gain vanishes under plane removal plus local scaling, or if global exemplar retrieval matches it.