Self-Similar Completion of Thin Transparent Geometry

Depth cameras largely fail on glass and clear plastic. The light they emit passes through the object or glances off it, so the sensor reports the table behind a beaker, or nothing at all. The parts that fail worst are usually the thin ones: rims, stems, handles and walls a millimetre or two thick. These are also the parts a robot gripper touches. Current fixes bring in knowledge from outside the scene. They train a depth-completion network on synthetic glassware, or they "opaquify" the image by painting a fake opaque texture onto the glass so that ordinary depth networks can read it. Both approaches weaken when the object looks unlike their training data.

This note proposes a different source of knowledge: the object itself. A wine glass that is broken in the sensor's view is rarely broken everywhere. Its rim catches a highlight, one side of the stem survives, and the base's contact edge is visible. The same local shapes recur around the glass and across scales: the curve of the rim at full resolution resembles the curve of the bowl at a coarser one. We will train a small generative model at test time on the reliable local geometry of one object, across several scales. We then use it to fill in the unreliable thin parts, and extract the result with a surface representation that does not inflate thin walls into blobs.

Core Claim

Internal geometric recurrence is enough to reconstruct the thin parts of transparent objects that externally trained priors miss. Its advantage should be largest on objects not seen during training. The mechanism is the 3D counterpart of a classical 2D observation. Super-resolution from a single image found that most small patches in a natural image have near-duplicates elsewhere in the same image at other scales. Multi-Scale Weighted Nuclear Norm Image Restoration showed that downsampling acts as a low-pass filter: it removes random corruption while keeping consistent structure. Both motivate the proposal, and neither has been applied to 3D geometry.

The closest existing uses of "cross-scale" in generative reconstruction do something else. KernelFusion trains a diffusion model (a generator that learns to turn noise into samples by repeated denoising) on 15×15 crops of one low-resolution image. It is purely 2D and takes about 20 minutes per image. PSHuman generates full-body and face-crop views together, but its scales are 2D image crops anchored to an external human body model, and it assumes opaque surfaces. We borrow KernelFusion's recipe of a single-instance model with a small receptive field. We then move it to local 3D point patches and aim it at the failure mode that the transparent-object literature leaves open.

The result would be new if a test-time, instance-specific prior closes most of the gap between known and novel transparent objects on thin regions, where external priors do not. If an explicit symmetry fit achieves the same, the claim fails, and the note says below how that will be checked.

Research Program

The pipeline has four stages. Each can be removed on its own for ablation.

Stage 1: initial point cloud with confidence. We start from a GHOST-opaquified depth map. GHOST hallucinates opaque texture onto transparent surfaces so that off-the-shelf depth models such as DepthAnythingV3 can process them. We back-project this depth map into a point cloud. Each point receives a confidence score, which combines three signals:

  • whether it agrees with any valid raw sensor depth;
  • how close it is to GHOST's predicted boundaries and specular highlights;
  • in multi-frame settings, whether it stays consistent across views registered with the camera trajectory.

The confidence score matters because GHOST's own weak-refraction assumption is a known source of bias, so not every hallucinated point can be trusted.

Stage 2: a multi-scale patch pyramid. We voxel-downsample the cloud to four or five resolutions. From each resolution we extract high-confidence local patches of a fixed number of points. Each patch is normalized to a unit-radius ball, so that a coarse-scale patch and a fine-scale patch are directly comparable. Random rotations during training let the model absorb rotational recurrence without being told an object is symmetric.

Stage 3: an instance-specific patch diffusion model. We train a compact point–voxel denoiser on these patches only. The architecture is taken from 3D Shape Generation and Completion through Point-Voxel Diffusion, which combines a voxel branch for spatial locality with a point branch for precise coordinates. It is shrunk to a local receptive field in the spirit of KernelFusion. Low-confidence regions are then completed with PVD's conditional reverse diffusion: observed points stay fixed while missing points are sampled. We run this over overlapping patches, averaging across overlaps. The process runs coarse-to-fine, so that each completed coarse scale conditions the next finer one. PVD motivates the choice of a generative model over a deterministic one, because a partial patch can be completed in several plausible ways, and deterministic completion averages them into a blur.

Stage 4: surface extraction. We fit Metric–Phase Fields to the completed, unoriented point cloud. This method separates an unsigned distance field from a smooth sign field. That lets it extract thin open sheets with Marching Cubes, avoiding the bulging of watertight signed distance functions and the fragmentation of unsigned ones. It reports a Chamfer distance of 0.672, against 5.591 for SDF baselines and 4.977 for UDF baselines, on thin synthetic models.

Its known weakness is that open surfaces come out as narrow double-sided shells. For glassware this is arguably a strength: a real glass wall has two refracting interfaces a small distance apart. We will test whether the shell thickness recovered after completion tracks true wall thickness.

Two operating modes. In single-frame mode, one RGB-D image supplies the patches. This matches the ClearGrasp and TransCG evaluations. In multi-frame mode, a short sequence is fused before Stage 2. This matches ClearPose, whose frames come with SLAM trajectories.

Compute budget. Our target is under five minutes per object on one GPU. This is plausible because point patches are far smaller than KernelFusion's image crops. The budget is a stated constraint, not an assumption, and we will report actual wall-clock times.

Baselines

Naive reference point. Raw sensor depth, with missing pixels filled by classical nearest-neighbour inpainting inside the transparent mask, followed by the same Metric–Phase surface extraction. This measures what any learned prior adds at all.

Learned single-view completion. ClearGrasp segments transparent regions, predicts normals and boundaries, and globally optimizes depth. It reports RMSE 0.039 on its Real-Known split, against 0.270 for direct monocular estimation with DenseDepth. TransDiff is an externally trained conditional depth diffusion model. It reports RMSE 0.032 and MAE 0.027 on ClearGrasp, and RMSE 0.028 and MAE 0.019 on TransCG. It is the strongest external generative prior in the file, which is exactly what our internal prior claims to beat on novel objects.

Strongest existing method: GHOST followed by DepthAnythingV3. GHOST reports DA3 RMSE dropping from 0.023 to 0.018 with its preprocessing in the ClearGrasp-based evaluation. It also reports DUSt3R reconstruction F-score rising from 77.36 to 88.66. Because our Stage 1 starts from GHOST's output, any gain we show is additive over it, and the comparison isolates Stages 2–4.

Numbers not available. We found no reported result on Lacuna for any of these methods on the ClearGrasp Real-Novel split, or on thin-region-only metrics. We will therefore rerun all baselines ourselves from released code, on identical inputs and masks. Each baseline receives the same hyperparameter-tuning budget on a held-out validation subset of synthetic scenes. Metrics are computed with a single shared script.

Ablations. Four ablations isolate the key ingredients:

  1. The same pipeline with Stage 3 removed, so Metric–Phase is fitted directly to the confidence-filtered cloud.
  2. Stage 3 replaced by a PVD model trained externally on synthetic transparent CAD models (ClearGrasp's synthetic set) at matched parameter count. This is the internal-versus-external test.
  3. Stage 3 restricted to same-scale patches. This tests the cross-scale ingredient.
  4. Metric–Phase replaced by a standard signed distance field fit. This tests the representation.

What counts as winning. Beating the strongest baseline means lower thin-region Chamfer distance and higher thin-region F-score than GHOST+DA3 on the Real-Novel split and on ClearPose. Global depth RMSE must be no worse than GHOST+DA3's.

Why not just…?

Why not just run GHOST with DUSt3R or DepthAnythingV3 and stop? GHOST's own reported gains show that opaquification helps. But the method assumes weak, alpha-blended refraction and produces a single opaque shell. It says nothing about wall thickness, and nothing about thin parts it paints incorrectly. Whether those errors are concentrated on thin regions is an empirical question. Our thin-region metrics answer it directly by scoring GHOST+DA3 on rims, stems and handles separately from bulk surfaces. If GHOST already does well there, the proposal's contribution shrinks to the representation stage, and we will report that.

Why not just fine-tune an external diffusion prior like TransDiff on more synthetic glassware? This is the most natural competitor, and ablation 2 tests it directly at matched capacity. The file gives indirect reasons to expect it to fall short on new shapes. In KernelFusion's 2D setting, supervised super-resolution models did worse than bicubic interpolation when the real degradation did not match their training data, while the internal model did not. TransDiff also operates only on 2D depth maps and degrades when its inference schedule changes. The decisive number is the Known-to-Novel performance drop. If the external prior's drop is as small as ours, internal learning adds nothing, and we would say so.

Why not just fit a surface of revolution, since most glassware is rotationally symmetric? This is a serious threat. Many ClearGrasp and ClearPose objects are bottles and cups, and an explicit axis-plus-profile fit is fast. We will implement it as a baseline and stratify every result by symmetry class: rotationally symmetric, mirror-symmetric only, and asymmetric (handles, spouts, irregular containers). The internal prior's claimed value is that it captures partial and local recurrence, such as a handle's cross-section repeating along its length, which parametric symmetry cannot. If the symmetry fit ties us on symmetric objects and we win only on the others, the paper's claim narrows to that. If the fit ties us everywhere, we would drop the diffusion stage.

Why not just use a stronger surface representation such as αSurf, without any prior? αSurf separates geometry from opacity on a grid and reaches Chamfer 0.284, against 0.454 for the next best method, on Thin Blender. However, it needs dense multi-view capture with NeRF initialization, and it cannot recover surfaces that no view observed reliably. Our setting is a single RGB-D frame or a short sequence. We will nonetheless run αSurf on ClearPose sequences where enough views exist, to check whether multi-view evidence alone removes the need for a prior.

Decisive Experiments

Test 1: novel-object gap on ClearGrasp. On ClearGrasp's 286 real images, we will report depth RMSE, MAE and relative error on the Real-Known and Real-Novel splits. We will also report Chamfer distance and F-score on the thin region: points of the aligned ground-truth CAD model where local wall thickness or part radius falls below a fixed threshold. The key statistic is the Known-to-Novel degradation for each method. The hypothesis is supported if our degradation is less than half that of TransDiff and of the external-PVD ablation, while global RMSE stays within GHOST+DA3's range.

Test 2: controlled thin-wall and clutter conditions on TransCG and ClearPose. On TransCG we repeat the depth metrics. On ClearPose we run multi-frame mode on the clean splits and on the translucent-cover split. That dataset reports 6D pose accuracy (ADD-S) collapsing from about 58% to 13% on the translucent-cover split. As a downstream check, we feed completed point clouds to one fixed pose estimator and report ADD-S. We will not claim a gain there unless it is measured.

Test 3: controlled synthetic study. Using ClearGrasp's Blender pipeline, we render glassware with controlled wall thickness (0.5–5 mm) and controlled symmetry. This tests two things:

  • whether the recovered Metric–Phase shell thickness correlates with true wall thickness;
  • whether gains grow as walls thin, which would support the thin-structure mechanism rather than a general smoothing effect.

Test 4: does cross-scale recurrence exist in 3D? We measure, for transparent objects and for opaque ShapeNet shapes, the fraction of fine-scale local patches with a near-duplicate at another scale of the same object. This parallels the patch-recurrence statistic from Super-resolution from a single image. If recurrence is weak, the premise fails regardless of downstream numbers.

Failure conditions. The idea is refuted if either of these holds:

  • the same-scale ablation matches the full model and the symmetry fit matches it on asymmetric objects;
  • the internal prior's advantage on novel objects vanishes at matched compute.