Suppose you want to morph one 3D object into another, and the two endpoints come from different kinds of input. One might be a photo of a stool, the other the phrase "a high-backed armchair." A 3D generator can be driven by a vision-language model (VLM, a model that reads both images and text). This works by passing the VLM's internal features through a learned translator, called a cross-modal projector, into the image features the 3D generator expects. That pipeline offers several places where one could draw a line between the two endpoints: in the VLM's semantic space, in the projected image-feature grid, or in the 3D generator's own latent space. We do not know which of these gives intermediate shapes that are both plausible objects and semantically sensible steps between the endpoints.
We expect a specific failure. A straight line in the semantic space probably keeps the meaning coherent, but midway points get projected into image-feature grids that no real image would produce, so the resulting geometry breaks. A straight line in the 3D generator's latent space probably keeps the geometry valid, but the midpoints are geometric blends with no semantic sense. This proposal measures that trade-off. It then tests one fix: bend the semantic-space path so that every projected intermediate stays in the high-density region of real image-feature grids. Density is judged by a discriminator, a small network trained to tell real feature grids from generated ones.
Core Claim
Interpolation through a cross-modal projector fails the same way feature matching fails. The file calls this the "paradox of proximity." Robust Vision-Language Models via Manifold-Adversarial Adapters showed that pulling features together in Euclidean distance pushes them off the data manifold a frozen downstream model relies on. On the R-Bench-Dis corruption benchmark, naive mean-squared-error (MSE) adaptation lowered a base LLaVA score from 58.79 to 57.98. Their manifold-adversarial discriminator raised it to 59.39. We claim that straight interpolation paths in VLM space are the generative analogue of this effect. We further claim that relaxing those paths under a density term measured in the projector's output space should give morphs that beat interpolation in the 3D generator's own latent space on semantic coherence while matching it on geometric validity. If interpolating directly in the 3D latent does equally well on both, the cross-modal route is not worth taking. The experiments are designed to show that outcome plainly if it is the true one.
Relation to Prior Work
The mechanism we build on is GAP3D. It trains a rectified-flow transformer, a generator that learns nearly straight transport paths from noise to data and can therefore be sampled in a few steps. This transformer maps VLM latents into the full patch-level feature grid of DINOv2 (a self-supervised image encoder), including its class, register and spatial tokens. That grid then conditions a frozen TRELLIS 3D generator. GAP3D evaluates only single-prompt generation and never studies paths between prompts. We take its projector as given and study its behaviour along paths.
On the geometry side, several papers show that interpolation succeeds only in the right space and fails on raw parameters. DeepSDF and implicit field decoders avoid the ghosting of voxel blending by warping a continuous surface boundary. Learning Unified Representation of 3D Gaussian Splatting reports 63.4 versus 44.7 PSNR (peak signal-to-noise ratio) on ShapeSplat for its field-based encoding against raw parameters. These papers motivate our premise that the choice of interpolation space dominates results. Shape Unicode interpolates in a shared voxel/point/view space but has no language interface, so it cannot accept a text endpoint.
Two papers constrain our design. SharpShooter uses linear latent interpolation, and its validity falls to about 50% on UCI Credit, where the data forms disconnected clusters. This warns that straight lines fail across disconnected regions, which in our setting means category changes and changes in topological genus (for example, gaining a hole). LatentLens finds that projected visual tokens follow context-dependent trajectories rather than static linear maps. That finding gives reason to doubt that linearity in the input space carries over to linearity in the output space. Our method borrows its density term from the manifold-adversarial adapter, but applies it to a path of generations rather than to restoration of a single corrupted input.
Research Program
We fix a frozen VLM encoder, the GAP3D-style projector $P$, and frozen TRELLIS. For a pair of endpoints $z_0, z_K$, each obtained by encoding an image, a text prompt, or an image plus text, we discretize a path into points $z_0, \dots, z_K$ in VLM latent space. The two endpoints stay fixed and the interior points are optimized:
$$\min_{z_1,\dots,z_{K-1}} \sum_{k=0}^{K-1} \lVert z_{k+1}-z_k \rVert^2 \;-\; \lambda \sum_{k=1}^{K-1} \log D\big(P(z_k;\epsilon)\big)$$
Here $\epsilon$ is a projector noise sample shared across the whole path. $D$ is a discriminator trained to output the probability that a DINOv2 patch grid comes from a real rendered object rather than from projected interpolants. $\lambda$ trades path shortness against density. The first term alone recovers the straight line.
Rectified flow can be sampled in a few steps, so gradients through $P$ can be computed with a short sampler. We will report the cost of doing so.
The analysis side of the program tracks per-step statistics of the projected grids along straight and relaxed paths. These are discriminator score, distance to the nearest real grid, and spatial coherence of the patch tokens. The goal is to locate where and for which endpoint pairs the paths leave the manifold.
We group pairs into four categories: same category and same topology; same category with a genus change; cross-category; and mixed-modality (image to text). This lets us test whether off-manifold excursions concentrate where SharpShooter's result predicts, namely in disconnected regions.
Baselines
The naive reference point is a straight line in VLM latent space passed through the projector with shared noise and decoded by TRELLIS. We use no relaxation and the same $K$ steps. This is the default a user of GAP3D would try.
The strongest competitor is interpolation in TRELLIS's own latent space. We generate both endpoints through the pipeline, take their 3D latents, and interpolate between them. We test both a straight line and spherical interpolation, and report whichever is better. This baseline is strong because it works where the geometry decoder has been trained, which is exactly where the geometry literature says interpolation should succeed. For fairness, we also give it the same relaxation budget, with a discriminator trained on its latents and the same number of optimization steps and wall-clock time. Any win for our method must then come from the space and not from the extra optimization.
A third baseline interpolates directly in the DINOv2 patch-grid space between the two projected endpoint grids. This isolates whether the semantic space adds anything beyond the dense feature space.
Two ablations isolate the key ingredient. The first replaces the discriminator term with a Euclidean pull toward the nearest real patch grid. This is the MSE-style constraint the paradox-of-proximity result predicts will fail. The second sets $\lambda = 0$ with path smoothing only.
All methods use identical endpoint sets, identical TRELLIS weights and seeds, and equal tuning grids for $\lambda$ or its counterparts, chosen on a held-out split.
No reported result found on Lacuna for interpolation quality on Toys4K or Objaverse-XL for any of these methods. GAP3D reports only that its single-prompt Fréchet and Kernel Distances approach native TRELLIS, without figures in the file. Beating the strongest baseline therefore means two things. First, higher semantic monotonicity with intermediate realism no worse than TRELLIS-latent interpolation. Second, statistically better realism than the naive straight line on cross-category and genus-change pairs.
Why not just…?
Why not just interpolate in the 3D generator's latent space? This is the real threat. If mixed-modality endpoints are first turned into shapes, the problem reduces to geometric morphing, where TRELLIS latents are the natural home. Our bet is that such paths are geometrically valid but semantically blind, much like the plausible hybrids in Learning to generate chairs. A stool-to-armchair morph should pass through recognizable chairs, not through fused blobs. The matched-budget comparison settles this. If TRELLIS-latent paths tie on semantic monotonicity, we will report that the cross-modal route adds nothing.
Why not just use spherical interpolation with fixed noise, the standard diffusion trick? Spherical interpolation corrects norm shrinkage at midpoints. It does not know where the projector's output grids are dense. We include it within both straight-path baselines. Its gap to our relaxed path on genus-change and cross-category pairs will show whether norm correction is enough.
Why not just interpolate the DINOv2 patch grids directly? Averaging two spatial grids superimposes two layouts. That is the feature-space version of voxel ghosting described in the DeepSDF and implicit-field papers. The patch-space baseline tests this directly, and we expect duplicated parts at the midpoints.
Proposed Experiments
Endpoints come from Toys4K objects and captions. We use 500 pairs per category group, and the mixed-modality pairs combine a rendered view with the caption of a different object. The discriminator trains on DINOv2 grids of the roughly 60,000 Objaverse-XL canonical renders used by GAP3D, and is kept disjoint from the evaluation objects.
Metrics are computed per path with $K = 9$ interior steps:
- Intermediate realism: Fréchet Distance of the midpoint shapes' renders against Toys4K.
- Smoothness: variance of Chamfer distance between consecutive shapes, which penalizes jumps.
- Geometric validity: watertightness and the count of spurious disconnected components.
- Semantic monotonicity: Spearman correlation between step index and the CLIP-score difference to the two endpoints.
- Endpoint fidelity.
- Human preference: a small two-alternative forced-choice study on 200 paths.
The decisive checks are three. The first is whether the discriminator score along straight VLM paths dips at the midpoints, and dips more on cross-category and genus-change pairs, which would confirm the diagnosis. The second is whether relaxation closes that dip while the nearest-neighbour Euclidean ablation does not, which would show that density rather than proximity is the working ingredient. The third is whether, at matched compute, relaxed cross-modal paths beat TRELLIS-latent paths on semantic monotonicity without losing validity. If the third check fails, the paper's contribution becomes the diagnostic map of where cross-modal interpolation breaks. That is still useful, but it is a smaller claim, and we would state it as such.
References
- Robust Vision-Language Models via Manifold-Adversarial Adapters
- GAP3D: Generative Alignment of VLM Latents to Patch-Level Embeddings for 3D Generation
- DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
- Learning Implicit Fields for Generative Shape Modeling.
- Learning Unified Representation of 3D Gaussian Splatting
- Shape Unicode: A Unified Shape Representation
- Counterfactual Explanations via Latent Space Projection and Interpolation
- LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs
- Learning to generate chairs with convolutional neural networks