When a 3D mesh is fitted to photographs, each pixel's error is pushed back to the vertices through a renderer. Any shape change can be written as a sum of smooth "vibration modes" of the surface, running from broad bulges to fine wrinkles. Those modes are the eigenfunctions of the Laplace-Beltrami operator, the surface analogue of a Fourier basis. We propose to measure, for each mode, how strongly it shows up in the rendered image. That one measurement predicts three things. It predicts which modes the optimizer learns quickly and which it learns slowly. It predicts which shape changes are effectively invisible to the camera. And it predicts how an image loss defined on frequencies, rather than raw pixels, should be weighted to speed up the slow modes. The tool we use to find the invisible changes comes from adversarial attacks on 3D shapes. There, perturbing a few low-frequency surface modes produces large but smooth deformations that detectors miss. We point that same idea at the reconstruction pipeline itself.
The opportunity exists because the four literatures have not met. Coordinate networks are known to fit coarse structure before fine detail, a property called spectral bias. Explicit mesh vertices have no such network and no positional encoding to tune, and nobody has charted how they converge when gradients pass through the occlusion-boundary approximations of a rasterizer. Image losses defined in Fourier, wavelet or Sobolev space are used for super-resolution, never to drive vertex optimization. Spectral mesh attacks are evaluated only against 3D networks, never through a renderer. Our claim is that one linearized object, the rendering-to-spectrum transfer operator, ties these together. It explains the convergence pathology, it locates the reconstruction blind spots, and it tells us when a frequency-domain loss can help and when no loss can.
Core Claim
Let $\Phi \in \mathbb{R}^{N \times k}$ hold the first $k$ Laplace-Beltrami eigenfunctions of a template mesh with $N$ vertices. Let $c$ be the shape's coefficients in that basis, $J$ the Jacobian of the rendered images with respect to vertex positions under a chosen rasterizer gradient, $F$ the 2D discrete Fourier transform, and $W$ a diagonal weighting over image frequencies. We define the per-mode kernel
$$K_W = \Phi^\top J^\top F^{*} W F J \Phi .$$
$K_W$ plays the role that the neural tangent kernel plays for coordinate networks. Under linearized gradient descent on the loss $\|W^{1/2}F(\text{render}(V)-I)\|^2$, the error in eigen-direction $u$ of $K_W$ decays at a rate proportional to its eigenvalue. We make three predictions.
The first prediction concerns plain pixel losses ($W$ constant). Silhouette gradients from edge-smoothing rasterizers are localized at occlusion boundaries. We therefore expect $K_W$ to give large eigenvalues to high-order modes near the silhouette, and small ones to low-order modes that change interior depth, such as bulges, concavities and foreshortening. That ordering is inverted relative to the coordinate-network picture.
The second prediction is that the smallest-eigenvalue directions inside the low-frequency band are spectral adversarial perturbations against reconstruction. They are large, smooth 3D changes that barely alter the images. Reconstruction error of standard pipelines should concentrate along them.
The third prediction separates what a loss can fix from what it cannot. Choosing $W$ to flatten the spectrum of $K_W$ should accelerate modes that are poorly conditioned but still visible. No choice of $W$ can recover directions in the near-null space of $J\Phi$. The method therefore separates "slow" from "unidentifiable", and existing pipelines conflate the two.
Relation to Prior Work
Three spectral attack papers supply the parameterization and the adversarial machinery. SAGA perturbs low-frequency Laplace-Beltrami coefficients so that a mesh autoencoder reconstructs a different face, and drives a geometric detector to about 53% accuracy. Universal spectral attacks perturb eigenvalues and rebuild shapes from a band-limited basis ($k=60$, $b=20$). GSDA perturbs graph Fourier coefficients of point clouds. All three attack 3D networks under 3D losses. We reuse their bases, but we search for perturbations that minimize change in rendered images rather than maximize a classifier's loss. The victim is the inverse renderer, not a recognizer.
Zhou and Lähner compute Laplace-Beltrami bases on Gaussian splats and observe that geometry keeps changing after image error plateaus. That observation motivates our work: it is direct evidence of modes that are slow or invisible to photometric loss. We turn it into a quantitative, per-mode account.
The rasterizer papers define the operators whose transfer functions we measure. N3MR uses a hand-crafted blurred gradient. SoftRas uses sigmoid edge probabilities. Rasterize-then-splat runs a non-differentiable oracle followed by differentiable splatting. Lin et al. run a hardware-rasterized pipeline. Each defines a different $J$, and none studies frequency-dependent convergence. DDSL rasterizes meshes through an analytic Fourier transform in image space. That is a Euclidean spectrum, not the intrinsic surface spectrum. We use it as one more renderer whose $K_W$ can be compared.
The spectral bias work supplies the analysis template. Wang et al. explain slow high-frequency fitting through the neural tangent kernel eigensystem, and NeRF counters it with Fourier features. Explicit vertex tensors have no inputs to encode, so the analogous fix has to come from the loss metric or a preconditioner.
The frequency-domain supervision work supplies candidate weightings $W$:
- amplitude and phase FFT losses (Xu et al.);
- radial soft weighting (UltraHR-100K);
- Sobolev $H^s$ norms as in ASASR, which also constrains us through its reported default $s=1.5$.
These papers also warn that over-weighting high frequencies causes ringing. That warning is exactly why we derive $W$ from $K_W$ instead of fixing it by hand.
Research Program
Measuring the operator. We compute $J\Phi$ with $k$ forward-mode Jacobian-vector products per view, one rendered derivative image per mode. Truncating to $k \le 200$ keeps this cheap at template sizes like CoMA's 5,023 vertices. We then form $K_W$ for each rasterizer (N3MR-style, SoftRas at several sigmoid widths, rasterize-then-splat, and the antialiased hardware rasterizer used by Lin et al.) and plot how its eigenvalues are distributed across mode index. This directly answers the open question in the file of whether edge smoothing attenuates or passes low-frequency intrinsic perturbations.
Testing the prediction. We fit meshes from multi-view renders and track error in each mode over iterations. We then check whether the measured decay rates match the eigenvalues of $K_W$ computed at initialization and recomputed periodically.
The adversarial probe. Inside the band of low-order modes, we solve for $\delta$ that maximizes 3D displacement subject to a rendered-image budget. We compute $\delta$ through the soft surrogate and then verify it under the exact hard forward rasterizer. A perturbation that is invisible in the surrogate but visible in the true render exposes a gap in the gradient approximation itself. We then measure what fraction of each baseline's final reconstruction error lies in the span of these blind directions.
The fix. We choose a diagonal $W$, parameterized by a smooth radial profile over image frequencies, to minimize the condition number of $K_W$ restricted to its visible subspace. We refit $W$ a few times during optimization. The blind subspace is reported rather than forced; it can be regularized with a shape prior.
Baselines
The naive reference point is the SoftRas recipe applied to direct vertex optimization: Adam on vertex positions, pixel and silhouette losses, and the spatial Laplacian smoothing term. The SoftRas paper reports a mean 3D IoU more than 2.1% above N3MR on ShapeNet. N3MR reports 0.6172 IoU on airplanes. Those numbers come from a learned single-view setting, so we use them only to sanity-check our reimplementation, not as targets.
The strongest existing method on real data is the pipeline of Lin et al., with its released code (diff). It reports a mean Chamfer distance of 0.68 mm and a PSNR of 26.04 on DTU after about 30 minutes on one GPU. We will swap only its image loss for our $K_W$-derived frequency loss, keeping its geometry stages, iteration count and hardware fixed. Beating it means a lower mean Chamfer distance at equal wall-clock time with PSNR no worse.
The strongest competitor on the conditioning question itself is gradient preconditioning in parameter space. It multiplies vertex gradients by $(I+\lambda L)^{-1}$, where $L$ is the mesh Laplacian. Equivalently, it optimizes in the Laplace-Beltrami basis with per-mode step sizes, unlocking modes coarse-to-fine as the band-limited reconstruction of Rampini et al. suggests. We found no reported result on Lacuna for this baseline on CoMA or DTU, so we run it ourselves. It gets the same tuning budget as our method: an identical grid size over $\lambda$, step size and unlocking schedule.
Two ablations isolate the key ingredient. The first replaces our derived $W$ with fixed hand-designed weightings: a radial soft weighting, and Sobolev $H^s$ with $s=1.5$ as reported by ASASR. If those match ours, the transfer-function analysis is not what drives the improvement. The second recomputes everything under each rasterizer's gradient. This tests whether the inverted bias is a property of silhouette-driven rendering in general or of a particular smoothing approximation.
Why not just…?
Why not just add spatial Laplacian smoothing or precondition the gradients? This is the most serious threat. A Laplacian preconditioner equalizes step sizes over surface modes, but it is blind to the camera. It speeds every low mode equally, including the blind directions it cannot actually identify, and it leaves unchanged any imbalance that comes from what the camera can see. If $J\Phi$ is close to diagonal with smoothly decaying gains, preconditioning will match our loss, and we will say so. The experiment that settles this compares per-mode convergence and final Chamfer distance at equal tuning budget. We also test the combination, since a preconditioner and an image-side metric act on different factors of $K_W$ and could be complementary.
Why not just use an image pyramid or a perceptual loss? A coarse-to-fine pyramid reweights image frequencies too, but in a fixed, monotone way. Our prediction is that the image bands that matter for slow interior modes are shading frequencies that do not order monotonically by scale. MDPG and ASASR argue that spatial and perceptual objectives are dominated by low-frequency energy. We include a pyramid baseline with matched compute. If it conditions $K_W$ as well as our derived $W$, the derivation is unnecessary.
Why not just reparameterize in the spectral basis and truncate, as the attack papers do? Truncation discards high modes. It cannot speed slow low modes, and it says nothing about which modes are unidentifiable. We include band-limited fitting as a baseline. The blind-direction analysis tests directly whether its residual error lies in the near-null space, where truncation is irrelevant.
Why not just use NeRF or Gaussian splats and avoid rasterization? Explicit meshes remain the representation that renders in real time and supports editing, and the file documents splats lacking connectivity (Zhou and Lähner). This proposal studies the mesh regime on purpose. Extending the operator to splats is natural future work, not a competitor here.
Proposed Experiments
Primary testbed: CoMA. The CoMA dataset has 20,466 faces registered to one 5,023-vertex template, so ground-truth per-mode coefficients are exact. We render each target mesh from 4 to 16 known views, fit from a neutral template, and report per-band coefficient error, vertex error in millimetres, and the fraction of vertices within 1 mm. The SMAL animal meshes provide a second template family with articulated limbs.
Real-data benchmarks. DTU is the real-data benchmark, using the protocol and code of Lin et al. ShapeNet airplanes and chairs test transfer to man-made shapes with sharp features.
Decisive checks. The analysis succeeds if three things hold.
- The eigenvalues of $K_W$ predict measured per-mode convergence rates, with a rank correlation reported per rasterizer.
- Adversarial blind perturbations computed through the surrogate change hard-rasterized images by less than their budget while moving vertices by millimetres. Those directions must also absorb a majority of the naive baseline's residual error.
- The derived $W$ reduces error in the visible-but-slow modes faster than both the preconditioner and the fixed weightings at equal tuning budget.
If the predicted inversion does not appear, meaning plain pixel losses already favour low modes, then the spectral-bias part of the thesis is wrong for meshes, and that is a publishable negative. If preconditioning matches the derived $W$ on every check, we keep the diagnostic and blind-direction results and drop the loss as a contribution.
References
- SAGA: Spectral Adversarial Geometric Attack on 3D Meshes
- Universal Spectral Adversarial Attacks for Deformable Shapes
- Exploring the Devil in Graph Spectral Domain for 3D Point Cloud Attacks
- Laplace-Beltrami Operator for Gaussian Splatting
- Neural 3D Mesh Renderer
- Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction
- Differentiable Surface Rendering via Non-Differentiable Sampling
- Multiview Textured Mesh Recovery by Differentiable Rendering
- DDSL: Deep Differentiable Simplex Layer for Learning Geometric Signals
- On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
- NeRF: representing scenes as neural radiance fields for view synthesis
- Boosting the Performance of Video Compression Artifact Reduction With Reference Frame Proposals and Frequency Domain Information
- UltraHR-100K: Enhancing UHR Image Synthesis with A Large-Scale High-Quality Dataset
- Coloring the Noise: Adversarial Sobolev Alignment for Faithful Image Super Resolution
- MDPG: Multi-domain Diffusion Prior Guidance for MRI Reconstruction
- Generating 3D Faces Using Convolutional Mesh Autoencoders
- 3D Menagerie: Modeling the 3D Shape and Pose of Animals