Scanning tunneling microscopes map how electron waves in a material scatter off atomic defects. The resulting ripple patterns, analyzed by quasiparticle interference (QPI), reveal the material's band structure and, more delicately, two kinds of physics. The first is whether the electrons' rotational symmetry is broken, a state called nematicity. The second is whether the superconducting order parameter changes sign. Neural networks now pull clean single-defect "kernels" out of cluttered multi-defect images in one forward pass. Running them cheaply, at low numerical precision, is attractive for spectroscopic grids with many energies and for closed-loop microscopes. The risk is specific: rounding a network's weights to 4 or 2 bits produces errors that are deterministic and spatially lopsided. Those errors can tilt a decoded fourfold-symmetric kernel into a twofold one, and a physicist would read that tilt as a discovery.
We propose to compress QPI inversion networks with mixed precision, meaning different bit widths for different layers, where the bits are allocated to protect physical observables rather than reconstruction loss. These observables are a nematic anisotropy measure and the sign of the energy-antisymmetrized phase-referenced response. The key observation is simple. Standard sensitivity scores use loss curvature because the trained loss sits at a minimum, where its gradient vanishes. A physical observable is not what the network was trained to minimize, so its gradient does not vanish. Rounding therefore shifts it at first order, and curvature-based allocation can be blind to exactly this drift.
Core Claim
At a fixed average bit budget, allocating precision by the first-order sensitivity of physics observables will keep false symmetry-breaking and phase-sign errors near the full-precision noise floor. It will do so on regimes where loss-curvature allocation in the style of HAWQ-V2 matches it on reconstruction error but not on the observables. The claim is falsifiable in both directions. If observable drift tracks reconstruction error closely under every allocation, the observable-aware score adds nothing and we will say so.
Research Program
The model under study is the two-step latent-alignment variational autoencoder of Ji et al.. A variational autoencoder is a network that compresses an image to a small code and decodes it back. Ji et al. first train one on isolated kernels, with a rotational-symmetry penalty, and then train a second encoder that maps noisy multi-defect images plus defect-location maps into that code through the frozen decoder. It plays the role of the method supplier: we reuse it unchanged at full precision as our reference. It also motivates the gap, because it never examines precision. We rebuild its synthetic data following the described recipe: 50,000 samples from 100 kernel types with 2-, 4- and 6-fold symmetry, polynomial background drift and Gaussian noise. We hold out kernel types for out-of-distribution tests.
We define two observables computed on the decoded kernel $K$. The first is a nematic anisotropy $\eta(K) = (P_x - P_y)/(P_x + P_y)$, where $P_x$ and $P_y$ are Fourier power integrated along the two lattice axes. The second is the sign $s(K)$ of the energy-antisymmetrized phase-referenced response at the characteristic scattering vectors, the quantity Cheung et al. used to argue for sign-changing pairing in NaFe$_{1-x}$Co$_x$As. Computing $s$ requires decoding kernels at paired energies $\pm E$. These are exactly the outputs a physicist would report, and both are lost or corrupted by the phase noise that plagues ordinary Fourier analysis.
For a layer $l$ quantized with step size $\Delta_l(b)$, which scales as the layer's weight range times $2^{-b}$ for $b$ bits, the expected shift in an observable $O$ is dominated by the first-order term:
$$\mathbb{E}\,[\delta O]^2 \approx \frac{\Delta_l(b)^2}{12}\,\big\|\nabla_{w_l} O\big\|^2 \;+\; \big(\nabla_{w_l} O^\top \bar{\epsilon}_l(b)\big)^2 .$$
Here $\nabla_{w_l} O$ is the gradient of the observable with respect to layer $l$'s weights, averaged over calibration kernels. The factor $1/12$ is the variance of uniform rounding noise. The vector $\bar{\epsilon}_l(b)$ is the actual deterministic rounding residual of layer $l$ at $b$ bits. The second term matters because rounding produces a fixed bias rather than zero-mean noise, and a fixed bias in $\eta$ is precisely a false nematic signal. For $\eta$ we compute the gradient on calibration kernels whose true symmetry is fourfold or sixfold, so that any first-order shift is spurious by construction.
The allocation objective adds these per-layer scores, weighted against reconstruction curvature, and solves the bit assignment under an average-bit budget. We use the same Pareto dynamic program that HAWQ-V2 and ZeroQ use, so that only the sensitivity score differs between methods. ZeroQ's data-free calibration is not used: it depends on batch-normalization statistics these networks lack, and real calibration kernels are cheap here because they are synthetic.
A secondary analysis addresses a question the file leaves open. We compare the sensitivity profiles of the alignment encoder and the frozen generative decoder, and ask whether observable sensitivity concentrates in the decoder's final upsampling layers, which carry the fine fringe structure.
Baselines
The naive reference is the full-precision network itself. Its observable spread across noise realizations defines the noise floor. A quantized model "matches" the reference when its false-nematic rate and sign-flip rate fall inside the FP32 bootstrap interval.
The cheapest real competitor is uniform post-training quantization at 8, 4 and 2 bits, with per-channel scales and the same calibration set as every other method. It is the default any practitioner would try first.
The strongest existing allocation method is HAWQ-V2 (Dong et al.), which scores layers by Hutchinson-estimated Hessian trace of the training loss and picks bit widths along a Pareto frontier. Its reported results are 75.92% top-1 on ImageNet for ResNet50 at 7.99 MB, and 34.8 mAP for RetinaNet on COCO. There is no reported result on Lacuna for any quantization method on QPI kernel extraction, so the comparison must be produced here. We run it with the same calibration data, the same number of Hutchinson probes as our gradient samples, and the same dynamic program and budgets.
We also include two output-based scores that are closer in spirit to ours. The first is first-order Taylor channel sensitivity of the loss (Zhao et al.). The second is divergence of decoded outputs, adapted from the KL criterion of Wang et al. as mean-squared kernel deviation. The second is important because it measures output change directly, and it would win if observables simply follow pixel error.
The key-ingredient ablations remove the bias term $\bar{\epsilon}_l$ from our score, and replace observable gradients with observable curvature. Beating the baselines means statistically fewer false-nematic calls and sign flips at equal average bits, without worse reconstruction error. A secondary win is reaching the FP32 noise floor at a lower budget.
Why not just…?
Why not just use uniform INT8 and be done? This is a real threat. INT8 may well preserve both observables, and if so, mixed precision is irrelevant for any hardware that runs INT8 well. Our first experiment measures this directly. The proposal's value then rests on the 4-bit and 2-bit regime needed for small embedded accelerators beside the microscope controller. If observables survive uniform 4-bit too, we will report quantization as safe for QPI and stop there.
Why not just run full-precision models offline on a workstation? For post-hoc analysis this is fine, and we do not claim otherwise. The case for compact on-instrument models comes from closed-loop operation like DeepSPM, which made real-time decisions over 16,000 images and 117 probe repairs in 86 hours. A QPI model in that loop could decide where to spend expensive spectroscopy time. The safety result matters independently of deployment: anyone who compresses a scientific inversion network needs to know whether compression fabricates symmetry breaking.
Why not just symmetrize the decoded kernel over the rotation group? Averaging outputs over rotations removes spurious anisotropy exactly, but it also erases genuine nematicity. It presupposes the answer to the question the measurement asks. FeSe is the test case, as is underdoped NaFe$_{1-x}$Co$_x$As, where Cheung et al. found nematicity that vanishes at optimal doping. Symmetrization is still included as a diagnostic on known-isotropic Ag(111) data.
Why not just skip networks and use sparse blind deconvolution? Cheung et al. recover phase-sensitive kernels without learning, but they solve an iterative manifold optimization from scratch for every map and energy. We use it as an independent physics reference on NaFe$_{1-x}$Co$_x$As rather than as a competitor for throughput.
Decisive Experiments
The first experiment sweeps uniform quantization from 8 to 2 bits on the synthetic held-out kernels. For each bit width we plot reconstruction error against $|\delta\eta|$ and the sign-flip rate of $s$. The decisive question is whether the observables degrade before pixel error does. If they never decouple, the core claim fails and the paper becomes a negative result.
The second experiment compares all allocation methods at average budgets of 3, 4 and 5 bits. It measures false-nematic rate on 4-fold and 6-fold kernels and missed-nematic rate on 2-fold kernels, using thresholds set from the FP32 noise floor. It also measures sign-flip rate, reconstruction error and out-of-distribution performance on unseen kernel types.
The third experiment transfers allocations, chosen only on synthetic calibration data, to experimental maps. On Ag(111), any nematic reading is spurious. On FeSe, true twofold anisotropy must survive. On NaFe$_{1-x}$Co$_x$As, if the data are accessible, the quantized model's antisymmetrized sign must agree with the sparse-blind-deconvolution reference across doping. We will report model size and bit operations, and measured latency only on hardware we actually run. The result that would settle the proposal is a budget at which HAWQ-V2 and our method tie on reconstruction error, while only ours keeps Ag(111) free of false nematic signals and FeSe's genuine anisotropy intact.
References
- HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks
- Quasiparticle Interference Kernel Extraction with Variational Autoencoders via Latent Alignment
- Dictionary Learning in Fourier Transform Scanning Tunneling Spectroscopy
- ZeroQ: A Novel Zero Shot Quantization Framework
- Distribution-Aware Adaptive Multi-Bit Quantization
- Lossless 4-bit Quantization of Architecture Compressed Conformer ASR Systems on the 300-hr Switchboard Corpus
- Artificial-intelligence-driven scanning probe microscopy