Held-Out Photon Audits for Hallucinated Bands in Neural ARPES Processing

Angle-resolved photoemission spectroscopy (ARPES) shines light on a crystal and counts the electrons it knocks out, sorted by energy and exit angle. The result is an image of the material's electronic bands: which energies electrons can have at each momentum. Physicists increasingly clean these images with neural networks. Autoencoders remove noise, U-Nets erase detector grid lines, and self-supervised vision models map domains. All of these networks share a property that the deepfake community knows well. They are very good at producing smooth, plausible structure, and plausibility is not the same as evidence. A denoiser can close a small energy gap, draw a faint band where there were only a few counts, or shift a band edge by a few milli-electron-volts. In this field, a few meV can decide whether a material has crossed a topological transition. Deepfake forensics answered a similar problem by checking content against constraints that real recordings cannot violate. This proposal brings that stance to scientific images, but swaps the constraint. Facial biomechanics are replaced by the physics of photon counting.

The core idea fits in two sentences. Raw ARPES counts are Poisson-distributed, so each dataset can be split exactly into two statistically independent half-datasets by randomly assigning each counted electron to one half. If the neural processor runs on one half and its output is tested against the other half region by region, we get a calibrated map of where the cleaned image claims structure that the independent photons do not support. Treated this way, the audit becomes a localization problem with ground truth. We will build a benchmark of injected and erased band features and score detectors with the localization metrics developed for partial-video deepfakes.

Core Claim

Neural ARPES processing produces "inadvertent synthetic media": outputs that look like measurements but partly come from the network's prior. Reconstruction metrics cannot see this. A held-out-photon likelihood test is a model-agnostic auditor that can. It needs no training on examples of hallucination. It localizes unsupported or erased features in energy-momentum space at a controlled false-discovery rate. It also transfers to processors and feature types that supervised forgery detectors never saw.

The claim is aimed at a gap the research file documents. The ARPESNet autoencoder (An autoencoder for compressing angle-resolved photoemission spectroscopy data) is judged by pixel error, reporting MSE 0.38 and PSNR 41.34 at 128:1 compression. Neither number says whether a specific gap or kink survived. The twin U-Net de-gridder (Removing grid structure in angle-resolved photoemission spectra via deep learning method) explicitly relies on the network's preference for smooth, spatially correlated patterns to fill in signal under the grid. It validates this only by noting that dispersion fits look "nearly identical", and the file records no quantitative discrepancy. These two papers motivate the work: they show that neural cleaning is in use and that it is unaudited. They are also the processors we will audit.

Research Program

Let $N(E,k)$ be the raw count image over binding energy $E$ and parallel momentum $k$. Independent binomial thinning with probability $1/2$ gives two images $N_1$ and $N_2$. If the true intensity is $\lambda(E,k)$, these are independent Poisson images with mean $\lambda/2$ each. This holds exactly for pulse-counting detectors. For analog CCD detectors it holds only approximately, after a gain calibration we will estimate from dark and flat frames.

The processor $f$ runs on $N_1$ and produces an intensity estimate $\hat U = f(N_1)$. A deliberately conservative reference $R$ also runs on $N_1$. It is a kernel smoother matched to the instrument's energy and momentum resolution, so it can blur structure but cannot invent sharp structure. For a candidate region $\Omega$, the audit statistic is the Poisson log-likelihood ratio of the independent half under the two explanations:

$$ S(\Omega) = \sum_{(E,k)\in\Omega} \Big[ N_2 \log\frac{\hat U}{R} - (\hat U - R) \Big], $$

where both intensities are rescaled to the expected count level of $N_2$. Large negative $S$ means the processor's claimed structure in $\Omega$ is less consistent with fresh photons than a blur is. Such a region is either a hallucinated feature or an erased real one. We calibrate $S$ by parametric bootstrap, drawing Poisson images from $R$. We scan regions with a multiscale sliding window and aggregate the resulting p-values with Benjamini–Hochberg false-discovery control. Swapping the roles of the halves and averaging recovers most of the lost statistical power.

PIA (PIA: Deepfake Detection Using Phoneme-Temporal and Identity-Dynamic Analysis) supplies the guiding principle: audit generated content against an invariant that generators do not model. In PIA the invariant is lip closure on /m, b, p/, and removing the viseme stream cost "over 32%" AUC. Our invariant is counting statistics on independent photons. We add one domain-specific physics check: the processed Fermi edge must match a Fermi–Dirac step at the known sample temperature, broadened by the known resolution.

BA-TFD+ (Glitch in the matrix: A large scale benchmark for content driven audio-visual forgery detection and localization) supplies the evaluation design. It reframed deepfake detection from classifying whole videos to localizing edited segments and scoring them with average precision (AP) at increasing IoU thresholds, where IoU (intersection over union) measures how well a predicted region overlaps the true one. We use the same metric family on 2D energy-momentum regions, and extend it to 4D spatial scans in micro-ARPES.

The benchmark, which we call ARPES-Halluc, is built in three tiers. In the first tier, we simulate spectral functions with known band parameters at controlled count levels, then add realistic grid artifacts and Poisson noise. Ground-truth masks mark features that are present in the truth but absent in a paired variant: gaps a few meV wide, band splittings, kinks, flat bands, and thermally populated states just above the Fermi level.

In the second tier, we run real low-count measurements through processors. These are a reimplemented ARPESNet-style autoencoder, a twin U-Net de-gridder, and a diffusion-based super-resolver. Reference labels come from long-exposure acquisitions of the same cut, which serve as a high-count ground truth.

In the third tier, we create adversarial forgeries by generative inpainting of features, following the inpainting style of OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers, into processed images. This tier is a stress test, not the main target. Its scope is set by the limitation stated in the Why-not section on provenance: a forger who also samples fake raw counts from the fabricated intensity defeats any count-statistics test.

Baselines

The naive reference point is to trust the processed image and flag nothing. We also run the common sanity check of thresholding the residual between the processed image and the same raw data it was computed from. We expect this residual to be biased toward zero exactly where the processor overfits. The fairness condition is that it gets the same images and the same region scanner as our method. Beating it means higher localization AP at matched false-discovery rate.

The strongest learned competitor is a supervised forgery localizer transplanted from deepfake forensics. It is a U-Net or MViTv2-style segmentation network in the spirit of BA-TFD+, trained on tier-1 injected features to output region masks. For reference, BA-TFD+ reaches [email protected] of 96.30 on LAV-DF but only [email protected] of 04.44 (Glitch in the matrix, ControlNet/LAV-DF). Those are video numbers, cited only to show how much precise localization degrades at strict thresholds. No reported result was found on Lacuna for any hallucination or forgery detector on ARPES data, so there is no in-domain number to beat. We give this baseline the same simulated data, the same compute, and an equal hyperparameter search budget. Beating it means matching it on feature families and processors it was trained on, and clearly exceeding it on held-out ones.

The third baseline is processor uncertainty: the per-pixel variance of an ensemble of five independently trained processors, plus Monte Carlo dropout. It is tuned on the same validation split.

The fourth baseline is an analytic physics checker alone. It fits the Fermi edge and tests energy-momentum conversion consistency, with no held-out photons.

Two ablations isolate the key ingredient. In the first, $S$ is computed with $N_2$ replaced by $N_1$ (no independence). In the second, $R$ is replaced by a constant local mean (no conservative reference).

Why not just…?

Why not just train a classifier on fake versus real spectra? It is the most serious competitor, and on in-distribution injections it may well match or beat the audit. Its weakness is that it learns the signatures of the forgeries it was shown. Hallucinations depend on the processor, and new processors appear constantly. The held-out-family and held-out-processor splits in our experiments settle this question directly. If the supervised localizer keeps its AP on processors it never saw, the audit's main advantage disappears and we will report that.

Why not just use ensemble uncertainty? Ensemble variance measures disagreement between networks, not error. Networks trained on the same data share the same smoothness prior, which is precisely the bias that the de-gridding paper relies on. So they can agree confidently on an invented band. The tier-2 comparison will show whether variance tracks the long-exposure ground truth as well as $S$ does.

Why not just sign raw data at the instrument and require deposition? For deliberate fraud, cryptographic provenance is the right answer and beats any statistical test. A forger who samples fake raw counts from a fabricated intensity is undetectable by our method. We state this plainly and treat provenance as complementary. Signing does nothing, however, about honest pipelines whose processing step hallucinates, and that is the case this proposal targets.

Why not just compare the output to the raw data directly? The processed image was computed from those same photons. A residual against them rewards a processor that has absorbed the noise into fake structure. The no-independence ablation measures exactly how much this double-dipping costs.

Proposed Experiments

The first decisive check is calibration. On tier-1 images with no injected change, the empirical false-discovery rate must stay at or below its nominal level across count levels spanning two orders of magnitude. If it fails, the bootstrap null is wrong.

The second check is localization. We report AP at IoU 0.5, 0.75, and 0.95 on ARPES-Halluc, split by feature family and by processor. The audit must beat every baseline on held-out processors.

The third check is scientific. We take a feature whose exact value matters and measure how often the processed image moves it without the audit noticing. The case modeled on Change of Fermi-surface topology in... is the cuprate antibonding band sitting at −7 meV versus +2 meV, which decides whether the Fermi surface is hole-like or electron-like. That case recovered states only about 60 meV above the chemical potential, by dividing out the Fermi function at 20 meV resolution. The case modeled on Observation of Fermi arc and its connection with bulk states in the candidate type-II Weyl semimetal WTe2 is the thermally populated surface state at 165 K. In simulations matched to these parameters, we measure the rate of undetected sign flips in the band position and undetected arc creation or erasure, as a function of count level.

The fourth check covers 4D micro-ARPES spatial scans of the kind used in Transfer Learning Application of Self-supervised Learning in ARPES, where DINO reached 92.94% domain-classification accuracy but unsupervised clustering reached only 50–60%. We test whether audit flags on the processed scans change the domain map that downstream tools produce.

The idea fails if a supervised localizer generalizes equally well across processors, or if CCD gain miscalibration breaks the Poisson null on real data.