Many machine-learning papers in chemistry, physics and 3D vision state that their model is "equivariant." An equivariant model is one whose output rotates or shifts in lockstep with its input, so rotating a molecule rotates the predicted forces by the same amount. These statements are rarely checked after publication. A careless preprocessing step, a grid interpolation, or a wrong sign convention can silently break the property while the code still runs and the headline numbers still look good. When this happens, later work builds on a claim that is false, which is exactly the uncorrected-error problem that scientific-record correction mechanisms exist to catch.
We propose an executable audit for one narrow, checkable class of published claims: statements that a model respects a continuous symmetry, such as 3D rotations or rotations plus translations. The audit does not ask a language model to judge the claim. It runs the released model. It searches for the worst rotation of the worst input, parameterizing rotations through their flat "Lie algebra" coordinates (the small-step generators that the exponential map turns into rotations). It calibrates the violation it finds against a numerical noise floor. It then emits a structured, versioned correction record stating whether the claim holds exactly, holds only on average, or fails, and by how much.
Core Claim
Our central claim is that worst-case search beats random spot-checks at finding real equivariance violations. Random spot-checks evaluate the model under a handful of random rotations, which is what a careful reviewer would do today. A worst-case search over rotations, parameterized in the Lie algebra, should find violations that a matched random-sampling budget misses.
The theory in the file motivates this. Achieving Approximate Symmetry Is Exponentially Easier than Exact Symmetry proves that for finite groups, approximate symmetry measured on average (an $L^2$ sense) needs only $O(\log |G|)$ random group elements, while exact symmetry needs on the order of $|G|$. The file flags two open questions about that result. Uniform, worst-case ($L^\infty$) symmetry is suspected to lose the logarithmic shortcut, and the result has not been extended to continuous groups. Published papers almost always assert exact equivariance, which is a worst-case statement. Random spot-checks therefore certify the wrong quantity: they can pass a model that is equivariant on average and badly wrong on a thin set of orientations.
Our proposal is the empirical counterpart of that open problem, applied to auditing the literature. We measure how far average-case testing and worst-case testing diverge on real released checkpoints, and we turn the gap into correction records.
Concretely, for a model $f$, an input $x$, and a rotation $g = \exp(\xi)$ generated by a Lie-algebra vector $\xi$, the violation is
$$\delta(x, \xi) = \frac{\| f(\exp(\xi)\cdot x) - \rho(\exp(\xi))\, f(x) \|}{\| f(x) \|},$$
where $\rho$ is the way outputs are supposed to transform. For example, $\rho$ is the identity for an invariant energy, the rotation matrix itself for forces, and Wigner D-matrices for higher-order quantities such as Hamiltonian blocks. A random audit reports the mean of $\delta$ over sampled $\xi$. Our audit reports $\max_{x,\xi} \delta$, found by gradient ascent on $\xi$ and on small perturbations of $x$ within the data distribution. Searching in flat Lie-algebra coordinates avoids the singular points of Euler-angle parameterizations, the same reason Diffusion Generative Modeling on Lie Group Representations works in the algebra rather than on the curved group. That paper supplies the method idea; we use it for search rather than generation.
Research Program
The audit separates three kinds of claim that papers usually blur together, because each needs a different test and a different correction.
The first kind is a property claim about the model ("our network is SE(3)-equivariant"). This is tested by the worst-case search above. A raw maximum is meaningless without a reference point: correctly built equivariant networks show violations around float32 rounding error, and steerable architectures using spherical-harmonic tensor products accumulate more. We therefore calibrate each model against itself. We rerun the same architecture in float64 and also evaluate it on exactly representable rotations (multiples of 90°), which gives a per-model noise floor $\epsilon_0$. A violation is reported as a failure only when it exceeds $\epsilon_0$ by a margin set from a held-out set of known-correct models, at a fixed 1% false-alarm rate.
The second kind is an assumption claim about the data ("the task is rotation-invariant"). The third is a benefit claim ("equivariance improves accuracy"). For these we reuse the diagnostics of To Augment or Not to Augment? Diagnosing Distributional Symmetry Breaking. That paper introduced a two-sample classifier score $m(p_X)$ for whether a dataset is canonically aligned and a task-dependent score $t(p_{X,Y})$ for whether orientation carries label information. It showed that forcing symmetry on aligned data such as ModelNet40, where $m(p_X)$ is about 94%, can hurt accuracy. That paper constrains our design: a benefit claim made on an aligned benchmark cannot be taken at face value even when the property claim holds. Its gap is also our opening, since it diagnoses datasets but never connects the diagnosis to specific published claims or issues corrections.
The output is a correction record modeled on the version-control view of Treat societally impactful scientific insights as open-source software artifacts. That paper argues that scientific claims should carry continuous-integration tests and versioned status, but it stays at the workflow level with no formal test for any mathematical claim. Each record names the claim, its type, the checkpoint hash, the calibrated worst-case violation with the input and rotation that witness it, the dataset scores, and a status: holds, approximately holds, or refuted. A refuted claim comes with a reproducible counterexample, which addresses the transparency gap reported in Retractions relevant to COVID-19: does the retraction rate jump during the pandemic?, where over 30% of retraction notices gave no clear reason.
Severity matters as much as existence. A violation of $10^{-4}$ in an energy prediction may be scientifically irrelevant. We therefore attach to every refutation the downstream effect of the violation: the change in task error when inputs are presented in the witnessed bad orientations rather than in their native frames.
Baselines
The naive reference point is to accept the published claim, which amounts to auditing nothing. Every audit method is scored against this on how many planted and real violations it surfaces.
The strongest competitor is a random-rotation spot-check. It draws $K$ rotations uniformly from the Haar measure (the uniform distribution over rotations), evaluates them on held-out inputs, and reports the mean and maximum violation. This is what a diligent reviewer would run. The file's theory suggests it is very efficient for average-case questions: on a synthetic 20-dimensional sign-flip task with about a million group elements, Achieving Approximate Symmetry Is Exponentially Easier than Exact Symmetry reports that 32 random elements captured almost the entire benefit of symmetry. That number concerns training benefit, not detection, so no reported result was found on Lacuna for random-sampling detection of equivariance bugs. For fairness, the spot-check gets exactly the same number of model evaluations as our search, including the gradient evaluations counted at their cost in forward passes. It uses the same held-out inputs and the same noise-floor calibration, and we sweep $K$ up to the full budget. Our method beats it if, at a matched budget and a matched 1% false-alarm rate, it detects a significantly larger share of planted violations, especially the localized ones described below. If the two tie, that is itself a publishable answer to the open worst-case question.
The third baseline is a frontier language-model reviewer. It receives the paper text, the code, and the ability to execute code, and must decide whether each symmetry claim holds. There is no reported result on Lacuna for this exact task. The nearest evidence is weak verification on neighboring tasks: 0.501 on the Verification subtask of AutoSupervision: Closing the Feedback Loop in Scientific Workflows with Grounded Revision Verification, and 77.7% accuracy for o4-mini against 93.8% for human experts on SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification. The agent gets the same compute budget for code execution as our audit.
Two ablations isolate the key ingredients. The first replaces Lie-algebra search with Euler-angle gradient search. The second removes noise-floor calibration and uses a single global threshold instead.
Why not just…?
Why not just run a few random rotations? This is the real threat, and the experiment is designed to settle it. Random sampling should catch bugs that break symmetry everywhere, such as uncentered coordinates or a global axis swap. It should miss bugs that break symmetry only on a small set of orientations. Examples include voxel or grid interpolation artifacts aligned with the axes, which Romero et al. identify as a source of broken equivariance in discretized group averaging, and cutoff or neighbor-list discontinuities that only appear for particular orientations. If the planted-bug suite shows random sampling matching our search on these localized bugs as well, we will drop the search, recommend the cheaper test, and keep only calibration and the claim typology as contributions.
Why not just ask a frontier model to read the paper and code? The cited benchmarks show that such models are weakest exactly where this task lives. They judge whether evidence is sufficient at about 0.501 on AutoSupervision Verification, and they lean toward "Supported" verdicts and toward text over contrary evidence, as reported in MuSciClaims: Multimodal Scientific Claim Verification, where most models score 0.3 to 0.5 F1. An equivariance claim is a numerical property of a checkpoint, and reading the paper cannot settle it. Our agent baseline still lets the model run code, so if it learns to write a good test it is credited for that.
Why not just use a data-science linter? PYRA : A high-level linter for data science software analyzes code and dataset structure to catch type and plotting incompatibilities. It has no notion of group actions. Many symmetry bugs are also numerically legal code, so no static rule fires on them. We include PYRA-style static rules (for example, detecting absolute-coordinate features) as a cheap pre-filter and report how many planted bugs they catch on their own.
Proposed Experiments
The first experiment is a planted-violation suite built on public equivariant baselines. One is QHNet on QH9: A Quantum Hamiltonian Prediction Benchmark for QM9 Molecules, where QHNet reports a mean absolute error of $76.31 \times 10^{-6}$ Hartree on QH9-stable-id and outputs transform by Wigner D-matrices. The others are equivariant and augmentation-trained point-cloud and molecular models on ModelNet40, QM9 and OC20, the datasets used in the distributional symmetry paper. We inject bug families of graded locality: a global frame leak, a Wigner D-matrix convention error affecting only higher-order output blocks, axis-aligned grid interpolation, orientation-dependent neighbor cutoffs, and float16 accumulation. We measure detection recall at a 1% false-alarm rate as a function of evaluation budget, separately for global and localized bugs.
The decisive check is whether the curve for worst-case search and the curve for random sampling separate on localized bugs while converging on global ones. That pattern would be the empirical sign of the suspected gap between average-case and worst-case testing on a continuous group.
The second experiment audits real released checkpoints from the same benchmarks. For every refuted property claim we report the witnessed rotation and the downstream error change. For benefit claims made on ModelNet40 or similar aligned data, we report $m(p_X)$ and $t(p_{X,Y})$.
The third experiment is a human check. Domain experts review every refutation and every "holds" verdict near the threshold. Precision of refutations is the metric that decides whether these records could responsibly be attached to papers as corrections.
References
- Achieving Approximate Symmetry Is Exponentially Easier than Exact Symmetry
- Diffusion Generative Modeling on Lie Group Representations
- To Augment or Not to Augment? Diagnosing Distributional Symmetry Breaking
- Treat societally impactful scientific insights as open-source software artifacts
- Retractions relevant to COVID-19: does the retraction rate jump during the pandemic?
- AutoSupervision: Closing the Feedback Loop in Scientific Workflows with Grounded Revision Verification
- SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification
- Co-Attentive Equivariant Neural Networks: Focusing Equivariance On Transformations Co-Occurring in Data
- MuSciClaims: Multimodal Scientific Claim Verification
- <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" altimg="si1.svg"> <mml:mrow> <mml:mi mathvariant="normal">PYRA</mml:mi> </mml:mrow> </mml:math> : A high-level linter for data science software
- QH9: A Quantum Hamiltonian Prediction Benchmark for QM9 Molecules