Replaying the Whole Pipeline: Honest Significance for Periodicities in Gapped, Red-Noise Light Curves

Many reported periodic signals in astronomy rest on the same three steps. The analyst computes a periodogram (a plot of how strongly the data match a sine wave at each trial frequency). They then estimate the background noise spectrum. Finally, they simulate fake light curves to ask how often noise alone produces a peak that tall. The background is often "red noise," which means random fluctuations that are stronger at low frequencies. The simulations pass through the observatory's real gaps (daylight, moonlight, satellite orbits). This is better than textbook formulas, but it is not the analysis that was actually done on the real data. The real analysis fitted the noise spectrum to the same data that contain the candidate. It often chose the data segment where the signal looked strongest. Increasingly, it filled gaps with a learned generative model before searching. Each of these choices is a degree of freedom that the null simulations usually leave out. Each one can turn a 2σ fluctuation into an apparent 4σ "detection."

This proposal treats the analysis pipeline itself as the thing to be simulated. Under the null hypothesis of no periodic signal, every simulated light curve goes through the identical sequence of steps: the same observing gaps, the same refitting of the noise model, the same segment selection, the same gap-filling, and the same search over frequencies. The p-value is read from the resulting distribution of maximum peak heights. Doing this exactly at the 3–5σ level used to be too expensive. A fast, accurate periodogram changes that. We plan to measure, across real observing cadences, how much significance current practice inflates, and to release a calibrated tool for this regime.

Core Claim

The claim has two parts. First, for gapped light curves dominated by red noise, the standard "plug-in" Monte Carlo test overstates significance in the 3–5σ gray zone by an amount we expect to be practically important. That test fits the noise spectrum once, simulates through the observing mask, and compares peak heights. The overstatement grows with gap severity, with how flexibly the noise spectrum is fitted, and with any data-dependent choice of segment or imputation. Second, full pipeline replay restores nominal false-positive rates at affordable cost. It is made affordable by GPU periodograms plus a fitted extreme-value tail for the rarest outcomes. The first claim is falsifiable: if inflation stays below about 0.2σ across realistic masks, the honest conclusion is that current practice is adequate, and we will report that.

Why This Gap, and Who Already Got Close

The motivating precedent comes from particle physics. Neutrino-4 anomaly: Oscillations or fluctuations? re-analyzed a claimed reactor oscillation signal. The apparent significance was 3.2σ when detector energy resolution was ignored. It fell to 2.7σ once resolution was included and significance was computed with Wilks' theorem (the large-sample shortcut that says likelihood-ratio statistics follow a chi-squared distribution). It collapsed to 2.2σ when the null distribution was simulated explicitly. The lesson is that every shortcut between the real analysis and its null simulation moved the answer in the same, flattering direction. We carry that lesson from a binned counting experiment into continuous time series with gaps. The Neutrino-4 analysis does not address that setting.

Three astronomy papers come closest, and each leaves out a different piece. Revisit the periodicity of SGR J1935+2154 bursts with updated sample passed synthetic Poisson burst sequences through the Fermi/GBM and GECAM visibility windows. It showed that a previously favored 238-day period, along with a 55-day peak, came from the windows themselves, and it moved the preferred period to 126.88 ± 2.05 days. That work supplies our core technique, forward simulation through the empirical mask. However, it is built for one pair of instruments and uses a Poisson rather than red-noise background. Optical Quasi-Periodic Oscillation of Blazar PKS 1440-389 in the TESS Light Curve is the closest instance of current best practice. It fitted the power spectrum, simulated 10,000 matched light curves, and reported 3.1-day and 3.7-day periods above 4σ (99.99%). It also noted that the signal appeared only in part of each sector. Three issues follow. A 99.99% threshold from 10,000 simulations sits at the resolution limit of the simulation set. The noise spectrum was not refitted inside each simulation. The localization to one segment adds trials that the global test does not count. That paper is our main target. A New Period Determination Method for Periodic Variable Stars filtered daily and annual aliases with a window function and analytic false-alarm probabilities that assume white noise. Its period accuracy for contact binaries and semi-regular variables dropped to about 50% and 53% when moved from ASAS-SN/OGLE to ZTF DR2 cadences. That result motivates testing calibration across surveys, not only within one.

Two machine-learning papers constrain the design and enter as pipeline steps to be audited. CSDI fills gaps with a conditional diffusion model, a generator that learns to produce samples by gradually removing noise from random inputs. LSCD adds a differentiable periodogram layer and a spectral-consistency loss, and itself flags the risk of hallucinated oscillations. Neither paper has asked what happens to downstream p-values. Our framing gives a direct answer: if you impute the real data, you must impute the null simulations too. We then measure what goes wrong when people do not. nifty-ls supplies the computational enabling step. It computes periodograms with a non-uniform fast Fourier transform at about 10⁸ frequencies per second on GPU, with median relative errors of 10⁻¹¹ to 10⁻⁸. The commonly used Astropy approximation has 99th-percentile errors that can reach 10% or more, and errors of that size in peak height directly corrupt tail probabilities.

Research Program

The central object is the test statistic produced by the full pipeline. For a light curve $y$ observed at times $t$, call it $T(y, t)$. It is the maximum periodogram power over the searched frequencies and, where the analyst searches them, over candidate time segments. That maximum is taken after the noise model has been refitted to $y$ and after any imputation. Its null distribution is obtained by simulating red-noise realizations $y^{(b)}$ from a noise model fitted to the real data, sampling them at the real times $t$, and recomputing $T(y^{(b)}, t)$ through every step. When fewer than a handful of simulations exceed the observed value $T_{\mathrm{obs}}$, we estimate the tail by fitting a generalized Pareto distribution to the simulated values above a high threshold $u$. That distribution is the standard extreme-value model for exceedances of a maximum. The estimate is

$$ \hat p = \frac{N_u}{B}\left(1 + \xi\,\frac{T_{\mathrm{obs}} - u}{\sigma_u}\right)^{-1/\xi}, $$

Here $B$ is the number of simulations, $N_u$ is the number exceeding $u$, and $\xi, \sigma_u$ are the fitted shape and scale. We use this form because a 5σ two-sided p-value of 2.87×10⁻⁷ cannot be resolved directly with fewer than roughly 10⁷ simulations. We will check the extrapolation against direct simulation down to about 4.5σ before trusting it further.

We will build this as an open tool on top of nifty-ls. The pipeline steps are modular stages that the user declares. One stage samples a power-law or bending-power-law noise process at arbitrary times. Another refits the noise spectrum using the same estimator used on the real data. A third searches segments using the same segmentation rule. An optional imputation stage calls CSDI or LSCD. Uncertainty in the noise model is handled with a parametric bootstrap. Each simulation draws its generating spectrum from the fit's uncertainty and is then refitted. This prevents the null from inheriting a too-confident noise estimate. A rough cost estimate: a TESS sector at 2-minute cadence has about 18,000 points and about 2×10⁵ trial frequencies at nifty-ls's default oversampling. That makes one periodogram take about 2 ms on GPU, so 10⁶ replays take under an hour, plus the cost of refitting the spectrum.

The analysis has two parts. The first is a factorial study that attributes inflation to each omitted step. The second is a cross-survey study using real observing masks: TESS sector gaps, ground-based daily and seasonal gaps from ASAS-SN and ZTF-like cadences, satellite orbit and South Atlantic Anomaly gaps of the kind documented for SGR J1935+2154, and Gamma-distributed spacings like the speleothem records in Comparison of correlation analysis techniques for irregularly sampled time series. That paper also showed that linear interpolation can overstate the persistence of irregular series by up to a factor of two. Persistence is exactly the red-noise property a periodicity test has to model correctly.

Baselines

The naive reference point is the generalized Lomb-Scargle periodogram with an analytic false-alarm probability that assumes white noise. This is how the variable-star pipeline in A New Period Determination Method for Periodic Variable Stars screens candidates. We run it on identical simulated and real light curves with no tuning. Beating it is expected and will not count as a contribution on its own. We include it to show the size of the problem on red-noise backgrounds.

The strongest existing method is plug-in Monte Carlo as performed in Optical Quasi-Periodic Oscillation of Blazar PKS 1440-389 in the TESS Light Curve. It fits the noise spectrum once, simulates light curves through the gaps, and compares peak heights over a fixed frequency grid. That paper reports peaks above 4σ (99.99%) from 10,000 simulations for both sectors. Lacuna has no reported result for the false-positive-rate calibration of this procedure under gapped red noise, so its calibration is something we must measure, not cite. To keep the comparison fair, we give it the same noise-model family, the same masks, the same frequency grid, and the same number of simulations as our replay, including the nifty-ls backend, so that only the omitted steps differ. Our method beats it if two things hold. Its empirical false-positive rate at nominal 3σ and 4σ must lie within the binomial error of the nominal level while plug-in's does not. And at matched empirical false-positive rate, our detection power must be no worse.

The mask-only forward simulation of Revisit the periodicity of SGR J1935+2154 bursts with updated sample, adapted to red noise, serves as an intermediate baseline. The ablations add replay ingredients one at a time on top of it: noise-model refitting, then the bootstrap over noise-parameter uncertainty, then segment search, then imputation. The decisive ablation is noise refitting alone. If adding it closes most of the calibration gap, then that is the key ingredient, and the rest of the machinery is secondary.

Why not just…?

Why not just use the analytic false-alarm probability, which already handles uneven sampling? Analytic bounds assume white noise and treat the window function only through an effective number of frequencies. The SGR J1935+2154 re-analysis shows the consequence: window-generated peaks at 238 and 55 days looked significant until the window was simulated explicitly (Revisit the periodicity of SGR J1935+2154 bursts with updated sample). Our first experiment measures this failure directly on red noise across each mask family.

Why not just run the plug-in Monte Carlo with more simulations? This is the real threat, because more simulations fix the resolution problem at 99.99%. They do not fix the mismatch between the real analysis and its null when the noise spectrum was fitted to the same data. We do not assume that mismatch matters. The ablation that isolates noise refitting decides it. If plug-in with 10⁶ simulations is already calibrated, the replay reduces to "plug-in plus tail extrapolation," and we will say so.

Why not just apply a Bonferroni correction for the number of segments searched? Bonferroni multiplies the p-value by the number of segments. That is valid, but it ignores the strong correlation between overlapping segments and between neighboring frequencies, so it should be conservative and lose power. We will report power at matched false-positive rate for Bonferroni versus replayed segment search. If Bonferroni loses little power for the two-to-three-segment searches typical of TESS sectors, it is the simpler recommendation for that case.

Why not just fill the gaps with a diffusion model and then use ordinary spectral tools? LSCD reports a PhysioNet MAE of 0.211 against CSDI's 0.219, and stable spectral error at 90% missingness (LSCD). These are reconstruction scores, not calibration of downstream tests, and LSCD's own spectral prior is exactly what could add power at preferred frequencies. We test this directly. We impute noise-only light curves with each model, run the plug-in test, and count false 3σ and 4σ detections. We then repeat with imputation inside the replay.

Why not just train an amortized neural posterior on simulated light curves? Compositional simulation-based inference for time series reports that global neural posterior estimation degrades as series grow long. Long series are this regime. A learned density is also least reliable in the far tail, which is precisely where a 5σ statement lives. We use direct simulation where it is affordable and parametric extreme-value extrapolation beyond, both of which can be checked.

Proposed Experiments

The first experiment is calibration on injected nulls. We generate pure red-noise light curves over a grid of spectral slopes, sampled with the TESS Sector 11 and 38 gap patterns, ASAS-SN and ZTF-like ground cadences, satellite orbit masks, and Gamma-distributed spacings. Each goes through every method above. The metric is the empirical false-positive rate at nominal 3σ, 4σ, and 4.5σ. For each test we also report the "effective σ," meaning the nominal threshold that actually achieves the target rate. The decisive check is whether plug-in Monte Carlo's effective σ falls more than 0.2σ below nominal. If it does not, the main claim fails.

The second experiment measures power. We inject sinusoids and transient quasi-periodic bursts, confined to one segment as in PKS 1440-389, at controlled amplitudes. We plot detection rate against false-positive rate for each method. This guards against a calibrated test that is simply blind.

The third experiment audits imputation. CSDI and LSCD are trained on gap-free stretches of red noise and applied to masked noise-only light curves. We count spurious detections with and without imputation inside the replay.

The fourth experiment validates the tail. Direct simulation with 10⁷ replays for a few configurations tests the extrapolated p-values down to about 4.5σ. It also tests the extrapolation's sensitivity to the threshold choice.

The final check applies the replay to the real TESS PKS 1440-389 sectors. Our hypothesis is that the significance will fall below the reported >4σ once noise refitting and segment search are replayed. The comparison is meaningful in either direction. If the significance holds, that is a stronger confirmation of the quasi-periodic oscillation than the original analysis provided. If it falls into the 2–3σ range, this becomes the time-series counterpart of the Neutrino-4 re-analysis.