Error-Tolerant Linear Complexity as a Forecastability Gauge for Solar Irradiance

Solar power forecasters keep building larger models: graph networks across hundreds of sites, sky-camera transformers, multi-stage decomposition ensembles. Yet a plain autoregressive model fitted after removing the daily sun curve often matches them. This proposal borrows one idea from stream-cipher cryptography to explain when, and to act on it. Cryptographers measure how predictable a sequence is by the size of the smallest linear recurrence that generates it, its linear complexity. They also ask how far that size collapses if a few entries are allowed to change. That second quantity, the k-error linear complexity, separates two kinds of hard sequences. In the first, a few points disguise an otherwise simple recurrence. In the second, the difficulty is spread evenly across the sequence.

We claim that the same distinction, carried over to real-valued irradiance, marks a difference that matters operationally. Some cloudy windows are "simple dynamics plus a few sharp ramps." There, a cheap linear forecaster is right almost everywhere and badly wrong at a handful of steps. Other windows are "diffusely turbulent," and no model does much better than persistence. Two such windows can have the same variance and the same average forecast error. They still call for different forecasters and different grid reserve policies. The goal is to compute this profile cheaply from recent history, use it to choose a forecaster and to size reserves, and check against physical measurements that the distinction holds outside software.

Core Claim

Let $x$ be a window of $n$ detrended irradiance samples. Detrending means subtracting the timestamp-wise historical mean, as in the Modified Auto Regressive (MAR) method. Let $H(x)$ be its Hankel matrix, the matrix whose rows are successive lagged copies of the window. Its numerical rank equals the order of the shortest linear recurrence that reproduces the window. We define the error-tolerant complexity profile as

$$ L_k(x) = \min_{e:\ \|e\|_0 \le k} \operatorname{rank}_\tau\big(H(x-e)\big), \qquad k = 0,1,\dots,K, $$

where $e$ is a correction that may alter at most $k$ samples, and $\operatorname{rank}_\tau$ counts singular values above $\tau$ times the largest one. This is the real-valued analogue of the $k$-error linear complexity studied for binary sequences. From the profile we take one summary, the early descent

$$ D = \frac{L_0 - L_{k_0}}{L_0} $$

for a small budget $k_0$ (two to four samples). $D$ is large when a few corrections collapse the apparent complexity, and small when complexity falls off gradually.

The claim has three testable parts. First, high-$D$ windows have forecast errors concentrated in a few steps, so their error tail is heavier than that of low-$D$ windows of matched variance. Second, among low-$L_{k_0}$ windows a linear model is enough, and heavier models gain only in high-$L_{k_0}$, low-$D$ windows. Third, conditioning reserve sizing on $D$ lowers curtailment at equal overvoltage risk compared with a single static error distribution.

Prior Work and Where This Sits

The discrete theory supplies the concept and an exact reference point. Efficient Algorithm for the Linear Complexity of Sequences and Some Related Consequences frames predictability as the smallest annihilating factor of a periodic sequence. Cube Theory and k-error Linear Complexity Profile and On the k-error linear complexity for pn-periodic binary sequences via hypercube theory decompose $2^n$- and $p^n$-periodic binary sequences into structured pieces and locate the descent points where complexity drops abruptly. These works are entirely combinatorial, cover only finite fields and periodic sequences, and stop giving closed forms beyond a few descent points. We use them as ground truth for a binary sanity check, not as the method itself.

The Eigenvalue Complexity of Sequences in the Real Domain carries a related complexity measure to real values with a distance threshold, and validates it only on synthetic chaotic maps. It notes that the threshold is arbitrary and that the measure captures one facet of randomness. We differ by tolerating a sparse set of errors rather than thresholding every distance, and by testing on measured weather data.

On the energy side, three papers motivate the proposal. Modified Auto Regressive Technique for Univariate Time Series Prediction of Solar Irradiance shows that a detrended AR model is competitive with CNNs and LSTMs, but it offers no way to tell in advance when that will hold. Solar Irradiance Anticipative Transformer shows that heavy models gain over persistence yet fail at irradiance extremes. Optimal Dispatch of Residential Photovoltaic Inverters Under Forecasting Uncertainties sizes curtailment through Conditional Value-at-Risk (CVaR, the average of the worst tail of errors) over one static error distribution, entirely in simulation. We reuse its dispatch formulation and replace the static distribution with one conditioned on $D$.

The Value of Multi-stage Stochastic Programming in Risk-averse Unit Commitment under Uncertainty acts as a constraint. It finds that richer uncertainty modelling pays off less as risk aversion grows, so we expect the dispatch gain to be largest at moderate confidence levels and will report it across levels.

Research Program

The first step is computation. Exact $L_k$ is combinatorial, so we trace the profile with a convex relaxation. It minimizes the nuclear norm of $H(x-e)$ plus $\lambda\|e\|_1$, sweeping $\lambda$ and then debiasing on the selected support. As a cheaper alternative, we use greedy removal of the sample whose correction most reduces the trailing singular values. Each run takes milliseconds on 64- to 128-sample windows, which is fast enough for a 1-minute rolling deployment.

We then check the relaxation against exact theory. We binarize windows into ramp-up and ramp-down symbols and treat each $2^n$-length window as one period. We then compare the profile shapes from the relaxation with the exact binary $k$-error profiles from the Games–Chan and cube decomposition. Agreement on descent locations is the test. This binary check is a validation of the tool, not the deployed method. Because real windows are not periodic, we report it as a consistency check rather than as proof.

The second step characterizes the data. We compute profiles on rolling windows of the radiometer streams from Chilbolton, SIRTA and NREL-TSI, using only radiometer data and no images. Where it can be obtained, we add the Peradeniya 10-minute series. The output is an empirical map of how $(L_{k_0}, D)$ is distributed by site, season and cloud regime.

The third step uses the profile. It drives a router that picks between a detrended AR model and a heavier learned model, and it feeds features into conditional quantile forecasts that set time-varying uncertainty sets for the CVaR inverter dispatch. The fourth step is physical validation, described under the decisive checks.

Baselines

The naive reference point is smart persistence: tomorrow's clear-sky-scaled value equals today's. Solar Irradiance Anticipative Transformer reports persistence at 93.33 W/m² one-step RMSE on SIRTA, against 76.94 W/m² for their transformer. On Chilbolton it reports a forecast skill of 21.45% over smart persistence for 15-minute-ahead prediction, versus 17.69% for ECLIPSE. We re-run persistence on our own splits so that every comparison uses the same windows.

The strongest cheap competitor is MAR, ensemble-mean detrending followed by univariate AR. Its reported 10-minute MAPE on Peradeniya is 14.20%, against 14.56% for a CNN and 14.71% for an LSTM. At 30 minutes and 1 hour the LSTM edges ahead, at 19.36% versus 19.94% and 22.18% versus 22.42% (Modified Auto Regressive Technique for Univariate Time Series Prediction of Solar Irradiance). This crossover by horizon is exactly what a router should exploit.

The heavy learned competitor is an LSTM and a small transformer trained on the radiometer series. We give them the same history length and a tuning budget equal to the total hyperparameter trials spent on the router plus AR. We do not retrain the full sky-image transformer, and we cite its numbers only as an external reference.

For dispatch, the baselines are the risk-neutral and static 95% CVaR policies from Optimal Dispatch of Residential Photovoltaic Inverters Under Forecasting Uncertainties. That paper reports 27.83 kWh and 44.05 kWh of curtailment reserve respectively on its 20-house radial network. We first reproduce these figures on the original hourly inputs, then drive the same network with minute-resolution measured irradiance.

Two ablations isolate the key ingredient. The first replaces the profile with $L_0$ alone, which is just robustly chosen AR order. This tests whether error tolerance adds anything over plain complexity. The second computes the profile on raw rather than detrended irradiance, to confirm that the diurnal cycle has to be removed first.

Beating a baseline means one of three things. The router lowers pooled RMSE below both always-AR and always-heavy on held-out sites. Profile-conditioned quantile forecasts lower pinball loss and CVaR calibration error below variance-conditioned ones. Profile-conditioned dispatch uses fewer kWh than static CVaR at an equal or lower measured overvoltage rate.

Why not just…?

Why not just condition on rolling variance or a variability index? This is the default and is cheap. Our central hypothesis is that variance confounds sparse ramp windows with diffusely turbulent ones. The test is direct. Within variance-matched bins, we compare the tail errors of AR forecasts for high-$D$ and low-$D$ windows. If the tails are indistinguishable, the profile is redundant with variance and the claim fails.

Why not just use permutation entropy or real-domain eigenvalue complexity? These measure overall irregularity in a single number. The Eigenvalue Complexity of Sequences in the Real Domain itself concedes that this single facet depends on an arbitrary threshold. Neither measure says whether the irregularity is concentrated in a few points, which is the quantity that bears on tail risk. We include both as router and quantile features, so the experiments show directly whether the profile adds information beyond them.

Why not just train a conditional quantile model on recent history and let it learn the tail? This is the real threat. A flexible quantile network could learn ramp-dominated regimes implicitly. We therefore treat it as a co-baseline rather than a straw man. We train it with and without the profile features under the same budget. If the profile gives no gain in pinball loss or tail calibration, we will report that its value is interpretability and cost rather than accuracy. That is a weaker but still useful result for distribution operators who cannot audit black-box reserves.

Why not stay in simulation? Measured irradiance already enters through the radiometer data. The open question is whether the simulated voltage response to ramp-dominated windows matches what real inverters with ramp-rate limits and control latency do. The experimental-validation literature repeatedly finds idealized models missing exactly this kind of boundary behaviour, for example the oscillatory tails in Effect of Reinforcing Bar on Rayleigh Wave Propagation on Concrete Structures. Hence the bench test below.

Proposed Experiments and Decisive Checks

The first decisive check is tail separation. Pooling Chilbolton and SIRTA radiometer windows, we bin by variance and compare the 95% CVaR of AR forecast error between the top and bottom $D$ quintiles. We use a permutation test and report effect size by site.

The second check is routing. A threshold rule on $(L_{k_0}, D)$, fitted on one site and frozen, chooses between AR and the heavy model on the other sites. It is compared with always-AR, always-heavy, a variance router, an entropy router and an oracle router, by RMSE and forecast skill at 10, 30 and 60 minutes. Recovering at least half the gap between the best single model and the oracle counts as success.

The third check is dispatch. On the reproduced 20-house network driven by minute-resolution measured irradiance, profile-conditioned CVaR is compared with static CVaR. We report curtailed kWh and overvoltage violation counts across confidence levels from 85% to 99%.

The fourth check is physical. We replay matched sets of high-$D$ and low-$D$ windows through a PV emulator feeding a commercial inverter with volt-var control. We compare measured terminal voltage excursions with the simulated network prediction for the same windows. The question is whether simulation error is larger in ramp-dominated windows. If it is, simulation-only reserve sizing is least trustworthy precisely where the profile flags risk.

A null result on the first check ends the project early and cheaply. That is the reason it comes first.