How Much to Whiten in Noisy Independent Component Analysis

Independent component analysis (ICA) recovers hidden source signals from their linear mixtures. It relies on the fact that the sources are non-Gaussian: peaky like speech or flat like a uniform carrier. Almost every ICA pipeline starts by "whitening" the data, which means linearly transforming it so its covariance becomes the identity. After that step only a rotation remains to be found. The difficulty is that the covariance also contains the sensor noise. Whitening with it therefore bends the geometry in the wrong way, and FastICA-style methods stay biased even with unlimited data. Voss, Belkin and Rademacher fixed this bias by replacing the covariance with matrices built from fourth-order cumulants, a higher-order statistic that is exactly zero for Gaussian noise (GI-ICA; PEGI). The fix has a price. Fourth-order estimates are far noisier than covariances, so the unbiased methods need roughly $10^4$–$10^5$ samples before they pay off. Practitioners are left with a binary choice: a biased method that is cheap in data, or an unbiased method that is hungry for it.

We propose to make that choice continuous and to let the data pick the setting. The preconditioner, meaning the matrix used to sphere the data before rotation, is a blend of the sample covariance and a fourth-order cumulant matrix. A single mixing weight trades noise bias against estimation variance. The weight is selected on held-out data using a score that Gaussian noise cannot fool: the leftover fourth-order cross-dependence between the separated outputs. The intended result is one procedure that matches classical whitening when samples are scarce or sources are heavy-tailed, and matches PEGI when samples are plentiful. In between, where real recordings such as EEG and audio usually sit, it should beat both.

Core Claim

Assume the data follow $x = As + n$, where $A$ is the unknown mixing matrix, $s$ the independent sources, and $n$ Gaussian noise with covariance $\Sigma_n$. The covariance of the data is $C_2 = AA^\top + \Sigma_n$. Voss et al. showed that the Hessian of the fourth cumulant along a direction $u$ has the form $C_4(u) = A D_u A^\top$. Here $D_u$ is diagonal, with entries proportional to each source's excess kurtosis $\kappa_i$ times $(a_i^\top u)^2$, and $a_i$ is the $i$-th column of $A$. Because Gaussian noise has zero fourth cumulant, no noise term appears in $C_4(u)$. Any matrix of the form $A D A^\top$ with diagonal $D$ makes the columns of $A$ orthogonal after sphering, which is what "quasi-orthogonalization" means. After rescaling $C_4$ to match the trace of $C_2$, the blended preconditioner is

$$M(\lambda) = (1-\lambda)\,\hat C_2 + \lambda\,\hat C_4(u) = A\big[(1-\lambda)I + \lambda D_u\big]A^\top + (1-\lambda)\,\Sigma_n + E(\lambda).$$

Here $\lambda \in [0,1]$ is the blend weight. $E(\lambda)$ is the finite-sample estimation error, and its size grows with $\lambda$, because fourth-cumulant estimates depend on eighth moments while covariance estimates depend only on fourth moments. The equation shows the trade-off directly. The structured bias that breaks orthogonality is $(1-\lambda)\Sigma_n$, which shrinks linearly in $\lambda$. The random error $E(\lambda)$ grows with $\lambda$. Classical whitening is $\lambda = 0$ and Voss-style quasi-orthogonalization is $\lambda = 1$.

Our claim is that a data-driven choice of $\lambda$ gives lower separation error than either endpoint across a wide middle range of sample sizes and noise levels. We select $\lambda$ by minimizing

$$J(\lambda) = \sum_{i \neq j} \widehat{\mathrm{cum}}_{\text{held-out}}\big(y_i, y_i, y_j, y_j\big)^2,$$

where $y = W(\lambda)x$ are the outputs after sphering with $M(\lambda)$ and running a standard fixed-point rotation, and the cumulants are computed on a held-out half of the data.

The criterion is chosen for three reasons. Cross-cumulants of independent outputs are zero. Gaussian noise passed through any linear demixer adds nothing to them. Computing them on held-out data penalizes the variance that a large $\lambda$ introduces. $J$ is therefore an honest proxy for interference, the one quantity both endpoints get wrong in opposite ways.

As a rough expectation, the bias at $\lambda=0$ produces an interference floor proportional to the noise-to-signal ratio that does not shrink with $N$. The error at $\lambda=1$ shrinks like $1/\sqrt{N}$, the standard rate for sample moments, with a constant set by the sources' eighth moments relative to their squared kurtosis. Setting these equal gives a crossover sample size that scales with the square of the ratio between the moment constant and the noise floor. The blend should win in a band around that crossover.

One case is especially sharp. Student-$t$ sources with 5 degrees of freedom appear in the Voss benchmark, and their eighth moment is infinite. The cumulant estimator then has unbounded variance at any sample size. The method should learn to keep $\lambda$ small for such mixtures, while PEGI cannot.

Research Program

The method is built on existing parts. The cumulant matrices and fixed-point rotation come from GI-ICA. For blends that become indefinite, which happens when sources of opposite kurtosis sign give $(1-\lambda)I + \lambda D_u$ negative entries, we use the pseudo-Euclidean iteration of PEGI. PEGI was introduced precisely to handle indefinite cumulant matrices and mixed sub- and super-Gaussian sources without switching heuristics. Our contribution is the one-parameter family, the noise-invariant held-out selection rule, and a finite-sample analysis of where the optimal $\lambda$ lies. The analysis covers both the bias–variance decomposition above and conditions under which $J$ is minimized near the interference-optimal $\lambda$.

There is a second observation, which motivates a smaller extension. PEGI's SINR-optimal demixer is $B = \tilde A^\top\,\mathrm{Cov}(X)^\dagger$, where SINR is the signal-to-interference-plus-noise ratio and $\tilde A$ is the recovered mixing matrix. That formula quietly reintroduces covariance inversion, which is exactly the operation Shiino et al. identify as the "whitening trap": dividing by small sample eigenvalues amplifies noise when dimension is high relative to sample size. We will test a shrunk covariance inside $B$, with the shrinkage chosen by the same held-out logic. This extension is secondary. The main claim does not depend on it.

Two other lines of work constrain the design rather than supply methods. Qiu, Chen and Chen show in errors-in-variables regression that prewhitening with an estimated covariance needs a sample budget growing like $p^2$, versus $\log p$ without it. That is evidence that whitening is a variance cost to be budgeted, not a free default. We will check whether a similar dimension dependence appears in the optimal $\lambda$. Ng et al. show that a structurally sparse mixing matrix allows identification even from purely Gaussian sources. Our method assumes non-Gaussianity and does not address that regime. We state that limitation openly rather than claim coverage.

Baselines

The do-nothing reference is standard covariance (PCA) whitening followed by a one-unit fixed-point rotation in the style of Hyvärinen and Oja. It is run both with a fixed cubic nonlinearity and with the kurtosis sign-switching rule of Gao et al., so that mixed-sign mixtures are handled fairly. This is the $\lambda=0$ endpoint and the method most practitioners actually use. JADE is included as the classical cumulant-after-whitening alternative. FastICA and JADE come from the comparisons in GI-ICA and PEGI, which report that both degrade badly under 25–50% Gaussian noise.

The strongest existing competitor is PEGI, followed by its SINR-optimal demixer. It is the $\lambda=1$ endpoint, it is already unbiased under arbitrary Gaussian noise, and it handles mixed kurtosis signs. Beating it at large $N$ is impossible by design. The question is whether we match it there and beat it at moderate $N$. GI-ICA is included as the older quasi-orthogonalization variant.

The file gives no textual numbers for either method on the synthetic Amari index or SINR benchmark: no reported result found on Lacuna for PEGI or GI-ICA Amari index or SINR values, only figure plots. The only textual facts are that GI-ICA converges to a $10^{-4}$ error threshold in about 4 iterations given enough samples, and processes 100,000 points in 5 dimensions in under a second. We will therefore rerun all baselines ourselves on identical draws of mixing matrices, sources and noise.

Fairness is enforced as follows. Every method sees the same total samples; our method spends half on selection, so every method gets the full set and ours is charged for the split. Every method gets the same fixed-point iteration cap and the same random restarts.

Two ablations isolate the key ingredient. The first is a discrete version that uses $J$ to choose only between $\lambda=0$ and $\lambda=1$. If it matches the full method, the continuum is unnecessary and a simple model-selection rule suffices. The second is an oracle-$\lambda$ version chosen with ground-truth Amari index, which shows how much is lost by the selection rule itself.

Winning means a lower median Amari index than the better of the two endpoints, over 50 random mixing matrices, at a strict majority of (noise level, sample size) cells in the $10^3$–$10^5$ range. It also means being statistically indistinguishable from PEGI at $10^6$ samples.

Why not just…?

Why not just regularize the covariance (shrinkage or adding $\epsilon I$) before whitening? Shrinkage reduces variance in the trailing eigenvalues, which is useful against the whitening trap. It does nothing about noise bias, because adding $\epsilon I$ pushes the matrix further toward $AA^\top + (\sigma^2+\epsilon)I$, away from the quasi-orthogonal form. We include Ledoit–Wolf-style shrunk whitening as a baseline. If it matches our method under nonzero noise, the claim that bias matters at moderate $N$ is wrong.

Why not just estimate the noise covariance and subtract it? When there are more sensors than sources, the trailing eigenvalues of $C_2$ estimate $\sigma^2$, and whitening with $C_2 - \hat\sigma^2 I$ removes the bias at second-order variance cost. That is a serious threat, and we expect it to win in clearly overdetermined, isotropic-noise settings. We run it as a baseline in an overdetermined arm. Our claim is restricted to square mixtures and to non-isotropic Gaussian noise, where no noise subspace exists to read $\Sigma_n$ from. The experiments report the boundary honestly.

Why not just always run PEGI, since it is asymptotically optimal? It is optimal only asymptotically. Voss et al. themselves note that its sample requirements exceed those of second-order methods and that it pulls ahead only in the $10^4$–$10^5$ range. The $t_5$ arm, with infinite eighth moment, is designed to show where the $\lambda=1$ endpoint fails at every $N$ we can afford.

Why not just separate with temporal correlations and skip higher-order statistics? The delayed-Hebbian rule of Clopath et al. avoids whitening altogether. It cannot separate temporally white sources, however, and our synthetic benchmark is i.i.d. by construction. For the speech experiment we report a second-order temporal baseline in that spirit, so readers can see whether time structure alone already suffices there.

Proposed Experiments

Main synthetic experiment. This reproduces the Voss benchmark. Sources are drawn from Laplace, Bernoulli($p=0.5$), $t$ with 5 degrees of freedom, exponential and uniform distributions, mixing super- and sub-Gaussian sources. Gaussian noise is set at 0%, 25%, 50% and 100% of signal variance, with both isotropic and randomly colored covariance. Sample sizes span $10^3$–$10^6$. Mixtures are square, with dimension 5, 10 and 20. Metrics are Amari index and output SINR.

Decisive checks. Three outcomes would settle the claim. First, the selected $\lambda$ should rise with $N$ and with noise level, and should fall when $t_5$ sources are present. If the selected $\lambda$ does not track these factors, the bias–variance account is wrong. Second, $J$ should rank values of $\lambda$ in nearly the same order as the oracle Amari index, measured by Spearman correlation across the $\lambda$ grid. Third, at zero noise the method should collapse to $\lambda \approx 0$ with no loss relative to FastICA.

Dimension sweep. A further sweep at fixed $N$ with growing dimension tests whether the optimal $\lambda$ follows the polynomial-in-dimension penalty suggested by Qiu et al.. The same sweep tests whether shrinking $\mathrm{Cov}(X)^\dagger$ inside PEGI's demixer improves SINR when dimension is a sizable fraction of $N$.

Real data. Two settings follow the file. The first mixes TSP speech recordings with uniform and Laplace interference, as in Bahroun et al., with added Gaussian sensor noise. Here ground truth is known and SINR can be computed. The second uses EEG artifact separation in the 19-channel, 10-second-epoch setup of Jung et al.. That paper reports only qualitative results: line noise isolated in a single sub-Gaussian component, and 8 Hz frontal alpha recovered. We therefore inject known synthetic artifacts into clean segments to obtain a measurable recovery score, and report the selected $\lambda$ alongside.

We expect EEG, with its moderate sample counts and heavy-tailed muscle artifacts, to select an intermediate $\lambda$. If real data instead always select an endpoint, the method has no practical niche, and we would report it as such.