Measuring Soil Metal Immobilization from Censored Data Without a Known Covariance

When a farmer spreads lime-like oyster shell powder or biochar on cadmium-polluted soil, success means that the fraction of metal a crop can absorb drops. Soil labs usually cannot measure that drop cleanly. Many readings fall below the instrument's detection limit, and large surveys often record only which regulatory class a sample falls into ("clean", "agricultural grade", "hazardous"). Remediation works by pushing metal out of the measurable mobile pool, so the better a treatment works, the more of its evidence disappears into "below detection." Recent theory shows how to recover a Gaussian mean exactly when you only see which bin each sample landed in, but only if the covariance of the data is known in advance. Covariance here means how the metals co-vary: cadmium, lead, and zinc readings move together in soil. That assumption never holds in soil.

We propose a two-stage estimator. It learns the covariance from a small set of fully measured samples, such as laboratory mass-spectrometry assays. It then runs a likelihood-based estimator on the much larger set of coarse or censored readings, treating the learned covariance as given. It reports a confidence region whose constants are written out and actually computed, instead of hidden in $O(\cdot)$ notation. The claim is that, with explicit constants, this region is informative at soil-study sample sizes: hundreds of sites and eight metals. We also claim it makes clear when a treatment's effect has become statistically invisible because too much of it sits below detection.

Core Claim

Mean Estimation from Coarse Data supplies the method we build on. It gives a polynomial-time algorithm for estimating a Gaussian mean from convex coarse observations: projected stochastic gradient descent on the negative log-likelihood, with sample clipping, at $\tilde{O}(d/\varepsilon^2)$ samples. It also proves a "Slab Theorem": a partition is non-identifiable exactly when almost every cell is an infinite slab along some direction. The algorithm requires the covariance to be known, and the constants in its local-growth analysis are left unsharpened. Soil censoring fits this setting well. Per-coordinate detection limits and regulatory thresholds cut space into axis-aligned rectangles, which are convex. Correlated metals, meanwhile, make the covariance far from the identity.

Our claim has three parts. First, the known-covariance requirement can be replaced by a covariance estimated from $m$ fully measured samples. The cost is an additive error term we can bound explicitly. Second, the Slab Theorem becomes quantitative for immobilization studies. As a treatment pushes nearly all samples of one metal below detection, the cells degenerate into slabs along that axis, and the confidence region along that axis must grow. We measure how fast. Third, an isotropic or diagonal covariance plug-in is not merely inefficient here. It is biased, because censoring one coordinate carries information about correlated coordinates only through the off-diagonal covariance.

Formally, let $x \in \mathbb{R}^d$ be the vector of log-concentrations at a site, with $x \sim \mathcal{N}(\mu, \Sigma)$. We observe $m$ exact draws and $n$ coarse draws, each reported only as the cell $S(x)$ of a known partition. We form $\hat\Sigma$ from the exact draws. We then estimate $\hat\mu$ by minimizing the coarse negative log-likelihood with $\hat\Sigma$ in place of $\Sigma$. The target guarantee is that, with probability at least $1-\delta$,

$$\|\hat\mu - \mu\|_{\Sigma^{-1}} \le \frac{1}{\lambda_S}\left( A\sqrt{\frac{d + \log(1/\delta)}{n}} + B\,\|\Sigma^{-1/2}\hat\Sigma\Sigma^{-1/2} - I\|_{\mathrm{op}} \right),$$

where $\|v\|_{\Sigma^{-1}}$ is the Mahalanobis norm (distance measured in units of the data's own spread). The quantity $\lambda_S$ is the local strong-convexity (curvature) constant of the coarse likelihood, which shrinks toward zero as the partition approaches a slab. The constants $A$ and $B$ are explicit numbers. The operator-norm term scales like $\sqrt{d/m}$ by standard Gaussian covariance concentration, which is where the precise subset's size enters. The contribution is not the $O$-rate, which is essentially forced. The contribution is computing $A$, $B$, and a data-dependent estimate of $\lambda_S$ tightly enough that the bound is non-vacuous at $d=8$, $n$ in the hundreds, and $m$ in the tens.

Research Program

The theoretical part has three pieces. We first carry the perturbation of $\Sigma$ through the local-growth argument of the coarse-data paper, tracking every constant. We then replace its generic concentration steps with the sharpest available one-dimensional tail bounds. We borrow the style of Learning with Square Loss: Localization through Offset Rademacher Complexity, which shows that a quadratic offset term removes peeling slack; its authors explicitly leave their own constant $c = 1/18$ unoptimized, and they rely on a lower-isometry condition on the design covariance. This paper supplies technique and a caution: our curvature constant $\lambda_S$ plays the role of that isometry condition and can degenerate. Finally, we use High-probability minimax lower bounds, which frames risk as a $(1-\delta)$ error quantile rather than an expectation, to derive a two-point lower bound for the censored-rectangle model. The ratio of our upper bound to this lower bound then states the remaining slack as a number.

The algorithmic part implements the two-stage estimator, a data-driven estimate of $\lambda_S$ taken from the observed Hessian of the coarse likelihood, and a certified confidence ellipsoid. For the two-sample question that matters to remediation, treated versus control plots, we apply the same machinery to the difference of means.

The empirical part tests whether the certified regions cover the truth at the stated rate, and how wide they are relative to what is actually achievable.

Baselines

The naive reference point is what environmental labs commonly do: substitute half the detection limit for non-detects (and bin midpoints for classed data), then take the sample mean. We run it on the same coarse and exact samples, pooled. It beats nothing if its bias along a censored metal exceeds our certified radius, which we expect precisely in strongly treated plots.

The second baseline is the sample mean of the precise subset alone, with a standard Hotelling-type confidence region. This is the honest "ignore the coarse data" competitor. Our method beats it only if its region is materially smaller at equal coverage. That requires the coarse data to add information beyond the covariance estimate, and we report the break-even ratio $n/m$.

The strongest existing method with a guarantee is the projected gradient estimator of Mean Estimation from Coarse Data, run with an identity or diagonal covariance, since practitioners lack the true one. It gets the same partition, samples, and iteration budget. We also run it with the oracle $\Sigma$ as an unattainable upper reference. That paper reports only rates, not numerical errors, so no reported result was found on Lacuna for its error on any environmental or multivariate benchmark. The comparison is therefore entirely our own runs.

The ablation isolating the key ingredient keeps our pipeline fixed and swaps the covariance plug-in across four choices: identity, diagonal from the precise subset, full $\hat\Sigma$, and oracle. Success means $\hat\Sigma$ closes most of the gap between diagonal and oracle, and diagonal shows measurable bias on correlated censored coordinates. A second ablation compares our explicit-constant region against the same region with textbook constants, to show the tightening is what makes it non-vacuous.

Why not just…?

Why not just fit the censored multivariate Gaussian jointly by expectation-maximization (EM), estimating mean and covariance together from all data? This is the real threat, and it is standard practice for non-detects. Joint EM uses the coarse data for the covariance as well, so in easy regimes it may match or beat our point estimate. We include it as a baseline with identical data. Where it plausibly falls short is in two places. Its likelihood is non-convex in $(\mu, \Sigma)$ jointly, with no finite-sample guarantee. And near-slab partitions are exactly where the Slab Theorem says the problem becomes ill-posed, so EM will return confident-looking but poorly supported answers. If EM's point accuracy equals ours everywhere, we will say so. The contribution then narrows to the certified region, and we would evaluate "EM point estimate plus our certificate" as the recommended pipeline.

Why not just use the precise subset and skip the coarse data? If the coarse readings carry little information, for example when nearly everything is below detection, this wins. Our bound predicts this, because $\lambda_S$ collapses. The experiment sweeps $n/m$ and censoring severity and reports where the crossover lies. That crossover is itself a useful design rule for how many expensive assays a field study needs.

Why not just assume identity covariance, as the existing theory does? Soil metals are strongly co-varying. That is why the multi-metal tensor model of A prediction model for soil heavy metal content based on improved tensor completion, which fills in a latitude by longitude by metal-type data cube by assuming it has low-rank structure, gains accuracy by exploiting cross-metal correlation. Under rectangular censoring, a misspecified correlation moves the likelihood's optimum, so identity is biased, not just noisy. The covariance-swap ablation measures this directly.

Proposed Experiments

The synthetic stage reuses the covariance regimes and heavy-tailed distributions of Robust High-Dimensional Mean Estimation With Low Data Size: identity, spherical, diagonal, and arbitrary covariance, plus multivariate $t$ and Laplace draws. That paper found that asymptotically calibrated thresholds break down at small $n$ and must be recalibrated. It motivates our insistence on explicit constants, and it warns that Gaussian-derived certificates may undercover on heavy tails. We impose detection-limit censoring at increasing quantiles and three-class binning, then measure Mahalanobis error, the empirical 95th-percentile error, coverage of the nominal 95% region, and the ratio of certified radius to empirical quantile. The target is coverage at or above nominal on Gaussian data with a ratio small enough to be useful. We report the undercoverage on $t$ data honestly.

The semi-synthetic stage uses fully measured multi-metal field data. One source is the 516-sample, eight-metal Huanghua survey from Estimation of Heavy Metal Content in Soil Based on Machine Learning Models, whose LASSO-GA-BPNN predictor reports MAPE below 13% for most metals. That figure is for site-level prediction, a different task, and is cited only to show the data carry a strong signal. The other source is the 319-site, eight-metal dataset of the tensor-completion paper. Availability of both must be confirmed with their authors. We take the full-data mean of log-concentrations as ground truth, designate a random precise subset of size $m$, and censor the rest at realistic detection limits and regulatory classes.

To mimic immobilization, we shift the cadmium coordinate downward, and correlated metals proportionally, by amounts matched to the five-year oyster-shell trial in Long-term oyster shell powder applications increase crop yields and control soil acidity and cadmium. That trial reports available Cd 7.96% to 34.65% below control and a decrease of 0.06 to 0.12 mg/kg per 0.1 pH unit. The shift pushes a growing share of samples below detection. We then estimate the treatment effect as a difference of means between shifted and unshifted halves. In this setting immobilization acts like the subtractive corruption that the low-data study found hardest for robust estimators. The results should show whether our certified region still contains the true effect.

The decisive checks are three. First, the full-$\hat\Sigma$ estimator must beat the diagonal and identity plug-ins in bias on correlated censored metals. Second, the certified region must be both valid and narrower than the precise-subset-only region over a stated range of $n/m$. Third, as censoring approaches a slab, the reported $\lambda_S$ and the region width must grow before the estimate's error does. The method should warn that it can no longer see the treatment effect, rather than silently report a false one. If joint EM matches on all three, the paper reframes around the certificate alone, as stated above.