Abstract
In this work, we construct a risk estimator for hard thresholding which can be used as a basis to solve the difficult task of automatically selecting the threshold. As hard thresholding is not even continuous, Stein's lemma cannot be used to get an unbiased estimator of degrees of freedom, hence of the risk. We prove that under a mild condition, our estimator of the degrees of freedom, although biased, is consistent. Numerical evidence shows that our estimator outperforms another biased risk estimator.
Introduction
Hard thresholding is a fundamental denoising technique in signal processing, yet it remains mathematically "stubborn" due to its sharp discontinuities. These jumps break standard risk estimation tools like Stein’s Unbiased Risk Estimate (SURE), which researchers typically use to optimize parameters without knowing the ground truth signal.
In practice, hard thresholding sets any data point smaller than a threshold $\lambda$ to zero while retaining others. To find the optimal $\lambda$, one must minimize the Risk (the mean squared error). However, SURE relies on the function being "weakly differentiable." Because hard thresholding has abrupt jumps at $\pm\lambda$, it is not differentiable, and standard SURE fails. This paper introduces SCORE (Stein COnsistent Risk Estimator), a novel formula that smooths these discontinuities to provide a consistent estimate of the error.
Methodology
The authors propose a clever workaround to the discontinuity problem by following a four-step process:
- Decomposition: They break the hard thresholding (HT) operator into two components: a soft thresholding (ST) part, which is continuous and compatible with SURE, and a jump part ($D$) that contains the problematic discontinuities.
- Gaussian Smoothing: To handle the jumps in $D$, they convolve it with a Gaussian kernel. This "blurs" the sharp edges, transforming the operator into an infinitely differentiable ($C^\infty$) function.
- Applying Stein’s Lemma: Because the blurred version is smooth, Stein’s Lemma can be applied to obtain an unbiased estimate of the smoothed operator's risk.
- Consistency Proof: The authors prove that as the signal length ($P$) grows, the bias of this estimator vanishes if the smoothing bandwidth ($h$) is reduced at an appropriate rate. This makes the estimator consistent, meaning it converges to the true risk in the limit.
The resulting SCORE formula for the Degrees of Freedom is computationally simple: it consists of the number of non-zero coefficients plus a correction term involving exponential functions centered at the threshold boundaries.
Experimental Results
The authors evaluated SCORE using synthetic numerical experiments on a "compressible" vector $x_0$ of length $P = 200,000$, where entry magnitudes decay following a power law. Additive White Gaussian Noise (AWGN) was added to achieve a Signal-to-Noise Ratio (SNR) of approximately 5.65dB.
As illustrated in the figure above, the SCORE curve almost perfectly overlaps with the "True Risk" curve across the entire range of threshold values. This is a significant improvement over previous methods, such as Jansen’s estimator, which tends to deviate wildly from the true risk at lower threshold values. By accurately tracking the true risk curve, SCORE allows for the automatic selection of the optimal $\lambda$ via a simple grid search.
Key Insights and Conclusion
A major contribution of this work is the demonstration that a consistent but biased estimator is often more useful in practice than an unbiased one when dealing with non-continuous operators. The SCORE formula reveals that the risk of hard thresholding is essentially the risk of soft thresholding plus a specific "penalty" for the jumps, which is highest for data points sitting right on the edge of the threshold.
While the current work focuses on hard thresholding in an identity basis, the authors suggest that this "smoothing" framework could be extended to more complex inverse problems, such as deblurring or tomography, and other non-continuous estimators. Future research will also look into more automated ways to select the smoothing bandwidth $h$ for finite-sized datasets.
Summary updated