TL;DR
This paper establishes a unified theoretical framework for Generalized Projected Gradient Descent (GPGD) that spans classical sparse recovery and deep Plug-and-Play (PnP) priors. By analyzing stability under noise, modeling discrepancies, and projection errors, the authors show how tailoring the back-projection operator can neutralize structured outliers and propose a Normalized Idempotent Regularization (NIPR) that stops deep PnP algorithms from drifting or diverging after convergence.
What Problem Does This Paper Address?
Solving underdetermined linear inverse problems $y = A\hat{x} + e$ requires prior assumptions on $\hat{x}$. In classical compressed sensing, this prior is typically sparsity, solved via Iterative Hard Thresholding (IHT). In modern imaging, deep neural networks (like denoisers or autoencoders) act as "projective priors" within Plug-and-Play (PnP) iterative schemes.
Despite their empirical success, deep projective priors often suffer from practical instability: as iterations continue, reconstructions frequently oscillate or blow up instead of stabilizing near the optimal solution. Furthermore, standard PGD uses a fixed back-projection $A^T(Ax - y)$ derived from an $\ell_2$ data-fit, which is fragile against heavy-tailed or structured corruptions (such as sensor saturation or dead pixels). The authors aim to theoretically unify these methods, quantify their stability limits, and devise practical fixes for divergence and outlier vulnerability.
Methods
The authors study generalized projected gradient iterations of the form: $$x_{n+1} = P(x_n) - \mu L(AP(x_n) - y)$$ where $P$ is an approximate projection onto a low-dimensional model $\Sigma$, and $L$ is a flexible linear back-projection operator (replacing the traditional $A^T$).
Key technical components include:
- Unified Convergence Theorem (Theorem 3.1): Proves linear convergence bounded by measurement noise $\|Le\|_2$, model approximation error $d_{P_\Sigma}(\hat{x}, \Sigma)$, and network projection error $\|R(x)\|_2 \le \eta$, driven jointly by the Restricted Isometry Constant $\delta(\mu LA)$ and the Restricted Lipschitz constant $\beta_\Sigma(P_\Sigma)$.
- Structured Noise Back-Projection: For $s$-sparse outliers (e.g., dead pixels), setting $L = A^T S$—where $S$ is a diagonal mask selecting uncorrupted measurements—mathematically zeroes out the outlier error term $\|Le\|_2 = 0$, balancing noise cancellation against measurement loss.
- Normalized Idempotent Regularization (NIPR): Standard neural networks are not true projections because they lack idempotence ($P(P(x)) \neq P(x)$). The authors introduce a regularizer during prior training: $$R_X(P) = \sum_{x \in X} \frac{\|P(P(x)) - P(x)\|_2}{\|P(x)\|_2}$$ Normalizing by $\|P(x)\|_2$ prevents the trivial failure mode where networks minimize idempotence error by collapsing output magnitudes to zero.
[ Noisy / Degraded Measurement y ]
│
▼
Residual: A P(x_n) - y
│
▼
Adapted Back-Projection: -μ L(·)
(e.g., L = A^T S masks out sparse outliers)
│
▼
Gradient Descent Update Step
│
▼
Projective Prior P(·) trained with NIPR
(||P(P(x)) - P(x)|| / ||P(x)|| prevents drift)
│
▼
[ Stable Iterate x_{n+1} ]
Datasets
- CelebA: 10,000 face images ($256 \times 256$) used to train a DRUNET denoiser for super-resolution ($\times 2$), inpainting, and deblurring. Evaluated on a 50-image test split.
- MNIST: 30,000 training images used to train an autoencoder prior evaluated on image inpainting.
- Synthetic Data: $k$-sparse vector recovery with random Gaussian measurement matrices ($m = 150, n = 300$).
- No new benchmark datasets were introduced.
Key Insights and Ideas
- Denoisers diverge because they are imperfect projectors: The approximate projection error $\eta$ acts as a persistent perturbation at each iteration. Without idempotence, GPGD cannot guarantee a strict fixed point, leading to late-stage divergence.
- Stability vs. Identifiability trade-off: Discarding measurements corrupted by outliers ($L = A^T S$) eliminates noise energy, but weakens the restricted isometry property of $SA$. Recovery exhibits a sharp phase transition once outlier count $s$ exceeds a threshold.
- Unnormalized idempotence is flawed: Simply penalizing $\|f(f(x)) - f(x)\|$ encourages the network toward the trivial solution $\alpha f(x) \to 0$. Normalization is essential for stable projective training.
Key Results
To quantify late-stage behavior after reaching the oracle optimal iteration $i_{\min}$, the authors evaluated Stability Metric 1 (SM1, maximum relative error increase) and Stability Metric 2 (SM2, trajectory oscillation):
- Super-Resolution ($\times 2$ on CelebA):
- Unregularized GPGD: PSNR 34.16 dB; at $i_{\min} + 100$, SM1 balloons to 10.25 and SM2 to 0.799 (severe visual breakdown).
- Stochastic Orthogonal Regularization (SOR): PSNR 32.86 dB; SM1 reaches 18.15.
- NIPR (Ours, $\lambda = 0.005$): PSNR 33.35 dB; SM1 remains constrained at 0.77 and SM2 at 0.095, preserving visual fidelity over extended iterations.
- Inpainting & Deblurring: NIPR achieved superior stability across both tasks (e.g., inpainting SM1 at $i_{\min}+100$ was 0.0017 for NIPR vs. 0.1376 for baseline, while improving PSNR to 36.76 dB vs. 35.62 dB).
- Dead Pixel Outliers: Using $L = A^T S$ maintained low reconstruction error up to ~2,000 corrupted pixels before the predicted phase transition degraded recovery.
Limitations and Future Work
- Known Outlier Support: The back-projection adaptation assumes the outlier mask $S$ can be identified beforehand (e.g., detecting saturated pixels), which may be challenging under non-extreme, arbitrary noise models.
- Hyperparameter Tuning: Coarse grid searches were used for the regularization weight $\lambda$; finer adaptive schedules were not explored.
- Future Directions: Combining NIPR with other constraints (such as stochastic orthogonal regularization) and automating noise-support estimation during reconstruction.