Abstract
Deep learning architectures often lack interpretability, functioning as "black boxes." This paper introduces the Frank-Wolfe Network (F-W Net), an interpretable deep structure derived by unrolling the Frank-Wolfe optimization algorithm for $L_p$-norm constrained coding. Unlike traditional sparse coding ($p \le 1$), F-W Net focuses on non-sparse cases ($p > 1$). By introducing a novel learnable nonlinear unit, pool_p, which unifies pooling, activation, and normalization, the network achieves superior performance in tasks like image denoising and super-resolution while maintaining mathematical transparency.
Introduction
In signal processing, coding involves transforming a signal into a representation that satisfies specific constraints. While sparse coding (using $L_0$ or $L_1$ norms) has been extensively studied, many real-world signals are better represented through non-sparse coding ($p > 1$), where information is distributed more evenly across the representation. However, solving $L_p$-norm constrained problems is computationally challenging because standard projection-based methods lack simple analytical solutions for general $p$.
To address this, the authors propose the Frank-Wolfe Network (F-W Net). By "unrolling" the Frank-Wolfe optimization algorithm into a neural network architecture, they create a model that is both fast and interpretable, with structural priors that heuristic models lack.
Methodology
The F-W Net is constructed by mapping $T$ iterations of the Frank-Wolfe algorithm to a $T$-layer feed-forward network. This "unrolling" allows the network to learn optimal parameters via backpropagation, compensating for the truncation error of using a small number of iterations.
The pool_p Unit
The core innovation is the pool_p unit, a nonlinear operator derived from the Frank-Wolfe linear minimizer. It consists of two steps:
- p-conjugate normalization: Normalization based on the Hölder conjugate of $p$.
- p-exponential neuron: An element-wise activation function parameterized by $p$.
This unit is significant because it unifies several classic operations: it behaves like max-pooling when $p=1$, root-mean-square pooling when $p=2$, and average pooling as $p \to \infty$.
Learnable Hyper-parameters
Unlike traditional iterative algorithms where parameters like the norm $p$ and step size $\gamma$ are fixed, F-W Net treats them as learnable. This allows the network to discover the most appropriate $L_p$ prior for a given dataset automatically. For image-based tasks, the architecture is extended to a Convolutional F-W Net, where fully connected layers are replaced by convolutional layers.
Experimental Results
The F-W Net was evaluated on synthetic data, MNIST digit recognition, image denoising (BSD-500/68), and super-resolution (Set-5/14).
- Convergence and Accuracy: On synthetic data, F-W Net converged significantly faster than the original Frank-Wolfe algorithm and outperformed LISTA.
- Image Restoration: A 4-layer Convolutional F-W Net outperformed the classic BM3D algorithm and a 4-layer DnCNN in denoising tasks. In super-resolution, it achieved higher PSNR/SSIM scores than SRCNN and VDSR by learning the optimal $p$ value.
Conclusion
The Frank-Wolfe Network bridges the gap between optimization theory and deep learning. By unrolling a projection-free algorithm, the authors developed an interpretable architecture that excels at non-sparse coding tasks. While currently limited to convex sets ($p \ge 1$), future work aims to extend this framework to non-convex cases and other structural constraints like low-rank priors.
Summary updated