Differentiable Forward and Backward Fixed-Point Iteration Layers

Younghan Jeon, Minsik Lee, Jin Young ChoiIEEE Access 2021 · OpenReview

TL;DR

This paper introduces the Fixed-Point Iteration (FPI) layer, a novel neural network layer that allows for the integration of complex iterative algorithms, like optimization problems, directly into deep learning models. They also propose an efficient backward pass for FPI layers, avoiding explicit Jacobian calculations and enabling practical application to real-world tasks like image denoising and optical flow.

What Problem Does This Paper Address?

Deep neural networks traditionally rely on simple, feedforward operations. However, many real-world problems involve complex iterative processes or constraints that are difficult to encode with standard layers. While some research has explored incorporating optimization problems into neural networks (e.g., differentiable optimization layers), these methods are often premature, requiring complex derivations of backpropagation formulas (like KKT conditions) for each new problem. This makes them difficult to implement and limits their practical applicability. The authors aim to bridge this gap by providing a general, modular, and efficiently trainable layer for incorporating iterative algorithms into deep networks.

Methods

The core contribution is the Fixed-Point Iteration (FPI) layer. This layer takes an input and iteratively applies a parametric function $g(x, z; \theta)$ until it converges to a fixed point $x^*$. This fixed point $x^*$ becomes the output of the FPI layer. The key innovation lies in how they handle backpropagation. Instead of explicitly computing the Jacobian of $g$ for each iteration (which is computationally expensive and memory-intensive), they propose a backward FPI layer. This backward layer, inspired by Recurrent Back-Propagation (RBP), uses a small network module to compute gradients without direct Jacobian inversion. This modularity allows the FPI layer to be treated as a single node in a computational graph, simplifying integration with existing deep learning frameworks and enabling efficient higher-order differentiation.

They also present two specific instantiations:

  • FPI_NN: Here, the function $g$ is an arbitrary neural network module. This allows for complex, learned iterative behaviors.
  • FPI_GD: This uses a gradient descent step as the iterative function $g$, making it suitable for training energy function networks where the goal is to find the minimum of a cost function.

Datasets

  • Flying Chairs dataset: Used for image denoising and optical flow experiments. For denoising, images were cropped and converted to grayscale. For optical flow, the original split was used.
  • Bibtex dataset: Used for multi-label text classification.

Key Insights and Ideas

  • Generalizing iterative algorithms as layers: The FPI layer provides a unified framework to integrate a wide range of iterative algorithms (optimization, numerical methods) into deep networks, making them more powerful and versatile.
  • Efficient backpropagation without explicit Jacobians: The backward FPI layer is a significant contribution, enabling practical training of FPI layers by avoiding the computational burden of explicit Jacobian calculations, a major hurdle in previous related work.
  • Modularity and abstraction: The implementation is highly modular, allowing easy integration with existing layers (convolution, ReLU, etc.) and facilitating experimentation with different iterative functions $g$.
  • Bridging deep learning and traditional methods: The FPI layer offers a way to incorporate prior knowledge and constraints from traditional algorithms into deep learning models, potentially improving performance and interpretability.

Key Results

The paper demonstrates the effectiveness of FPI layers across several tasks:

  • Image Denoising: A single FPI_NN layer outperformed a standard feedforward network with the same structure, especially in noisier conditions. Performance was measured by Peak Signal-to-Noise Ratio (PSNR).
  • Optical Flow: Attaching a small FPI layer to the end of FlowNet [12] resulted in noticeable performance improvements (measured by average end-point error, EPE) with minimal additional parameters and computation.
  • Multi-label Classification: FPI_GD and FPI_NN layers achieved state-of-the-art performance among methods that did not use adversarial data augmentation, outperforming other structured prediction methods like SPEN and ICNN.

Limitations and Future Work

  • Convergence Guarantees: While the paper proves convergence for both forward and backward FPI layers under mild conditions (e.g., $g$ being a contraction mapping), ensuring these conditions are met in practice, especially with learned $g$, might require careful initialization or architectural design.
  • Practical Implementation of $g$: The performance and convergence heavily depend on the choice and implementation of the iterative function $g$. Designing effective $g$ for specific tasks remains an area for exploration.
  • Future Work: The authors suggest that the FPI layer can be applied to a broader range of problems and could potentially enable deep learning in resource-limited environments due to its iterative nature and potential for constant memory usage. They also highlight the potential for FPI to expand the application of deep learning beyond GPU environments.

Summary updated

Abstract

Recently, several studies have proposed methods to utilize some classes of optimization problems in designing deep neural networks to encode constraints that conventional layers cannot capture. However, these methods are still in their infancy and require special treatments, such as the analysis of the Karush-Kuhn-Tucker (KKT) condition, to derive the backpropagation formula. In this paper, we propose a new formulation called the fixed-point iteration (FPI) layer, which facilitates the use of more complicated operations in deep networks. The backward FPI layer, which is motivated by the recurrent backpropagation (RBP) algorithm, is also proposed. However, in contrast to RBP, the backward FPI layer yields the gradient using a small network module without explicitly calculating the Jacobian. In actual applications, both forward and backward FPI layers can be treated as nodes in the computational graphs. All the components of our method are implemented at a high level of abstraction, which allows efficient higher-order differentiations on the nodes. In addition, we present two practical methods, the neural net FPI (FPI_NN) layer and the gradient descent FPI (FPI_GD) layer, whereby the FPI update operations are a small neural network module and a single gradient descent step based on a learnable cost function, respectively. FPI_NN is intuitive and simple, while FPI_GD can be used to efficient train energy function networks that have been studied recently. While RBP and related studies have not been applied to practical examples, our experiments show that the FPI layer can be successfully applied to real-world problems such as image denoising, optical flow, and multi-label classification.
BibTeX
@inproceedings{jeon2021differentiable,
  author = {Younghan Jeon and Minsik Lee and Jin Young Choi},
  title = {{Differentiable Forward and Backward Fixed-Point Iteration Layers}},
  booktitle = {IEEE Access},
  year = {2021},
  url = {https://openreview.net/forum?id=YLOMuYaVgMU},
}
Figure