TorchDEQ: A Library for Deep Equilibrium Models

Zhengyang Geng, J. Zico KolterarXiv 2023 · DOI

TL;DR

Deep Equilibrium Models (DEQs) offer a powerful way to build deep learning models with "infinite depth" but are notoriously difficult to train and deploy. This paper introduces TorchDEQ, a comprehensive PyTorch library that standardizes and simplifies the implementation of DEQs, making them more accessible and efficient. By providing a unified framework and a "DEQ Zoo" of existing models, TorchDEQ significantly improves DEQ performance, stability, and training speed across various tasks.

What Problem Does This Paper Address?

Deep Equilibrium Models (DEQs) are a class of neural networks that define their output as a fixed point of a nonlinear system, offering benefits like parameter efficiency and rich representations. However, the practical application of DEQs is hindered by a lack of standardized tools and best practices. Training DEQs can be unstable, backpropagation methods are diverse and complex, and implementing them often requires ad-hoc solutions spread across various research papers. This fragmentation makes it challenging for researchers and practitioners to leverage the full potential of DEQs.

Methods

The core contribution of this paper is the TorchDEQ library, a modular and feature-rich PyTorch framework for building and deploying DEQs. Key aspects include:

  • Unified Interface: TorchDEQ provides a consistent interface for defining DEQ models, abstracting away the complexities of fixed-point solvers and backward pass computations.
  • Flexible Solvers: It supports various solvers for finding the fixed point, including Fixed Point Iteration, Anderson Acceleration, and Broyden's method, allowing users to choose the best fit for their problem.
  • Multiple Backward Pass Options: TorchDEQ implements both Implicit Differentiation (IFT) and Phantom Gradients (PG) for efficient and memory-conscious backpropagation.
  • Normalization and Regularization: The library integrates techniques like Weight Normalization, Spectral Normalization, Jacobian Regularization, and Fixed Point Correction, which are crucial for stabilizing DEQ training and improving performance.
  • DEQ Zoo: A collection of six published implicit models (DEQ Transformer, MDEQ, IGNN, DEQ-Flow, DEQ-INR, DEQ-DDIM) reimplemented using TorchDEQ, serving as benchmarks and examples.

Datasets

The paper evaluates TorchDEQ and the DEQ Zoo models on a variety of datasets across different domains:

  • Language Modeling: WikiText-103
  • Computer Vision: CIFAR-10, ImageNet, KITTI, Sintel
  • Image Generalization, Video Representation, Audio: Not explicitly named, but evaluated using metrics like PSNR.
  • Optical Flow: Sintel, KITTI

Key Insights and Ideas

  • Standardization is Key: The primary insight is that a unified, well-designed library like TorchDEQ can significantly lower the barrier to entry for DEQ research and application, leading to improved performance and stability.
  • Smoothness Matters: The paper emphasizes the importance of a smooth equilibrium landscape, achieved through normalization and regularization techniques, which makes fixed-point solving more stable and efficient, even with simpler solvers.
  • Solver Choice is Crucial: Different solvers and their configurations have a significant impact on performance and efficiency, especially for highly structured or challenging equilibrium systems.
  • DEQ-DDIM Unification: The work demonstrates a novel way to unify diffusion models with DEQs, enabling parallel sampling and faster inversion by treating the diffusion process as a fixed-point problem.

Key Results

  • Performance and Stability Improvements: Across the six models in the DEQ Zoo, TorchDEQ consistently achieved better performance, improved training stability, and increased efficiency compared to their original implementations.
  • DEQ-Transformer: Achieved state-of-the-art perplexity on WikiText-103 with significantly reduced training time (1/3 of original) and memory usage.
  • MDEQ: Demonstrated over 30% time savings and enabled training with fixed-point iteration on ImageNet, leading to a 40% reduction in training time.
  • DEQ-Flow: Achieved state-of-the-art results on optical flow estimation, showing strong out-of-distribution generalization with reduced training iterations.
  • DEQ-DDIM: Showed over 2x speedup in inversion for a complex multi-variate system.

Limitations and Future Work

  • Underestimated Equilibrium Landscape: The authors note that the importance of a smooth equilibrium landscape is often underestimated in prior literature, suggesting future work should focus more on this aspect.
  • Library Extension: TorchDEQ is designed for extensibility, allowing users to register custom solvers, normalization, and regularization techniques, paving the way for future contributions and integrations.
  • Community Building: The paper aims to foster a thriving DEQ community by making these models more accessible and encouraging further research.

Summary updated

Abstract

Deep Equilibrium (DEQ) Models, an emerging class of implicit models that maps inputs to fixed points of neural networks, are of growing interest in the deep learning community. However, training and applying DEQ models is currently done in an ad-hoc fashion, with various techniques spread across the literature. In this work, we systematically revisit DEQs and present TorchDEQ, an out-of-the-box PyTorch-based library that allows users to define, train, and infer using DEQs over multiple domains with minimal code and best practices. Using TorchDEQ, we build a ``DEQ Zoo'' that supports six published implicit models across different domains. By developing a joint framework that incorporates the best practices across all models, we have substantially improved the performance, training stability, and efficiency of DEQs on ten datasets across all six projects in the DEQ Zoo. TorchDEQ and DEQ Zoo are released as \href{https://github.com/locuslab/torchdeq}{open source}.
BibTeX
@misc{geng2023torchdeq,
  author = {Zhengyang Geng and J. Zico Kolter},
  title = {{TorchDEQ: A Library for Deep Equilibrium Models}},
  year = {2023},
  eprint = {2310.18605},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.18605},
}
Figure