Revisiting Implicit Models: Sparsity Trade-offs Capability in Weight-tied Model for Vision Tasks

Haobo Song, Soumajit Majumder, Tao LinarXiv 2023 · DOI · arXiv

Abstract

Deep Equilibrium Models (DEQs) have gained attention for their memory efficiency and theoretical "infinite depth," yet they remain difficult to train and computationally expensive. This paper revisits the simpler weight-tied model architecture for vision tasks. We show that weight-tied models, when enhanced with a novel "multi-masking" technique, significantly outperform DEQs in both accuracy and inference speed. Multi-masking applies different random sparse masks to shared weights at each iteration, effectively increasing model capacity without adding parameters.

Introduction: The Rise and Challenges of Implicit Models

In the quest for more efficient neural architectures, the machine learning community has explored implicit models, most notably Deep Equilibrium Models (DEQs). DEQs represent an "infinite-depth" network by solving for the fixed point of a single layer's transformation. While theoretically elegant and memory-efficient, DEQs are notoriously difficult to optimize, often requiring complex root-finding algorithms that are computationally expensive and prone to instability.

This research asks whether the simpler precursor to DEQs—weight-tied models—might actually be a more practical choice for computer vision. In a weight-tied model, a single layer is reused for a fixed number of iterations ($K$). While easier to train, these models traditionally suffer from limited expressive power because the weights remain identical across all steps.

Methodology: Weight-Tying and Multi-Masking

The authors propose a refined weight-tied architecture designed to overcome capacity bottlenecks. The core innovation is Multi-Mask Weight-Tying.

Comparison of DEQ and Weight-tied Model structures.

As illustrated in the structural comparison above, while DEQs seek an equilibrium point $z^*$, weight-tied models iterate a layer a fixed number of times. To boost capacity, the authors introduce static, non-trainable boolean masks applied to the weights at each iteration. For a weight matrix $W$, the operation at iteration $i$ becomes $W \odot Mask_i$. These masks are generated deterministically using random seeds, requiring no additional storage. This approach forces the shared layer to extract different features at each step, effectively mimicking the behavior of a deeper, non-tied network while maintaining a low parameter count.

Experimental Results

The proposed models were evaluated on CIFAR-10 and ImageNet32 across three major architectures: Multi-Layer Perceptrons (MLP), ResNets, and Compact Convolutional Transformers (CCT).

Performance comparison (Accuracy vs. Runtime) across MLP, ResNet, and CCT architectures.

The results demonstrate a clear advantage for weight-tied models. Across all tested architectures, weight-tied models achieved higher accuracy than their DEQ counterparts while being significantly faster (often 5x to 10x) in both training and inference. For instance, a weight-tied ResNet-Small with only 35k parameters matched the performance of a standard ResNet-20, which has nearly double the parameters.

Analysis using Centered Kernel Alignment (CKA) confirmed that standard weight-tied models often suffer from high "layer similarity," where features become redundant after a few steps. Multi-masking effectively breaks this pattern, allowing the model to continue learning new features throughout its depth.

Discussion and Conclusion

The findings suggest that for vision tasks, the "infinite depth" of DEQs may be an unnecessary complexity. Simple weight-tied models, especially when augmented with multi-masking, provide a more stable and efficient alternative. Furthermore, the multi-masking approach is compatible with modern hardware-accelerated sparsity patterns (e.g., NVIDIA's 2:4 structured sparsity), making it highly practical for real-world deployment.

Future work includes scaling these experiments to full-resolution ImageNet and exploring whether learned or optimized masks can further enhance performance. Additionally, investigating the applicability of these insights to Natural Language Processing (NLP) remains a promising direction.

Summary updated

Abstract

Implicit models such as Deep Equilibrium Models (DEQs) have garnered significant attention in the community for their ability to train infinite layer models with elegant solution-finding procedures and constant memory footprint. However, despite several attempts, these methods are heavily constrained by model inefficiency and optimization instability. Furthermore, fair benchmarking across relevant methods for vision tasks is missing. In this work, we revisit the line of implicit models and trace them back to the original weight-tied models. Surprisingly, we observe that weight-tied models are more effective, stable, as well as efficient on vision tasks, compared to the DEQ variants. Through the lens of these simple-yet-clean weight-tied models, we further study the fundamental limits in the model capacity of such models and propose the use of distinct sparse masks to improve the model capacity. Finally, for practitioners, we offer design guidelines regarding the depth, width, and sparsity selection for weight-tied models, and demonstrate the generalizability of our insights to other learning paradigms.
BibTeX
@misc{song2023revisiting,
  author = {Haobo Song and Soumajit Majumder and Tao Lin},
  title = {{Revisiting Implicit Models: Sparsity Trade-offs Capability in Weight-tied Model for Vision Tasks}},
  year = {2023},
  eprint = {2307.08013},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.08013},
}
Figure