Towards Model-Size Agnostic, Compute-Free, Memorization-based Inference of Deep Learning

Davide Giacomini, Maeesha Binte Hashem, Jeremiah Suarez, Amit Ranjan Trivedi

TL;DR

This paper introduces Memorization-based Inference (MBI), a novel compute-free approach for deep learning inference that stores pre-computed predictions in a lookup table (LUT). MBI offers model-size agnostic inference with constant time and storage, making it ideal for resource-constrained edge devices, and achieves significant energy efficiency improvements over traditional compute-in-memory methods.

What Problem Does This Paper Address?

The rapid growth in deep learning model complexity, particularly in terms of parameters and computational requirements, poses a significant challenge for deployment on edge devices. These devices have limited power, memory, and processing capabilities, making it difficult to run sophisticated neural networks for tasks like image recognition or real-time decision-making. While existing methods like Compute-in-Memory (CIM) aim to reduce energy consumption, they often still rely on substantial computations and can be tied to specific model architectures or precision levels. This paper addresses the need for a truly compute-free, model-size agnostic inference method that can operate efficiently on edge hardware.

Methods

The core of this work is the Memorization-based Inference (MBI) approach. Instead of performing computations during inference, MBI distills the predictions of a pre-trained deep learning model into a lookup table (LUT). This LUT stores key-value pairs, where keys represent input features (like glimpses of an image) and values represent the corresponding predictions.

Key techniques employed to make MBI practical include:

  • Recurrent Attention Model (RAM): The authors leverage the RAM architecture, which processes only small "glimpses" of the input at a time. This significantly reduces the dimensionality of the input processed at each step, thereby minimizing the size of the LUT required.
  • Incomplete Tables and Bayesian Optimization: To further reduce LUT size, MBI uses incomplete tables, meaning it doesn't need to store every possible input combination. Instead, it searches for the closest match to an input query. Bayesian optimization is used to find optimal distance metrics for this search, improving accuracy by 3-4%.
  • Hierarchical Clustering: To efficiently search the LUT for the closest match, a hierarchical clustering method is used to organize the key-value pairs into a tree structure. This allows for a faster search by narrowing down the possibilities.
  • In-Memory Computing Circuits: The paper proposes specialized analog-domain in-memory computing circuits to perform the distance calculations and searches directly within the memory array. This minimizes data movement, a major source of energy consumption and latency in traditional systems.

Datasets

The primary dataset used for evaluation in this paper is MNIST, a benchmark dataset for handwritten digit recognition. The authors also mention its applicability to more complex models like ResNet20 and MLP-CIM, implying that the underlying principles could be applied to other datasets like ImageNet, though specific results on those are not detailed.

Key Insights and Ideas

  • Compute-Free Inference is Achievable: The most significant insight is that deep learning inference can be made entirely compute-free by pre-computing and memorizing results, moving the computational burden to the training/distillation phase.
  • Model-Size Agnosticism: MBI's reliance on a LUT of distilled predictions, rather than the model's internal parameters, makes it inherently agnostic to the original model's size and complexity. This is a major advantage for edge deployment where model size is a critical constraint.
  • The Power of "Good Enough" Matching: By accepting the "closest match" rather than an exact match in the LUT, MBI significantly reduces the storage requirements, making the memorization approach feasible. Bayesian optimization plays a crucial role in ensuring this "good enough" match still yields high accuracy.
  • In-Memory Computing is Key for Efficiency: The proposed in-memory search circuits are critical for realizing the energy efficiency gains of MBI. By performing computations directly where data is stored, they bypass the energy-intensive data movement between memory and processing units.

Key Results

  • Energy Efficiency: MBI demonstrates substantial energy efficiency improvements:
  • ~2.7x more energy-efficient than multilayer perceptron (MLP)-CIM.
  • ~83x more energy-efficient than ResNet20-CIM.
  • Projected energy per inference for MBI is 13.16 nJ (at 16nm technology node), significantly lower than CIM approaches.
  • Accuracy: For MNIST character recognition, MBI achieves competitive accuracy, especially when using a mixed approach (MBI for simpler cases, traditional DNN for harder ones). The paper notes that accuracy can be maintained even with extreme quantization (1-bit for hidden state, 2-bit for patch vectors). Bayesian optimization improved accuracy by 3-4%.
  • Storage: While MBI's storage overhead for MNIST is noted as worse than some CIM approaches, the paper argues it becomes more storage-efficient for larger, more complex models where CIM would require storing vast numbers of parameters.

Limitations and Future Work

  • Storage Overhead for Simple Tasks: For simple tasks like MNIST, the LUT size can be larger than traditional CIM approaches. The paper suggests that for more complex models, MBI's storage efficiency becomes a significant advantage.
  • Mixed-MBI Thresholding: The accuracy of the mixed-MBI approach is dependent on the threshold value used to decide whether to use MBI or a traditional DNN. A more stringent threshold leads to higher accuracy but less MBI usage, and vice-versa.
  • Future Work: The authors suggest exploring MBI for more complex tasks beyond character recognition, such as object localization or ImageNet classification, which would require larger LUTs but could showcase MBI's scalability advantages. Further optimization of the in-memory computing circuits and exploration of different clustering and optimization techniques are also implied.

Summary updated

BibTeX
@inproceedings{giacomini2024towards,
  author = {Davide Giacomini and Maeesha Binte Hashem and Jeremiah Suarez and Amit Ranjan Trivedi},
  title = {{Towards Model-Size Agnostic, Compute-Free, Memorization-based Inference of Deep Learning}},
  booktitle = {VLSID},
  year = {2024},
  doi = {10.1109/vlsid60093.2024.00036},
}
Figure