TL;DR
This paper introduces COMPASS, a novel compiler framework designed to enable large deep learning models to run on resource-constrained in-memory computing (IMC) accelerators. By intelligently partitioning neural networks and managing weight reloading from external memory, COMPASS significantly boosts throughput and reduces energy consumption compared to existing methods.
What Problem Does This Paper Address?
Deep learning models are growing rapidly in size and complexity, demanding significant computational power and memory bandwidth. Traditional Von Neumann architectures struggle with this due to the energy and time overhead of moving data between processing units and memory. In-memory computing (IMC) architectures, particularly those using crossbar arrays, offer a promising solution by performing computations directly where data is stored, leading to higher throughput and energy efficiency. However, a major limitation of current IMC accelerators is their limited on-chip memory capacity. This prevents them from running larger, state-of-the-art neural networks that exceed this footprint. Existing solutions often assume the entire model fits on-chip or lack sophisticated compiler support for handling external memory access. This paper tackles the challenge of running large DNNs on IMC accelerators with limited on-chip memory by introducing a compiler framework that manages partitioning and weight reloading.
Methods
The core of COMPASS is a compiler framework that intelligently partitions a DNN model into smaller, manageable units that can fit within the IMC accelerator's on-chip memory. This is crucial for handling models larger than the available memory. The framework operates in three main stages:
- Partition Generator: This component takes the network model and hardware configuration as input. It decomposes the model into "partition units" (based on kernel sizes) and then groups these units into "partition groups." This process considers data dependencies between layers and aims to fit these partitions within the chip's memory constraints, including mechanisms for weight replication where beneficial. A "validity map" is used to efficiently find valid partition boundaries.
- Partition Optimizer: This is where the novel Genetic Algorithm (GA) comes into play. The GA iteratively refines the partition groups to optimize for user-defined metrics like throughput or energy-delay product (EDP). It uses a "fitness function" that evaluates the performance of each partition and the overall partition group. The GA explores various mutation schemes (merge, split, move, fixed-random) to discover optimal partitioning strategies.
- Scheduler: Finally, the scheduler generates the necessary instructions for the hardware to execute the partitioned model. This includes instructions for loading weights, processing data, and storing intermediate results, all while managing the weight replacement process between partitions.
A key innovation is the "weight replacement" mechanism, which allows the accelerator to load weights for one partition, execute it, then replace those weights with those for the next partition, enabling sequential execution of larger models.
Datasets
The paper evaluates COMPASS using three representative Convolutional Neural Network (CNN) models: VGG16, ResNet18, and SqueezeNet. These models vary in size and complexity. The evaluation is performed on simulated hardware configurations with different memory footprints (labeled S, M, and L).
Key Insights and Ideas
- Bridging the Gap: The most significant insight is that sophisticated compiler support, specifically for partitioning and weight management, is essential for unlocking the potential of IMC accelerators for larger, more complex DNNs.
- Intelligent Partitioning is Key: Simply splitting layers randomly or greedily isn't optimal. COMPASS's GA-driven approach finds a balance between pipeline depth (how many layers can run concurrently or in sequence) and weight replication (duplicating weights to avoid reloading) to maximize performance.
- Weight Reloading is a Feature, Not a Bug: The paper reframes the necessity of accessing external memory (weight reloading) from a limitation to a manageable aspect of the execution flow, enabling larger models.
- Batch Size Trade-offs: The study highlights the critical trade-off between batch size, throughput, and latency. Larger batch sizes improve throughput by amortizing weight replacement overhead but can increase end-to-end latency. COMPASS aims to find an optimal balance.
Key Results
COMPASS demonstrates significant improvements over baseline partitioning methods (greedy and layerwise):
- Throughput: Achieves 1.78X higher throughput on average across different networks and chip configurations. It outperforms the greedy scheme by up to 2.24X and the layerwise scheme by up to 1.98X.
- Energy-Delay Product (EDP): Provides 1.28X savings in EDP on average compared to baseline methods.
- Model Accommodation: COMPASS enables the execution of models that would otherwise be too large for the IMC accelerator's on-chip memory.
- Latency: The latency breakdown shows COMPASS achieving speed-ups of 2.26X and 1.67X compared to greedy and layerwise partitioning, respectively, for the "ResNet18-M-16" configuration.
Limitations and Future Work
- Hardware Specificity: While the approach is designed to be general, the evaluation is performed on an SRAM-based architecture. The authors suggest that the approach can be extended to other emerging non-volatile memory (eNVM) technologies like ReRAM and MRAM, though specific considerations for their unique characteristics (e.g., write endurance for ReRAM, write latency for MRAM) would be needed.
- GA Parameter Tuning: The performance of the GA is dependent on its parameters (population size, number of generations, mutation schemes). While they present their chosen parameters, further exploration might yield even better results.
- Future Work: The authors implicitly suggest future work in extending the framework to other PIM technologies and potentially exploring more complex network architectures or hardware configurations. The paper also notes that the GA algorithm explores the design space, implying that further optimization of the GA itself could be a direction.
Summary updated