Abstract
The data partitioning and scheduling strategies used by DNN accelerators to leverage reuse and perform staging are known as dataflow, and they directly impact the performance and energy efficiency of DNN accelerator designs. An accelerator microarchitecture dictates the dataflow(s) that can be employed to execute a layer or network. Selecting an optimal dataflow for a layer shape can have a large impact on utilization and energy efficiency, but there is a lack of understanding on the choices and consequences of dataflows, and of tools and methodologies to help architects explore the co-optimization design space. In this work, we first introduce a set of data-centric directives to concisely specify the space of DNN dataflows in a compiler-friendly form. We then show how these directives can be analyzed to infer various forms of reuse and to exploit them using hardware capabilities. We codify this analysis into an analytical cost model, MAESTRO (Modeling Accelerator Efficiency via Spatio-Temporal Reuse and Occupancy), that estimates various cost-benefit tradeoffs of a dataflow including execution time and energy efficiency for a DNN model and hardware configuration. We demonstrate the use of MAESTRO to drive a hardware design space exploration (DSE) experiment, which searches across 480M designs to identify 2.5M valid designs at an average rate of 0.17M designs per second, including Pareto-optimal throughput- and energy-optimized design points.
Introduction
In Deep Neural Network (DNN) accelerators, "dataflow" is the strategy governing how computations are scheduled and how data is mapped across processing elements (PEs) and memory hierarchies. Since data movement from off-chip memory consumes orders of magnitude more energy than arithmetic operations, maximizing data reuse is critical for efficiency. However, the design space is vast, involving millions of combinations of buffer sizes, interconnect bandwidths, and loop transformations. Traditional "compute-centric" loop-nest representations make it difficult to reason about data movement directly, and cycle-accurate hardware simulations (RTL) are often too slow for large-scale design space exploration.
Data-Centric Directives
To address these challenges, MAESTRO introduces a set of data-centric directives that treat data movement as a first-class citizen. Instead of traditional code-like loops, architects use three primary directives:
- SpatialMap: Describes how data is distributed across different PEs to enable parallelism.
- TemporalMap: Describes how data is reused within a single PE over time (stationarity).
- Cluster: Enables hierarchical modeling of hardware, such as grouping PEs into clusters with dedicated local memory.
These directives provide a concise, compiler-friendly way to represent complex dataflows, allowing for rapid analysis of data orchestration and reuse patterns.
The MAESTRO Cost Model
The MAESTRO analytical engine takes a DNN model, a dataflow description, and hardware parameters as input. It employs mathematical analysis to count data reuses and calculate required bandwidth, estimating execution time and energy without step-by-step simulation. A key feature of MAESTRO is its formalization of reuse into four categories: spatial/temporal multicasting (sending data to multiple locations) and spatial/temporal reduction (combining partial results).
Evaluation and Results
MAESTRO was evaluated using state-of-the-art DNN models including ResNet50, MobileNetV2, and UNet. Its accuracy was validated against RTL implementations of the MAERI and Eyeriss accelerators.
- Performance: MAESTRO is 1,000x to 4,000x faster than cycle-accurate RTL simulations, evaluating a design in roughly 10ms.
- Accuracy: It maintains high fidelity, with an average error of only 3.9% compared to RTL simulations for total execution cycles.
- Design Space Exploration (DSE): In a DSE experiment searching 480 million designs, MAESTRO identified 2.5 million valid designs at a rate of 0.17 million designs per second, successfully locating Pareto-optimal points for throughput and energy.
Key Insights and Future Work
A significant finding is that no single dataflow is optimal for all layers. Early layers (wide and shallow) benefit from different strategies than late layers (narrow and deep). Implementing "adaptive" dataflows that change per layer can reduce runtime by 37%.
Future work includes extending MAESTRO to support non-uniform sparsity distributions common in optimized AI models and developing an "auto-tuner" to automatically discover optimal dataflows for given hardware configurations. The ultimate goal is to link MAESTRO to automated hardware generation flows, enabling a direct path from high-level dataflow description to synthesizable chip design.
Summary updated