An Analytic Model for Cost-Benefit Analysis of Dataflows in DNN Accelerators

Hyoukjun Kwon, Michael Pellauer, Tushar Krishna

Abstract

The data partitioning and scheduling strategies used by DNN accelerators to leverage reuse and perform staging are known as dataflow, and they directly impact the performance and energy efficiency of DNN accelerator designs. An accelerator microarchitecture dictates the dataflow(s) that can be employed to execute a layer or network. Selecting an optimal dataflow for a layer shape can have a large impact on utilization and energy efficiency, but there is a lack of understanding on the choices and consequences of dataflows, and of tools and methodologies to help architects explore the co-optimization design space. In this work, we first introduce a set of data-centric directives to concisely specify the space of DNN dataflows in a compiler-friendly form. We then show how these directives can be analyzed to infer various forms of reuse and to exploit them using hardware capabilities. We codify this analysis into an analytical cost model, MAESTRO (Modeling Accelerator Efficiency via Spatio-Temporal Reuse and Occupancy), that estimates various cost-benefit tradeoffs of a dataflow including execution time and energy efficiency for a DNN model and hardware configuration. We demonstrate the use of MAESTRO to drive a hardware design space exploration (DSE) experiment, which searches across 480M designs to identify 2.5M valid designs at an average rate of 0.17M designs per second, including Pareto-optimal throughput- and energy-optimized design points.

Introduction

Designing a Deep Neural Network (DNN) accelerator is a complex balancing act. Architects must manage billions of operations within the constraints of limited on-chip memory (SRAM) and strict power budgets. The core of this challenge lies in the dataflow: the specific strategy for partitioning, tiling, and scheduling operations. Dataflow determines critical efficiency metrics, such as DRAM access frequency, processing element (PE) utilization, and inter-unit data sharing.

Historically, dataflows have been described using qualitative terms like "Weight Stationary" or "Output Stationary." However, the field has lacked a formal language to describe these patterns and a rapid method to predict their performance. Traditional cycle-accurate simulators, while precise, are prohibitively slow, often taking days to evaluate a single design point. This bottleneck makes it nearly impossible to explore the vast design space required to find optimal hardware-software configurations.

The MAESTRO Framework

To address these challenges, we present MAESTRO (Modeling Accelerator Efficiency via Spatio-Temporal Resource Occupancy). MAESTRO is an analytical framework designed to model data movement and hardware constraints mathematically, enabling rapid design space exploration.

Overview of the MAESTRO framework, showing inputs (NN structure, architecture, dataflow) and analytical outputs.

The framework consists of two primary components:

  1. A Domain-Specific Language (DSL): Dataflows are described using five concise "pragmas" or directives: SpatialMap (parallel distribution across PEs), TemporalMap (local data reuse over time), Unroll/Tile (loop management), and Cluster (hierarchical PE grouping).
  2. An Analytical Engine: This engine processes the DSL directives alongside hardware specifications (e.g., buffer sizes, NoC bandwidth) to calculate "data occupancy." It identifies three fundamental types of reuse: Temporal (stationary data), Spatial (multicasted data), and Spatio-temporal (data forwarded between neighbors).

By analyzing these patterns, MAESTRO predicts execution time, energy consumption, and bandwidth requirements without the need for time-consuming simulations.

Dataflow as Loop Transformation

A key insight of this work is that every complex dataflow is essentially a specific transformation of the nested loops that define a convolution. A standard convolution involves six dimensions: K (filters), C (channels), R/S (filter height/width), and Y/X (output height/width).

The 6-dimensional nested loop structure of a standard convolution operation.

By formalizing dataflow as loop transformations, MAESTRO can systematically explore how different nesting orders and parallelization strategies impact performance. This approach reveals that there is no "one size fits all" dataflow; a strategy that is energy-efficient for early, input-heavy layers may be inefficient for later, weight-heavy layers.

Evaluation and Results

MAESTRO was evaluated using diverse networks including VGG16 and PWCNet. The framework's predictions were validated against MAERI, a hardware-accurate simulator.

  • Accuracy: MAESTRO achieved 94.1% accuracy compared to cycle-accurate RTL simulations.
  • Speed: The tool can analyze approximately 170,000 design points per second. In a large-scale experiment, it searched 480 million designs to identify Pareto-optimal points.
  • Adaptive Gains: By enabling "adaptive dataflows"—selecting the optimal flow for each specific layer—researchers demonstrated a 60.5% reduction in runtime on flexible hardware compared to fixed-dataflow designs like Eyeriss.

Conclusion

MAESTRO provides a powerful tool for the co-design of DNN models and hardware. While currently focused on convolutional layers, its analytical approach offers a scalable path toward modeling more complex operators. Future work includes integrating MAESTRO into automated Neural Architecture Search (NAS) tools to further optimize the synergy between AI models and their underlying hardware.

Summary updated

Abstract

The data partitioning and scheduling strategies used by DNN accelerators to leverage reuse and perform staging are known as dataflow, and they directly impact the performance and energy efficiency of DNN accelerator designs. An accelerator microarchitecture dictates the dataflow(s) that can be employed to execute a layer or network. Selecting an optimal dataflow for a layer shape can have a large impact on utilization and energy efficiency, but there is a lack of understanding on the choices and consequences of dataflows, and of tools and methodologies to help architects explore the co-optimization design space. In this work, we first introduce a set of data-centric directives to concisely specify the space of DNN dataflows in a compilerfriendly form. We then show how these directives can be analyzed to infer various forms of reuse and to exploit them using hardware capabilities. We codify this analysis into an analytical cost model, MAESTRO (Modeling Accelerator Efficiency via Spatio-Temporal Reuse and Occupancy), that estimates various cost-benefit tradeoffs of a dataflow including execution time and energy efficiency for a DNN model and hardware configuration. We demonstrate the use of MAESTRO to drive a hardware design space exploration (DSE) experiment, which searches across 480M designs to identify 2.5M valid designs at an average rate of 0.17M designs per second, including Pareto-optimal throughput- and energy-optimized design points.
BibTeX
@misc{kwon2018analytic,
  author = {Hyoukjun Kwon and Michael Pellauer and Tushar Krishna},
  title = {{An Analytic Model for Cost-Benefit Analysis of Dataflows in DNN Accelerators}},
  year = {2018},
  eprint = {1805.02566},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.02566},
}
Figure