Understanding Reuse, Performance, and Hardware Cost of DNN Dataflows: A Data-Centric Approach

Hyoukjun Kwon, Prasanth Chatarasi, Michael Pellauer, Angshuman Parashar, Vivek Sarkar, Tushar Krishna

Abstract

The data partitioning and scheduling strategies used by DNN accelerators to leverage reuse and perform staging are known as dataflow, and they directly impact the performance and energy efficiency of DNN accelerator designs. An accelerator microarchitecture dictates the dataflow(s) that can be employed to execute a layer or network. Selecting an optimal dataflow for a layer shape can have a large impact on utilization and energy efficiency, but there is a lack of understanding on the choices and consequences of dataflows, and of tools and methodologies to help architects explore the co-optimization design space. In this work, we first introduce a set of data-centric directives to concisely specify the space of DNN dataflows in a compiler-friendly form. We then show how these directives can be analyzed to infer various forms of reuse and to exploit them using hardware capabilities. We codify this analysis into an analytical cost model, MAESTRO (Modeling Accelerator Efficiency via Spatio-Temporal Reuse and Occupancy), that estimates various cost-benefit tradeoffs of a dataflow including execution time and energy efficiency for a DNN model and hardware configuration. We demonstrate the use of MAESTRO to drive a hardware design space exploration (DSE) experiment, which searches across 480M designs to identify 2.5M valid designs at an average rate of 0.17M designs per second, including Pareto-optimal throughput- and energy-optimized design points.

The Challenge of Dataflow Optimization

In the design of Deep Neural Network (DNN) hardware, "dataflow" is the critical strategy for partitioning, scheduling, and moving data between processing elements (PEs) and memory hierarchies. Because data movement often consumes significantly more energy than actual computation, a suboptimal dataflow can render even the most powerful chip inefficient.

The challenge lies in the astronomical design space. Architects must decide how to tile loops, which data to keep stationary in local buffers, and how to parallelize work across hundreds of PEs. Traditionally, evaluating these choices required slow, low-level hardware simulations (RTL) or "black-box" compilers that offered little intuition. There has been a lack of a unified language to describe these dataflows or a fast way to predict their performance across diverse neural network layers.

MAESTRO: A Data-Centric Analytical Framework

The authors introduce MAESTRO (Modeling Accelerator Efficiency via Spatio-Temporal Reuse and Occupancy). Its core innovation is a set of data-centric directives that describe dataflow as first-class entities rather than side effects of complex loop nests.

MAESTRO uses data-centric directives like SpatialMap and TemporalMap to describe data movement and reuse patterns.

The framework utilizes several primary directives to define the dataflow:

  • SpatialMap: Defines how data is distributed across different PEs (parallelism).
  • TemporalMap: Defines how data is distributed across time steps within a single PE (reuse).
  • Cluster: Groups PEs into hierarchies to model multi-level hardware structures.
  • Data Movement Order: Dictates the sequence of these maps.

Instead of cycle-by-cycle simulation, MAESTRO mathematically analyzes "data reuse" based on these directives. It calculates the frequency of data reads from buffers and multicasting across wires, allowing for near-instantaneous estimation of execution time, energy, and area costs.

The target hardware architecture modeled by MAESTRO includes shared buffers, networks-on-chip, and processing elements with private buffers and ALUs.

Key Insights and Results

MAESTRO was evaluated using iconic DNN models like VGG16, ResNet50, and MobileNetV2, representing a wide variety of computational "shapes." Several key insights emerged:

  • Data-Centric vs. Compute-Centric: By focusing on data movement rather than loop nests, MAESTRO is 1,000x to 4,000x faster than traditional RTL simulations, analyzing designs in roughly 10ms with 90-95% accuracy.
  • Adaptive Dataflow: Dataflows that work well for early layers (large activations, few channels) often perform poorly for late layers (small activations, many channels). Switching dataflows between layers can reduce runtime by 37% and energy by 10%.
  • Hardware-Dataflow Co-optimization: Hardware and scheduling must be optimized together. MAESTRO enabled a search of 480 million designs, identifying 2.5 million valid configurations at a rate of 170,000 designs per second.

Conclusion and Future Directions

MAESTRO provides a powerful tool for architects to navigate the complex DNN accelerator design space. While currently focused on dense tensors, future work aims to incorporate stochastic sparsity patterns. The authors also envision extending the framework toward auto-tuning optimal dataflows and automatically generating RTL code from the identified optimal design points.

Summary updated

Abstract

The data partitioning and scheduling strategies used by DNN accelerators to leverage reuse and perform staging are known as dataflow, and they directly impact the performance and energy efficiency of DNN accelerator designs. An accelerator microarchitecture dictates the dataflow(s) that can be employed to execute a layer or network. Selecting an optimal dataflow for a layer shape can have a large impact on utilization and energy efficiency, but there is a lack of understanding on the choices and consequences of dataflows, and of tools and methodologies to help architects explore the co-optimization design space. In this work, we first introduce a set of data-centric directives to concisely specify the space of DNN dataflows in a compilerfriendly form. We then show how these directives can be analyzed to infer various forms of reuse and to exploit them using hardware capabilities. We codify this analysis into an analytical cost model, MAESTRO (Modeling Accelerator Efficiency via Spatio-Temporal Reuse and Occupancy), that estimates various cost-benefit tradeoffs of a dataflow including execution time and energy efficiency for a DNN model and hardware configuration. We demonstrate the use of MAESTRO to drive a hardware design space exploration (DSE) experiment, which searches across 480M designs to identify 2.5M valid designs at an average rate of 0.17M designs per second, including Pareto-optimal throughput- and energy-optimized design points.
BibTeX
@misc{kwon2018understanding,
  author = {Hyoukjun Kwon and Prasanth Chatarasi and Michael Pellauer and Angshuman Parashar and Vivek Sarkar and Tushar Krishna},
  title = {{Understanding Reuse, Performance, and Hardware Cost of DNN Dataflows: A Data-Centric Approach}},
  year = {2018},
  eprint = {1805.02566},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.02566},
}
Figure