Introduction: The Memory Wall in Scientific Computing
While Deep Learning (DL) accelerators have seen explosive growth, traditional High-Performance Computing (HPC) applications—such as the Conjugate Gradient (CG) solver—remain notoriously difficult to accelerate. The primary challenge lies in the nature of scientific workloads, which often involve "skinny" or skewed matrices rather than the dense, cubic matrices found in deep learning. This structural difference leads to inherently low "intra-operation" data reuse, forcing hardware to spend more time moving data from memory than performing calculations.
To bridge this gap, we present CELLO, a hardware-software co-design that exploits data reuse across complex operation graphs. By pairing a specialized software scheduler (SCORE) with a hybrid implicit-explicit buffer (CHORD), CELLO achieves significant improvements in both performance and energy efficiency.
The Challenge of Skewed Tensors and Complex DAGs
Traditional accelerators struggle with HPC workloads for two main reasons. First, the skewed tensor shapes (e.g., a $10^6 \times 8$ matrix) result in extremely low arithmetic intensity. In a regular GEMM (General Matrix Multiplication), data is reused many times within a single operation. In skewed GEMMs, the datapath remains idle most of the time because the data is used only once or twice before being discarded.
Second, HPC workloads are structured as complex Directed Acyclic Graphs (DAGs) rather than simple linear chains. Existing systems struggle to manage "inter-operation" reuse—keeping data on-chip for multiple downstream operations—without incurring massive manual programming overhead or inefficient hardware cache management.
Hardware-Software Co-Design: SCORE and CHORD
The CELLO architecture addresses these roadblocks through a unique contract between the compiler and the hardware.
SCORE: The Software Scheduler
SCORE analyzes the entire DAG of tensor operations at compile-time. It categorizes dependencies into specific types, such as Pipelineable, Delayed Hold, and Delayed Writeback. By optimizing loop orders and minimizing costly data layout transformations (swizzling), SCORE ensures that data stays on-chip for as long as possible, maximizing reuse across different operations in the graph.
CHORD: The Hybrid Hardware Buffer
CHORD is a novel memory mechanism that combines the benefits of scratchpads and caches. It is "explicit" because it receives metadata from SCORE regarding tensor addresses and reuse distances, but it is "implicit" because it manages cycle-level data placement and replacement in hardware.
CHORD utilizes two primary policies:
- PRELUDE: Manages the staging of tensors from DRAM.
- RIFF: Uses reuse frequency and distance metadata to decide which tensors to keep or evict. Unlike traditional caches that track 64-byte lines, CHORD operates at the tensor level, reducing tag area overhead from ~30% to just 0.01%.
Evaluation and Results
CELLO was evaluated across a variety of high-value workloads, including HPC solvers (CG, BiCGStab), fluid dynamics simulations, and Graph Neural Networks (GNNs).
- Performance: CELLO provides a 4x geomean speedup compared to state-of-the-art accelerators like Flexagon or FLAT.
- Efficiency: By reducing DRAM accesses by 64% to 83%, the system achieves a 4x improvement in energy efficiency.
- Area Overhead: The CHORD controller adds only 2% area overhead to the data array, significantly lower than the ~33% overhead typical of traditional cache controllers.
- Versatility: While optimized for HPC, CELLO also matches or outperforms specialized DL accelerators on complex networks like ResNet by more gracefully handling skip-connection dependencies.
Conclusion and Future Directions
CELLO demonstrates that the key to accelerating memory-bound scientific applications is not just faster compute, but smarter data management through hardware-software co-design. By moving from line-level cache management to compiler-informed tensor-level management, it overcomes the limitations of low arithmetic intensity. Future work will explore extending this approach to more irregular sparse-sparse operations (SpGEMM) and scaling the architecture for multi-node HPC environments.
Summary updated