COMET: A Domain-Specific Compilation of High-Performance Computational Chemistry

Erdal Mutlu, Ruiqin Tian, Bin Ren, Sriram Krishnamoorthy, Roberto Gioiosa, Jacques A. Pienaar, Gokcen Kestor

TL;DR

COMET is a novel domain-specific compiler and programming language designed to bridge the gap between high-level computational chemistry notations and low-level hardware performance. By leveraging the Multi-Level Intermediate Representation (MLIR) framework, it automates complex tensor algebra optimizations, achieving up to 1.98x speedup over hand-tuned libraries and up to 23.9x speedup for multi-operand expressions.

What Problem Does This Paper Address?

In computational chemistry, simulations of chemical reactions rely heavily on tensor contractions—high-dimensional versions of matrix multiplication. Historically, scientists faced a "struggle for balance": they could either use high-level languages that are easy to write but slow, or write highly specialized, low-level code that is fast but incredibly difficult to maintain and port to new hardware (like GPUs or TPUs).

Existing solutions often rely on "black-box" libraries. While these libraries are fast for a single operation, they can't "see" the bigger picture. For example, if a scientist wants to multiply four tensors together, a library might just do them one by one in the order they were written. However, changing that order can sometimes reduce the total number of calculations by orders of magnitude. COMET addresses this by providing a compiler that understands the math (Einstein notation) and the hardware simultaneously.

Methods

The authors built COMET using MLIR (Multi-Level Intermediate Representation), a framework that allows a compiler to look at code through different "lenses" or dialects.

  1. High-Level (TA Dialect): The compiler starts with a Domain-Specific Language (DSL) that looks like standard math. Here, it performs "algorithmic" optimizations, such as finding the most efficient order to perform a chain of contractions.
  2. Intermediate (LinAlg/Loop Dialects): It then lowers the code to more general operations. It uses the TTGT (Transpose-Transpose-GEMM-Transpose) method, which reformulates complex tensor contractions into standard matrix multiplications (GEMM) that hardware is already very good at.
  3. Low-Level (LLVM/Micro-kernels): Finally, it generates machine code optimized for specific hardware features like AVX-512 vector instructions or custom hardware accelerators.
The COMET compilation pipeline showing progressive lowering from high-level DSL to machine code.

A unique feature of COMET is its support for hardware-software co-design. By pairing the compiler with a hardware simulator (Aladdin), researchers can test how their chemistry code would perform on hypothetical custom chips before they are even built.

Datasets

The researchers evaluated COMET using:

  • TCCG Benchmark Suite: 30 tensor contractions representing common workloads in machine learning and quantum chemistry (e.g., CCSD methods).
  • NWChem Expressions: 118 complex multi-operand tensor expressions from the NorthWest Chemistry computational package, involving up to 417 total contractions.

Key Insights and Ideas

  • The Power of Global Optimization: Unlike standard libraries that optimize one operation at a time, COMET’s compiler-based approach analyzes entire expressions. This allows it to perform "multi-operand reordering," which can drastically reduce the computational footprint of a simulation.
  • Permutation is Key: In the TTGT method, how you "flatten" a high-dimensional tensor into a 2D matrix matters. COMET uses a cost model to automatically select the optimal index permutation, which can improve performance by an average of 1.38x (and up to 4.36x) before other optimizations are even applied.
  • Hardware Tailoring: The study found that "bigger isn't always better" for hardware accelerators. For certain workloads, a medium-sized accelerator (64x64 tiles) outperformed a larger one (256x256) because the larger one overwhelmed the system's cache, leading to memory bottlenecks.

Key Results

COMET's automated approach consistently matched or beat state-of-the-art, manually optimized libraries like BLIS and HPTT.

  • Speedup: COMET achieved an average of 1.22x and a peak of 1.98x speedup compared to the TCCG benchmarks on standard x86 CPUs.
  • Multi-Operand Efficiency: For complex chains of operations, COMET achieved up to 23.9x speedup simply by reordering the sequence of contractions to minimize work (see Table 1).
  • Peak Performance: On x86, it reached ~51 GFLOPS. When targeting simulated custom hardware accelerators, performance jumped to 156 GFLOPS, a 3.1x improvement over "soft" vector optimizations.
Performance breakdown showing the incremental benefit of different optimization layers in COMET.
Comparison of COMET against the TCCG library across various tensor contractions.

Limitations and Future Work

The current version of COMET focuses primarily on dense tensors, where every position in the tensor contains data. However, many real-world chemistry problems involve sparse tensors (mostly zeros). The authors identify adding support for sparse tensor algebra as a primary goal for future versions. They also plan to expand the compiler's backends to support more diverse hardware architectures and eventually open-source the project to the broader scientific community.

Summary updated

Abstract

The computational power increases over the past decades havegreatly enhanced the ability to simulate chemical reactions andunderstand ever more complex transformations. Tensor contractions are the fundamental computational building block of these simulations. These simulations have often been tied to one platform and restricted in generality by the interface provided to the user. The expanding prevalence of accelerators and researcher demands necessitate a more general approach which is not tied to specific hardware or requires contortion of algorithms to specific hardware platforms. In this paper we present COMET, a domain-specific programming language and compiler infrastructure for tensor contractions targeting heterogeneous accelerators. We present a system of progressive lowering through multiple layers of abstraction and optimization that achieves up to 1.98X speedup for 30 tensor contractions commonly used in computational chemistry and beyond.
BibTeX
@inproceedings{mutlu2020comet,
  author = {Erdal Mutlu and Ruiqin Tian and Bin Ren and Sriram Krishnamoorthy and Roberto Gioiosa and Jacques A. Pienaar and Gokcen Kestor},
  title = {{COMET: A Domain-Specific Compilation of High-Performance Computational Chemistry}},
  booktitle = {LCPC},
  year = {2020},
  doi = {10.1007/978-3-030-95953-1\_7},
}

Loading related content...

Figure