Introduction
Modern computing faces a significant challenge known as the "memory wall," where CPU processing speeds have vastly outpaced the rate at which data can be transferred from main memory. For data-intensive applications—such as those utilizing NumPy, Pandas, or Intel MKL—the primary performance bottleneck is often the constant shuffling of data between RAM and the CPU rather than the calculations themselves.
To address this, optimizing compilers like Weld or XLA attempt to "fuse" operations to keep data within CPU registers. However, these solutions encounter a "usability wall": they require library developers to rewrite functions in specialized intermediate representations (IR), discarding years of hand-optimized SIMD and assembly code. This paper introduces Mozart, a system that achieves compiler-level performance optimizations for existing black-box libraries without requiring code rewrites.
Methodology: Split Annotations and the Mozart System
The core innovation of Mozart is the use of Split Annotations (SAs). Instead of rewriting library internals, developers "tag" existing functions to describe how data can be partitioned (e.g., splitting a matrix by rows or columns).
The Mozart architecture consists of four primary components:
- Split Types: Parameterized types describing data partitioning strategies.
- Splitting API: A minimal set of functions (
Split,Merge, andInfo) that define how to slice and join data structures. - Mozart Client Library: This layer captures a dataflow graph of function calls lazily. In C++, it utilizes OS-level memory protection (SIGSEGV handling) to detect when a program attempts to read a "lazy" value, triggering execution.
- Mozart Runtime: The runtime partitions data into cache-sized chunks based on L2 cache heuristics. It then pipelines these chunks through original, unmodified library functions across multiple threads, ensuring data remains in the CPU cache across successive calls.
Evaluation
The authors evaluated Mozart across 15 benchmarks, including numerical simulations (Black Scholes, nBody), data science tasks (Pandas joins), and machine learning (spaCy NLP). Mozart was compared against unmodified base libraries, JIT compilers like Numba, and the Weld optimizing compiler.
Key Results
- Significant Speedups: Mozart achieved up to a 15x speedup over single-threaded libraries and up to a 5x speedup over already-parallelized libraries like Intel MKL by mitigating the data movement bottleneck.
- Compiler-Competitive Performance: In the Black Scholes benchmark, Mozart was 2x faster than the Weld compiler because it could leverage MKL’s hand-tuned vectorization while Weld generated its own, less-optimized code.
- Efficiency: Hardware counters indicated that Mozart’s pipelining reduced Last Level Cache (LLC) misses by 2x. Furthermore, the overhead for planning and managing the dataflow graph was negligible, accounting for less than 0.5% of total execution time.
- Reduced Engineering Effort: Integrating Mozart into an existing library required approximately 17x less code than integrating a full optimizing compiler.
Conclusion and Future Work
Mozart demonstrates that high-performance pipelining is possible without access to function source code. By treating functions as black boxes and focusing on data movement through Split Annotations, Mozart preserves expert-written optimizations while overcoming the memory wall.
Current limitations include a focus on "pure" functions without side effects and challenges with complex boundary conditions in certain algorithms. Future research may explore automated tools to formally verify the safety of annotations, further reducing the potential for human error in the optimization process.
Summary updated