PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems

Yunseong Lee, Alberto Scolari, Byung-Gon Chun, Marco Domenico Santambrogio, Markus Weimer, Matteo Interlandi

Introduction

Machine learning (ML) model deployment typically relies on "black-box" serving, where models are encapsulated in containers (e.g., Docker) and treated as opaque binaries. While this approach simplifies deployment, it introduces significant inefficiencies at scale. PRETZEL is a "white-box" prediction serving system designed to overcome these limitations by decomposing ML pipelines into an intermediate representation, enabling aggressive cross-model parameter sharing and end-to-end operator fusion.

The Inefficiency of Black-Box Serving

Current serving systems face three primary challenges:

  1. Memory Redundancy: Many models share identical featurizers, such as word embedding dictionaries. In a black-box setup, each model instance loads its own copy, leading to massive memory waste.
  2. The "Cold Start" Penalty: To conserve resources, systems often unload idle models. Re-initializing these models involves JIT compilation and reflection, causing significant latency spikes for the first request.
  3. Sub-optimal Execution: Standard ML libraries often execute pipelines "operator-at-a-time," which involves frequent data movement between memory and CPU caches, missing opportunities for hardware-level optimizations like operator fusion.

The PRETZEL Architecture

PRETZEL treats ML pipelines as structured dataflows rather than opaque binaries. Its architecture consists of an off-line preparation phase and an on-line execution phase.

The PRETZEL compilation pipeline: (1) Flour transforms models into a logical DAG, (2) Optimization identifies shared components, and (3) Compilation produces a physical Model Plan.

The system operates through three main components:

  • Flour: A dataflow API that translates pipelines into a Directed Acyclic Graph (DAG) of logical transformations.
  • Oven (Optimization): An optimizer that performs "multi-query" style optimizations. It identifies common sub-expressions across different models and fuses operators into "stages" to improve data locality.
  • Object Store: A shared repository for model parameters. If multiple models use the same featurizer, they reference the same memory address, drastically reducing the overall footprint.

Key Insights: Redundancy as a Feature

In production environments—such as those involving A/B testing or personalized models—pipelines are rarely unique; they are often slight variations of one another. PRETZEL leverages this redundancy by sharing underlying components.

Visualization of component sharing across 250 Sentiment Analysis models. Shared featurizers (like c1 or w1) allow multiple models to reference the same memory-intensive objects.

By treating an ML pipeline as a query plan, PRETZEL avoids redundant execution. Furthermore, by fusing operators into stages, the system keeps data in the L1/L2 cache longer, addressing the "memory wall" that often limits throughput in traditional systems.

Evaluation and Results

PRETZEL was evaluated using 500 production-like pipelines from Microsoft, covering Sentiment Analysis (text-heavy) and Attendee Count (structured data) tasks.

  • Memory Efficiency: PRETZEL achieved a 25x reduction in memory footprint for Sentiment Analysis models. It successfully hosted all 500 models in memory, whereas traditional systems like Clipper or standard ML.Net crashed or required disk swapping.
  • Latency Improvements: The system reduced 99th percentile (tail) latency by 5.5x. Crucially, Ahead-Of-Time (AOT) compilation eliminated the "cold start" problem, making initial predictions nearly as fast as subsequent ones.
  • Throughput: PRETZEL increased total throughput by 4.7x, demonstrating linear scaling with CPU cores by minimizing cache contention.

Limitations and Future Directions

While PRETZEL demonstrates significant gains, it is currently tightly coupled with the ML.Net framework. Future work aims to integrate with the ONNX (Open Neural Network Exchange) format to support a broader range of frameworks like TensorFlow and PyTorch. Additionally, the authors plan to automate the definition of logical and physical stages and extend the system to support distributed "scale-out" across multiple nodes.

Summary updated

BibTeX
@inproceedings{lee2018pretzel,
  author = {Yunseong Lee and Alberto Scolari and Byung-Gon Chun and Marco Domenico Santambrogio and Markus Weimer and Matteo Interlandi},
  title = {{PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems}},
  booktitle = {OSDI},
  year = {2018},
  url = {https://openreview.net/forum?id=wSwmE-0Oj_7w},
}
Figure