SysML: The New Frontier of Machine Learning Systems.

Alexander Ratner, Dan Alistarh, Gustavo Alonso, David G. Andersen, Peter Bailis, Sarah Bird, Nicholas Carlini, Bryan Catanzaro, Eric Chung, Bill Dally, Jeff Dean, Inderjit S. Dhillon, Alexandros G. Dimakis, Pradeep Dubey, Charles Elkan, Grigori Fursin, Gregory R. Ganger, Lise Getoor, Phillip B. Gibbons, Garth A. Gibson, Joseph E. Gonzalez, Justin Gottschlich, Song Han, Kim M. Hazelwood, Furong Huang, Martin Jaggi, Kevin G. Jamieson, Michael I. Jordan, Gauri Joshi, Rania Khalaf, Jason Knight, Jakub Konecný, Tim Kraska, Arun Kumar, Anastasios Kyrillidis, Jing Li, Samuel Madden, H. Brendan McMahan, Erik Meijer, Ioannis Mitliagkas, Rajat Monga, Derek Gordon Murray, Dimitris S. Papailiopoulos, Gennady Pekhimenko, Theodoros Rekatsinas, Afshin Rostamizadeh, Christopher Ré, Christopher De Sa, Hanie Sedghi, Siddhartha Sen, Virginia Smith, Alex Smola, Dawn Song, Evan R. Sparks, Ion Stoica, Vivienne Sze, Madeleine Udell, Joaquin Vanschoren, Shivaram Venkataraman, Rashmi Vinayak, Markus Weimer, Andrew Gordon Wilson, Eric P. Xing, Matei Zaharia, Ce Zhang, Ameet TalwalkararXiv 2019 · OpenReview

TL;DR

This position paper outlines the necessity of a new, dedicated research community at the intersection of computer systems and machine learning. To bridge the gap between isolated academic models and complex real-world deployments, the authors introduce the MLSys conference, establishing a roadmap for full-stack research targeting hardware/software co-design, efficiency, privacy, and accessibility.

What Problem Does This Paper Address?

While machine learning (ML) models have achieved historic breakthroughs on curated benchmarks, deploying them in production remains a massive bottleneck. The authors argue that modern ML represents a radical paradigm shift—often called "Software 2.0"—where developers spend more time collecting, labeling, and reshaping data than writing traditional code.

Traditional ML research focuses almost exclusively on maximizing predictive accuracy. However, real-world deployment introduces severe full-stack bottlenecks that accuracy alone cannot solve:

  • Skyrocketing Costs: Compute costs for training large models are immense, and data annotation can cost millions.
  • Resource Constraints: Deploying models on edge devices requires strict adherence to latency, power, and memory limits.
  • Trust and Safety: Production systems must navigate complex, system-wide concerns like privacy, adversarial robustness, fairness, and interpretability.

Because these challenges span both hardware and software, they cannot be solved by the traditional ML community or the traditional systems community working in isolation.

Methods

As a position paper, this work does not introduce a new algorithm or technical artifact. Instead, it proposes an organizational and intellectual framework to catalyze a new field. The authors define a research agenda partitioned into three core pillars:

  1. Software Systems for ML: Designing tools to support the entire ML lifecycle, including high-level programming interfaces, data preprocessing pipelines, debugging tools, and production monitoring.
  2. Hardware Systems for ML: Developing specialized, heterogeneous hardware and distributed architectures tailored to the unique data access patterns and stochastic nature of ML workloads.
  3. Optimizing for Metrics Beyond Accuracy: Engineering systems that inherently satisfy constraints like power efficiency, federated privacy, cost, and fairness.

Additionally, the paper highlights the reciprocal domain of Machine Learning for Systems—applying ML techniques to replace heuristics and optimize low-level system components like compilers, operating systems, and databases.

To operationalize this, the authors established the MLSys conference, featuring a program committee split evenly between systems and ML experts to ensure true cross-disciplinary peer review and a strong emphasis on artifact evaluation for reproducibility.

Datasets

Not discussed in the paper (as this is a position and community-building white paper, no experimental evaluations or datasets were utilized).

Key Insights and Ideas

  • The "Software 2.0" Reality: Programming in the ML era is fundamentally about data curation and pipeline management rather than writing explicit logic. Systems must adapt to treat data, features, and labels as first-class citizens.
  • Full-Stack Bottlenecks Require Full-Stack Solutions: Issues like model latency or training cost cannot be solved purely at the algorithmic level; they require co-designing hardware, compilers, and frameworks simultaneously.
  • A Two-Way Street: The relationship between systems and ML is symbiotic. We need better systems to train and serve ML, but we also need ML to build smarter, self-tuning computer systems.

Key Results

The primary result of this paper is the successful institutionalization of the MLSys community. Originally emerging from successful satellite workshops at venues like NeurIPS, ICML, OSDI, and SIGMOD, the standalone MLSys conference was formally established in 2018. The paper outlines its structural success, noting a highly selective review process and the widespread adoption of modern artifact evaluation to spur rapid, reproducible progress in the field.

Limitations and Future Work

The authors frame the entire paper as an open-ended invitation to tackle future challenges. They explicitly note that the community must eventually look beyond single-model pipelines and address larger, systemic issues:

  • Ecosystem Management: How do we manage, maintain, and evaluate entire ecosystems of models that interact with each other in complex, unpredictable ways?
  • Long-Term Dynamics: How do we monitor and safely maintain systems designed to pursue long-term, evolving goals?
  • Societal Scale: How can we facilitate the sharing and reuse of data and models at a societal scale while navigating strict economic, legal, and privacy boundaries?

Summary updated

Abstract

Machine learning (ML) techniques are enjoying rapidly increasing adoption. However, designing and implementing the systems that support ML models in real-world deployments remains a significant obstacle, in large part due to the radically different development and deployment profile of modern ML methods, and the range of practical concerns that come with broader adoption. We propose to foster a new systems machine learning research community at the intersection of the traditional systems and ML communities, focused on topics such as hardware systems for ML, software systems for ML, and ML optimized for metrics beyond predictive accuracy. To do this, we describe a new conference, MLSys, that explicitly targets research at the intersection of systems and machine learning with a program committee split evenly between experts in systems and ML, and an explicit focus on topics at the intersection of the two.
BibTeX
@misc{ratner2019sysml,
  author = {Alexander Ratner and Dan Alistarh and Gustavo Alonso and David G. Andersen and Peter Bailis and Sarah Bird and Nicholas Carlini and Bryan Catanzaro and Eric Chung and Bill Dally and Jeff Dean and Inderjit S. Dhillon and Alexandros G. Dimakis and Pradeep Dubey and Charles Elkan and Grigori Fursin and Gregory R. Ganger and Lise Getoor and Phillip B. Gibbons and Garth A. Gibson and Joseph E. Gonzalez and Justin Gottschlich and Song Han and Kim M. Hazelwood and Furong Huang and Martin Jaggi and Kevin G. Jamieson and Michael I. Jordan and Gauri Joshi and Rania Khalaf and Jason Knight and Jakub Konecný and Tim Kraska and Arun Kumar and Anastasios Kyrillidis and Jing Li and Samuel Madden and H. Brendan McMahan and Erik Meijer and Ioannis Mitliagkas and Rajat Monga and Derek Gordon Murray and Dimitris S. Papailiopoulos and Gennady Pekhimenko and Theodoros Rekatsinas and Afshin Rostamizadeh and Christopher Ré and Christopher De Sa and Hanie Sedghi and Siddhartha Sen and Virginia Smith and Alex Smola and Dawn Song and Evan R. Sparks and Ion Stoica and Vivienne Sze and Madeleine Udell and Joaquin Vanschoren and Shivaram Venkataraman and Rashmi Vinayak and Markus Weimer and Andrew Gordon Wilson and Eric P. Xing and Matei Zaharia and Ce Zhang and Ameet Talwalkar},
  title = {{SysML: The New Frontier of Machine Learning Systems.}},
  year = {2019},
  eprint = {1904.03257},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.03257},
}
Figure