DLSpec: A Deep Learning Task Exchange Specification

Abdul Dakkak, Cheng Li, Jinjun Xiong, Wen-mei W. Hwu

Introduction

The "reproducibility crisis" in Deep Learning (DL) is a significant hurdle for research and deployment. Currently, sharing a new model or optimization typically involves providing a README and a collection of ad-hoc scripts. This approach forces practitioners to manually replicate software environments, manage specific dataset versions, and hope for hardware parity. Such lack of standardization makes it difficult to compare results fairly, port tasks across different architectures (e.g., from X86 to PowerPC), and isolate performance bottlenecks within complex pipelines.

To address these challenges, we present DLSpec, a standardized, agnostic exchange specification designed to make DL tasks—including training and inference—truly reproducible and portable. By decoupling tasks into modular components, DLSpec allows researchers to share and execute innovations across diverse environments without relying on fragile, custom scripts.

The DLSpec Architecture

At the core of DLSpec is the principle of decoupling. Rather than a single monolithic configuration, the specification utilizes four distinct, versioned manifests that define the entire execution context.

Overview of the DLSpec manifest structure and its integration with the runtime environment.

As illustrated in the figure above, the four manifests are:

  1. Hardware: Specifies CPU/GPU architectures, memory requirements, and low-level system configurations (e.g., disabling Intel Turbo Boost) to ensure benchmarking consistency.
  2. Software: Defines the framework (TensorFlow, PyTorch, etc.), versioning, and the specific container environment required.
  3. Dataset: References external resources via URLs with associated checksums to ensure data integrity and versioning.
  4. Model: Contains the core logic, including the model graph, weights, and the execution pipeline.

Methodology

DLSpec introduces several key innovations to ensure robustness and performance:

  • Pipeline Demarcation: Tasks are explicitly split into three stages: pre-processing, run, and post-processing. These are defined as Python functions embedded within the Model manifest, allowing for clear separation of concerns.
  • In-Memory Execution: To minimize the overhead of disk serialization, DLSpec runtimes utilize Python sub-interpreters to pass data between pipeline stages directly in memory.
  • Artifact Content Addressing: All external resources are referenced using content-addressed URLs, ensuring that the exact intended data is used for every run.
  • Agnosticism and Auditability: DLSpec acts as a "universal glue" for MLOps, providing a blueprint for execution that is independent of specific frameworks or hardware. Every experiment generates a "reference log" that records the manifest IDs and resulting metrics, creating a verifiable audit trail.

Evaluation and Results

The utility of DLSpec was demonstrated by specifying and running hundreds of DL tasks across a wide array of frameworks, including Caffe, Caffe2, CNTK, MXNet, PyTorch, TensorFlow, and TensorRT. The specification was validated on diverse hardware architectures, including X86, ARM, and PowerPC, proving its effectiveness in handling cross-platform portability.

While the paper does not introduce new datasets, it provides a library of manifests for industry standards such as ImageNet (ILSVRC 2012), COCO, Pascal VOC, and CIFAR. By isolating the run stage, the authors implemented a runtime for DL inference benchmarking that provides significantly more consistent and reproducible results than traditional, script-based methods.

Discussion and Future Work

While DLSpec currently focuses heavily on inference, extending this rigor to complex, distributed training setups is a primary goal for future development. The success of the specification depends on community adoption; the authors are maintaining the spec at dlspec.netlify.com and refining it for evolving tasks like Reinforcement Learning and Generative AI. Furthermore, while the specification itself is agnostic, the development of more open-source runtime implementations will be critical for DLSpec to become a true industry standard for deep learning task exchange.

Summary updated

Abstract

Deep Learning (DL) innovations are being introduced at a rapid pace. However, the current lack of standard specification of DL tasks makes sharing, running, reproducing, and comparing these innovations difficult. To address this problem, we propose DLSpec, a model-, dataset-, software-, and hardware-agnostic DL specification that captures the different aspects of DL tasks. DLSpec has been tested by specifying and running hundreds of DL tasks.
BibTeX
@inproceedings{dakkak2020dlspec,
  author = {Abdul Dakkak and Cheng Li and Jinjun Xiong and Wen-mei W. Hwu},
  title = {{DLSpec: A Deep Learning Task Exchange Specification}},
  booktitle = {OpML},
  year = {2020},
  url = {https://openreview.net/forum?id=MtLcC6eAWJ},
}
Figure