The GA4GH Task Execution API: Enabling Easy Multi Cloud Task Execution

Alexander Kanitz, Matthew H. McLoughlin, Liam Beckman, Venkat S. Malladi, Kyle Ellrott

TL;DR

The GA4GH Task Execution Service (TES) API is a standardized, cloud-agnostic framework designed to run batch genomic analysis tasks across any infrastructure—from local HPC clusters to major cloud providers. By providing a common "language" for task execution, TES enables researchers to bring their compute to the data, bypassing the massive bottleneck of moving multi-hundred-gigabyte genomic files.

What Problem Does This Paper Address?

Genomic research is a logistical nightmare. Analyzing a single human genome generates hundreds of gigabytes of data, making it impractical to move files around for analysis. Furthermore, the field relies on a "fragmented" software ecosystem: researchers use a mix of decades-old command-line tools and cutting-edge scripts, each with unique library dependencies and hardware requirements.

Historically, if you wanted to run a workflow on a local cluster (using SLURM or Grid Engine) and then move it to the cloud (AWS or Azure), you had to rewrite your execution logic. There was no "universal remote" for compute. The authors address this fragmentation by introducing a standardized API that abstracts away the underlying infrastructure, allowing the same task definition to run anywhere.

Methods

The authors present the TES API, a RESTful specification built using OpenAPI. Unlike previous attempts at standardization (like DRMAA, which required specific C library bindings), TES is network-based, making it easy to integrate with modern web services and various programming languages.

Key technical components of the TES approach include:

  • Atomic Task Messages: A single JSON object contains everything needed to run a job: the Docker image (environment), resource requirements (CPU/RAM/Disk), input/output locations (URLs), and the actual command lines.
  • Sequential Executors: A task can contain multiple "executors" that run in order within the same environment, allowing for setup, processing, and teardown steps to be bundled together.
  • Infrastructure Abstraction: TES acts as a translation layer. A "TES Server" receives the standardized request and translates it into the specific commands needed for the backend (e.g., submitting a job to an Azure Batch pool or a Kubernetes cluster).
  • Conformance Testing: To prevent "standard fragmentation," the community developed a YAML-based test suite that ensures any new TES implementation actually adheres to the specification.

Datasets

As this paper describes a technical standard and API rather than a specific biological discovery or machine learning model, it does not use training or evaluation datasets in the traditional sense. However, it references Whole Genome Sequencing (WGS) data—specifically files exceeding 200GB—as the primary use case and driver for the API's design.

Key Insights and Ideas

  • "Compute to the Data": This is the core philosophy. Instead of downloading a 200GB file to your local machine, TES allows you to send a small "task" message to a server sitting right next to the data.
  • Containerization as the Great Equalizer: By mandating (or strongly encouraging) the use of Docker images, TES solves the "dependency hell" of bioinformatics. The API doesn't care what's inside the container; it just ensures the container runs with the requested resources.
  • Federated Analysis: TES is a critical building block for "data sovereign" projects. For example, it allows researchers to run analyses across multiple national data centers in Europe without the raw genomic data ever leaving its country of origin.
  • Workflow Engine Agnostic: TES isn't a workflow manager itself; it's a target for them. Major engines like Nextflow, Snakemake, and Cromwell can now "speak" TES, instantly granting them compatibility with any TES-compliant backend.

Key Results

The paper highlights the successful adoption and "ecosystem" of TES:

  • Server Implementations: Multiple production-ready servers exist, including Funnel (for HPC/Cloud), TES-Azure (native Microsoft Azure support), and TESK (Kubernetes-native).
  • Workflow Integration: The API has been successfully integrated into the "Big Four" of bioinformatics workflows: Galaxy, Cromwell, Nextflow, and Snakemake.
  • Real-World Deployment: TES is currently being used to power the European Genomic Data Infrastructure, a massive project connecting institutions across 20 different countries for federated genomic analysis.

Limitations and Future Work

While TES provides a robust framework for execution, the authors acknowledge several areas for growth:

  • Security and Auth: Passing storage credentials and compute authorizations through multiple layers of infrastructure remains a complex challenge that needs further standardization.
  • Beyond Docker: While currently Docker-centric, future versions need better native support for other container runtimes like Singularity (common in HPC) or Podman, as well as environment managers like Conda.
  • Advanced Compute Patterns: There is ongoing discussion about extending the API to support distributed computing frameworks like Spark or Hadoop, and enabling privacy-preserving federated learning.

Summary updated

Abstract

The Global Alliance for Genomics and Health (GA4GH) Task Execution Service (TES) API is a standardized schema and API for describing and executing batch execution tasks. It provides a common way to submit and manage tasks to a variety of compute environments, including on premise High Performance Compute and High Throughput Computing (HPC/HTC) systems, Cloud computing platforms, and hybrid environments. The TES API is designed to be flexible and extensible, allowing it to be adapted to a wide range of use cases, such as "bringing compute to the data" solutions for federated and distributed data analysis or load balancing across multi cloud infrastructures. This API has been adopted by a number of different service providers and utilized by several workflow engines. Using its capabilities, genomes research institutes are building hybrid compute systems to study life science.
BibTeX
@misc{kanitz2024ga,
  author = {Alexander Kanitz and Matthew H. McLoughlin and Liam Beckman and Venkat S. Malladi and Kyle Ellrott},
  title = {{The GA4GH Task Execution API: Enabling Easy Multi Cloud Task Execution}},
  year = {2024},
  eprint = {2405.00013},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.00013},
}
Figure