When building large-scale data pipelines, machine learning models, or scientific simulations, developers face a persistent tension: they must write the core algorithmic logic while simultaneously managing the complex infrastructure where that logic executes. Historically, these two concerns have been tightly coupled. Code frequently contains hardcoded references to specific cloud APIs, cluster schedulers, or local hardware configurations.
As Romain Reuillon notes in Reproducible Large-Scale Neuroimaging Studies with the OpenMOLE Workflow Management System, the manual effort required to adapt computational workflows for different execution environments introduces massive engineering overhead and stifles reproducibility. When the underlying hardware changes, the application code breaks.
To solve this, a prominent research direction focuses on infrastructure-agnostic orchestration. The core insight is simple yet powerful: by completely separating the logical definition of a workflow from its physical execution mechanism, the same application can run unmodified across local laptops, high-performance computing (HPC) clusters, and multi-cloud environments.
The Core Mechanism: Separating What from How
The fundamental architecture of an infrastructure-agnostic system relies on a strict separation of concerns. The developer defines what the computation is—typically as a directed acyclic graph (DAG) of tasks or a declarative topology—while a separate execution layer determines how and where those tasks run.
+-------------------------------------------------------+
| Application Logic |
| (Python Scripts, ML Pipelines, Declarative DAGs) |
+-------------------------------------------------------+
|
v [Abstract Logical Graph]
+-------------------------------------------------------+
| Intermediate Abstraction |
| (TOSCA Templates, Unified APIs, IR) |
+-------------------------------------------------------+
|
v [Environment Translation]
+-------------------------------------------------------+
| Pluggable Execution Layer |
| (Slurm, Kubernetes, AWS Lambda, Local CPU) |
+-------------------------------------------------------+
For example, in Parsl: Scalable Parallel Scripting in Python, Yadu Babuji et al. introduce a modular execution layer that abstracts hardware by separating task definition from resource management. As further detailed by Kyle Chard et al. in Extended Abstract, program logic is written using standard Python decorators that implicitly construct a dynamic dependency graph. The actual execution environment is defined in an entirely separate configuration object. Swapping this object allows the exact same script to scale from a single workstation to a national supercomputer without modifying a single line of application code.
This pattern shifts the developer's operational burden. As shown by Brian F. Cooper et al. in Building a Cloud for Yahoo! and Yu‐Feng Hu et al. in Daisy: Data analysis integrated software system for X-ray experiments, insulating domain experts from the complexities of distributed systems—such as container orchestration, data partitioning, and cluster scheduling—allows them to focus exclusively on algorithm development.
Intermediate Representations and Standardized Topologies
To bridge the gap between abstract logic and concrete infrastructure, these frameworks rely on Intermediate Representations (IR) or standardized modeling languages. These formats act as a universal translation layer.
A widely adopted standard for cloud applications is TOSCA (Topology and Orchestration Specification for Cloud Applications). In INDIGO-DataCloud: a Platform to Facilitate Seamless Access to E-Infrastructures, Davide Salomoni et al. use TOSCA to specify resource requirements in a provider-agnostic manner. Similarly, Marica Antonacci et al. demonstrate in Digital repository as a service: automatic deployment of an Invenio-based repository using TOSCA orchestration and Apache Mesos that defining application architectures by the relationships between components rather than provider-specific server scripts allows complex distributed systems to be deployed consistently across highly fragmented cloud infrastructures.
In the serverless domain, where vendor lock-in is notoriously severe, intermediate syntaxes prevent API fragmentation. Larissa Schmid et al. introduce a platform-agnostic approach in SeBS-Flow: Benchmarking Serverless Cloud Function Workflows by utilizing a unified configuration syntax. This intermediate representation is automatically compiled into provider-specific formats (such as AWS Step Functions JSON or Azure imperative code), allowing developers to benchmark and deploy workflows across disparate clouds seamlessly.
The Translation Layer: Pluggable Backends
Why does this decoupling work in practice? It succeeds because the orchestration engine acts as a late-binding compiler. The high-level workflow is compiled into concrete execution steps only at the moment of deployment, using a pluggable connector layer.
In TORCH: a TOSCA-Based Orchestrator of Multi-Cloud Containerised Applications, the authors implement a modular architecture where a pluggable connector layer translates generic resource commands into specific API calls for different cloud providers or container orchestrators. This minimizes the implementation cost of adding support for new infrastructure.
This translation can happen through various mechanisms:
- Automated Script Generation: As demonstrated by Philipp Koehn in An Experimental Management System, high-level pipeline definitions can be automatically compiled into executable shell scripts tailored for specific target environments.
- Configuration Encapsulation: Antonio Corradi et al. show in Towards an Automated BPEL-based SaaS Provisioning Support for OpenStack IaaS that wrapping software configuration scripts within an intermediate abstraction layer ensures deployment workflows remain consistent even when the underlying virtualized resources change.
- Event Mediation: To decouple event-driven behavior from proprietary cloud APIs, Giuliano Casale et al. design an event gateway abstraction in RADON: rational decomposition and orchestration for serverless computing that acts as a mediator, translating generic triggers into vendor-specific events.
Application to Modern Data and AI Pipelines
The practical impact of decoupling compute logic from infrastructure is most visible in modern, data-intensive fields like machine learning and foundation model research.
In large-scale machine learning, frameworks like the one proposed by Pierre Andrews et al. in stopes - Modular Machine Translation Pipelines allow massive data pipelines to be developed locally on small samples and then dispatched to large-scale clusters via simple configuration changes. Similarly, Jeremy Wohlwend et al. demonstrate in Flambé: A Customizable Framework for Machine Learning Experiments that decoupling experiment code from infrastructure management reduces the engineering overhead required to scale hyperparameter searches across multiple cloud nodes.
This paradigm is pushed to its logical extreme when dealing with frontier-scale AI models. In NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals, Jaden Fiotto-Kaufman et al. decouple the experimental interpretability logic from the model's physical runtime. Researchers can design and debug complex "surgical" interventions on massive models using a local laptop; the framework packages this logic into an intervention graph and offloads the actual execution to a remote high-performance cluster where the model parameters are pre-loaded and sharded.
Ultimately, by treating infrastructure as an interchangeable backend rather than a foundational constraint, these systems ensure that computational logic remains portable, reusable, and resilient to the shifting landscapes of underlying hardware.