Declarative Configuration for Machine Learning Experiments

88 concepts66 papers1 proposal

When reproducing a machine learning paper, researchers often encounter a familiar frustration: a labyrinth of imperative scripts, undocumented command-line flags, and hyperparameters scattered across deep directory trees. This "glue code" crisis occurs because traditional machine learning code tightly couples what an experiment is trying to achieve with how it is executed by the underlying hardware.

To resolve this, a major research direction has emerged around declarative experiment configuration. Instead of writing imperative code to orchestrate data loading, model building, and evaluation, researchers define the entire experimental pipeline as a static, structured data asset (such as a YAML or JSON file).

By treating the experiment pipeline as data rather than code, this paradigm establishes a formal contract for machine learning workflows, making them highly reproducible, composable, and scalable.

The Anatomy of a Consolidated Artifact

The core intuition behind declarative configuration is the prevention of information fragmentation. In traditional setups, an experiment's state is split between the codebase, environment variables, and runtime arguments. If any of these pieces are lost or modified, the experiment becomes impossible to replicate.

As argued by Iordanis Fostiropoulos et al. in Reproducibility Requires Consolidated Artifacts, centralizing all experimental settings into a single, machine-readable file is vital for scientific integrity. This "consolidated artifact" acts as a complete recipe for the experiment.

                       IMPERATIVE APPROACH
 [ Python Scripts ] + [ CLI Arguments ] + [ Hardcoded Seeds ] = Fragile Run

                       DECLARATIVE APPROACH
 [ Unified Config (YAML/JSON) ] ───> [ Execution Engine ] ───> Deterministic Run

This concept traces back to early foundational work in the field. For instance, Hendrik Blockeel et al. proposed the use of structured relational schemas in Experiment Databases: Towards an Improved Experimental Methodology in Machine Learning to store exact algorithm versions, parameter configurations, and random seeds. Modern frameworks have adapted this idea into human-readable formats. In recommender systems research, Vito Walter Anelli et al. demonstrated in Elliot: a Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation that an entire pipeline—from data filtering to statistical significance testing—can be cleanly orchestrated through a single configuration file.

Decoupling Architecture from Implementation

Why does this separation of concerns work so effectively? By decoupling the logical structure of an experiment from its algorithmic implementation, developers can modify system behavior without touching the core source code.

Consider the challenge of moving a pipeline from an experimental notebook to a production environment. In an imperative setup, this requires rewriting the "glue code" that connects data preprocessing to model training. However, as shown by Doruk Kilitçioglu et al. in Representing the Unification of Text Featurization using a Context-Free Grammar, representing pipelines as declarative JSON schemas provides a formal contract that ensures identical execution across both research and live production environments.

This decoupling provides several practical advantages:

Hierarchical Scaling and Automation

Declarative configurations truly shine when scaling up experiments. When hyperparameter tuning or multi-task learning requires running thousands of variations, manual script management breaks down.

Because a declarative configuration file is structured data, it can be parsed, manipulated, and generated programmatically. This enables hierarchical configuration management, where a base configuration defines the standard pipeline, and small, nested overrides specify variations.

[ Base Config: BERT Model ]
       │
       ├───> [ Override A: Learning Rate = 1e-5 ] ───> Run Experiment A
       │
       └───> [ Override B: Learning Rate = 5e-5 ] ───> Run Experiment B

Pierre Andrews et al. leveraged this approach in stopes - Modular Machine Translation Pipelines to scale data processing pipelines across hundreds of languages without modifying the underlying source code. Similarly, Heiko Paulheim et al. showed in NewsRecLib: A PyTorch-Lightning Library for Neural News Recommendation how hierarchical configurations allow researchers to cleanly mix-and-match model components and track complex hyperparameter combinations.

This structure also makes automated iteration straightforward. As demonstrated by Gregor Karetka et al. in RAGthoven: A Configurable Toolkit for RAG-enabled LLM Experimentation, embedding hyperparameter grids directly within a structured configuration file allows an orchestration engine to automatically iterate over pipeline variations, such as varying retrieval depths or language model backends, through a single interface.

The Tension: Code vs. Configuration

While declarative configuration offers massive benefits for reproducibility and scaling, it introduces a fundamental trade-off: it trades the infinite flexibility of a programming language for the rigid structure of a data schema.

When an experiment requires a highly non-standard operation—such as a custom loss function that changes dynamically based on the epoch—a purely declarative framework can become a hindrance. The researcher is forced to either extend the underlying execution engine or abuse the configuration schema.

Recognizing this tension, some framework designers choose the opposite path. For example, Carlo D'Eramo et al. explain in MushroomRL: Simplifying Reinforcement Learning Research that their framework deliberately avoids external configuration files like YAML or JSON. Instead, they rely on pure Python scripts to define experiment parameters and architectures. The authors argue that keeping the setup within an executable programming environment simplifies debugging and version control, as researchers can use standard Python debuggers and avoid the overhead of learning a specialized configuration syntax.

Ultimately, the choice between imperative scripts and declarative configurations depends on the goal of the project. When a researcher is in the chaotic, early prototyping phase of a novel algorithm, the flexibility of pure code is often preferred. However, once the research shifts toward benchmarking, hyperparameter sweeping, and public sharing, declarative configuration becomes an essential tool for ensuring that findings are robust, verifiable, and built to last.

Key Foundational Work

DKPro Similarity: An Open Source Framework for Text Similarity (2013) — contributed 2 concept elements to this direction.
Experiment Databases: Towards an Improved Experimental Methodology in Machine Learning (2007) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_declarative_experiment_configuration_6e5ee6d1,
  title = {declarative experiment configuration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/declarative-experiment-configuration/rdr_6e5ee6d1efc940ca9471df65249be47c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_declarative_experiment_configuration_6e5ee6d1_v1,
  title = {declarative experiment configuration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/declarative-experiment-configuration/rdr_6e5ee6d1efc940ca9471df65249be47c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake