The Dagstuhl beginners guide to reproducibility for experimental networking research

Vaibhav Bajpai, Anna Brunström, Anja Feldmann, Wolfgang Kellerer, Aiko Pras, Henning Schulzrinne, Georgios Smaragdakis, Matthias Wählisch, Klaus Wehrle

TL;DR

This guide provides practical advice for researchers in experimental networking to design their experiments for reproducibility. It emphasizes documenting every step, using version control, and sharing artifacts to ensure that others can reliably repeat, replicate, and build upon their work, fostering a more robust scientific ecosystem.

What Problem Does This Paper Address?

In experimental fields like networking research, achieving reproducibility is a significant challenge. Unlike theoretical fields, experimental work relies on complex setups, software, and data that are often difficult to fully document and share. This lack of reproducibility hinders the scientific process by making it hard for other researchers to verify findings, build upon existing work, or even for the original authors to revisit their own past experiments. The paper aims to bridge this gap by offering a comprehensive guide on how to design and conduct experiments with reproducibility as a core principle from the outset.

Methods

The paper doesn't introduce novel algorithms or techniques in the traditional sense. Instead, its "method" is a structured approach to scientific practice. It advocates for:

  • Clear Terminology: Adopting ACM's definitions for repeatability, replicability, and reproducibility to ensure consistent understanding.
  • Early Planning and Design: Emphasizing hypothesis formulation, experimental design, and soliciting feedback before extensive work begins.
  • Comprehensive Documentation: Stressing the importance of detailed "lab notebooks" (electronic or otherwise) that record every step, observation, and even mistakes.
  • Version Control: Recommending the use of systems like Git for all code, documentation, and experimental artifacts.
  • Metadata as Data: Treating metadata as crucial information that describes how data was created, its contents, and how to recreate it.
  • Tool Recommendations: Suggesting the use of common, readily available tools and packaging experiments using containers (Docker, Kubernetes) or virtual machines (Vagrant) to manage dependencies.
  • Data Handling Best Practices: Addressing data privacy, anonymization, and ethical considerations, as well as ensuring data integrity.
  • Methodology-Specific Guidance: Providing tailored advice for simulations, system prototyping, real-world measurements, and human subject experiments.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Reproducibility is a Design Choice, Not an Afterthought: The paper strongly argues that reproducibility must be considered from the very beginning of an experiment, not as a final step.
  • The "Why" is as Important as the "What": Understanding why certain parameters were chosen, why a specific tool was used, or why a particular methodology was adopted is crucial for others to reproduce the work.
  • Documentation is an Active Process: Simply writing down notes isn't enough; it needs to be structured, detailed, and easily accessible, akin to a shared electronic lab notebook.
  • Version Control is Essential for Sanity: Using version control for everything (code, data, documentation) helps track changes, revert to previous states, and understand how modifications might have affected results.
  • External Factors are Key: In experimental networking, external factors like system dynamism, network conditions, and even the time of day can significantly impact results and must be accounted for and documented.
  • Don't Reinvent the Wheel (Unless Necessary): Leveraging existing tools and platforms (like RIPE Atlas, PlanetLab) can simplify reproducibility efforts.

Key Results

This paper is a guide, not an empirical study, so it doesn't present quantitative results in terms of accuracy or performance improvements. Its "results" are the principles and practices it outlines. The key qualitative findings are the detailed recommendations for each stage of experimental research, from problem formulation to data analysis and presentation, across various methodologies.

Limitations and Future Work

The authors acknowledge that this is an editorial note and has not undergone peer review. They also implicitly highlight the challenges of reproducibility in dynamic environments like the Internet, where complete control is often impossible. Future work is implied in the ongoing efforts of communities and institutions to promote reproducibility through artifact evaluation committees and open science frameworks. The paper serves as a foundational guide, and its impact will be measured by its adoption and the subsequent increase in reproducible research in the field.

Summary updated

BibTeX
@article{bajpai2019dagstuhl,
  author = {Vaibhav Bajpai and Anna Brunström and Anja Feldmann and Wolfgang Kellerer and Aiko Pras and Henning Schulzrinne and Georgios Smaragdakis and Matthias Wählisch and Klaus Wehrle},
  title = {{The Dagstuhl beginners guide to reproducibility for experimental networking research}},
  journal = {Comput. Commun. Rev.},
  year = {2019},
  doi = {10.1145/3314212.3314217},
}
Figure