Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization

Leonard Papenmeier, Luigi Nardi

Abstract

Benchmarking black-box optimization (BBO) algorithms is often hindered by "dependency hell," where conflicting software requirements and outdated library versions make reproducible comparisons difficult. We present Bencher, a modular framework that isolates benchmarks within containerized environments. By decoupling optimization logic from benchmark execution via a unified gRPC interface, Bencher enables seamless evaluation across over 80 diverse benchmarks without environment conflicts.

Introduction

In Black-Box Optimization (BBO), researchers seek to find the best input for functions where the internal mechanics are unknown, relying solely on input-output observations. Validating new algorithms requires testing against a broad spectrum of real-world benchmarks. However, these benchmarks often have incompatible dependencies—for instance, one might require Python 3.8 and specific versions of MuJoCo, while another needs the latest scientific stack.

This fragmentation leads to "software rot" and inconsistent experimental setups, making it nearly impossible to compare results across different papers fairly. Bencher addresses this by treating benchmarks as isolated, version-agnostic services rather than local library imports.

System Architecture

Bencher employs a client-server architecture to ensure strict isolation and portability. The core of the system is a Coordination Server that manages requests from a client-side optimizer and delegates them to specific Benchmark Servers.

The Bencher architecture isolates the client from the benchmark servers via a coordination layer.

Key Components:

  • Isolation via Containers: Each benchmark environment is encapsulated in a Docker or Singularity container. This ensures that the execution environment is identical across different machines and research groups.
  • gRPC Communication: Communication between the optimizer (Client) and the benchmarks occurs via gRPC. This allows the optimizer to run in a modern environment (e.g., Python 3.12) while interacting with benchmarks trapped in legacy environments.
  • Lightweight Client: The bencherscaffold package provides a minimal interface for users to evaluate points, removing the need to install heavy benchmark dependencies locally.

Supported Benchmarks

Bencher provides a unified portal to over 80 benchmarks spanning continuous, categorical, and binary domains. These range from synthetic functions to complex robotics simulations and high-dimensional hyperparameter optimization tasks.

A comprehensive list of benchmarks supported by Bencher across various domains and sources.

Notable inclusions include:

  • LassoBench: High-dimensional problems ($d$ up to 19,959).
  • MuJoCo: Standard robotics tasks like Ant, Hopper, and Walker.
  • BBOB & IOHexperimenter: Widely used synthetic suites for algorithm profiling.
  • PBO: A suite of 23 binary domain benchmarks.

To further simplify algorithm development, all continuous benchmarks are automatically normalized to a unit hypercube $[0, 1]^d$.

Insights and Reproducibility

The fundamental shift in Bencher is treating benchmarks as external services. This "clean abstraction boundary" prevents local environment pollution and ensures long-term stability. By hosting pre-built images on Docker Hub, Bencher provides "Reproducibility as a Service," guaranteeing that a specific benchmark version remains consistent for all users.

Furthermore, the framework is designed for High-Performance Computing (HPC) environments. It integrates with Slurm and Singularity, allowing researchers to scale evaluations across large clusters without manual environment configuration on every node.

Conclusion and Future Work

Bencher currently focuses on unconstrained, single-objective optimization. While it covers a significant portion of the BBO landscape, future iterations aim to expand its capabilities. Planned developments include support for multi-objective and multi-fidelity optimization, constrained problems, and more diverse search domains such as graph-based benchmarks. By providing a stable, containerized foundation, Bencher aims to become the standard for fair and reproducible BBO research.

Summary updated

BibTeX
@inproceedings{papenmeier2025bencher,
  author = {Leonard Papenmeier and Luigi Nardi},
  title = {{Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization}},
  booktitle = {CODEML @ ICML},
  year = {2025},
  url = {https://openreview.net/forum?id=1N7Na3O8ak},
}
Figure