Abstract
Benchmarking black-box optimization (BBO) algorithms is often hindered by "dependency hell," where conflicting software requirements and outdated library versions make reproducible comparisons difficult. We present Bencher, a modular framework that isolates benchmarks within containerized environments. By decoupling optimization logic from benchmark execution via a unified gRPC interface, Bencher enables seamless evaluation across over 80 diverse benchmarks without environment conflicts.
Introduction
In Black-Box Optimization (BBO), researchers seek to find the best input for functions where the internal mechanics are unknown, relying solely on input-output observations. Validating new algorithms requires testing against a broad spectrum of real-world benchmarks. However, these benchmarks often have incompatible dependencies—for instance, one might require Python 3.8 and specific versions of MuJoCo, while another needs the latest scientific stack.
This fragmentation leads to "software rot" and inconsistent experimental setups, making it nearly impossible to compare results across different papers fairly. Bencher addresses this by treating benchmarks as isolated, version-agnostic services rather than local library imports.
System Architecture
Bencher employs a client-server architecture to ensure strict isolation and portability. The core of the system is a Coordination Server that manages requests from a client-side optimizer and delegates them to specific Benchmark Servers.
Key Components:
- Isolation via Containers: Each benchmark environment is encapsulated in a Docker or Singularity container. This ensures that the execution environment is identical across different machines and research groups.
- gRPC Communication: Communication between the optimizer (Client) and the benchmarks occurs via gRPC. This allows the optimizer to run in a modern environment (e.g., Python 3.12) while interacting with benchmarks trapped in legacy environments.
- Lightweight Client: The
bencherscaffoldpackage provides a minimal interface for users to evaluate points, removing the need to install heavy benchmark dependencies locally.
Supported Benchmarks
Bencher provides a unified portal to over 80 benchmarks spanning continuous, categorical, and binary domains. These range from synthetic functions to complex robotics simulations and high-dimensional hyperparameter optimization tasks.
Notable inclusions include:
- LassoBench: High-dimensional problems ($d$ up to 19,959).
- MuJoCo: Standard robotics tasks like Ant, Hopper, and Walker.
- BBOB & IOHexperimenter: Widely used synthetic suites for algorithm profiling.
- PBO: A suite of 23 binary domain benchmarks.
To further simplify algorithm development, all continuous benchmarks are automatically normalized to a unit hypercube $[0, 1]^d$.
Insights and Reproducibility
The fundamental shift in Bencher is treating benchmarks as external services. This "clean abstraction boundary" prevents local environment pollution and ensures long-term stability. By hosting pre-built images on Docker Hub, Bencher provides "Reproducibility as a Service," guaranteeing that a specific benchmark version remains consistent for all users.
Furthermore, the framework is designed for High-Performance Computing (HPC) environments. It integrates with Slurm and Singularity, allowing researchers to scale evaluations across large clusters without manual environment configuration on every node.
Conclusion and Future Work
Bencher currently focuses on unconstrained, single-objective optimization. While it covers a significant portion of the BBO landscape, future iterations aim to expand its capabilities. Planned developments include support for multi-objective and multi-fidelity optimization, constrained problems, and more diverse search domains such as graph-based benchmarks. By providing a stable, containerized foundation, Bencher aims to become the standard for fair and reproducible BBO research.
Summary updated