Abstract
Featuremetric benchmarking is a novel framework for evaluating quantum hardware by mapping performance against high-dimensional circuit features. By utilizing Gaussian Process regression and quasirandom sampling, this approach creates predictive models that outperform traditional volumetric benchmarking in both data efficiency and accuracy. This framework allows for the creation of "capability regions" that account for the complex, multi-dimensional nature of quantum circuit execution.
Introduction: The Limits of Volumetric Benchmarking
As quantum hardware scales, the industry has relied heavily on volumetric benchmarking, which plots success rates against circuit width ($w$) and depth ($d$). However, width and depth are often insufficient predictors of success. Two circuits with identical $(w, d)$ shapes can exhibit vastly different performance if one contains a high density of error-prone two-qubit gates while the other consists primarily of single-qubit gates.
Traditional volumetric benchmarking is descriptive—it records past performance—but lacks predictive power for novel circuit configurations. This creates a critical gap in our ability to estimate how a specific algorithm will perform without running it directly on the hardware.
Methodology: Featuremetric Mapping
To address these limitations, the authors propose a generalized framework called featuremetric benchmarking. The process involves identifying a set of features (e.g., width, depth, two-qubit gate density) and using Sobol sequences—a type of quasirandom sampling—to efficiently "fill up" the multi-dimensional feature space without the data-heavy requirements of exhaustive grid searches.
The "secret sauce" of the framework is Monotonic Gaussian Process (GP) Regression. By enforcing the physical intuition that quantum performance should generally decrease as circuits become deeper or denser, the model can interpolate between measured points with high accuracy, even in data-sparse regions. This prevents the model from making physically impossible predictions and significantly improves data efficiency.
Experimental Validation and Results
The framework was validated using datasets from IBM Q Montreal, IBM Q Algiers, and IonQ Forte1. A key insight from the IBM Q Algiers study was the massive impact of two-qubit gate density. A circuit with a specific width and depth saw its success probability plummet from 51% to 7% simply by increasing the density of two-qubit gates from 0 to 0.25.
By accounting for this third dimension, the "noise" typically seen in 2D volumetric plots is revealed to be the deterministic effect of an ignored variable. Furthermore, the study demonstrated that a GP model trained on just 20% of the data could accurately predict the performance of the remaining 80%, with prediction error converging to approximately 3% after only 75 training points.
Conclusion and Future Work
Featuremetric benchmarking provides a more faithful representation of quantum hardware capabilities by moving beyond simple width-depth metrics. By incorporating physical priors like monotonicity and using efficient sampling techniques, it offers a path toward automated, predictive hardware characterization. Future iterations may utilize active learning, where the model automatically identifies areas of highest uncertainty in the feature space and requests specific hardware executions to improve its accuracy most efficiently.
Summary updated