Qoncord: A Multi-Device Job Scheduling Framework for Variational Quantum Algorithms

Meng Wang, Poulami Das, Prashant J. NairMICRO 2024

TL;DR

Qoncord is a new scheduling framework that dramatically speeds up Variational Quantum Algorithms (VQAs) by breaking the "one job, one device" rule. By running noise-resilient early training on idle, low-fidelity machines and reserving high-fidelity hardware for final fine-tuning, it achieves results up to 17.4x faster than traditional cloud scheduling.

What Problem Does This Paper Address?

We are currently in the NISQ (Noisy Intermediate-Scale Quantum) era. Quantum computers are powerful but prone to errors (noise) and are a scarce resource. Most users access them via the cloud (IBM, AWS, etc.), creating a massive logistical headache.

Variational Quantum Algorithms (VQAs)—the most promising apps for today's hardware—are particularly demanding. They are iterative: they run a quantum circuit, use a classical optimizer to tweak parameters, and repeat hundreds of times. Because noise can trap these algorithms in "local optima" (bad solutions), researchers often use restarts, running the entire process multiple times from different starting points to find the best result.

Currently, cloud schedulers are "dumb." A user picks a high-fidelity device, and their job sits in a long queue while lower-fidelity devices sit idle. This leads to a "high-fidelity bottleneck" where everyone waits for the same few machines, even though not every part of the VQA process actually requires premium hardware.

Methods

The authors introduce Qoncord, an automated framework that treats VQA training as a multi-stage journey rather than a single monolithic task. It relies on two key insights:

  1. Phase Splitting: VQA training has two phases. The Exploration phase (early iterations) is surprisingly resilient to noise. Qoncord runs this phase on "low-fidelity, low-load" devices that have shorter queues. Once the algorithm nears a solution, it enters the Fine-tuning phase, where Qoncord migrates the job to a "high-fidelity" device for the final, precise adjustments.
  2. Smart Restart Management: Instead of running every random restart to completion, Qoncord evaluates them early. By looking at "Intermediate Value Clusters," it identifies which restarts are likely to lead to the global optimum and kills the "bad" ones at roughly 40% completion, saving massive amounts of compute time.

To make this work, Qoncord uses an Execution Fidelity Estimator to rank available devices and an Adaptive Convergence Checker that monitors both the "Expectation Value" (the score) and Shannon Entropy (a measure of output randomness) to decide exactly when to switch devices or terminate a restart.

Datasets

The researchers evaluated Qoncord using simulated noisy environments based on real-world error profiles from:

  • IBM Quantum systems: Including the 27-qubit ibmq_kolkata (high fidelity) and ibmq_toronto (low fidelity), as well as the 127-qubit ibm_kyoto.
  • IonQ systems: Using the Forte device noise profile for sensitivity studies.
  • Benchmarks: They tested two primary VQA types: QAOA (Quantum Approximate Optimization Algorithm) for solving Max-Cut problems on random graphs, and VQE (Variational Quantum Eigensolver) for finding the ground state energy of a Hydrogen ($H_2$) molecule.

Key Insights and Ideas

  • The "Entropy Arc": One of the coolest findings is the relationship between Shannon Entropy and the algorithm's progress. Early on, entropy is high (randomness). As the VQA finds a "good" region, entropy often peaks before dropping as it settles into an optimum. Qoncord uses this "arc" to trigger the migration from cheap to premium hardware.
  • Exploration is Robust: The paper quantitatively proves that you don't need the world's best qubits to find the neighborhood of a solution. This allows "junk" hardware to become useful "scouts" for the "elite" hardware.
  • Incentivizing Providers: This framework gives cloud providers a reason to keep older, lower-fidelity machines online, as they can now contribute meaningfully to high-end research tasks rather than just gathering dust.

Key Results

  • Massive Speedups: Qoncord achieved similar solution quality 17.4x faster than the baseline approach of running everything on a single high-fidelity device.
  • Better Solutions: When given the same "time budget" as the baseline, Qoncord produced solutions that were 13.3% more accurate.
  • Throughput: In a simulated cloud environment with 1,000 jobs, Qoncord significantly outperformed "Least Busy" or "Best Fidelity" policies by balancing the load across the entire fleet.
  • Efficiency: For a 50-restart QAOA task, Qoncord filtered out 31 low-performing restarts early, executing 70% of the total iterations on low-fidelity devices without sacrificing the final result.

Limitations and Future Work

  • Calibration Drift: The framework relies on device calibration data (error rates). In the real world, these rates drift over time. While the authors suggest periodic "re-calibration" jobs, the system's performance is tied to the accuracy of these hardware snapshots.
  • Migration Overhead: While the paper accounts for queuing, the actual "cost" of moving a job's state between different cloud providers or different types of quantum backends (e.g., moving from a superconducting qubit to a trapped-ion system) could introduce latencies not fully explored here.
  • Real Hardware Validation: Most results are based on high-fidelity noise simulations. Testing Qoncord on a live, multi-vendor cloud (like AWS Braket) with real-time fluctuating queues is the logical next step.

Summary updated

Abstract

Quantum computers face challenges due to limited resources, particularly in cloud environments. Despite these obstacles, Variational Quantum Algorithms (VQAs) are considered promising applications for present-day Noisy Intermediate-Scale Quantum (NISQ) systems. VQAs require multiple optimization iterations to converge on a globally optimal solution. Moreover, these optimizations, known as restarts, need to be repeated from different points to mitigate the impact of noise. Unfortunately, the job scheduling policies for each VQA task in the cloud are heavily unoptimized. Notably, each VQA execution instance is typically scheduled on a single NISQ device. Given the variety of devices in the cloud, users often prefer higher-fidelity devices to ensure higher-quality solutions. However, this preference leads to increased queueing delays and unbalanced resource utilization. We propose Qoncord, an automated job scheduling framework to address these cloud-centric challenges for VQAs. Qoncordleverages the insight that not all training iterations and restarts are equal, Qoncord strategically divides the training process into exploratory and fine-tuning phases. Early exploratory iterations, more resilient to noise, are executed on less busy machines, while fine-tuning occurs on high-fidelity machines. This adaptive approach mitigates the impact of noise and optimizes resource usage and queuing delays in cloud environments. Qoncord also significantly reduces execution time and minimizes restart overheads by eliminating low-performance iterations. Thus, Qoncord offers similar solutions 17.4x faster. Similarly, it can offer 13.3% better solutions for the same time budget as the baseline.
BibTeX
@inproceedings{wang2024qoncord,
  author = {Meng Wang and Poulami Das and Prashant J. Nair},
  title = {{Qoncord: A Multi-Device Job Scheduling Framework for Variational Quantum Algorithms}},
  booktitle = {MICRO},
  year = {2024},
}
Figure