Efficiently Controlling Multiple Risks with Pareto Testing

Bracha Laufer-Goldshtein, Adam Fisch, Regina Barzilay, Tommi S. Jaakkola

Abstract

Machine learning applications frequently come with multiple diverse objectives and constraints that can change over time. Accordingly, trained models can be tuned with sets of hyper-parameters that affect their predictive behavior (e.g., their run-time efficiency versus error rate). As the number of constraints and hyper-parameter dimensions grow, naively selected settings may lead to sub-optimal and/or unreliable results. We develop an efficient method for calibrating models such that their predictions provably satisfy multiple explicit and simultaneous statistical guarantees (e.g., upper-bounded error rates), while also optimizing any number of additional, unconstrained objectives (e.g., total run-time cost). Building on recent results in distribution-free, finite-sample risk control for general losses, we propose Pareto Testing: a two-stage process which combines multi-objective optimization with multiple hypothesis testing. The optimization stage constructs a set of promising combinations on the Pareto frontier. We then apply statistical testing to this frontier only to identify configurations that have (i) high utility with respect to our objectives, and (ii) guaranteed risk levels with respect to our constraints, with specifiable high probability. We demonstrate the effectiveness of our approach to reliably accelerate the execution of large-scale Transformer models in natural language processing (NLP) applications. In particular, we show how Pareto Testing can be used to dynamically configure multiple inter-dependent model attributes -- including the number of layers computed before exiting, number of attention heads pruned, or number of text tokens considered -- to simultaneously control and optimize various accuracy and cost metrics.

Introduction

Modern machine learning models, particularly large Transformers, come with many "knobs" or hyperparameters that affect the trade-off between accuracy and computational cost. For instance, practitioners may want to deploy a model that is as fast as possible, provided its error rate stays below a certain threshold. However, as the number of tuning dimensions (e.g., token pruning, early exiting, and head pruning) and constraints grows, finding the optimal configuration becomes a "needle in a haystack" problem.

Overview of the Transformer pruning application where multiple inter-dependent attributes like tokens, layers, and heads are configured to meet specific accuracy and cost constraints.

Existing frameworks for "risk control" often lose effectiveness in these high-dimensional spaces. They either become computationally prohibitive or too conservative, failing to find useful configurations because they spread their statistical confidence too thin across too many candidates. This paper addresses this by focusing statistical power on the most promising candidates.

Methods: The Pareto Testing Framework

The authors propose Pareto Testing, a two-stage process that leverages the "Learn Then Test" (LTT) framework but enhances its efficiency:

  1. Stage 1: Constructing the Pareto Frontier: The calibration data is split into two parts. The first part ($D_{opt}$) is used to solve an unconstrained multi-objective optimization problem. This identifies the "Pareto frontier"—the set of configurations that offer the best possible trade-offs between all objectives (e.g., accuracy vs. latency). This drastically narrows the search space from thousands of possibilities to a handful of promising candidates.
  2. Stage 2: Sequential Statistical Testing: The second part of the data ($D_{testing}$) is used to perform rigorous hypothesis testing only on the configurations found in Stage 1. By ordering these candidates from "most likely to pass" to "least likely," the framework uses Fixed Sequence Testing (FST) to identify the highest-utility configuration that provably satisfies the user's risk constraints with high probability (e.g., $1-\delta$).
Illustration of the Pareto Testing process: identifying the Pareto frontier and sequentially testing hypotheses (H0 to H5) until a risk threshold is exceeded.

This approach is distribution-free and provides finite-sample guarantees, meaning it does not make assumptions about the underlying data distribution and works reliably even with limited calibration samples.

Key Results and Insights

The method was evaluated on five diverse NLP text classification tasks (IMDB, AG News, QNLI, QQP, and MNLI) using a BERT-base model. The results highlight several key advantages:

  • Superior Efficiency: Pareto Testing consistently found configurations with lower computational costs than existing risk-control baselines (like Split FST or 3D SGT) while maintaining the same strict error bounds.
  • Significant Speedups: On tasks like IMDB and AG News, the method achieved between 2x and 4.8x speedups in FLOPs while guaranteeing that the accuracy drop remained within a tiny user-defined margin (e.g., 2.5%).
  • Reliable Safety: In experiments with 100 random trials, Pareto Testing never violated the user-specified risk levels more than the allowed tolerance ($\delta$), whereas non-risk-controlling optimization methods frequently failed.
  • Multi-Dimensional Synergy: The framework successfully navigated the complex performance landscape where the "best" threshold for one dimension (e.g., token pruning) depends heavily on others (e.g., head pruning).

Conclusion

Pareto Testing provides a robust way to manage multiple risks in high-dimensional hyperparameter spaces. While it requires a held-out calibration dataset, it offers a principled path toward deploying efficient and safe machine learning models. Future work may extend this framework to computer vision or to enforce fairness and algorithmic recourse constraints in sensitive deployment scenarios.

Summary updated

Abstract

Machine learning applications frequently come with multiple diverse objectives and constraints that can change over time. Accordingly, trained models can be tuned with sets of hyper-parameters that affect their predictive behavior (e.g., their run-time efficiency versus error rate). As the number of constraints and hyper-parameter dimensions grow, naively selected settings may lead to sub-optimal and/or unreliable results. We develop an efficient method for calibrating models such that their predictions provably satisfy multiple explicit and simultaneous statistical guarantees (e.g., upper-bounded error rates), while also optimizing any number of additional, unconstrained objectives (e.g., total run-time cost). Building on recent results in distribution-free, finite-sample risk control for general losses, we propose Pareto Testing: a two-stage process which combines multi-objective optimization with multiple hypothesis testing. The optimization stage constructs a set of promising combinations on the Pareto frontier. We then apply statistical testing to this frontier only to identify configurations that have (i) high utility with respect to our objectives, and (ii) guaranteed risk levels with respect to our constraints, with specifiable high probability. We demonstrate the effectiveness of our approach to reliably accelerate the execution of large-scale Transformer models in natural language processing (NLP) applications. In particular, we show how Pareto Testing can be used to dynamically configure multiple inter-dependent model attributes -- including the number of layers computed before exiting, number of attention heads pruned, or number of text tokens considered -- to simultaneously control and optimize various accuracy and cost metrics.
BibTeX
@inproceedings{laufergoldshtein2023efficiently,
  author = {Bracha Laufer-Goldshtein and Adam Fisch and Regina Barzilay and Tommi S. Jaakkola},
  title = {{Efficiently Controlling Multiple Risks with Pareto Testing}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=cyg2YXn_BqF},
}
Figure