In traditional machine learning, we train models to be "accurate." We minimize the distance between a prediction and a ground-truth value, often using metrics like Mean Squared Error (MSE). However, in many industrial applications, a prediction is just the first step. The prediction (e.g., tomorrow’s traffic or energy prices) serves as an input to an optimization problem (e.g., finding the shortest delivery route or the cheapest production schedule).
The core insight of Decision-Focused Learning (DFL)—also known as "Predict-and-Optimize"—is that a statistically accurate model is not necessarily a useful one. As Jayanta Mandi, Ferdinando Fioretto, and Tias Guns argue in their comprehensive survey, DFL shifts the training objective from minimizing prediction error to minimizing downstream regret: the difference in cost between the decision made using predicted parameters and the decision that would have been made with perfect foresight.
The Accuracy-Utility Gap
Why does MSE fail us? Consider a delivery route. A small prediction error on a highway might not change the optimal path at all. However, the same small error on a bridge—perhaps causing the model to think the bridge is closed—could force the optimizer to choose a massive, costly detour. Standard loss functions treat these two errors as identical because they have the same magnitude. DFL, by contrast, "sees" the bridge and the highway through the lens of the optimizer, penalizing errors that lead to poor decisions more heavily than errors that are decision-irrelevant.
This paradigm shift is explored across various domains, from health supply chains by Hamsa Bastani to inventory routing by Azmine Toushik Wasi. In each case, the goal is to prevent small statistical errors from snowballing into expensive real-world failures.
The Computational Bottleneck
If DFL is so effective, why isn't it the default? The challenge lies in the training loop. To compute the "regret" for a single training instance, the model must:
- Predict the parameters.
- Solve the optimization problem using those predictions.
- Compare that solution to the optimal one.
As noted by Jayanta Mandi and Emir Demirovic in Smart Predict-and-Optimize, many real-world problems are NP-hard. Solving them thousands of times per epoch creates a massive computational bottleneck that prevents scaling to large datasets. Furthermore, combinatorial solvers are typically discrete and non-differentiable, making it impossible to use standard backpropagation.
Overcoming the Solver Bottleneck
Researchers have developed several clever strategies to bypass the need for a full solver call during every gradient update:
1. Learning to Rank
Instead of predicting exact cost coefficients, we can treat the problem as a ranking task. Jayanta Mandi and Maxime Mulamba demonstrate in Predict and Optimize: Through the Lens of Learning to Rank that if a model can correctly rank the "best" feasible solution above sub-optimal ones, it will naturally minimize regret. This approach is often more computationally efficient and GPU-friendly than direct optimization.
2. Solution Caching and Surrogates
Rather than solving the problem from scratch, we can maintain a "pool" of previously found solutions. Maxime Mulamba and Michele Lombardi use noise-contrastive estimation to treat these cached solutions as negative examples, effectively training the model to "push away" from bad decisions. Others, like Gaetano Signorelli, use Gaussian Process surrogates to mimic the solver's behavior at a fraction of the cost.
3. Geometric and Solver-Free Approaches
Some of the most recent work seeks to eliminate the solver entirely during training. Senne Berden and Tias Guns introduced LAVA (Loss via Adjacent Vertex Alignment) in Solver-Free Decision-Focused Learning. LAVA uses the geometric properties of linear programs—specifically the relationship between a predicted cost vector and the vertices of the feasible region—to create a differentiable loss function that requires no solver calls during the training phase.
A Unified Perspective on Regret
The field is currently moving toward a unified understanding of these diverse methods. Stefano Teso and Michele Lombardi suggest that regret minimization serves as a "Rosetta Stone," linking decision-focused learning with structured-output prediction and surrogate-based optimization.
While benchmarks by Haoyu Geng and Runzhong Wang show that there is no "silver bullet" for every problem, the evidence is clear: when machine learning models are aware of the decisions they support, they produce significantly better real-world outcomes.