Abstract
MathOptAI.jl is an open-source Julia library designed to bridge the gap between machine learning (ML) and mathematical optimization. By allowing users to embed pre-trained ML models—such as neural networks, decision trees, and Gaussian processes—directly into JuMP optimization models, the library facilitates the use of ML surrogates in complex engineering and scientific workflows. This paper details the library's architecture, highlighting its unique "gray-box" formulation that enables GPU-accelerated gradient calculations via Python-based tools like PyTorch, while maintaining the flexibility of the Julia/JuMP ecosystem.
Introduction
In modern engineering and scientific research, we frequently encounter subsystems that are too complex to be described by simple, closed-form physics equations. To manage this complexity, researchers often employ machine learning models as "surrogates" to approximate these processes. However, a significant challenge arises when these surrogates must be integrated into a larger mathematical optimization problem—for instance, optimizing a design where the performance metric is the output of a deep neural network.
Most mathematical optimization solvers require explicit algebraic forms or high-quality derivatives (gradients and Hessians) to find optimal solutions. Manually translating a complex model, like a deep neural network, into solver-friendly equations is both tedious and computationally inefficient. While tools exist in the Python ecosystem, such as OMLT, there has been a notable gap for a high-performance tool within the Julia/JuMP ecosystem capable of handling both simple algebraic reformulations and massive, high-parameter ML models. MathOptAI.jl was developed to fill this gap.
Methodology
MathOptAI.jl introduces three distinct strategies for representing an ML predictor within a JuMP optimization model, allowing users to choose the approach that best fits their solver and model size:
- Full-space Representation: In this approach, every intermediate step of the ML model (e.g., every layer and activation in a neural network) is added as an individual decision variable and constraint. This maintains a "sparse" problem structure that many solvers can navigate efficiently, though it can lead to a massive increase in the number of variables for deep models.
- Reduced-space Representation: The entire ML model is collapsed into a single, complex mathematical expression. This minimizes the number of variables in the optimization problem but creates "dense" expressions that can become computationally overwhelming for certain solvers.
- Gray-box Formulation: This is the library's most innovative feature. Instead of explicitly writing out the equations, the library provides the solver with "callback oracles." When the solver requires the value or gradient of the ML model, MathOptAI.jl calls an external library (such as PyTorch or Flux.jl) to calculate it on the fly.
The Gray-Box Advantage and GPU Acceleration
The gray-box approach allows MathOptAI.jl to leverage Julia’s Python interoperability to trigger PyTorch’s GPU-accelerated automatic differentiation. This creates a powerful hybrid workflow: the optimization solver runs on the CPU within the Julia environment, while the heavy lifting of the neural network's gradient calculations is offloaded to the GPU in Python. This ensures that the optimization model size only scales with the inputs and outputs of the ML model, rather than its internal parameter count.
Key Insights and Features
MathOptAI.jl is built on the principle of modularity and extensibility. Key insights from its development include:
- Predictors as Compositions: The library treats all ML components as "predictors." For example, a neural network layer is viewed as a composition of an
Affinepredictor and aReLUpredictor. This modular design makes the codebase maintainable and easy to extend. - Extensibility via Multiple Dispatch: Leveraging Julia’s multiple dispatch system, adding support for new ML libraries (like
GLM.jl) is straightforward, often requiring only a few lines of code to define how a new model type should be handled. - Solver Independence: Unlike solver-specific tools (e.g., Gurobi’s ML library), MathOptAI.jl is compatible with any solver supported by JuMP, including Ipopt for nonlinear problems and HiGHS for mixed-integer programming.
Comparative Analysis
Compared to existing tools like OMLT or Gurobi’s internal ML integration, MathOptAI.jl offers unique capabilities. It is currently the only general-purpose library that supports the gray-box formulation alongside GPU acceleration for ML predictors within an optimization model. It supports a wide array of layers (Linear, ReLU, Sigmoid, Softmax, Tanh) and model types, including Decision Trees, Random Forests, Gradient Boosted Trees, and Gaussian Processes.
Limitations and Future Work
While MathOptAI.jl provides a robust framework, it currently has some limitations. All inputs and outputs must be flattened into one-dimensional vectors, as the library does not yet natively support multi-dimensional tensors (e.g., 3D images). Furthermore, the gray-box approach is primarily suitable for "local" solvers like Ipopt. Because the solver cannot "see" the internal algebraic structure of the ML model in gray-box mode, it cannot perform the spatial branching required for "global" optimization.
Future development will focus on creating decomposition algorithms that can exploit the specific structure of ML predictors to solve these hybrid models more efficiently, as well as expanding support for higher-dimensional data structures.
Conclusion
MathOptAI.jl provides a flexible, high-performance bridge between the worlds of machine learning and mathematical optimization. By offering multiple representation strategies and a unique GPU-accelerated gray-box formulation, it enables researchers to incorporate massive, complex ML surrogates into their optimization workflows without sacrificing performance or flexibility.
Summary updated