TL;DR
S2MPJ is a new, open-source decoder that brings the massive CUTEst optimization test set—the "gold standard" for benchmarking—directly into native Matlab, Python, and Julia. By bypassing the traditional requirement for Fortran compilers and complex MEX-file interfaces, it allows researchers to run over 1,000 standard optimization problems with just a few lines of code.
What Problem Does This Paper Address?
For nearly 30 years, the CUTEst (and its predecessors CUTE and CUTEr) collection has been the primary resource for testing continuous optimization software. However, it has a "Fortran problem." The test problems are written in an obscure "Standard Input Format" (SIF) that requires a Fortran-based decoder to function.
For modern researchers working in Matlab, Python, or Julia, using CUTEst usually means wrestling with MEX files, Docker containers, or complex cross-language interfaces that are difficult to maintain and restrict the ability to modify problem parameters. S2MPJ (SIF to Matlab, Python, and Julia) eliminates this friction by translating SIF files into native, executable scripts in the user's language of choice.
Methods
The authors developed a framework that reads SIF files and translates them into a single executable output file (.m, .py, or .jl).
Key technical features include:
- Native Implementation: It does not "wrap" Fortran; it generates actual code in the target language that utilizes specialized libraries (
s2mpjlib) for evaluation. - GPS Structure Preservation: It maintains the "group-partially-separable" (GPS) structure of the original problems. This structure is crucial because it allows for the efficient computation of gradients and Hessians by breaking the objective function into smaller, manageable "groups" and "elements."
- Reduced-Precision Support: In the Matlab version, S2MPJ optionally supports reduced-precision computations (via the Symbolic Math Toolbox), allowing researchers to test how optimization algorithms behave under lower numerical accuracy—a hot topic in modern large-scale computing.
Datasets
The tool was evaluated using the CUTEst collection, which incorporates several famous subsets:
- The Argonne test set
- Toint’s collection
- Buckley’s problems
- COPS
- The Hock and Schitkowski collection
- The Lukšan and Vlček test set
In total, the authors successfully tested the decoder on 1,075 problems from this collection.
Key Insights and Ideas
- The "Three-Line" Goal: The authors prioritized ease of use. A user can decode a problem, set up its data structure, and evaluate the objective function/constraints in roughly three commands.
- Precision Discrepancies: A fascinating "aha moment" in the paper involves the discovery that the original Fortran CUTEst tools only store real numbers with eight significant digits in their intermediate data files. S2MPJ, by being fully executable, can often maintain higher precision for constants (like mesh sizes) computed during the setup phase.
- Language-Specific Nuances: The paper details how the decoder handles the different philosophies of the three languages—treating Python problems as classes, while Matlab and Julia use function-based interfaces with persistent structures.
Key Results
- Broad Compatibility: S2MPJ successfully decoded and ran 1,075 problems across all three target languages.
- Numerical Accuracy: When compared against the original Fortran-based evaluations (via the
pycutestwrapper), S2MPJ showed excellent coherence, with relative differences typically falling below $10^{-14}$. - Open Access: The decoder and the entire library of 1,075 pre-decoded problems are publicly available on GitHub, significantly lowering the barrier to entry for optimization benchmarking.
Limitations and Future Work
While S2MPJ is a major step forward, the authors acknowledge several limitations:
- Unsupported SIF Features: It cannot currently handle SIF files that call external Fortran subroutines or those written in "Free Form" SIF.
- Performance: Because the generated code is "native" and interpreted/JIT-compiled, it is intrinsically slower than the original compiled Fortran counterparts, especially for problems with a massive amount of data.
- Memory Usage: The 'setup' action in Julia can be memory-intensive for very large problems.
- Future Directions: The authors aim to implement automatic code vectorization and better storage for precompiled problem files to improve execution speed.
Summary updated