Abstract
Benchmarking continuous optimization algorithms requires access to diverse and standardized test problems. For decades, the CUTEst collection has served as the industry standard, yet its reliance on the legacy Standard Input Format (SIF) and Fortran-based decoders has created significant barriers for modern researchers. This paper introduces S2MPJ (SIF to Matlab, Python, and Julia), an open-source decoder that translates SIF problems into native, executable scripts. By eliminating the need for Fortran compilers or complex MEX-file interfaces, S2MPJ enables the direct computation of objective values and derivatives for over 1,000 problems within the user's preferred high-level environment. Furthermore, the Matlab implementation introduces support for reduced-precision computations, facilitating research into numerical robustness.
Introduction
In the field of continuous optimization, the CUTEst (Constrained and Unconstrained Testing Environment with extras) collection is the "gold standard" for software benchmarking. It encompasses a vast array of problems ranging from simple quadratic programs to complex non-linear constraints. However, the collection’s reliance on the Standard Input Format (SIF) presents a significant hurdle. Traditionally, using these problems in modern environments like Matlab, Python, or Julia required a Fortran-based decoder and the maintenance of fragile cross-language interfaces (such as MEX files or Docker containers).
S2MPJ addresses this "Fortran problem" by providing a framework that translates SIF files directly into native code. This approach allows researchers to run standard optimization benchmarks with minimal setup, fostering greater reproducibility and ease of use in the optimization community.
The S2MPJ Framework
S2MPJ is designed as a standalone decoder that reads a SIF file and generates a single executable output file (e.g., .m, .py, or .jl). Unlike previous attempts that merely wrapped Fortran binaries, S2MPJ produces actual source code in the target language. This code utilizes a specialized library, s2mpjlib, to handle the evaluation of functions and their derivatives.
Group-Partially-Separable (GPS) Structure
A core technical achievement of S2MPJ is its preservation of the group-partially-separable (GPS) structure inherent in SIF problems. In this structure, the objective function is decomposed into a sum of "element" functions, each transformed by a "group" function. By maintaining this decomposition, S2MPJ allows for the efficient computation of gradients and Hessians. Rather than treating the problem as a black box, the generated code exploits the underlying sparsity and structure of the problem, which is essential for large-scale optimization.
Language-Specific Implementations
The decoder adapts to the specific paradigms of its target languages:
- Matlab and Julia: These versions utilize function-based interfaces and persistent data structures to manage problem states.
- Python: Problems are treated as classes, aligning with Python’s object-oriented philosophy.
- Reduced-Precision Support: The Matlab version integrates with the Symbolic Math Toolbox to support reduced-precision computations. This allows researchers to simulate how optimization algorithms perform under lower numerical accuracy, a critical area of study for modern large-scale and hardware-constrained computing.
Evaluation and Results
The authors validated S2MPJ using the comprehensive CUTEst collection, which includes subsets such as the Argonne test set, Toint’s collection, and the Hock and Schitkowski problems.
Broad Compatibility and Accuracy
S2MPJ successfully decoded and executed 1,075 problems across all three target languages. To ensure numerical integrity, the results were compared against the original Fortran-based evaluations via the pycutest wrapper. The comparison revealed excellent coherence, with relative differences typically remaining below $10^{-14}$.
Precision Insights
An interesting finding during development involved the precision of constants. The original Fortran CUTEst tools often store real numbers with only eight significant digits in intermediate data files. Because S2MPJ generates fully executable code, it can maintain higher precision for constants (such as mesh sizes) calculated during the problem's setup phase, potentially providing more accurate benchmarks than the legacy tools.
Limitations and Future Work
While S2MPJ significantly lowers the barrier to entry for using CUTEst, it has some trade-offs:
- Performance: As interpreted or JIT-compiled native code, S2MPJ is generally slower than compiled Fortran, particularly for data-intensive problems.
- Memory: The setup phase in Julia can be memory-intensive for exceptionally large problem sets.
- SIF Coverage: The decoder does not currently support SIF files that require external Fortran subroutines or those written in "Free Form" SIF.
Future development aims to implement automatic code vectorization to bridge the performance gap and improve storage efficiency for precompiled problem files.
Conclusion
S2MPJ modernizes the use of the CUTEst collection by providing a native, easy-to-use interface for the three most popular languages in numerical computing. By achieving the "three-line" goal—allowing users to decode, set up, and evaluate a problem in just a few commands—S2MPJ ensures that the optimization community can continue to rely on these classic benchmarks without the technical debt of legacy Fortran dependencies. The decoder and the library of 1,075 pre-decoded problems are now publicly available on GitHub.
Summary updated