GENO - Optimization for Classical Machine Learning Made Fast and Easy

Sören Laue, Matthias Mitterreiter, Joachim Giesen

Abstract

GENO (GENeric Optimization) is a framework designed to bridge the gap between easy-to-use modeling languages and high-performance, hand-coded solvers. By allowing users to define optimization problems in a simple, MATLAB-like syntax, GENO automatically generates Python solvers that are as fast as—and often faster than—specialized, state-of-the-art implementations.

Introduction

Most classical machine learning tasks, such as Support Vector Machines (SVMs), elastic net, or matrix factorization, are fundamentally optimization problems. Currently, researchers and practitioners face a difficult trade-off when implementing these models. Standard toolboxes like scikit-learn are fast but rigid, making it difficult to add custom constraints. Conversely, modeling languages like CVXPY or JuMP offer flexibility but often suffer from significant computational overhead because they transform specific problem instances into standard forms, leading to representational bloat. The third option—hand-writing solvers—is efficient but requires weeks of expert design and debugging.

GENO aims to resolve this tension by providing the flexibility of a modeling language with the performance of a hand-engineered solver.

The GENO Framework

The GENO pipeline consists of two tightly coupled components: a high-level modeling language and an automatic solver generation engine.

Modeling Language

Users specify their objective functions and constraints using vectorized linear algebra expressions, similar to MATLAB or NumPy. A key advantage of GENO is that users do not need to specify dimensions or ensure the problem is convex. As shown in the examples below, the syntax is intuitive and handles various constraints seamlessly.

Examples of optimization problems defined in GENO's MATLAB-like syntax, including Rayleigh quotient, non-negative least squares, and basis pursuit.

Automatic Solver Generation

Unlike traditional modeling stacks that transform a specific problem instance, GENO transforms the entire class of the problem into a general standard form. This "class-level transformation" is the core innovation that avoids representational inefficiency. The complexity of the generated solver does not grow with the size of the data.

The generation process involves:

  • Automatic Differentiation: A specialized matrix and tensor calculus algorithm computes gradients automatically.
  • Optimization Engine: The core solver utilizes the L-BFGS-B algorithm, a quasi-Newton method suited for large-scale problems. To handle non-smooth functions and constraints, GENO extends this with an Augmented Lagrangian approach.

The result is a tailored Python solver produced in just a few milliseconds.

Performance Evaluation

GENO's performance has been evaluated against a variety of baselines, including standard libraries (LIBSVM, LIBLINEAR), recent research-specific solvers from ICML and NeurIPS, and modeling stacks like CVXPY paired with commercial solvers (Gurobi, Mosek).

Key results include:

  • Performance Parity: GENO-generated solvers match the efficiency of well-engineered, specialized solvers like LIBSVM.
  • Superiority over Specialized Research: In many instances, GENO's automatically generated solvers outperformed specialized solvers published at top-tier conferences by significant margins.
  • Massive Speedups: Compared to traditional modeling approaches (CVXPY + Gurobi), GENO is often orders of magnitude faster due to its efficient problem representation.

Conclusion and Future Work

GENO provides a powerful tool for rapid prototyping and deployment of machine learning models. While it currently guarantees only local optima for non-convex problems and is limited to its implemented syntax, its performance is already competitive with pure C++ implementations. Future work involves extending the underlying matrix/tensor calculus to handle even more complex mathematical structures and expanding the library of supported operators.

Summary updated

Abstract

Most problems from classical machine learning can be cast as an optimization problem. We introduce GENO (GENeric Optimization), a framework that lets the user specify a constrained or unconstrained optimization problem in an easy-to-read modeling language. GENO then generates a solver, i.e., Python code, that can solve this class of optimization problems. The generated solver is usually as fast as hand-written, problem-specific, and well-engineered solvers. Often the solvers generated by GENO are faster by a large margin compared to recently developed solvers that are tailored to a specific problem class.An online interface to our framework can be found at http://www.geno-project.org.
BibTeX
@inproceedings{laue2020geno,
  author = {Sören Laue and Matthias Mitterreiter and Joachim Giesen},
  title = {{GENO - Optimization for Classical Machine Learning Made Fast and Easy}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i09.7097},
}
Figure