Symbolic Regression on FPGAs for Fast Machine Learning Inference

Ho Fung Tsoi, Adrian Alan Pol, Vladimir Loncar, Ekaterina Govorkova, Miles Cranmer, Sridhara Dasu, Peter Elmer, Philip Harris, Isobel Ojalvo, Maurizio Pierini

TL;DR

This paper introduces an end-to-end pipeline that compiles Symbolic Regression (SR) equations into FPGA firmware using the hls4ml framework. By replacing a 3-layer deep neural network with compact algebraic expressions, the authors achieved up to a 13x reduction in inference latency (down to 5 ns) while retaining over 90% accuracy for a critical real-time particle physics task.

What Problem Does This Paper Address?

High-energy physics (HEP) experiments, such as those at the CERN Large Hadron Collider (LHC), generate data at staggering rates. They process proton-proton collisions at 40 MHz, producing tens of terabytes of raw data per second. To handle this firehose, real-time "trigger systems" must filter data at the edge with an ultra-low maximum latency of $O(1)$ microsecond.

Field-Programmable Gate Arrays (FPGAs) are ideal for this environment. However, while deep neural networks (DNNs) offer high accuracy, deploying them on resource-constrained FPGAs is challenging due to their large memory and computational footprints. Finding the optimal trade-off between physics performance and hardware resource consumption remains a major hurdle.

Methods

Instead of traditional deep learning, the authors propose using Symbolic Regression (SR). SR searches the mathematical equation space to find simple analytic expressions that fit a dataset.

The key components of their novel end-to-end procedure include:

  • PySR + hls4ml Integration: They use PySR (an open-source tool driven by an evolutionary algorithm) to discover equations, and extend the hls4ml library to parse these expressions into High-Level Synthesis (HLS) code for Xilinx FPGAs.
  • Lookup Table (LUT) Approximations: To avoid expensive hardware implementations of complex functions (like sin, tan, log, and exp), they implement LUT-based approximations directly into the HLS pipeline.
  • Latency-Aware Training (LAT): Instead of treating all mathematical operations equally during the evolutionary search, they penalize operations based on their actual clock-cycle cost on an FPGA (e.g., tan costs significantly more than sin).
Figure 1. The sine (left) and tangent (right) functions evaluated with and without the use of LUTs, implemented in HLS with precision ⟨12, 6⟩, i.e., 12 bits variable with 6 integer bits. The LUT notation reads: [range start, range end; table size] for table definition. The lower panel shows the function deviation from the truth.

Datasets

The method is validated on a standard high-energy physics benchmark: multiclass jet tagging using a publicly available LHC dataset from Zenodo. The task involves classifying simulated particle jets into five categories (quarks, gluons, W bosons, Z bosons, or top quarks) based on 16 physics-motivated expert features.

Key Insights and Ideas

  • Direct Pareto Optimization: Unlike neural networks where architecture search is discrete and massive, SR naturally produces a Pareto front of models. This allows engineers to directly optimize and choose the best performance-to-resource trade-off.
  • Hardware-Informed Evolution: By feeding FPGA clock cycle costs back into the PySR evolutionary algorithm (via LAT), the algorithm naturally discovers equations that are inherently faster to execute in hardware.
  • LUTs for Speed: Replacing exact math library calls with small lookup tables drastically cuts down latency and resource usage with negligible impact on physics performance.

Key Results

  • Inference Latency: The SR models reduced execution time down to just 5 ns (1 clock cycle) for exponential and logarithmic models using LUTs, compared to 65 ns (13 clock cycles) for the baseline neural network—a 13-fold speedup.
  • Accuracy: The symbolic models successfully approximated the baseline 3-layer fully connected neural network, maintaining over 90% relative accuracy across various bit widths.
  • Resource Reduction: The implementation drastically minimized the utilization of Digital Signal Processors (DSPs) and hardware LUTs compared to the baseline network.
Figure 2. Relative accuracy as a function of bit width, for polynomial (top left), trigonometric (top right), exponential (bottom left), and logarithmic (bottom right) models. The relative accuracy is eval- uated with respect to the baseline QAT NN trained and implemented at corresponding precision. The number of integer bits is fixed at I = 12 for the exponential model and at I = 6 for other models.

Limitations and Future Work

  • High-Dimensional Scaling: Evolutionary algorithms for symbolic regression scale poorly to high-dimensional datasets. To mitigate this, the authors had to pre-select 6 out of the 16 available features using a random forest regressor before running PySR.
  • Future Directions: The authors suggest further optimization of the performance-resource trade-off and exploring other application domains beyond jet tagging where ultra-fast FPGA inference is required.

Summary updated

Abstract

The high-energy physics community is investigating the potential of deploying machine-learning-based solutions on Field-Programmable Gate Arrays (FPGAs) to enhance physics sensitivity while still meeting data processing time constraints. In this contribution, we introduce a novel end-to-end procedure that utilizes a machine learning technique called symbolic regression (SR). It searches the equation space to discover algebraic relations approximating a dataset. We use PySR (a software to uncover these expressions based on an evolutionary algorithm) and extend the functionality of hls4ml (a package for machine learning inference in FPGAs) to support PySR-generated expressions for resource-constrained production environments. Deep learning models often optimize the top metric by pinning the network size because the vast hyperparameter space prevents an extensive search for neural architecture. Conversely, SR selects a set of models on the Pareto front, which allows for optimizing the performance-resource trade-off directly. By embedding symbolic forms, our implementation can dramatically reduce the computational resources needed to perform critical tasks. We validate our method on a physics benchmark: the multiclass classification of jets produced in simulated proton-proton collisions at the CERN Large Hadron Collider. We show that our approach can approximate a 3-layer neural network using an inference model that achieves up to a 13-fold decrease in execution time, down to 5 ns, while still preserving more than 90% approximation accuracy.
BibTeX
@inproceedings{tsoi2024symbolic,
  author = {Ho Fung Tsoi and Adrian Alan Pol and Vladimir Loncar and Ekaterina Govorkova and Miles Cranmer and Sridhara Dasu and Peter Elmer and Philip Harris and Isobel Ojalvo and Maurizio Pierini},
  title = {{Symbolic Regression on FPGAs for Fast Machine Learning Inference}},
  booktitle = {EPJ},
  year = {2024},
  doi = {10.1051/epjconf/202429509036},
}
Figure