Machine learning models are typically born in flexible, high-level environments like Python, using frameworks such as PyTorch or Keras. These environments prioritize developer velocity, rapid prototyping, and easy debugging. However, production environments—such as real-time physics simulators, database engines, mobile applications, or embedded systems—demand low latency, minimal memory footprints, and strict predictability.
Running a heavy Python runtime in these production environments introduces a massive "Python tax" in the form of context-switching overhead, large dependency trees, and sub-optimal hardware utilization. To bridge this gap, a prominent line of research focuses on decoupling model training from model execution. Instead of forcing production environments to host research frameworks, this approach treats a trained model as an abstract computational graph that can be compiled, serialized, or transpiled into native, platform-independent representations.
The Blueprint: Intermediate Representations and Serialization
How do we extract a model from its training framework without losing its functional essence? The answer lies in standardized intermediate representations (IRs) and serialization formats.
As shown by Hongji Wang et al. in Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit, exporting models into unified formats like ONNX or TensorRT allows developers to preserve the core mathematical logic of a network while stripping away the training-specific boilerplate. This serialization acts as a universal bridge. For instance, L. K. Graczykowski et al. demonstrate in Using Machine Learning for Particle Identification in ALICE how standardized model exchange formats allow complex neural networks trained in Python to be executed natively in high-throughput C++ environments, achieving the extreme inference speeds required for real-time particle identification.
By serializing the model's weights and architecture into a platform-independent format, the model effectively becomes portable data rather than framework-bound code. In his work on Déployer des LLMs en local, Yannis Bendi-Ouis highlights how unified serialization formats enable large language models to be executed across diverse hardware environments (CPUs and GPUs) via efficient memory mapping. Similarly, Chen Chu et al. leverage this strategy in CityFlowER: An Efficient and Realistic Traffic Simulator with Embedded Machine Learning Models to embed pre-trained neural networks directly into a C++ traffic simulator core, bypassing the performance bottlenecks of interpreted languages entirely.
From Graphs to Source Code: Heterogeneous Transpilation
While serialization formats work well when an interoperable runtime engine is available, some legacy or highly specialized environments cannot afford even the minimal overhead of an external runtime engine. In these scenarios, researchers use heterogeneous transpilation: converting the machine learning model directly into the native source code or data structures of the target environment.
Consider scientific computing, which is heavily anchored in legacy languages like Fortran. To solve this, Jordan Ott et al. developed A Fortran-Keras Deep Learning Bridge for Scientific Computing, which parses high-level Keras model definitions and weights directly into native Fortran data structures. This allows high-performance climate emulations to run deep learning components at native execution speeds without manual code translation.
When processing data at extreme scales within physics pipelines or databases, models can be compiled into low-level languages or domain-specific expressions. L. Anderlini et al. transpile neural network simulation pipelines directly into optimized C code in The LHCb ultra-fast simulation option, Lamarr design and validation to eliminate Python context-switching overhead in particle physics workflows. Taking this concept to data management systems, Kwanghyun Park et al. demonstrate in End-to-end Optimization of Machine Learning Prediction Queries that machine learning models can be transpiled directly into native SQL expressions for in-database execution. This flexibility allows the database engine to optimize data retrieval and model inference as a single, unified query.
[ Research Framework ] (PyTorch / Keras)
│
▼ [ Automated Transpilation Layer ]
│
┌───────┼──────────────────────────┐
│ ▼ ▼
[ Native C/Fortran Code ] [ Native SQL Expressions ]
(Physics / Simulations) (In-Database Queries)
Navigating Edge and Mobile Fragmentation
The edge ecosystem is notoriously fragmented. Deploying a model to mobile devices requires navigating a maze of hardware delegates (GPUs, NPUs) and platform-specific frameworks like Android's TensorFlow Lite and iOS's Core ML.
To prevent developers from having to manually re-implement models for every target device, automated conversion pipelines abstract away these platform-specific complexities. For example, Sichang He et al. introduce a unified pipeline in Fedkit: Enabling Cross-Platform Federated Learning for Android and iOS that converts high-level research prototypes into formats compatible with both Android and iOS, enabling cross-platform federated learning without requiring deep device-specific expertise.
The maturation of these automated tools has dramatically accelerated deployment timelines. As noted by the authors of RGB Photo Enhancement on Mobile GPUs, Mobile AI 2025 Challenge: Report, direct translation of research frameworks into mobile-optimized inference formats eliminates manual re-engineering overhead. Furthermore, in Learned Smartphone ISP on Mobile GPUs, Mobile AI 2025 Challenge: Report, the authors highlight how specialized plugins can bypass intermediate exchange formats entirely, translating high-level framework code directly into mobile GPU and NPU execution formats to achieve sub-15ms latencies for complex tasks like computational photography.
However, because different mobile runtimes exhibit massive performance variances, automated frameworks like the one proposed by Qiyang Zhang et al. in A Comprehensive Benchmark of Deep Learning Libraries on Mobile Devices systematically convert models into multiple library-specific formats (such as TFLite, MNN, or ncnn) to benchmark and select the optimal engine for a given architecture.
Cross-Framework Code Translation
Sometimes the deployment challenge isn't moving from Python to C++, but rather moving between competing high-level Python libraries (e.g., PyTorch to TensorFlow or vice versa) due to production ecosystem constraints. This requires translating the source code itself while maintaining functional equivalence.
Weixiang Yan et al. address this in CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation by mapping high-level library calls and tensor operations across distinct APIs, ensuring that the underlying computational graph remains identical despite differences in syntax. This automated framework-to-framework translation, as explored by Wenxin Jiang et al. in Challenges and practices of deep learning model reengineering: A case study on computer vision, leverages compilers and intermediate representations to mitigate the interface and API defects that frequently plague manual model reengineering.
Why It Works: Mathematical Invariance
At its core, cross-environment model deployment works because a trained neural network is ultimately just data: a structured collection of weights and a directed acyclic graph (DAG) of deterministic mathematical operations.
By treating the model as an abstract mathematical specification rather than a piece of framework-dependent software, compilation tools can re-target that specification to any environment. Whether compiling a graph into native C arrays, embedding it as a serialized payload in a C++ runtime, or translating it into SQL expressions, the underlying logic remains invariant. This decoupling unlocks the best of both worlds: infinite flexibility during research and optimal, native performance during deployment.
Key Foundational Work
MLModelCI: An Automatic Cloud Platform for Efficient MLaaS (2020) — contributed 1 concept elements to this direction.
Real-time, low-latency closed-loop feedback using markerless posture tracking (2020) — contributed 1 concept elements to this direction.
A Fortran-Keras Deep Learning Bridge for Scientific Computing (2020) — contributed 1 concept elements to this direction.