When building a deep learning model, we rarely just write mathematics. Instead, we express that mathematics through the lens of a specific software ecosystem—whether it is PyTorch's imperative design, TensorFlow's static or deferred execution graphs, or JAX's functional, composable transformations.
This creates a fundamental problem: deep learning frameworks are not merely libraries; they are isolated execution environments with distinct tensor primitives, memory layouts, and API paradigms. As a result, the modern AI landscape suffers from what Anush Sankaran et al. describe as a "Tower of Babel" problem in Democratization of Deep Learning Using DARVIZ. A model or tool written for one ecosystem becomes locked into that environment, creating immense friction for researchers and engineers who operate across different software platforms.
To overcome this ecosystem lock-in, a major research direction focuses on cross-framework interoperability. The goal is to decouple a model's mathematical intent from its underlying execution engine, allowing neural network architectures, optimization techniques, and diagnostic tools to move seamlessly across software boundaries.
The Cost of Ecosystem Lock-In
Why does framework dependency matter? When a specialized tool or mathematical operation is tightly coupled to a specific framework's primitives, its utility is restricted to a fraction of the research community.
Consider the following dimensions where framework lock-in creates bottlenecks:
- Specialized Mathematical Operations: Implementing advanced mathematical transforms often requires writing custom low-level kernels. As noted by Patrick Kidger et al. in Signatory: differentiable computations of the signature and logsignature transforms, on both CPU and GPU, when these specialized operations are deeply integrated with a single tensor library's primitives, they cannot be easily ported, limiting the adoption of novel mathematical techniques across the broader community.
- Domain-Specific Toolkits: Scientific and specialized software packages are frequently built on top of a single deep learning library. For instance, Nils Lehmann et al. point out in Lightning-UQ-Box: A Comprehensive Framework for Uncertainty Quantification in Deep Learning that relying heavily on one ecosystem (such as PyTorch) restricts a toolkit's direct use in alternative environments like TensorFlow or JAX.
- Interpretability and Diagnostics: Tools designed to explain or map neural network behaviors often face severe adoption barriers. As argued by Maximilian Alber et al. in iNNvestigate neural networks!, interpretability frameworks are frequently restricted to specific ecosystems, preventing standardized analysis methods from being applied universally.
- Model Infrastructure and Versioning: Even infrastructure tasks like model version control suffer. Nikhil Kandpal et al. observe in Git-Theta: A Git Extension for Collaborative Development of Machine Learning Models that tracking parameter-level changes across models requires framework-specific adapters to interpret internal structures, highlighting the lack of universal serialization standards.
Mechanism 1: Declarative, Framework-Agnostic Notations
One of the most elegant ways to bypass framework dependency is to change how we write tensor manipulations. Traditionally, simple operations like reshaping, stacking, or broadcasting require framework-specific syntax (e.g., torch.permute vs. tf.transpose), which Alex Rogozhnikov notes creates fragmented codebases.
In Einops: Clear and Reliable Tensor Manipulations with Einstein-like Notation, Alex Rogozhnikov demonstrates that tensor manipulations can be expressed using a minimalist, Einstein-inspired string notation that describes the structural relationship between input and output dimensions. Because this notation is purely declarative, a single string pattern can be parsed and executed across PyTorch, TensorFlow, NumPy, and JAX alike.
Building on this philosophy, Florian Fervers et al. introduce einx, a Universal Notation for Tensor Operations. By treating complex tensor operations as vectorized versions of elementary functions, the authors replace inconsistent broadcasting and axis-specific rules with a unified language.
Standard PyTorch:
x = x.mean(dim=1, keepdim=True)
Universal Notation (einx):
x = einx.mean("b s d -> b 1 d", x)
The intuition here is powerful: by abstracting the structural intent away from the framework's imperative API, the code becomes completely framework-agnostic. The underlying library is reduced to a mere execution backend, while the developer writes clean, portable mathematical descriptions.
Mechanism 2: Abstract Graphs and Pivot Models
When dealing with existing codebases or models described in literature, manual rewriting is often impractical. Researchers have therefore turned to automated translation mechanisms that rely on intermediate representations.
In DLPaper2Code: Auto-Generation of Code From Deep Learning Research Papers, Akshay Sethi et al. demonstrate that design artifacts like tables and architectural diagrams can be extracted into a library-agnostic "abstract computational graph." This graph serves as a neutral intermediary from which ready-to-run code for target frameworks (like Keras or Caffe) can be generated.
Similarly, when migrating code directly between source files, Nadia Daoudi et al. leverage an intermediate "pivot" model in Towards Migrating Neural Network Implementations. Instead of building direct translators for every pair of frameworks—which would scale quadratically ($O(N^2)$) as new libraries emerge—code is translated into a standardized pivot representation before being compiled down to the target framework ($O(N)$ complexity).
[PyTorch Source] ------> [Intermediate Pivot Model] ------> [TensorFlow Source]
This compiler-like approach decouples the specification of the neural network from its implementation details, ensuring functional equivalence across platforms.
Mechanism 3: Multi-Backend Abstraction Layers
For large-scale scientific software, being locked into a single framework can prevent users from leveraging hardware optimizations or specialized model architectures available only in alternative ecosystems. To solve this, modern toolkits are engineered from the ground up with multi-backend abstraction layers.
A prime example of this evolution is detailed by Jinzhe Zeng et al. in DeePMD-kit v3: A Multiple-Backend Framework for Machine Learning Potentials. Originally a TensorFlow-centric tool, the framework was re-architected to support PyTorch, JAX, and PaddlePaddle simultaneously.
This is achieved by isolating the core domain logic (e.g., molecular dynamics equations) from the tensor operations. The framework defines an internal interface for tensor primitives; each supported deep learning library then implements this interface. As a result, researchers can switch backends seamlessly via configuration files, allowing them to run the exact same scientific model using whichever framework provides the best performance or hardware compatibility for their specific infrastructure.
Why Interoperability Works
Ultimately, cross-framework interoperability works because it recognizes a fundamental truth about deep learning: the underlying mathematics of a neural network are universal, while the software frameworks are ephemeral.
Whether a model computes a convolution in PyTorch or TensorFlow, the underlying tensor contractions and gradient derivations remain identical. By introducing declarative notations, abstract computational graphs, and multi-backend interfaces, interoperability research successfully separates the what (the mathematical model) from the how (the framework-specific execution), paving the way for a more open, reproducible, and collaborative machine learning ecosystem.
Key Foundational Work
DeepDSL: A Compilation-based Domain-Specific Language for Deep Learning (2017) — contributed 2 concept elements to this direction.
DLPaper2Code: Auto-Generation of Code From Deep Learning Research Papers (2018) — contributed 1 concept elements to this direction.
Democratization of Deep Learning Using DARVIZ (2018) — contributed 1 concept elements to this direction.