In modern high-performance computing (HPC), we are hitting a physical wall. While our capacity to generate data through massive parallel simulations has grown exponentially, our ability to write that data to persistent storage has not kept pace. High-fidelity simulations in physics, biology, and climate science can easily generate petabytes of data at a rate of terabytes per second. If a simulation must pause to dump this raw data to a disk filesystem before any analysis can happen, the entire pipeline grinds to a halt.
To bypass this storage bottleneck, researchers are shifting from traditional post-hoc analysis (where data is saved to disk and analyzed later) to processing data "in-flight." This paradigm manifests in two primary architectural patterns: in-situ and in-transit processing. By keeping data entirely within system memory and analyzing it as it is generated, these techniques eliminate intermediate files and unlock real-time scientific discovery.
The Architectural Split: In-Situ vs. In-Transit
The fundamental goal of in-flight processing is to intercept data before it ever touches a hard drive. However, how we allocate compute resources to handle this data creates a distinct architectural trade-off.
Post-Hoc: [Simulation Nodes] ----(Slow Disk I/O)----> [Storage Layer] --------> [Analysis Nodes]
In-Situ: [Simulation + Analysis shared on same Compute Nodes]
In-Transit: [Simulation Nodes] ----(High-Speed Network)----> [Dedicated Analysis Nodes]
In-Situ Processing
In an in-situ workflow, data analysis happens on the exact same compute nodes that are running the simulation. As noted by Alexander et al., this approach compresses and analyzes data while it remains directly in system memory, completely bypassing slow disk I/O speeds.
The primary advantage of in-situ processing is efficiency; there is zero network overhead because the analysis code has direct access to the simulation's memory space. The downside, however, is resource contention. The analysis tasks must steal CPU cycles and memory capacity away from the primary numerical solver, which can destabilize simulation timelines.
In-Transit Processing
To resolve this resource contention, in-transit architectures decouple data generation from data consumption. Instead of analyzing data on the simulation nodes, the simulation streams its outputs over a high-speed network fabric directly to a separate, dedicated cluster of analysis nodes.
As Gueroudji et al. argue, this separation is increasingly vital for exascale computing because it allows analysis tasks to run concurrently with data generation without slowing down the primary simulation. This architecture enables the use of modern distributed data frameworks; for example, Zanúz et al. successfully leveraged Apache Flink to perform real-time molecular dynamics analysis in-transit, significantly reducing the I/O burden on traditional HPC storage systems.
Intercepting the Data Stream Without Code Modification
One of the largest practical hurdles to adopting in-flight processing is legacy code. Scientific simulations are often massive, decades-old codebases written in Fortran or C++ that expect to write data to standard file formats like HDF5. Rewriting these engines to support network streaming is often intractable.
To solve this, modern workflow systems use API interception. As demonstrated by Yildiz et al., frameworks can intercept standard storage API calls at runtime. When the legacy simulation attempts to execute a "write to disk" command, the underlying workflow system catches the call and transparently redirects the data into a high-performance memory bus or network stream. This allows legacy applications to participate in advanced in-situ and in-transit workflows with zero modifications to their original source code.
Mechanisms for In-Flight Data Reduction
Once data is accessible in memory, the next challenge is reduction. We cannot simply stream every raw variable indefinitely; we must distill the stream into actionable insights or compact representations at the point of ingestion.
Online Statistical and Dimensionality Reduction
Instead of saving every time-step of a simulation to calculate statistics later, algorithms can update mathematical models incrementally. Aldinucci et al. show that integrating data reduction directly into a simulation loop allows for the real-time distillation of statistical values like mean and variance from asynchronous streams, bypassing storage bottlenecks entirely.
For more complex structural data, Schwerdtner et al. developed a one-pass greedy method for constructing quadratic manifolds from streaming data. By incrementally updating a truncated Singular Value Decomposition (SVD) alongside numerical solvers, the authors achieved nonlinear dimensionality reduction on petabyte-scale datasets in a single pass, eliminating the need for massive batch-processing memory allocations.
Edge-of-Ingestion Filtering and Acceleration
When dealing with high-velocity data streams—such as real-time mobility forecasting or experimental physics data—reduction must happen the millisecond data enters the system. Doulkeridis et al. and Vouros et al. demonstrate that applying primitive operators, cleaning, and compression at the point of data ingestion drastically reduces downstream computational load and network bandwidth consumption.
When data rates scale to extreme levels, hardware acceleration becomes necessary. In the context of particle physics triggers, Aaij et al. demonstrate that integrating GPU accelerators directly into data ingestion servers allows for massive data filtering and reduction before transmission across the primary network. This approach can reduce required network bandwidth by factors of 30 to 60, preventing network saturation. Similarly, in database streaming contexts, Sintos et al. emphasize that performing sampling directly at the ingestion source minimizes the network overhead required to maintain dynamic data synopses.
In-Transit Machine Learning and Deep Surrogates
The ultimate expression of the in-transit paradigm is moving beyond simple data reduction to train complex machine learning models on live streams. This is particularly powerful for creating deep surrogates—neural networks trained to mimic expensive numerical solvers at a fraction of the computational cost.
[Numerical Solvers] --(Direct Stream)--> [Reservoir Buffer] ----> [ML Training Server]
^ |
|----(Active Steering)----|
Overcoming Streaming Bias with Buffers
Training a neural network on a live stream introduces a fundamental machine learning problem: data bias. Standard deep learning assumes data is independent and identically distributed (i.i.d.), but a running simulation produces highly correlated, sequential time-steps.
To solve this, Meyer et al. developed an in-transit training architecture that streams simulation outputs directly to training nodes over a network. To break the sequential data bias, the authors introduced a strategic "Reservoir" memory buffer on the training server. This buffer shuffles and samples incoming data points in real-time, allowing the model to achieve predictive accuracy comparable to traditional offline methods without ever writing a single file to disk.
Global Sensitivity Analysis and Active Steering
This real-time connection between simulations and analytics opens up new ways to control experiments. Terraz et al. introduced the Melissa framework, which processes large-scale simulation ensemble outputs in transit to iteratively update global sensitivity indices. By avoiding intermediate files, the authors enabled sensitivity analysis at an unprecedented scale.
Building on this concept, Dymchenko et al. combined the Melissa framework with active learning in a system called Breed. Instead of just passively analyzing the stream, the training server uses the incoming data to evaluate where the machine learning model is most uncertain. It then sends real-time feedback to "steer" the numerical solvers, forcing them to generate data for the most difficult-to-learn scenarios.
This closed-loop, diskless architecture represents the future of AI-for-Science workflows. As demonstrated by Rogers et al. in plasma physics simulations, and supported by early visions of direct correlation analysis from Su et al., streaming data directly from generation to machine learning models circumvents filesystem I/O limitations entirely. This allows scientists to train highly accurate models on massive datasets that would otherwise be completely unmanageable due to storage capacity and bandwidth constraints.
Key Foundational Work
StochKit-FF: Efficient Systems Biology on Multicore Architectures (2010) — contributed 2 concept elements to this direction.
Supporting correlation analysis on scientific datasets in parallel and distributed settings (2014) — contributed 1 concept elements to this direction.