Deep Set Neural Networks for forecasting asynchronous bioprocess timeseries

M. Borisyak, Stefan Born, Peter Neubauer, Nicolás Cruz-BournazouarXiv 2023 · DOI · arXiv

Abstract: Cultivation experiments often produce sparse and irregular time series. Classical approaches based on mechanistic models, like Maximum Likelihood fitting or Monte-Carlo Markov chain sampling, can easily account for sparsity and time-grid irregularities, but most statistical and Machine Learning tools are not designed for handling sparse data out-of-the-box. Among popular approaches there are various schemes for filling missing values (imputation) and interpolation into a regular grid (alignment). However, such methods transfer the biases of the interpolation or imputation models to the target model. We show that Deep Set Neural Networks equipped with triplet encoding of the input data can successfully handle bio-process data without any need for imputation or alignment procedures. The method is agnostic to the particular nature of the time series and can be adapted for any task, for example, online monitoring, predictive control, design of experiments, etc. In this work, we focus on forecasting. We argue that such an approach is especially suitable for typical cultivation processes, demonstrate the performance of the method on several forecasting tasks using data generated from macrokinetic growth models under realistic conditions, and compare the method to a conventional fitting procedure and methods based on imputation and alignment.

The Challenge of Asynchronous Bioprocess Data

In bioprocessing, monitoring a bioreactor involves multiple sensors and sampling techniques that rarely align. For instance, dissolved oxygen might be logged every second, while biomass (OD600) or metabolite concentrations (like glucose or acetate) are measured manually every few hours. This results in asynchronous time series where measurement points do not share a common time grid across different variables.

Standard Machine Learning models, such as Recurrent Neural Networks (RNNs) or Convolutional Neural Networks (CNNs), typically require data to be structured on a fixed, regular grid. To accommodate these models, researchers often resort to preprocessing steps like imputation (filling in missing values) or alignment (interpolating data onto a consistent grid). However, these steps are inherently problematic: they introduce the biases and errors of the interpolation model into the downstream forecasting model, potentially obscuring the true biological dynamics.

Figure 1: Comparison of linear and RBF interpolation on sparse, irregular bioprocess data points. Traditional methods often rely on these interpolations to create a regular grid for standard ML models.

Proposed Method: Deep Sets and Triplet Encoding

To address these limitations, we propose a method that processes raw, irregular observations directly. The core of this approach is the Deep Set Neural Network, an architecture designed to handle sets of data where the order of elements is not fixed (permutation invariance).

Triplet Encoding

Instead of representing data as a table with missing entries, we treat every single measurement as a "triplet":

  1. Timestamp ($t$): The exact time of the measurement.
  2. Channel Indicator: A categorical variable identifying which sensor or variable was measured.
  3. Value ($x$): The numerical result of the measurement.

This encoding preserves the original information without any artificial smoothing or guessing of missing values.

Figure 6: Illustration of how asynchronous measurements across multiple channels are mapped to a structured matrix of triplets (time, channel, value) for processing by the Deep Set network.

Architecture

The model consists of an Extractor ($f$) and an Aggregator ($g$). The Extractor processes each triplet individually into a latent representation. These representations are then summed by the Aggregator. Because the summation operation is commutative and can handle any number of inputs, the model is naturally agnostic to the number of observations and their timing. To generate a forecast, the Aggregator is provided with a target timestamp ($ au$) to predict the state of the system at that future point.

Experimental Results

We evaluated the performance of the Deep Set model against traditional mechanistic model fitting using two simulated biological systems: Michaelis–Menten Kinetics (MMK) and a complex E. coli growth model involving overflow metabolism. Performance was measured using the $R^2$ coefficient on forecasting tasks.

Method MMK ($R^2$) E. coli ($R^2$)
Mechanistic Model (Fitted) 0.9073 0.387
Deep Sets + Linear Splines 0.9431 0.848
Deep Sets + Triplet Encoding 0.9481 0.856

The Deep Set model with triplet encoding consistently outperformed the alternatives. Most notably, in the complex E. coli model, it achieved an $R^2$ of 0.856, vastly superior to the traditional mechanistic fitting approach (0.387), which often struggled with local minima in the optimization landscape.

Conclusion

Deep Set Neural Networks with triplet encoding provide a robust framework for bioprocess forecasting that avoids the pitfalls of data imputation and alignment. By treating measurements as sets rather than grids, the model remains flexible and highly accurate even with sparse, noisy data. Future work will explore the application of this architecture to real-time optimal control and online monitoring in industrial bioreactors.

Summary updated

Abstract

Cultivation experiments often produce sparse and irregular time series. Classical approaches based on mechanistic models, like Maximum Likelihood fitting or Monte-Carlo Markov chain sampling, can easily account for sparsity and time-grid irregularities, but most statistical and Machine Learning tools are not designed for handling sparse data out-of-the-box. Among popular approaches there are various schemes for filling missing values (imputation) and interpolation into a regular grid (alignment). However, such methods transfer the biases of the interpolation or imputation models to the target model. We show that Deep Set Neural Networks equipped with triplet encoding of the input data can successfully handle bio-process data without any need for imputation or alignment procedures. The method is agnostic to the particular nature of the time series and can be adapted for any task, for example, online monitoring, predictive control, design of experiments, etc. In this work, we focus on forecasting. We argue that such an approach is especially suitable for typical cultivation processes, demonstrate the performance of the method on several forecasting tasks using data generated from macrokinetic growth models under realistic conditions, and compare the method to a conventional fitting procedure and methods based on imputation and alignment.
BibTeX
@misc{borisyak2023deep,
  author = {M. Borisyak and Stefan Born and Peter Neubauer and Nicolás Cruz-Bournazou},
  title = {{Deep Set Neural Networks for forecasting asynchronous bioprocess timeseries}},
  year = {2023},
  eprint = {2312.02079},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.02079},
}
Figure