A priori compression of convolutional neural networks for wave simulators

Hamza Boukraichi, Nissrine Akkari, Fabien Casenave, David Ryckelynck

Abstract

Convolutional neural networks are now seeing widespread use in a variety of fields, including image classification, facial and object recognition, medical imaging analysis, and many more. In addition, there are applications such as physics-informed simulators in which accurate forecasts in real time with a minimal lag are required. The present neural network designs include millions of parameters, which makes it difficult to install such complex models on devices that have limited memory. Compression techniques might be able to resolve these issues by decreasing the size of CNN models that are created by reducing the number of parameters that contribute to the complexity of the models. We propose a compressed tensor format of convolutional layer, a priori, before the training of the neural network. 3-way kernels or 2-way kernels in convolutional layers are replaced by one-way fiters. The overfitting phenomena will be reduced also. The time needed to make predictions or time required for training using the original Convolutional Neural Networks model would be cut significantly if there were fewer parameters to deal with. In this paper we present a method of a priori compressing convolutional neural networks for finite element (FE) predictions of physical data. Afterwards we validate our a priori compressed models on physical data from a FE model solving a 2D wave equation. We show that the proposed convolutinal compression technique achieves equivalent performance as classical convolutional layers with fewer trainable parameters and lower memory footprint.

Introduction

In the world of physics-informed simulators, speed and memory efficiency are paramount. While Convolutional Neural Networks (CNNs) excel at learning complex patterns in physical data, such as fluid dynamics or wave propagation, they are often computationally heavy. Modern architectures frequently include millions of parameters, complicating deployment on memory-constrained edge devices or in real-time forecasting scenarios where latency must be minimized.

Traditional approaches rely on a posteriori compression—training a large model first and then pruning or quantizing it. This paper proposes an alternative: a priori compression. By designing lean models from the outset, researchers can emulate Finite Element (FE) models without the massive memory footprint, effectively addressing the "curse of dimensionality" in physics-informed deep learning.

Methods: A Priori Tensor Decomposition

The core innovation is the replacement of standard $N$-way convolutional kernels (e.g., $3 \times 3$ or $3 \times 3 \times 3$) with a succession of 1D filters using a rank-one approximation.

Canonical Polyadic Decomposition (CPD) of a 3D tensor into rank-one components.

Key technical components include:

  • CPD/SVD Approximation: Singular Value Decomposition (SVD) is used to approximate Canonical Polyadic Decomposition (CPD). This transforms a single complex convolution into a sequence of simpler, one-dimensional operations, reducing complexity from exponential to linear relative to the dimension.
  • Weight Sharing (Layer Coupling): For simulations predicting multiple linked fields (e.g., displacement and velocity), the model shares initial feature-extraction layers, halving the parameter count for those layers.
  • Euler Regularization: To maintain physical consistency, a time-regularization term is introduced in the loss function. This forces the derivative of predicted displacement to match predicted velocity via a finite difference (Euler) scheme.

Datasets and Validation

The approach was validated using synthetic data from a Finite Element (FE) model solving a 2D wave equation. The dataset included 100 training and 25 test samples, each representing the time-space evolution of a wave from a specific source point.

Visualization of the displacement field (U) and a zoomed-in region of interest.

Results and Discussion

The a priori compressed models demonstrated significant advantages:

  • Parameter Reduction: In 2D wave propagation tests, parameter counts were slashed from 7 million to approximately 1 million without sacrificing performance.
  • Accuracy Parity: The "Conv3D" model with Batch Normalization (BN) emerged as the top performer, achieving high precision in predicting both displacement and velocity fields.
  • Overfitting Mitigation: By limiting trainable parameters from the start, the models showed natural resistance to overfitting, a common issue when training deep networks on limited physical datasets.
  • Boundary Generation: The models effectively functioned as "boundary generators," emulating parametric predictions of full FE models within specific zones of interest with minimal error.

Conclusion

While the proposed technique achieves equivalent performance with a lower memory footprint, current training times per epoch are higher due to unoptimized low-level implementations of custom decomposed layers in standard libraries. Future work will focus on optimizing these 1D filter successions to translate parameter savings into direct wall-clock speedups and extending the method to higher-dimensional physical data.

Summary updated

Abstract

Convolutional neural networks are now seeing widespread use in a variety of fields, including image classification, facial and object recognition, medical imaging analysis, and many more. In addition, there are applications such as physics-informed simulators in which accurate forecasts in real time with a minimal lag are required. The present neural network designs include millions of parameters, which makes it difficult to install such complex models on devices that have limited memory. Compression techniques might be able to resolve these issues by decreasing the size of CNN models that are created by reducing the number of parameters that contribute to the complexity of the models. We propose a compressed tensor format of convolutional layer, a priori, before the training of the neural network. 3-way kernels or 2-way kernels in convolutional layers are replaced by one-way fiters. The overfitting phenomena will be reduced also. The time needed to make predictions or time required for training using the original Convolutional Neural Networks model would be cut significantly if there were fewer parameters to deal with. In this paper we present a method of a priori compressing convolutional neural networks for finite element (FE) predictions of physical data. Afterwards we validate our a priori compressed models on physical data from a FE model solving a 2D wave equation. We show that the proposed convolutinal compression technique achieves equivalent performance as classical convolutional layers with fewer trainable parameters and lower memory footprint.
BibTeX
@article{boukraichi2023priori,
  author = {Hamza Boukraichi and Nissrine Akkari and Fabien Casenave and David Ryckelynck},
  title = {{A priori compression of convolutional neural networks for wave simulators}},
  journal = {Engineering Applications of AI},
  year = {2023},
  doi = {10.1016/j.engappai.2023.106973},
}
Figure