Optimization of geological carbon storage operations with multimodal latent dynamic model and deep reinforcement learning

Zhongzheng Wang, Yuntian Chen, Guodong Chen, Dongxiao Zhang

Abstract

Geological Carbon Storage (GCS) is a critical technology for climate mitigation, yet optimizing injection operations remains computationally prohibitive due to the complexity of subsurface physics. This paper introduces the Multimodal Latent Dynamic (MLD) model, a deep learning framework that learns compressed representations of reservoir states. By serving as a high-speed surrogate environment for reinforcement learning agents, the MLD model enables a 60% reduction in computational costs while maximizing storage performance and economic value across diverse geological scenarios.

Introduction

To make GCS projects commercially viable, engineers must optimize "well controls"—the time-varying rates of $CO_2$ injection and production—to maximize Net Present Value (NPV) while ensuring safety. Traditional optimization relies on high-fidelity numerical simulators that solve complex physics equations. However, a single simulation can take hours, and optimization algorithms often require thousands of iterations.

Existing surrogate models often suffer from two primary issues: temporal consistency, where errors accumulate over long time horizons, and poor generalization, where models must be retrained for every new reservoir. This research addresses these challenges by combining a novel world model with deep reinforcement learning.

Methodology

The Multimodal Latent Dynamic (MLD) Model

The MLD model avoids predicting high-dimensional reservoir states (like 3D pressure maps) directly. Instead, it compresses this information into a low-dimensional latent space. The architecture consists of three primary components:

  1. Representation Module: A fused encoder using CNNs for spatial data and MLPs for vector data to generate a latent vector $z$.
  2. Transition Module: Predicts the evolution of the latent state over time based on current control actions.
  3. Prediction Module: Translates the latent state into observable flow responses, such as $CO_2$ injection rates.
The MLD model architecture uses a latent space and Joint-Embedding Consistency (JEC) loss to predict system evolution efficiently.

A core innovation is the Joint-Embedding Consistency (JEC) loss. Unlike standard models that attempt to reconstruct full reservoir images, JEC ensures that the predicted future latent state matches the encoded version of the actual future state. This approach maintains temporal grounding over long horizons without the overhead of full-scale reconstruction.

Deep Reinforcement Learning (MSDRL)

The MLD model acts as a "flight simulator" for a Soft Actor-Critic (SAC) agent. The agent interacts with this latent environment to learn an optimal control policy. Because the MLD model is purely mathematical and bypasses physics-based solvers, the agent can simulate decades of operation in seconds.

The standard Markov Decision Process (MDP) framework where the RL agent learns through interaction with the MLD-based environment.

Experimental Setup

The researchers utilized the industry-standard ECLIPSE simulator to generate training data across two primary cases:

  • Deterministic Case: 500 simulation runs of a specific 3D aquifer model to evaluate raw optimization efficiency.
  • Generalizable Case: 2,000 simulation runs where permeability, porosity, and fluid properties were varied. This tested the model's ability to handle "unseen" reservoirs, with 1,600 runs used for training and 400 for testing.

Key Results

  • Superior Performance: The proposed MSDRL approach achieved a higher NPV than traditional methods like Differential Evolution (DE) and surrogate-assisted evolutionary algorithms.
  • Computational Efficiency: The framework reduced required computational resources by over 60%, saving approximately 30 hours of compute time in test cases. Once trained, the MLD model evaluated flow responses $10^4$ times faster than the full physics simulator.
  • Zero-Shot Generalization: In the generalizable case, the agent provided significantly better control schemes for brand-new reservoir scenarios compared to unoptimized baselines, demonstrating that the model learned transferable physical principles rather than just memorizing specific maps.

Conclusion

The MLD framework represents a significant step forward in making GCS optimization practical for real-world engineering. By prioritizing latent consistency over full-scale reconstruction and utilizing multimodal inputs, the model achieves both speed and generalization. Future work will focus on handling more complex, "messy" geological distributions and integrating hard non-linear state constraints, such as safety pressure thresholds, directly into the reinforcement learning framework.

Summary updated

Abstract

Maximizing storage performance in geological carbon storage (GCS) is crucial for commercial deployment, but traditional optimization demands resource-intensive simulations, posing computational challenges. This study introduces the multimodal latent dynamic (MLD) model, a deep learning framework for fast flow prediction and well control optimization in GCS. The MLD model includes a representation module for compressed latent representations, a transition module for system state evolution, and a prediction module for flow responses. A novel training strategy combining regression loss and joint-embedding consistency loss enhances temporal consistency and multi-step prediction accuracy. Unlike existing models, the MLD supports diverse input modalities, allowing comprehensive data interactions. The MLD model, resembling a Markov decision process (MDP), can train deep reinforcement learning agents, specifically using the soft actor-critic (SAC) algorithm, to maximize net present value (NPV) through continuous interactions. The approach outperforms traditional methods, achieving the highest NPV while reducing computational resources by over 60%. It also demonstrates strong generalization performance, providing improved decisions for new scenarios based on knowledge from previous ones.
BibTeX
@inproceedings{wang2024optimization,
  author = {Zhongzheng Wang and Yuntian Chen and Guodong Chen and Dongxiao Zhang},
  title = {{Optimization of geological carbon storage operations with multimodal latent dynamic model and deep reinforcement learning}},
  booktitle = {Geoenergy},
  year = {2024},
  doi = {10.1016/j.geoen.2024.213407},
}
Figure