Revealing Robust Oil and Gas Company Macro-Strategies using Deep Multi-Agent Reinforcement Learning

Dylan R. Radovic, Lucas Kruitwagen, Christian Schröder de Witt, Ben Caldecott, Shane Tomlinson, Mark WorkmanarXiv 2022 · DOI · arXiv

Introduction

International Oil Companies (IOCs) are currently navigating an existential crisis driven by the global shift toward decarbonization. While the necessity of an energy transition is clear, the optimal strategic path remains a subject of intense debate. Should firms capitalize on high oil prices in the short term or pivot immediately to renewable energy? Traditional financial models often rely on static scenario analysis that ignores competitive dynamics—how one company’s strategic shift influences the market and its rivals. This research addresses the need for "robust" strategies: those that perform well across a vast array of potential futures and against aggressive competitors.

The 2DP Wargame Framework

To analyze these complex dynamics, researchers developed the 2 Degrees Pathways (2DP) wargame. This high-dimensional simulation environment allows agents representing IOCs to make annual decisions regarding hydrocarbon production, exploration, low-carbon investments, dividends, and debt management over a 30-year horizon.

Overview of the 2 Degrees Pathways (2DP) wargame simulation loop, integrating energy data and corporate decision-making.

To solve the game, the study employed Deep Multi-Agent Reinforcement Learning (MARL), specifically Proximal Policy Optimization (PPO) and Advantage Actor-Critic (A2C) algorithms. A "League Training" approach, inspired by systems like DeepMind’s AlphaStar, was used to train "Main Agents" against a diverse pool of "Exploiter Agents." These exploiters were designed to identify weaknesses by adhering to business-as-usual (BAU) or delayed transition paths. Agents improved through self-play and Prioritized Fictitious Self-Play (PFSP), learning to counter strategies that were previously difficult to beat.

Experimental Setup and Data

The model was trained and evaluated using 408 unique energy futures from the IAMC/IIASA 1.5°C explorer database and the IEA’s Net-Zero report, covering temperature outcomes from 1.5°C to over 4°C. Initial agent balance sheets were calibrated using the 2019/2020 annual reports of the six "Majors": ExxonMobil, Chevron, BP, Total, Shell, and Eni. Market dynamics were further informed by historical OPEC production trends and industry capital cost data.

Strategic Robustness and Portfolio Evolution

The core finding of the simulation is that robustness outweighs optimization. The most successful agents did not merely seek maximum profit in a single scenario; instead, they adopted strategies that prevented bankruptcy and "debt engulfment" across all possible futures.

Ternary plots illustrating the strategic shift from oil-heavy to low-carbon-dominant portfolios across simulated scenarios.

Key insights from the emergent robust strategies include:

  • The 2024 Threshold: In 76% of robust strategies, agents began allocating the majority of their capital to low-carbon assets by 2024.
  • Asset Monetization: Robust agents treated existing oil reserves as "cash cows" to fund the transition, halting new exploration to focus on efficient depletion of current assets.
  • Early Mover Advantage: Early low-carbon (E-LC) strategies consistently outperformed laggards, even if oil demand remained high. These early movers secured lower-cost assets and avoided the late-game "asset stranding" that plagued BAU companies.

In terms of performance, E-LC strategies yielded higher cumulative dividend payouts across 11,424 simulated matches. By the simulation's end, winning IOCs had transitioned their portfolios from approximately 65% oil value to over 80% low-carbon value. Furthermore, the model suggests these robust IOCs could contribute between 1.87% and 2.48% of the total global capital required for a 1.5°C transition.

Conclusion and Future Directions

While the study provides a compelling case for an aggressive low-carbon pivot, it acknowledges several limitations. The focus on investor-owned IOCs excludes National Oil Companies (NOCs), which operate under different mandates. Additionally, while the model incorporates broad energy scenarios, it simplifies localized policy shifts and sudden geopolitical shocks. Future work aims to integrate NOC dynamics and more granular geopolitical modeling to further refine these strategic insights.

Summary updated

BibTeX
@misc{radovic2022revealing,
  author = {Dylan R. Radovic and Lucas Kruitwagen and Christian Schröder de Witt and Ben Caldecott and Shane Tomlinson and Mark Workman},
  title = {{Revealing Robust Oil and Gas Company Macro-Strategies using Deep Multi-Agent Reinforcement Learning}},
  year = {2022},
  eprint = {2211.11043},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.11043},
}
Figure