Multi-Agent Reinforcement Learning Controller to Maximize Energy Efficiency for Multi-Generator Industrial Wave Energy Converter

Soumyendu Sarkar, Vineet Gundecha, Alexander Shmakov, Sahand Ghorbanpour, Ashwin Ramesh Babu, Paolo Faraboschi, Mathieu Cocho, Alexandre Pichard, Jonathan FievezAAAI 2022 · DOI

Abstract

Wave energy remains a largely untapped renewable resource due to the high Levelized Cost of Energy (LCOE) and the mechanical complexity of capturing energy from chaotic ocean environments. This paper presents a novel Multi-Agent Reinforcement Learning (MARL) controller designed for the CETO 6 wave energy converter. By employing Proximal Policy Optimization (PPO) and recurrent neural networks, the proposed system achieves significant improvements in energy capture efficiency—up to 19.9%—while drastically reducing mechanical stress. These results demonstrate the potential of AI-driven control systems to enhance the commercial viability of offshore renewable energy.

Introduction

The transition to a sustainable energy future requires diversifying renewable sources beyond solar and wind. Wave energy offers a dense and predictable power source, yet it remains a difficult engineering challenge. The primary hurdle is the Levelized Cost of Energy (LCOE); to be competitive, Wave Energy Converters (WECs) must maximize energy extraction from variable ocean swells while surviving extreme conditions that can damage sensitive machinery.

The CETO 6 wave energy converter system, showing the buoyant actuator, power take-off (PTO) units, and mooring system.

Modern WECs, such as the three-legged CETO 6 platform shown above, are structurally complex. They utilize multiple "legs" or tethers connected to independent generators (Power Take-Off units) that interact in highly non-linear ways. Traditional control strategies, often based on rigid "Spring Damper" models, struggle to account for these complex interactions, leading to suboptimal energy capture and increased mechanical wear.

The Complexity of Wave Energy Control

Traditional physics-based controllers are often too inflexible to handle the stochastic nature of ocean waves. In a multi-generator system like the CETO 6, the motion of the buoyant actuator is influenced by the combined reactive forces of all three legs. A key issue is "yaw"—the twisting motion of the buoy. Excessive yaw not only reduces the efficiency of energy capture but also causes significant mechanical stress on the tethers and foundations, leading to high maintenance costs and shorter operational lifespans.

The research team from HPE and Carnegie Clean Energy sought an AI-driven approach capable of "learning" the optimal control policy through interaction with the environment. The goal was to maximize power output while simultaneously minimizing the rotational motions that lead to mechanical fatigue.

Multi-Agent Reinforcement Learning Approach

The researchers implemented a Multi-Agent Reinforcement Learning (MARL) framework utilizing the Proximal Policy Optimization (PPO) algorithm. This architecture treats each of the three generators as an independent agent (or groups them into two agents to exploit symmetry), each controlling the reactive force of a specific leg.

The MARL framework architecture, illustrating the interaction between the wave environment, sensors, reward function, and the multi-agent controller.

Technical Innovations

Several key technical components were integrated to ensure the controller's effectiveness:

  • Recurrent Neural Networks (LSTM): Because ocean waves are inherently time-series data, the agents utilize Long Short-Term Memory (LSTM) layers. This allows the system to "remember" previous states and anticipate future wave motions using a 10-second look-ahead provided by wave sensors.
  • Reward Shaping and Team Coefficients: To balance individual generator performance with the overall health of the system, a "team coefficient" was introduced. This parameter allows agents to cooperate or compete for power capture depending on the incoming wave direction, ensuring the system acts as a cohesive unit.
  • Safety and Stability: To prevent the policy from reaching unsafe mechanical limits, "action clipping" was implemented. Furthermore, the researchers found that a smaller-than-standard clipping parameter in the PPO algorithm was essential to maintain training stability in the sensitive marine environment.

Experimental Methodology

The study utilized a high-fidelity simulator of the CETO 6 WEC platform. To ensure the results were representative of real-world conditions, the wave environment was modeled using statistical distributions (height, frequency, and spectrum) collected from three distinct locations: Albany (Western Australia), Armintza (Spain), and Wave Hub (UK).

The standard Jonswap Spectrum was used to generate heterogeneous wave episodes. The evaluation set consisted of 1,000 episodes per principal wave period, ranging from 6 to 16 seconds, covering a wide variety of sea states.

Performance and Efficiency Gains

The MARL controller demonstrated a clear advantage over the baseline Spring Damper controller across all tested scenarios.

Energy Capture

The system achieved an average 19.9% gain in energy capture for frontal waves. Even in more complex "angled wave" configurations (30°), the controller maintained a 13.9% gain. Interestingly, the RL agent often took a "long-term view," occasionally compromising immediate power gains to better position the buoy for larger energy capture later in the wave cycle—a behavior not seen in traditional controllers.

Stress Reduction

One of the most significant findings was the impact on mechanical health. The RL controller reduced yaw movement to between 1.2% and 6.3% of the baseline levels. Initially, the researchers feared that penalizing yaw would hurt power production; however, they discovered that stabilizing the buoy's rotation actually allowed the agent to enter a more stable regime, facilitating higher energy capture.

Survivability

In simulated "disaster" scenarios with 7-meter high waves, the RL controller successfully mitigated yaw build-up that caused the baseline controller to fail. It maintained positive power production while keeping the system within safe operational limits.

Conclusion and Future Outlook

The application of MARL to wave energy conversion represents a significant step toward making offshore renewables cost-competitive. The PPO-based controller proved to be both highly efficient and mechanically protective, with a computational footprint small enough to run on embedded edge devices at sea.

Future research will focus on scaling these results to "WEC farms." Deploying multiple units in close proximity introduces "cross-talk"—hydrodynamic interference between units—that will require even more sophisticated multi-agent coordination. Additionally, extending the wave prediction horizon beyond 10 seconds could unlock further efficiency gains, bringing the rhythm of the ocean closer to a reliable, global power source.

Summary updated

BibTeX
@inproceedings{sarkar2022multi,
  author = {Soumyendu Sarkar and Vineet Gundecha and Alexander Shmakov and Sahand Ghorbanpour and Ashwin Ramesh Babu and Paolo Faraboschi and Mathieu Cocho and Alexandre Pichard and Jonathan Fievez},
  title = {{Multi-Agent Reinforcement Learning Controller to Maximize Energy Efficiency for Multi-Generator Industrial Wave Energy Converter}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21473},
}
Figure