G-Core: A Simple, Scalable and Balanced RLHF Trainer

Junyu Wu, Wonkeun Chang, Xiaotao Liu, Guojian He, Haokai Hong, Bowen Liu, Hao Tian, Tao Yang, Yi-Jin Shi, Feng Lin, Ting Yao

TL;DR

This paper introduces G-Core, a novel framework for training large language and diffusion models using Reinforcement Learning from Human Feedback (RLHF). G-Core addresses scalability and efficiency bottlenecks in existing RLHF systems by employing a parallel controller programming model and a dynamic resource placement strategy, enabling more robust and efficient training of human-aligned AI.

What Problem Does This Paper Address?

Training large-scale AI models, particularly with Reinforcement Learning from Human Feedback (RLHF), has become increasingly popular but faces significant hurdles. Existing RLHF systems struggle with:

  • Scalability: Handling complex, multi-model workflows, especially for multimodal data (images, videos) and diffusion models, can overwhelm single controllers, leading to communication or memory bottlenecks.
  • Dynamic Workloads: RLHF often involves dynamic sampling and generative reward modeling, where the overhead of switching between models and dealing with "straggler" tasks (tasks that take much longer than others) can lead to significant hardware idle time and reduced training efficiency.
  • Orchestration Complexity: Coordinating multiple models (actor, critic, reward models) within a single training pipeline is challenging, and current distributed training frameworks can become bottlenecks themselves.

These issues limit the ability to efficiently train and deploy large, human-aligned AI models.

Methods

G-Core tackles these challenges with two main innovations:

  1. Parallel Controller Programming Model: Instead of a single, centralized controller managing the entire RLHF workflow, G-Core distributes this control. Multiple controllers manage different subsets of data and resources, allowing for more flexible and efficient orchestration of complex workflows. This parallel approach avoids the bottlenecks associated with a single controller, especially when dealing with large data transfers or complex computational procedures.
  2. Dynamic Scaling Placement Schema: This schema adaptively partitions GPU resources and schedules workloads. It combines "co-existing" (models running in parallel on separate devices) and "co-location" (models sharing GPU memory) strategies. For early stages like generation and rewarding, models can co-exist to minimize swapping overhead. For later stages like training, co-location is used to maximize GPU utilization. This dynamic approach aims to minimize device idle time ("bubbles") and improve hardware utilization, even under highly variable training conditions and long-tail effects.

Datasets

Not discussed in the paper. The paper focuses on the framework's architecture and training methodology rather than specific datasets used for evaluation.

Key Insights and Ideas

  • Decentralizing Control is Key: The paper highlights that a single controller is a fundamental bottleneck in complex RLHF pipelines. Distributing control across multiple parallel controllers unlocks greater flexibility and scalability.
  • Dynamic Resource Allocation is Crucial for Efficiency: RLHF workflows are inherently dynamic. A static resource allocation strategy is insufficient. G-Core's dynamic placement schema, which adapts to workload fluctuations, is essential for maximizing hardware utilization and minimizing idle time.
  • Balancing Co-existence and Co-location: The paper demonstrates that a hybrid approach to model placement, strategically using both parallel execution (co-existence) and shared memory (co-location) based on the workflow stage, is more effective than a single strategy.
  • Real-World Deployment Matters: The successful application of G-Core in training models for WeChat product features underscores its practical effectiveness and robustness in a large-scale, production environment.

Key Results

The paper emphasizes that G-Core has successfully trained models that support WeChat product features, serving a large-scale user base. This real-world deployment demonstrates the framework's effectiveness and robustness. While specific quantitative metrics like accuracy improvements or speedups are not detailed in the provided text, the success in production implies significant gains in efficiency and scalability over existing methods. The authors state that G-Core advances the state of the art in RLHF training.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while G-Core is designed to be loosely coupled and interoperable with systems like MPI and SLURM, it currently uses WeChat's internal job scheduling system.
  • While the dynamic placement strategy is effective, the paper notes that predicting how model output length will change during training is challenging, making static estimation difficult.
  • Future Work:
    • The authors suggest that there remain open challenges in further optimizing resource allocation and supporting increasingly diverse model architectures.
  • The framework provides a solid foundation for future research and practical deployment of large-scale, human-aligned models.

Summary updated

Abstract

Reinforcement Learning from Human Feedback (RLHF) has become an increasingly popular paradigm for training large language models (LLMs) and diffusion models. While existing RLHF training systems have enabled significant progress, they often face challenges in scaling to multi-modal and diffusion workflows and adapting to dynamic workloads. In particular, current approaches may encounter limitations in controller scalability, flexible resource placement, and efficient orchestration when handling complex RLHF pipelines, especially in scenarios involving dynamic sampling or generative reward modeling. In this paper, we present \textbf{G-Core}, a simple, scalable, and balanced RLHF training framework designed to address these challenges. G-Core introduces a parallel controller programming model, enabling flexible and efficient orchestration of complex RLHF workflows without the bottlenecks of a single centralized controller. Furthermore, we propose a dynamic placement schema that adaptively partitions resources and schedules workloads, significantly reducing hardware idle time and improving utilization, even under highly variable training conditions. G-Core has successfully trained models that support WeChat product features serving a large-scale user base, demonstrating its effectiveness and robustness in real-world scenarios. Our results show that G-Core advances the state of the art in RLHF training, providing a solid foundation for future research and deployment of large-scale, human-aligned models.
BibTeX
@misc{wu2025g,
  author = {Junyu Wu and Wonkeun Chang and Xiaotao Liu and Guojian He and Haokai Hong and Bowen Liu and Hao Tian and Tao Yang and Yi-Jin Shi and Feng Lin and Ting Yao},
  title = {{G-Core: A Simple, Scalable and Balanced RLHF Trainer}},
  year = {2025},
  eprint = {2507.22789},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.22789},
  howpublished = {arXiv},
}

Loading related content...

Figure