Supervised Fine-Tuning (SFT) is remarkably efficient. Because the target outputs are fixed and known ahead of time, models can be trained using "teacher forcing," which allows the entire sequence of tokens to be processed in parallel. However, when we transition to Reinforcement Learning (RL) to optimize foundation models for complex reasoning, alignment, or interactive tool use, this parallelism breaks down, and the computational requirements skyrocket.
Understanding why RL is so resource-intensive requires looking under the hood at how the training loop changes from static data ingestion to dynamic exploration.
The Generation-in-the-Loop Bottleneck
In standard supervised learning, the model never has to generate text during training; it simply calculates the likelihood of the next token given a human-annotated prefix. As Hao et al. demonstrate in Teacher Forcing Recovers Reward Functions for Text Generation, while teacher forcing can be mathematically linked to reward recovery, its primary practical advantage is scalability.
RL completely upends this paradigm. To learn from trial and error, the model must actively explore its environment by generating its own sequences. As Sifa points out in Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization, this "generation-in-the-loop" requirement means the training process must pause to perform autoregressive token sampling before it can compute a single gradient step.
Because autoregressive generation is sequential—token by token—it cannot take advantage of the massive parallel processing capabilities of modern GPUs in the same way that forward-pass training does. Yin notes in Learning to Generalize for Sequential Decision Making that when large-scale pre-trained models require millions of iterations to explore complex environments, hosting these high-parameter models inside an interactive trial-and-error loop quickly becomes computationally prohibitive.
The Multi-Model Memory Tax
When training an agent via popular RL algorithms like Proximal Policy Optimization (PPO), the hardware infrastructure must support far more than just the weights of the model being optimized. To ensure the model doesn't degenerate or drift too far from its original capabilities, RL frameworks typically maintain multiple large model instances in memory simultaneously:
- The Active Policy: The model currently being trained and updated.
- The Reference Model: A frozen copy of the original model used to calculate a KL-divergence penalty, ensuring the active policy stays close to the base distribution.
- The Critic / Value Network: A model that estimates the expected future reward from any given state, guiding the policy updates.
- The Reward Model: A model that evaluates the final generated output to provide a scalar score.
The authors of P-TA: Using Proximal Policy Optimization to Enhance Tabular Data Augmentation via Large Language Models emphasize that running these multiple high-parameter instances concurrently makes RL pipelines vastly more resource-intensive than standard SFT. Even when researchers attempt to bypass full RL by incorporating reward learning directly into supervised frameworks, Li demonstrates in Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment that the need for on-the-fly response generation and multi-model memory allocation aligns the computational costs closely with full reinforcement learning from human feedback (RLHF).
The Latency of External and Multimodal Critics
The computational burden is further compounded when the reward function itself requires heavy processing. For specialized tasks like code generation or structured data extraction, rewards cannot be computed by simple string matching; they require external validation.
For example, Shojaee et al. show in Execution-based Code Generation using Deep Reinforcement Learning that execution-based feedback loops add significant training time because the system must compile and run code for every rollout. Similarly, Jana et al. demonstrate in CoTran: An LLM-based Code Translator using Reinforcement Learning with Feedback from Compiler and Symbolic Execution that combining RL with symbolic execution engines requires solving complex logical constraints mid-optimization, severely limiting training throughput.
In the multimodal domain, the critic is often another massive neural network. Yang et al. utilize large cross-modal models like CLIP as critics to evaluate visual accuracy in image captioning, which introduces significant cross-network overhead. When scaling these systems to autonomous web agents or vision-language tasks, Zhou et al. observe in Proposer-Agent-Evaluator (PAE): Autonomous Skill Discovery For Foundation Model Internet Agents that the repeated multimodal inference required to evaluate multi-turn rollouts creates an extreme computational bottleneck that restricts the scalability of autonomous discovery.
Prolonged Exploration and the Verification Crisis
Because RL agents must discover optimal strategies through extended exploration rather than mimicking a static dataset, training cycles are inherently prolonged. To expand the reasoning boundaries of large models, Liu et al. show in ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models that effective training recipes require thousands of GPU hours to allow models to discover novel reasoning paths.
This extreme resource requirement creates a significant barrier for the broader research community, leading to what can be described as a verification crisis. As Kumar et al. point out in Offline Q-learning on Diverse Multi-Task Data Both Scales And Generalizes, the sheer volume of gradient steps and computational power required to train high-capacity models makes running multi-seed experiments nearly impossible.
This sentiment is echoed by Lee et al. in Global Reward to Local Rewards: Multimodal-Guided Decomposition for Improving Dialogue Agents, who note that the high cost of RLHF frequently restricts researchers to a single training run. Without the ability to perform comprehensive ablation studies or verify stability across multiple seeds, iterating on training recipes remains an expensive privilege reserved for those with access to massive industrial GPU clusters.