Option-GRPO: Hierarchical RL Against AI Slop in Code Generation for Unseen SDKs

Prashank KadamarXiv 2026 · arXiv

TL;DR

Option-GRPO is a hierarchical reinforcement learning framework that enables LLMs to master new, unseen software libraries by mining the library's own source code for idiomatic patterns. By treating these patterns as "options" in a hierarchical RL setup, the system significantly reduces "AI slop"—verbose, redundant code—producing solutions that are 27% shorter and 34.7% more API-efficient than standard approaches.

What Problem Does This Paper Address?

Modern code LLMs suffer from two major issues when dealing with specialized software development kits (SDKs). First is the training cutoff problem: models perform poorly on libraries released or updated after they were trained. Second is the "AI Slop" problem: even when a model produces functional code, it is often "bloated, redundant, and non-idiomatic"—it works, but it’s ugly and inefficient.

Existing solutions like Retrieval-Augmented Generation (RAG) or simple fine-tuning help with the first problem but ignore the second. They focus on functional correctness (does it pass the test?) rather than elegance or idiomatic usage. Furthermore, most RL-based code generation methods require human-written test cases or usage examples, which are rarely available for brand-new SDKs.

Methods

The authors introduce Option-GRPO, which adapts a model to an unseen SDK using only the SDK's source code. The process follows a "curriculum-as-options" approach:

  1. Graph-Based Option Discovery: The system analyzes the SDK source code to build parameter-flow and semantic similarity graphs. It identifies frequently composed function sequences, which it defines as "code options" (temporally extended actions).
  2. Hierarchical RL (HRL): Instead of just predicting the next token, the model learns to select these high-level "options." This mirrors the Options Framework in classical RL.
  3. Group Relative Policy Optimization (GRPO): The authors use GRPO, a critic-free RL algorithm (recently popularized by DeepSeek-R1). It computes rewards by comparing a group of outputs for the same prompt, which is more stable than traditional actor-critic methods for code.
  4. Curriculum Learning: Training happens in three phases:
  5. Phase 1 (Level 0): Mastering individual, atomic API calls.
  6. Phase 2 (Level 1): Learning to use the discovered compositional patterns.
  7. Phase 3 (Level 2): Assembling these patterns into full programs.
  8. Automated Rewards: To avoid needing human-written tests, they use UnseenSDK-Bench, which auto-generates property-based tests from SDK type signatures and source code invariants.
Figure 1: Overview of Option-GRPO. The top row constructs three SDK graphs from source code and discovers a hierarchical option vocabulary A0 ⊂A1 ⊂A2 (Algorithm 1). The middle row runs three curriculum phases of GRPO training, each receiving the option vocabulary from above and the layered execution reward from below (dashed arrows). Successful trajectories from phase k contribute newly extracted options to phase k+1 (Algorithm 2). The trained policy πθ generates code at inference. Solid arrows denote training-stage progression; dashed arrows denote artifact dependencies.

Datasets

The primary evaluation is performed on UnseenSDK-Bench, a new benchmark created for this paper. It consists of:

  • 12 Unseen SDKs: Ranging from DataFrames (Daft) to Vector DBs (LanceDB) and document processing (Docling).
  • Task Levels: 1,871 atomic tasks, 3,742 compositional tasks, and 398 full-program tasks.
  • Property Tests: 7,484 auto-generated tests used for reward computation.

The authors verified "unseen-ness" using a memorization probe, confirming that the base model (Qwen3-32B) had very low prior knowledge of these specific libraries.

Key Insights and Ideas

  • SDKs are Natural Hierarchies: The core "aha moment" is that SDK source code inherently encodes a hierarchy. Type relationships and parameter flows dictate how functions should compose, providing a "free" curriculum for RL.
  • Credit Assignment via Options: In standard RL, if a 100-token program fails, it's hard to know which token was the problem. By using options, the model makes fewer, higher-level decisions. The authors prove that this improves the signal-to-noise ratio of the policy gradient by a factor of $\sqrt{T/K}$ (where $T$ is tokens and $K$ is options).
  • Compression as a Proxy for Quality: The paper theoretically links "option description length" to code conciseness. By forcing the model to use high-level idioms, the resulting code is naturally more efficient and less "sloppy."

Key Results

Option-GRPO was tested against several baselines, including RAG, ExploraCoder, and "flat" (non-hierarchical) GRPO.

  • Higher Correctness: Achieved 61.4% pass@1, outperforming flat GRPO by 8.7 points and the previous state-of-the-art (ReCode) by 6.3 points.
  • Slop Reduction: Generated code was 27% shorter and used 34.7% fewer API calls than flat GRPO.
  • Efficiency: Option-GRPO at 1/3 the compute budget outperformed flat GRPO at full compute.
  • Frontier Comparison: On SDKs truly unseen to all models, the 32B Option-GRPO model outperformed off-the-shelf frontier models like GPT-4o and Claude 3.5 Sonnet.
Table 1: Comparison with most relevant prior work. Option-GRPO uniquely combines hierarchical RL with structural SDK knowledge for example-free generation. †Voyager builds a skill library via iterative GPT-4 prompting, not gradient-based RL; we mark its RL column accordingly.

Limitations and Future Work

  • Training Cost: Adapting to a single SDK requires ~636 H100 GPU-hours. While the authors show this can be amortized via cross-SDK transfer, it remains a significant upfront cost.
  • Typing Assumption: The current discovery method relies on Python type annotations. Adapting this to untyped or dynamically typed languages would require an additional type-inference step.
  • Semantic Gaps: While property-based tests are powerful, they don't guarantee 100% semantic correctness. A program might pass all type and shape checks but still calculate the wrong value in a way the tests didn't catch.
  • Option Rigidity: The "options" are currently fixed after discovery. Future work could involve "Option-Critic" style architectures where the model learns to refine or discover new options end-to-end during training.

Summary updated

Abstract

Large language models increasingly produce AI slop in generated code: verbose, redundant, non-idiomatic implementations that pass functional checks but bloat codebases. The problem compounds on SDKs released after the model's training cutoff, where the model has neither idiomatic priors nor compositional understanding. We present Option-GRPO, a hierarchical reinforcement learning framework that adapts a code LLM to an unseen SDK from its source code alone, without human-written usage examples or test cases, and that explicitly optimizes for both functional correctness and code elegance. Our key insight is that SDK source code contains compositional structure that defines a hierarchy of reusable code patterns (code options) which simultaneously serve as the building blocks of a curriculum and the action space for efficient policy optimization. We show that code options induce a valid semi-Markov decision process, give conditional bounds under which shorter option description length implies more concise and, in expectation, more API-efficient programs, and analyze under stated assumptions how the option hierarchy improves credit assignment by a factor of $\sqrt{T/K}$. Using Group Relative Policy Optimization (GRPO) with execution-based rewards at each curriculum level, Option-GRPO discovers SDK idioms, composes them into patterns, and assembles patterns into complete programs. Experiments on 12 unseen SDKs with a 32B-parameter base model demonstrate 61.4% pass@1, outperforming flat GRPO by 8.7 points, ExploraCoder by 11.2 points, and ReCode by 6.3 points. Option-GRPO produces solutions that use 34.7% fewer API calls and are 27% shorter than flat-GRPO outputs while maintaining higher correctness, directly attacking the slop problem.
BibTeX
@misc{kadam2026option,
  author = {Prashank Kadam},
  title = {{Option-GRPO: Hierarchical RL Against AI Slop in Code Generation for Unseen SDKs}},
  year = {2026},
  eprint = {2611.23633},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23633},
}
Figure