Abstract
Reinforcement learning (RL) has shown promise for enhancing code generation capabilities in large language models (LLMs), yet its effectiveness critically depends on high-quality test suites for reliable reward signals. Current approaches suffer from inadequate test case quantity and quality, leading to false positives (incorrect solutions passing verification) and slow positives (valid but suboptimal implementations), which corrupt RL training dynamics. We address these challenges through three key contributions: (1) We systematically analyze how low-quality test suites degrade Code RL performance via reward misalignment; (2) We propose Themis, an automated framework that transforms test case generation into code synthesis—first extracting problem constraints via template-guided parsing, then generating executable test generators through LLM-powered code synthesis, and finally validating tests through constraint-aware filtering; (3) We develop an error-guided test case reduction method that preserves error detection efficacy while reducing test set cardinality, thereby enhancing reinforcement learning training efficiency. Evaluated on programming competition datasets, Themis achieves 95 percent error detection rates, outperforming original test suites in most of the cases. When integrated into RL pipelines, models trained with Themis-generated tests demonstrate consistent 3-5 percent improvements across HumanEval, MBPP, and LiveCodeBench compared to the baseline, matching performance levels achieved with manually curated test suites.
Introduction
Reinforcement Learning has become a critical component for scaling reasoning in state-of-the-art models. However, applying RL to code generation is notoriously difficult because it requires a reliable "judge" to provide rewards. Current test suites used for training often suffer from inadequate quantity, allowing incorrect code to pass (False Positives), and poor quality, failing to catch inefficient implementations (Slow Positives). These issues lead to "reward hacking" where models produce suboptimal but technically correct code. Furthermore, manually curating high-quality tests for large-scale RL datasets is prohibitively expensive. Themis addresses these challenges by automating the creation of adversarially validated test suites that ensure precise correctness and stable reward signals.
The Themis Framework
Themis moves away from simple prompting for test cases and instead treats test generation as an executable code synthesis task. The framework follows a multi-stage pipeline designed to ensure both semantic validity and high error-detection efficacy.
- Constraint Extraction: Using template-guided parsing, Themis extracts structured invariants from problem descriptions, such as numerical boundaries and data types.
- Test Code Generation: Themis uses an LLM to write a
TestGeneratorPython class. This class generates "basic," "corner," and "complex" inputs using seeded randomization, ensuring a diverse range of test scenarios. - Validation & Filtering: Generated inputs are executed against a reference solution and passed through a
Validatorclass to ensure they adhere to the problem's original constraints. - Error-Guided Reduction: To maintain RL training efficiency, Themis prunes redundant tests. It generates "synthetic erroneous solutions" using various LLM personas to simulate different mistake levels, then identifies the smallest subset of tests capable of catching all these errors. This method achieved a 5x compression of test sets without losing detection efficacy.
Experimental Results
The researchers evaluated Themis using datasets like TACO-verified and KodCode, testing final models on HumanEval, MBPP, and LiveCodeBench (v5). The core insight is that generating a program to create tests is far more robust than generating raw data, ensuring semantically valid tests that can scale infinitely.
Themis-generated tests achieved a 95% error detection rate, surpassing original human-curated test suites. When integrated into RL pipelines, models trained with Themis showed consistent 3–5% improvements across major benchmarks compared to standard baselines. The results demonstrate that reward misalignment is destructive; low-quality tests can actually degrade model performance by reinforcing incorrect or inefficient patterns. Themis provides the stable reward signal necessary for effective training.
Discussion and Future Work
While Themis significantly advances automated testing for RL, its performance still depends on the underlying LLM's coding capability for generator synthesis. Extremely complex problems may still pose challenges. Future work could expand the framework to handle API-driven development, tool-augmented execution, and formal verification domains. Additionally, while template-guided extraction is effective, highly abstract semantic constraints may require more advanced natural language understanding in future iterations.
Summary updated