Themis: Automated Constraint-Aware Test Synthesis Framework for Code Reinforcement Learning

S. Ye, Qi Liu, Yi Jiang, Zheng Zhang, Heng Yu, Zhenya HuangAAAI 2026 · DOI

Abstract

Reinforcement learning (RL) has shown promise for enhancing code generation capabilities in large language models (LLMs), yet its effectiveness critically depends on high-quality test suites for reliable reward signals. Current approaches suffer from inadequate test case quantity and quality, leading to false positives (incorrect solutions passing verification) and slow positives (valid but suboptimal implementations), which corrupt RL training dynamics. We address these challenges through three key contributions: (1) We systematically analyze how low-quality test suites degrade Code RL performance via reward misalignment; (2) We propose Themis, an automated framework that transforms test case generation into code synthesis—first extracting problem constraints via template-guided parsing, then generating executable test generators through LLM-powered code synthesis, and finally validating tests through constraint-aware filtering; (3) We develop an error-guided test case reduction method that preserves error detection efficacy while reducing test set cardinality, thereby enhancing reinforcement learning training efficiency. Evaluated on programming competition datasets, Themis achieves 95 percent error detection rates, outperforming original test suites in most of the cases. When integrated into RL pipelines, models trained with Themis-generated tests demonstrate consistent 3-5 percent improvements across HumanEval, MBPP, and LiveCodeBench compared to the baseline, matching performance levels achieved with manually curated test suites.

Introduction

Reinforcement Learning has become a critical component for scaling reasoning in state-of-the-art models. However, applying RL to code generation is notoriously difficult because it requires a reliable "judge" to provide rewards. Current test suites used for training often suffer from inadequate quantity, allowing incorrect code to pass (False Positives), and poor quality, failing to catch inefficient implementations (Slow Positives). These issues lead to "reward hacking" where models produce suboptimal but technically correct code. Furthermore, manually curating high-quality tests for large-scale RL datasets is prohibitively expensive. Themis addresses these challenges by automating the creation of adversarially validated test suites that ensure precise correctness and stable reward signals.

The Themis Framework

Themis moves away from simple prompting for test cases and instead treats test generation as an executable code synthesis task. The framework follows a multi-stage pipeline designed to ensure both semantic validity and high error-detection efficacy.

Overview of the Themis automated test synthesis framework, detailing constraint extraction, test code generation, validation, and reduction.
  1. Constraint Extraction: Using template-guided parsing, Themis extracts structured invariants from problem descriptions, such as numerical boundaries and data types.
  2. Test Code Generation: Themis uses an LLM to write a TestGenerator Python class. This class generates "basic," "corner," and "complex" inputs using seeded randomization, ensuring a diverse range of test scenarios.
  3. Validation & Filtering: Generated inputs are executed against a reference solution and passed through a Validator class to ensure they adhere to the problem's original constraints.
  4. Error-Guided Reduction: To maintain RL training efficiency, Themis prunes redundant tests. It generates "synthetic erroneous solutions" using various LLM personas to simulate different mistake levels, then identifies the smallest subset of tests capable of catching all these errors. This method achieved a 5x compression of test sets without losing detection efficacy.

Experimental Results

The researchers evaluated Themis using datasets like TACO-verified and KodCode, testing final models on HumanEval, MBPP, and LiveCodeBench (v5). The core insight is that generating a program to create tests is far more robust than generating raw data, ensuring semantically valid tests that can scale infinitely.

Comparison of reward dynamics during training and final Pass@1 performance on LiveCodeBench across different test suites.

Themis-generated tests achieved a 95% error detection rate, surpassing original human-curated test suites. When integrated into RL pipelines, models trained with Themis showed consistent 3–5% improvements across major benchmarks compared to standard baselines. The results demonstrate that reward misalignment is destructive; low-quality tests can actually degrade model performance by reinforcing incorrect or inefficient patterns. Themis provides the stable reward signal necessary for effective training.

Discussion and Future Work

While Themis significantly advances automated testing for RL, its performance still depends on the underlying LLM's coding capability for generator synthesis. Extremely complex problems may still pose challenges. Future work could expand the framework to handle API-driven development, tool-augmented execution, and formal verification domains. Additionally, while template-guided extraction is effective, highly abstract semantic constraints may require more advanced natural language understanding in future iterations.

Summary updated

Abstract

Reinforcement learning (RL) has shown promise for enhancing code generation capabilities in large language models (LLMs), yet its effectiveness critically depends on high-quality test suites for reliable reward signals. Current approaches suffer from inadequate test case quantity and quality, leading to false positives (incorrect solutions passing verification) and slow positives (valid but suboptimal implementations), which corrupt RL training dynamics. We address these challenges through three key contributions: (1) We systematically analyze how low-quality test suites degrade Code RL performance via reward misalignment; (2) We propose Themis, an automated framework that transforms test case generation into code synthesis—first extracting problem constraints via template-guided parsing, then generating executable test generators through LLM-powered code synthesis, and finally validating tests through constraint-aware filtering; (3) We develop an error-guided test case reduction method that preserves error detection efficacy while reducing test set cardinality, thereby enhancing reinforcement learning training efficiency. Evaluated on programming competition datasets, Themis achieves 95 percent error detection rates, outperforming original test suites in most of the cases. When integrated into RL pipelines, models trained with Themis-generated tests demonstrate consistent 3-5 percent improvements across HumanEval, MBPP, and LiveCodeBench compared to the baseline, matching performance levels achieved with manually curated test suites. Our constraint-aware test synthesis framework ensures full automation while preserving semantic validity—critical for scaling RL training to complex code generation tasks. The framework's modular design also enables seamless integration with existing code data synthesis frameworks.
BibTeX
@inproceedings{ye2026themis,
  author = {S. Ye and Qi Liu and Yi Jiang and Zheng Zhang and Heng Yu and Zhenya Huang},
  title = {{Themis: Automated Constraint-Aware Test Synthesis Framework for Code Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i40.40741},
}
Figure