Contain, Conform, Compare (Extended Version): A Red-Teamed Evaluation Kit for Small Coding Agents and the Full Record of Its Experiments

Nguyen Minh ThangarXiv 2026 · arXiv

This paper introduces drydock-eval, a specialized local evaluation kit designed for the development of small coding agents, specifically targeting the Gemma 4 Developer Agent competition.

TL;DR

The authors present a red-teamed evaluation suite that ensures execution safety, environment fidelity, and statistical rigor for AI coding agents. By analyzing a massive record of nearly 10,000 agent runs, they demonstrate that many supposed "prompting traps" have negligible impact on performance, while highlighting the significant risk of over-optimizing on small, noisy benchmarks.

What Problem Does This Paper Address?

Developing AI "coding agents"—models that can write and execute code to solve software issues—is notoriously difficult. In competitions, developers often rely on a single "hosted score" from a hidden test set. This score is a "noisy draw": a single number that doesn't tell you why an agent failed or if a small change actually improved things.

Local evaluation is the solution, but it brings three major headaches:

  1. Safety: Coding agents execute shell commands and install packages. Running untrusted, model-generated code on a local workstation is a security nightmare.
  2. Fidelity: Subtle differences in how a local server renders prompts or tokenizes text (e.g., adding a newline after a "thought" block) can silently change model behavior, making local results irrelevant to the real competition.
  3. Inference: Task difficulty varies wildly. A small gain on a few tasks might just be "luck." Developers need a way to statistically prove that one agent configuration is truly better than another.

Methods

The paper presents drydock-eval, which tackles these problems with three specialized modules:

  • Contain: A secure runner that executes each agent job in a fresh, network-isolated container. It uses resource caps (CPU, memory, disk) and a "forwarder" that only allows specific chat-completion requests. Crucially, the authors used AI agents to "red-team" this module, attempting to find escape vectors.
  • Conform: A suite of eight checks that compare local requests against the competition's "scorer path." It looks for "rendering traps" in tool roles, sampling settings, and chat templates, checking down to the exact token IDs generated by the server.
  • Compare: A statistical framework designed for repository-level evaluation. Since tasks from the same repository are often correlated, Compare uses clustered inference and a pre-registered "frozen pass rule" to ensure that reported gains are statistically significant and not just artifacts of selection bias.
Figure 1: The kit: where each module sits.

Datasets

The evaluation relies on several primary sources:

  • Competition Development Tasks: 70 tasks from 4 repositories provided by the Gemma 4 competition organizers.
  • Berth: A set of 579 validated Python tasks from 24 repositories, derived from the Berth dataset and formatted for the competition harness.
  • Task Panels: The authors organized these into various "panels" (Screen, Small, Large, Case-Study) to separate the tasks used for finding candidate improvements from those used for final confirmation.

Key Insights and Ideas

  • Prompting Traps are Often "Null": One of the most surprising findings was that four major prompt divergences—including removing the model's internal "reasoning" steps or changing the chat template—resulted in no detectable change in the solve rate for the small E4B model. This suggests that at this scale, model performance is more robust to formatting than previously thought.
  • The "Screen Lead" Mirage: The authors observed that agents that performed exceptionally well in small "screens" (e.g., 48 tasks) often failed to maintain that lead in larger confirmation runs. This "selection effect" means that without a rigorous "Compare" step, developers are likely to chase phantom improvements.
  • Small Models are Nearly Deterministic: For the E4B model, about two-thirds of tasks were either always solved or never solved across multiple runs. The "noise" in the score comes from a small minority of uncertain tasks.
  • Safety Requires Constant Vigilance: Even with containerization, red-teaming revealed high-severity vectors related to how the host system handles shared resources like file systems and model servers.

Key Results

  • Throughput: The kit is fast. On a single RTX 5090, it can process about 250 agent runs per hour.
  • Model Performance: The authors provide detailed serving benchmarks. For instance, the Gemma 4 E4B model decodes at 105.8 tokens per second on four L4 GPUs, nearly 2.8 times faster than the 31B variant.
Figure 2: Decode throughput on four L4 GPUs with a context of 2,048 tokens. Left: tokens per second of one stream as the number of concurrent streams grows. Right: tokens per second summed over the streams.
  • LoRA Costs: Loading a LoRA adapter (rank 8 or 16) slows down decoding by 9% to 13% and reduces the available KV cache by about 19%.

Limitations and Future Work

  • Parity Not Guaranteed: While Conform helps, the authors haven't definitively proven that a local "pass" perfectly predicts a hosted "pass" due to remaining environment differences (e.g., different GPU architectures).
  • Task Exhaustion: The massive scale of these experiments has nearly "used up" the available pool of validated tasks. Future research will need fresh, high-quality task sets.
  • Withheld Data: To protect the ongoing competition, the specific agent prompts and fine-tuned adapters are currently withheld, though they are scheduled for release in late 2026.

By releasing the full record of nearly 10,000 runs, the authors provide a valuable calibration for the community, showing exactly how much noise to expect and how to build more reliable AI coding agents.

Summary updated

Abstract

A hosted score in the Gemma 4 Developer Agent competition is one noisy number, so development needs a local evaluator that is fast, safe for agent-written code, and matching the hosted scorer's prompts. drydock-eval evaluates a small coding agent (Gemma 4 E4B) at about 250 runs per hour on one RTX 5090. Contain runs each job in a fresh container without network and passed 200 scripted escape tests; conform provides eight checks that compare a local stack's requests with the scorer's configuration, down to rendered tokens; compare provides paired statistics that respect repository clusters. AI agents built and attacked the kit; a human owner approved machine use and release. This extended version repeats the short paper and adds the full record of the experiments made with and around the kit: 9,660 rows of agent runs, 2,494 grading controls and 72 serving measurements, all released. On four L4 GPUs the 31B model decodes 38.1 tokens per second in one stream and E4B 105.8. Six configurations of a 31B agent solved between 37 and 44 of 70 development tasks, and no pairwise difference was detected. A pre-registered five-arm experiment (231 tasks, 4,620 jobs) measured four prompt divergences: no E4B solve-rate change was detected (largest estimate -2.1 points, 95% interval -4.4 to +0.4). Three changes of the E4B agent led their screens by +1.0 to +6.2 points; each was then tested once on new tasks under a frozen pass rule, and none was confirmed (gains -0.35, -0.39 and +4.10 points). The unchanged baseline, run again on the same 128 tasks, moved by 1.17 points, and 65% of these tasks were solved in none of eight runs. Prompts and agent bundles are withheld; every run-level result is public.
BibTeX
@misc{thang2026contain,
  author = {Nguyen Minh Thang},
  title = {{Contain, Conform, Compare (Extended Version): A Red-Teamed Evaluation Kit for Small Coding Agents and the Full Record of Its Experiments}},
  year = {2026},
  eprint = {2611.16485},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16485},
}
Figure