Abstract
As Large Language Models shift programming toward human-guided "vibe coding", agentic coding tools increasingly rely on models to self-diagnose and repair their own subtle faults—a capability central to autonomous software engineering yet never systematically evaluated. We present VIBEPASS, the first empirical decomposition that jointly evaluates two coupled tasks: Fault-Triggering Test Generation (FT-Test), constructing a discriminative witness that exposes a latent bug, and Fault-targeted Program Repair (FPR), repairing it under varying diagnostic conditions. Evaluating 12 frontier LLMs, we find that fault-targeted reasoning does not scale with general coding ability. Models produce syntactically valid test inputs at near-ceiling rates yet collapse on discriminative generation, with fault hypothesis generation—not output validation—as the dominant bottleneck. Test-guided repair reveals a complementary insight: when self-generated tests successfully witness a fault, the resulting repair matches or outperforms repair guided by externally provided tests, but tests that fail to witness the fault actively degrade repair below unguided baselines.
Introduction: The Era of "Vibe Coding"
We are entering the era of "vibe coding," where developers use agentic tools to generate large swaths of software with minimal oversight. While LLMs now boast 90%+ pass rates on standard benchmarks, these benchmarks usually provide clear specifications and ideal conditions. In the real world, code often "passes the vibe check"—it looks correct and passes basic tests—but fails silently on complex edge cases.
True autonomous software engineering requires fault-targeted reasoning: the ability to look at a near-correct program, hypothesize where it might fail, generate a concrete input to prove that failure (a "witness"), and then use that diagnosis to synthesize a repair. Current evaluations often skip the "diagnosis" step, leaving a massive blind spot in our understanding of how reliable these AI "vibe coders" actually are.
The VIBEPASS Framework
The VIBEPASS benchmark decomposes autonomous debugging into two coupled tasks:
- Fault-Triggering Test Generation (FT-Test): The model is given a problem description and a buggy solution. It must generate a "discriminative witness"—a test case (input and expected output) that the buggy code fails but a correct ("silver") solution passes.
- Fault-Targeted Program Repair (FPR): The model attempts to fix the bug under three guidance settings: NoTest (no feedback), ExtTest (guided by an external high-quality test), and IntTest (guided by a self-generated test).
The benchmark consists of 173 instances across 76 unique algorithmic problems sourced from LiveCodeBench to prevent data contamination. Problems are high-difficulty, focusing on dynamic programming, graph algorithms, and complex data structures.
Key Insights: Syntax vs. Reasoning
A core finding of VIBEPASS is that syntax is not reasoning. Models are nearly perfect at generating syntactically valid test inputs (86% success), but they collapse when those inputs need to actually trigger a specific bug (only 61% success). The primary failure mode is the "Hypothesis Generation" Bottleneck: the inability to even imagine the input that would cause a failure. This fault hypothesis gap is 2.7x larger than the output validation gap.
Furthermore, self-correction can be sabotage. If a model generates a valid, bug-exposing test, its repair performance improves. However, if it generates a "hallucinated" test that doesn't actually expose the bug, it performs worse than if it had no tests at all. The bad data actively misleads the repair process.
Performance Analysis
The study identified massive drops in success rates at two specific transitions: moving from general execution to fault localization (-14.7 percentage points) and moving from test generation to actual repair (-21.2 percentage points).
OpenAI models (specifically variants referred to as GPT-5 in the paper) maintained the highest stability and success rates (54.3%), while open-source models lagged significantly, particularly in the repair task (averaging only 12.1%). There is a near-perfect correlation ($r = 0.98$) between a model's ability to generate a fault-triggering input and its ability to predict the correct output, suggesting these are two sides of the same "causal reasoning" coin.
Conclusion
VIBEPASS reveals that the binding bottleneck for autonomous debugging is not code synthesis or test validity, but fault-target reasoning. While frontier models are excellent at writing code from scratch, they struggle with the causal reasoning required to identify and exploit edge-case failures. Future research should focus on improving the "internal world models" of LLMs, helping them better simulate program state transitions to identify where logic diverges from specifications.
Summary updated