CIVER: Causal Intervention Verification for Reliable Code Review

Xinyuan Song, Hasibul Haque, Jay Somasundaram, Ryan Turner, Bowen Zhu, Liang ZhaoarXiv 2026 · arXiv

TL;DR

CIVER is a novel framework that uses causal interventions to verify the reliability of LLM-generated code reviews. By testing whether a critique’s validity score drops when a targeted repair is applied—while remaining stable under unrelated code changes—CIVER effectively filters out false positives and significantly boosts the success rate of downstream autonomous coding agents.

What Problem Does This Paper Address?

The software engineering world is seeing a surge in code contributions generated by LLM-based agents (like SWE-agent or GitHub Copilot). To keep up, we are also deploying LLM-based reviewers. However, these automated reviewers suffer from a "hallucination" problem: they often produce lengthy, plausible-sounding critiques that are actually false positives or lack evidence.

For a human developer, these noisy reviews cause fatigue. For a downstream coding agent trying to fix bugs, a false positive is even worse—it can lead the agent to "fix" non-existent issues, potentially breaking perfectly good code. The core challenge is moving beyond just checking if a critique is plausible to verifying if it is causally linked to an actual defect in the code.

Methods

CIVER (Causal Intervention Verification) shifts the focus from static analysis to dynamic, intervention-based verification. The framework consists of three jointly optimized modules:

  1. Review Module ($F_\theta$): Transforms raw LLM reviews into structured candidate critiques, identifying the specific code location and context.
  2. Intervention Module ($R_\phi$): Generates and ranks different types of code modifications. It specifically looks for targeted repairs (edits that should fix the claimed bug) and unrelated modifications (distractors that change the code but shouldn't affect the bug).
  3. Verifier Module ($G_\psi$): This is the heart of the system. it evaluates reliability by comparing the critique's validity score across original and modified program states.
Figure 1: Overview of CIVER. (a) A frozen LLM reviewer generates code reviews, which are transformed into candidate critiques by the review module. (b) The intervention module selects relevant code modifications, including targeted repairs and unrelated perturbations. (c) The verifier module evaluates review reliability by comparing scores across original and modified program states. The three modules are jointly optimized through expert iteration to distinguish genuine defects from false positives, while pretrained models remain frozen.

The "causal" magic happens through specific training objectives:

  • Defect Removal: If a critique is valid, its score must decrease when the targeted repair is applied.
  • Unrelated Modification: The score should remain stable if an unrelated part of the code is changed.
  • Semantic Transformation: The score should remain invariant if the code is simply refactored (e.g., renaming variables) without changing behavior.

Datasets

The authors evaluated CIVER across several diverse benchmarks to ensure robustness:

  • SWE-Review-Bench: Used for constructing interventions and evaluating causal effects and downstream repair.
  • AACR-Bench: The primary benchmark for testing validity classification and filtering.
  • CRScore: Provides human-rated comments to evaluate review quality.
  • SWE-PRBench & CodeReviewBench: Real-world pull request data used to test how well CIVER generalizes to new, unseen repositories.
  • Review4Repair: Provided aligned review comments and code edits for building the intervention sets.

Key Insights and Ideas

  • Causality as a Filter: The most important insight is that a true bug report should be "sensitive" to a fix. If you "fix" the bug and the reviewer still complains with the same intensity, the original complaint was likely a hallucination or based on superficial patterns rather than logic.
  • The Importance of Negative Controls: By using unrelated modifications as a baseline, CIVER ensures that the verifier isn't just learning that "any code change equals a lower bug score." It forces the model to be specific.
  • Closing the Loop for Agents: Automated reviews are a critical feedback signal for autonomous agents. The paper shows that high-quality, verified feedback is a force multiplier for agent performance.

Key Results

CIVER demonstrated significant improvements over standard LLM-as-a-judge baselines:

  • Accuracy: It correctly identifies 77% of genuine defects through targeted repairs on SWE-Review-Bench.
  • Reliability: On the AACR-Bench, CIVER reduced false positives per pull request from 2.14 to 1.65 while maintaining high precision (0.77).
  • Downstream Success: Perhaps the most striking result is in downstream repair. As shown in the table below, providing an agent with CIVER-verified "True Reviews" increased its task resolution rate from 61% to 93%. Conversely, giving an agent "False Reviews" actually tanked its performance to 53%.
Table 6: Downstream repair with review guidance. Resolve denotes the final repair success rate, and Resolve Δ denotes the difference in resolution rate relative to No Review.
  • Generalization: CIVER generalized well to held-out projects like Django and scikit-learn, consistently outperforming base reviewers.

Limitations and Future Work

  • Intervention Quality: CIVER’s effectiveness is bottlenecked by the quality of the intervention generator. If the system cannot generate a valid "targeted repair" for a complex bug, it cannot verify the critique.
  • Computational Overhead: Performing multiple causal interventions (generating and scoring multiple code versions) for every critique is computationally more expensive than a single-pass review.
  • Complexity of Edits: The current implementation focuses heavily on identifier renaming and line shifts for semantic invariance. Future work could explore more complex, behavior-preserving refactorings to further harden the verifier against "brittle" LLM logic.

Summary updated

Abstract

Code review is essential for maintaining software repositories by identifying defects, improving code quality, and supporting reliable software evolution. With the growing adoption of LLM-based coding agents, an increasing number of code contributions and repository changes are generated by these agents. Consequently, LLM-based code review agents are becoming increasingly prevalent to evaluate these contributions and maintain software quality. However, LLM reviewers often generate lengthy reviews containing numerous potential issues, many of which are false positives or lack sufficient evidence. These unreliable reviews can mislead developers and downstream coding agents, motivating the need to distinguish actionable findings from unsupported claims. We propose CIVER, a causal intervention-based framework that identifies the core issues underlying code-review findings and distinguishes genuine defects from false positives. CIVER integrates three modules: a review module assesses findings, an intervention module generates repair-oriented perturbations, and a verifier module scores review reliability. Through joint optimization via expert iteration, the three modules identify genuine defects by responding to targeted repairs while remaining stable under unrelated changes. Extensive experiments on SWE-Review-Bench demonstrate that CIVER correctly identifies 77% of genuine defects through targeted repairs. Cross-project and downstream repair evaluations further demonstrate that CIVER generalizes across repositories and improves coding agents' task success rates, highlighting its effectiveness for reliable automated code review.
BibTeX
@misc{song2026civer,
  author = {Xinyuan Song and Hasibul Haque and Jay Somasundaram and Ryan Turner and Bowen Zhu and Liang Zhao},
  title = {{CIVER: Causal Intervention Verification for Reliable Code Review}},
  year = {2026},
  eprint = {2610.36010},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36010},
}
Figure