BitsAI-CR: Automated Code Review via LLM in Practice

Tao Sun, Jian Wen Xu, Yuanpeng Li, Yan Zhao, Ge Zhang, Lei Xie, Lu Geng, Zheng Wang, Yueyan Chen, Qin Lin, Wenbo Duan, Kaixin Sui

Abstract

Code review remains a critical yet resource-intensive process in software development, particularly challenging in large-scale industrial environments. While Large Language Models (LLMs) show promise for automating code review, existing solutions face significant limitations in precision and practicality. This paper presents BitsAI-CR, an innovative framework that enhances code review through a two-stage approach combining RuleChecker for initial issue detection and ReviewFilter for precision verification. The system is built upon a comprehensive taxonomy of review rules and implements a data flywheel mechanism that enables continuous performance improvement through structured feedback and evaluation metrics. Our approach introduces an Outdated Rate metric that can reflect developers' actual adoption of review comments, enabling automated evaluation and systematic optimization at scale. Empirical evaluation demonstrates BitsAI-CR's effectiveness, achieving 75.0% precision in review comment generation. For the Go language which has predominant usage at ByteDance, we maintain an Outdated Rate of 26.7%. The system has been successfully deployed at ByteDance, serving over 12,000 Weekly Active Users (WAU). Our work provides valuable insights into the practical application of automated code review and offers a blueprint for organizations seeking to implement automated code reviews at scale.

Introduction

Code review is a cornerstone of modern software engineering, ensuring code quality, security, and knowledge sharing. However, it is often a significant bottleneck. At ByteDance, internal surveys reveal that reviewers spend an average of 15 minutes per review, and 67% of engineers express a need for more efficient tooling.

While Large Language Models (LLMs) have demonstrated a remarkable ability to understand and generate code, their application in industrial code review has been hindered by three primary challenges: low precision (hallucinations), low practicality (focusing on "nits" that developers ignore), and static performance (the inability to learn from developer interactions). BitsAI-CR was developed to overcome these hurdles, moving AI-assisted review from a research prototype to a production-scale tool serving over 12,000 weekly active users.

System Architecture

BitsAI-CR utilizes a two-stage verification pipeline designed to prioritize precision over recall, ensuring that developers are not overwhelmed by "alert fatigue."

1. RuleChecker: Targeted Detection

The first stage, RuleChecker, utilizes a fine-tuned LLM (based on ByteDance’s Doubao-Pro) to identify potential issues. Rather than providing generic feedback, the model is guided by a structured taxonomy of 219 specific review rules. To improve accuracy, the system uses tree-sitter for context preparation, allowing the model to analyze full function definitions and relevant code blocks rather than isolated "diff" hunks.

2. ReviewFilter: Precision Verification

To mitigate hallucinations, the second stage—ReviewFilter—acts as a sanity check. This fine-tuned model evaluates the proposed comment against the code to determine its validity. Research during development indicated that a "Conclusion-First" reasoning pattern—where the LLM provides a binary decision before its explanation—offered the optimal balance of inference speed and filtering precision.

The Data Flywheel and the "Outdated Rate"

A core innovation of BitsAI-CR is its "Data Flywheel," a continuous feedback loop that allows the system to evolve based on real-world usage. This loop is powered by two primary data sources:

  • Direct User Feedback: Developers can "like" or "dislike" AI-generated comments, providing explicit labels for model tuning.
  • The Outdated Rate Metric: To measure actual adoption, the authors introduced the "Outdated Rate." This metric tracks whether a developer modified the specific lines of code flagged by the AI after the comment was posted. If the code is changed, the comment is considered "adopted." If the developer ignores the suggestion and merges the code as-is, the comment is marked as "outdated," suggesting the rule may be too noisy or irrelevant.

By decommissioning rules with high Outdated Rates and fine-tuning models on high-quality, adopted feedback, the system maintains a high standard of relevance.

Empirical Evaluation

Datasets

The system was trained on 120,000 internal code review comments from ByteDance repositories. This was refined into a high-quality training set of approximately 18,000 samples for Go and Frontend development, and 5,000 samples for other languages. Evaluation was conducted against a curated offline dataset of 1,397 cases, benchmarking BitsAI-CR against state-of-the-art models like Qwen2.5-Coder and DeepSeek-v2.5.

Key Results

  • Precision: The two-stage pipeline achieved a peak precision of 75.0%.
  • Developer Adoption: In Go, the system’s primary language, the Outdated Rate reached 26.7%. For comparison, human reviewers at ByteDance typically see Outdated Rates between 35% and 46%, indicating that BitsAI-CR’s suggestions are approaching human-level relevance.
  • Scale and Retention: The system serves over 12,000 Weekly Active Users and maintains a second-week retention rate of 61.64%, demonstrating sustained value for the developer community.

Discussion and Future Work

While BitsAI-CR represents a significant step forward, it currently focuses on function-level analysis. This limitation means the system can struggle with "cross-file" issues, where changes in one module impact logic in another. Future iterations will focus on expanding contextual depth to capture these complex architectural bugs and increasing language coverage to support the full breadth of the ByteDance ecosystem.

By combining a rigorous two-stage pipeline with a metric-driven feedback loop, BitsAI-CR demonstrates that LLMs can provide high-precision, actionable code reviews at an industrial scale.

Summary updated

Abstract

Code review remains a critical yet resource-intensive process in software development, particularly challenging in large-scale industrial environments. While Large Language Models (LLMs) show promise for automating code review, existing solutions face significant limitations in precision and practicality. This paper presents BitsAI-CR, an innovative framework that enhances code review through a two-stage approach combining RuleChecker for initial issue detection and ReviewFilter for precision verification. The system is built upon a comprehensive taxonomy of review rules and implements a data flywheel mechanism that enables continuous performance improvement through structured feedback and evaluation metrics. Our approach introduces an Outdated Rate metric that can reflect developers' actual adoption of review comments, enabling automated evaluation and systematic optimization at scale. Empirical evaluation demonstrates BitsAI-CR's effectiveness, achieving 75.0% precision in review comment generation. For the Go language which has predominant usage at ByteDance, we maintain an Outdated Rate of 26.7%. The system has been successfully deployed at ByteDance, serving over 12,000 Weekly Active Users (WAU). Our work provides valuable insights into the practical application of automated code review and offers a blueprint for organizations seeking to implement automated code reviews at scale.
BibTeX
@misc{sun2025bitsai,
  author = {Tao Sun and Jian Wen Xu and Yuanpeng Li and Yan Zhao and Ge Zhang and Lei Xie and Lu Geng and Zheng Wang and Yueyan Chen and Qin Lin and Wenbo Duan and Kaixin Sui},
  title = {{BitsAI-CR: Automated Code Review via LLM in Practice}},
  year = {2025},
  eprint = {2501.15134},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.15134},
}
Figure