Advantages of maintaining a multi-task project-specific bot: an experience report

Théo Zimmermann, Julien Coolen, Jason Gross, Pierre-Marie Pédrot, Gaëtan Gilbert

Abstract

Bots are becoming a popular method for automating basic everyday tasks in many software projects. This is true in particular because of the availability of many off-the-shelf task-specific bots that teams can quickly adopt (which are sometimes completed with additional task-specific custom bots). Based on our experience in the Coq project, where we have developed and maintained a multi-task project-specific bot, we argue that this alternative approach to project automation should receive more attention because it strikes a good balance between productivity and adaptibility. In this article, we describe the kind of automation that our bot implements, what advantages we have gained by maintaining a project-specific bot, and the technology and architecture choices that have made it possible. We draw conclusions that should generalize to other medium-sized software teams willing to invest in project automation without disrupting their workflows.

Introduction

Automation is essential for modern software development, but the "app store" model of GitHub bots—where projects adopt multiple specialized third-party tools—has significant drawbacks for established projects. While small teams can easily adopt off-the-shelf bots, larger projects often have complex, idiosyncratic workflows that generic tools fail to support.

The developers of the Coq proof assistant identified three main pain points with generic bots:

  1. Friction: Generic bots often lack social context and can disrupt existing developer workflows.
  2. Rigidity: Configuration files for off-the-shelf bots are often too limited to handle unique project requirements, such as specific PGP signing rules or complex CI bridging.
  3. Bot Fatigue: Interacting with multiple different bots for different tasks increases the cognitive load on contributors.

To address these issues, the Coq project developed Coqbot, a single, project-specific bot tailored to their specific needs.

Methodology and Architecture

Coqbot was designed using a trigger-action programming model, built around three core components:

  • Event Triggers: These listen to webhooks, such as new comments or Pull Request (PR) updates.
  • State Triggers: These query APIs for additional context, such as verifying if a user is a member of the core team.
  • Actions: These perform state-changing requests, such as adding labels or merging PRs.

A strategic decision was made to build Coqbot in OCaml, the same language used for the Coq proof assistant itself. This ensured that the project's maintainers were already familiar with the codebase, significantly lowering the barrier for contributions. Furthermore, the bot serves as a bridge between GitHub (hosting) and GitLab CI (testing), managing failure reporting and ensuring test validity through automated merge commits.

The Multi-Task Advantage

By consolidating many features into one "meta-bot," the team significantly reduced noise in PR threads. Developers only need to remember one prefix (@coqbot) to trigger various actions. This unified approach allows for synergy between features; for example, CI reporting logic was easily extended to support specialized test-case reduction tools.

Additionally, a project-specific bot allows for custom logic that generic bots cannot provide. For instance, instead of a standard "stale bot" that closes PRs based on time, Coqbot identifies staleness based on merge conflicts. This ensures that valuable but "quiet" PRs are preserved while technically broken ones are flagged for attention.

Results and Impact

Coqbot has become the primary mechanism for maintainers to merge code. It enforces complex requirements—such as checking for milestones, PGP signatures, and specific formatting—that the standard GitHub "Merge" button cannot handle.

Coqbot managing a merge request and automated project board updates.

Beyond merging, the bot has automated significant portions of project management. It tracks "backports" (applying fixes to older versions) by automatically moving cards on a GitHub Project board, a task that was previously manual and error-prone. The bot also manages the heavy lifting of mirroring PRs to GitLab CI and reporting back only relevant failures, preventing developers from being overwhelmed by logs from dozens of parallel test runs.

Conclusion

While developing a custom bot requires a significant upfront investment in time and engineering, the Coq project's experience demonstrates that the long-term benefits in productivity and workflow adaptability are substantial. For medium-to-large software teams, building a project-specific bot in the project's native language can turn automation into a first-class part of the source code. Future work for Coqbot includes integrating automated benchmarking and more sophisticated merging strategies, such as mandatory "cooling-off" periods after CI passes.

Summary updated

Abstract

Bots are becoming a popular method for automating basic everyday tasks in many software projects. This is true in particular because of the availability of many off-the-shelf task-specific bots that teams can quickly adopt (which are sometimes completed with additional task-specific custom bots). Based on our experience in the Coq project, where we have developed and maintained a multi-task project-specific bot, we argue that this alternative approach to project automation should receive more attention because it strikes a good balance between productivity and adaptibility. In this article, we describe the kind of automation that our bot implements, what advantages we have gained by maintaining a project-specific bot, and the technology and architecture choices that have made it possible. We draw conclusions that should generalize to other medium-sized software teams willing to invest in project automation without disrupting their workflows.
BibTeX
@inproceedings{zimmermann2022advantages,
  author = {Théo Zimmermann and Julien Coolen and Jason Gross and Pierre-Marie Pédrot and Gaëtan Gilbert},
  title = {{Advantages of maintaining a multi-task project-specific bot: an experience report}},
  booktitle = {IEEE Softw},
  year = {2022},
  url = {https://openreview.net/forum?id=2Dw9pItc5r},
}
Figure