Extending the team with a project-specific bot

Théo Zimmermann, Julien Coolen, Jason Gross, Pierre-Marie Pédrot, Gaëtan GilbertarXiv 2021 · OpenReview

Abstract

While every other software team is adopting off-the-shelf bots to automate everyday tasks, the Coq team has made a different choice by developing and maintaining a project-specific bot from the ground up. In this article, we describe the reasons for this choice, what kind of automation this has allowed us to implement, how the many features of this custom bot have evolved based on internal feedback, and the technology and architecture choices that have made it possible.

Introduction

Modern software development relies heavily on bots for automation, but generic, "off-the-shelf" bots often cause more friction than they solve. They can be rigid, forcing teams to change established workflows, or "noisy," lacking the social context of the project. For the Coq proof assistant team, these limitations were a significant hurdle. Instead of compromising their workflow, they built a custom, project-specific bot in OCaml. This "meta-bot" handles everything from cross-platform CI synchronization to tracking backports, proving that bespoke automation is a worthwhile investment for large-scale open-source projects.

Challenges in the Coq Development Workflow

By 2018, the Coq team faced several specific challenges that standard tools couldn't handle:

  • Complex CI Needs: They required "Reverse Dependency Compatibility Testing," which involved massive parallelization and artifact sharing across GitHub and GitLab. Standard integrations didn't support this level of cross-platform coordination.
  • Strict Merge Requirements: To maintain security and traceability, Coq requires PGP-signed merge commits with specific formatting. This was a high barrier for new maintainers who might not have their local environments perfectly configured.
  • Process Overhead: Tracking which pull requests (PRs) needed to be "backported" to older release branches was a manual, error-prone task for the Release Manager.

Architecture and Implementation

The Coq bot was developed as a multi-tasking agent built from the ground up. Key technical choices include:

  • Language Choice (OCaml): Since Coq itself is written in OCaml, the core team already possessed the expertise to maintain and extend the bot. This ensured that the bot wasn't a "black box" but a tool the developers could own.
  • Trigger-Action Programming: The bot’s architecture is divided into three components: Event Triggers (listening to webhooks), State Triggers (querying additional data like user membership), and Actions (state-changing requests like adding labels or merging).
  • GraphQL Integration: The bot uses GitHub’s GraphQL API via graphql-ppx to fetch data in a type-safe manner. This reduces API requests and increases reliability compared to traditional REST APIs.

Key Insights: The Bot as a Team Member

Rather than a tool that developers must serve, the bot acts as an assistant that understands the team's specific rules. It doesn't just execute commands; it enforces policy. For example, it won't merge a PR unless milestones and labels are correctly set, acting as a first line of defense for project organization.

The Coq bot enforcing merge rules and tracking backports.

This automation also lowers the barrier to entry. By handling PGP-signed merge commits, the bot allows the team to onboard new maintainers more easily. Furthermore, the bot automates "social" bookkeeping. It monitors PR milestones to automatically move cards on a GitHub Project board, tracking the status of backports (requested vs. shipped) without human intervention. Features evolved based on feedback; for instance, to combat "rebase fatigue," the bot was updated to automatically create merge commits for testing purposes.

Impact and Results

The bot has fundamentally changed how Coq is developed:

  • Dominant Workflow: The bot-based merge command is now the primary way maintainers integrate code.
  • Cross-Platform Synergy: The bot successfully bridges GitHub (code hosting) and GitLab (heavy-duty CI), handling the synchronization of PRs from forks—a feature GitHub's native GitLab integration lacked.
  • Ecosystem Adoption: The bot's core has been adopted by other major projects in the Coq ecosystem, such as the MathComp library.
  • Advanced Features: It now includes a "bug minimizer" that contributors can trigger via comments to help reduce complex bug reports into manageable test cases.

Limitations and Future Directions

Despite its success, the bot requires active maintenance to keep up with API changes from GitHub and GitLab. Currently, the library of bot components is tightly coupled with the Coq bot application. Future work aims to decouple these into a general-purpose OCaml library, allowing other projects to build their own project-specific bots more easily. While the bot is faster than GitHub Actions (which must spin up VMs), it still faces constraints related to external webhooks and API rate limits.

Summary updated

Abstract

While every other software team is adopting off-the-shelf bots to automate everyday tasks, the Coq team has made a different choice by developing and maintaining a project-specific bot from the ground up. In this article, we describe the reasons for this choice, what kind of automation this has allowed us to implement, how the many features of this custom bot have evolved based on internal feedback, and the technology and architecture choices that have made it possible.
BibTeX
@misc{zimmermann2021extending,
  author = {Théo Zimmermann and Julien Coolen and Jason Gross and Pierre-Marie Pédrot and Gaëtan Gilbert},
  title = {{Extending the team with a project-specific bot}},
  year = {2021},
  eprint = {2112.07365},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.07365},
}
Figure