Abstract
Bots are becoming a popular method for automating everyday tasks in many software projects, thanks to the availability of many off-the-shelf task-specific bots that teams can quickly adopt. Based on our experience, we argue that an alternative approach deserving more attention is to develop a multitask project-specific bot, because it strikes a good balance between productivity and adaptability.
Introduction
Modern software development relies heavily on automation to manage the complexities of large-scale projects. Bots are frequently deployed for tasks such as labeling issues, monitoring continuous integration (CI) status, and managing pull requests (PRs). However, the prevailing industry trend of adopting multiple "off-the-shelf," single-task bots often introduces significant friction. Generic bots typically force teams to adapt their established processes to the bot's rigid logic. Furthermore, as a project accumulates various bots—for stale issues, formatting, and CI—contributors face cognitive overload, struggling to remember the specific commands and behaviors of each tool.
In this experience report, members of the Coq proof assistant team describe an alternative: Coqbot, a single, multi-task, project-specific bot. By building a bespoke tool tailored to their unique requirements, the team has achieved superior flexibility and better alignment with complex development workflows.
System Architecture and Implementation
Coqbot has been maintained since 2018 and is built on several key technical pillars designed for long-term sustainability and integration:
- Language Familiarity: The bot is written in OCaml, the same language as the Coq proof assistant itself. This ensures that the core development team already possesses the necessary skills to maintain and extend the bot without learning a new ecosystem.
- Trigger-Action Programming: The bot utilizes a modular design. Event Triggers listen to webhooks (e.g., new comments or PR updates), while State Triggers query APIs for context (e.g., verifying maintainer status). These triggers then initiate specific Actions, such as labeling or merging.
- GraphQL Integration: By using GraphQL to interface with GitHub, Coqbot queries only the specific data required, reducing API overhead and leveraging type safety.
- Cross-Platform Bridge: A critical feature is the bot's ability to bridge GitHub (hosting the code) with GitLab CI (handling Coq's extensive CI requirements). It manages branch mirroring and reports granular failure logs back to the GitHub interface.
Strategic Advantages: Workflow-First Automation
The primary benefit of a project-specific bot is the ability to prioritize the team's workflow over the tool's limitations. Rather than changing their rules to fit a generic bot, the Coq team used Coqbot to enforce their existing, complex policies.
As illustrated in Figure 1, the bot acts as a gatekeeper, ensuring that PRs meet specific criteria—such as having a milestone set—before allowing a merge. This consolidation creates a "Meta-Bot" interface where developers only need to learn one set of commands (e.g., @coqbot: merge now) to interact with the entire automation suite. This significantly reduces the social friction and mental energy required for project participation.
Furthermore, maintaining a single codebase allows for feature synergy. For instance, the bot’s CI reporting logic provides the necessary data for its automated test-case reduction feature, which minimizes code to isolate bugs—a task that would be difficult to coordinate across multiple independent bots.
Impact and Results
Coqbot has become the primary method for merging PRs within the Coq project, successfully handling requirements that the standard GitHub "Merge" button cannot. Its impact extends across the Coq ecosystem, including the MathComp library and external organizations like SaltStack, which utilizes the bot's core logic across hundreds of repositories.
Key results include:
- Advanced CI Management: The bot manages "reverse dependency" testing, triggering builds for dozens of related projects whenever a Coq PR is updated to prevent breaking changes.
- Intelligent Stale Management: Unlike generic bots that close PRs based on simple inactivity timers, Coqbot uses a "needs: rebase" label to flag only those PRs that are technically broken by merge conflicts, a strategy developers find far more constructive.
- Maintenance Efficiency: Reusing components across different workflows has made the maintenance burden of one large bot lower than that of managing several disparate tools.
Discussion and Future Work
While highly effective, the project-specific approach requires a significant upfront investment in time and code, making it less suitable for very small projects. Additionally, while the strategy is generalizable, the current codebase is highly specific to the Coq environment.
Future efforts are focused on making the OCaml bot components more reusable for other projects and implementing even more advanced merging strategies, such as automated "final comment periods" and integrated performance benchmarking. Ultimately, the Coq team's experience suggests that for medium-to-large projects with specialized needs, the productivity and adaptability of a bespoke bot far outweigh the initial development costs.
Summary updated