The Present and Future of Bots in Software Engineering

Emad Shihab, Stefan Wagner, Marco Aurélio Gerosa, Mairieli Santos Wessel, Jordi Cabot

Abstract

We are witnessing a massive adoption of software engineering bots—applications that react to events triggered by tools and messages posted by users to run automated tasks. This paper explores the current state of these "digital teammates," synthesizing experiences and challenges from across the field. While bots have become essential for managing the complexity of modern development, significant hurdles remain in bot-to-bot coordination, social awareness, and security.

The Complexity Crisis in Software Engineering

Modern software development is currently grappling with an "essential complexity" crisis. Maintainers, particularly within the open-source ecosystem, are increasingly overwhelmed by repetitive, high-volume tasks: triaging bugs, managing pull requests (PRs), updating dependencies, and addressing recurring architectural queries. This administrative burden often leads to developer burnout and threatens the long-term sustainability of critical projects.

To combat this, "Software Engineering Bots" (DevBots) have emerged as a primary solution. However, the field remains fragmented. Until recently, we lacked a comprehensive understanding of how these bots are deployed, their actual impact on human workflows, and the technical friction—such as bot-on-bot conflicts—that prevents them from becoming truly intelligent collaborators.

Research Methodology

This synthesis draws from an editorial overview of several peer-reviewed studies featured in IEEE Software. The research utilizes a multi-pronged approach to map the bot landscape:

  • Case Studies: Longitudinal analysis of specific tools, such as "Coq bot" and "Dependabot," to track their evolution over time.
  • Repository Mining: Large-scale sampling of the top 1,000 most popular GitHub repositories to quantify adoption rates and task types.
  • Empirical Analysis: A review of approximately 500 projects to understand how developers customize bot configurations and the trade-offs involved.
  • Expert Synthesis: Identification of emerging challenges in security, coordination, and the socio-economics of automation.

Key Insights: From Mechanics to Butlers

The "Rule-Based" Ceiling

Most contemporary bots function as "mechanics" rather than "butlers." They operate on rigid, if-then logic and lack the interactive "soft skills" required to navigate nuanced human communication. While they excel at repetitive technical tasks, they often fail to integrate seamlessly into the social fabric of a development team.

The Identity and Configuration Paradox

Bots are currently among the most active accounts on platforms like GitHub, yet they are frequently poorly identified. This "ghost activity" can skew productivity analytics, as tools may struggle to distinguish between human and automated contributions. Furthermore, while customizing bots (like Dependabot) can reduce notification "noise," it often inadvertently breaks core features. Developers are frequently caught in a struggle to find the "sweet spot" between helpful automation and manual oversight.

Bots as Social Catalysts

There is significant potential for bots to move beyond code management and act as social catalysts. Future iterations could promote diversity by identifying and supporting contributions from underrepresented groups or using sentiment analysis to "read the room" before intervening in a heated developer discussion.

Research indicates that bot adoption is no longer a niche phenomenon; over 60% of popular open-source projects now utilize at least one bot. These tools rarely remain static; for instance, the Coq bot evolved from a simple repository synchronizer into a sophisticated system handling backporting and merge-conflict tracking based on direct developer feedback.

However, gaps remain. Despite the ubiquity of GitHub Actions, bots are still rarely utilized for continuous performance benchmarking. This represents a significant missed opportunity for teams to catch performance regressions early in the development cycle.

The Road Ahead: Challenges and Future Work

The integration of bots remains a "Wild West" with several unresolved issues:

  • Bot-on-Bot Conflict: There is currently no "traffic cop" to manage interactions between multiple bots in the same repository. Conflicting actions can lead to "edit wars" or broken builds.
  • Cognitive Testing: We lack frameworks to verify "cognitive bots." Ensuring that a bot’s conversational component aligns correctly with its technical logic remains a major hurdle.
  • Economic and Security Models: The industry lacks robust ROI models for automation; the "cost of learning" and configuration often offsets the perceived benefits of "free" bots. Additionally, there is a pressing need for "minimum permission" protocols to ensure bots do not accidentally leak sensitive project data or become vectors for supply-chain attacks.

As bots transition from simple scripts to autonomous teammates, the focus must shift from mere task execution to sophisticated, socially aware coordination.

Summary updated

Abstract

We are witnessing a massive adoption of software engineering bots, applications that react to events triggered by tools and messages posted by users and run automated tasks in response, in a variety of domains. This thematic issues describes experiences and challenges with these bots.
BibTeX
@inproceedings{shihab2022present,
  author = {Emad Shihab and Stefan Wagner and Marco Aurélio Gerosa and Mairieli Santos Wessel and Jordi Cabot},
  title = {{The Present and Future of Bots in Software Engineering}},
  booktitle = {IEEE Softw},
  year = {2022},
  url = {https://openreview.net/forum?id=Oid0AsVqpY},
}

Loading related content...

Figure