Idiolect: A Reconfigurable Voice Coding Assistant

Breandan Considine, Nicholas Albion, Xujie Si

Abstract

This paper presents Idiolect, an open source (https://github.com/OpenASR/idiolect) IDE plugin for voice coding and a novel approach to building bots that allows for users to define custom commands on-the-fly. Unlike traditional chatbots, Idiolect does not pretend to be an omniscient virtual assistant but rather a reconfigurable voice programming system that empowers users to create their own commands and actions dynamically, without rebuilding or restarting the application. We offer an experience report describing the tool itself, illustrate some example use cases, and reflect on several lessons learned during the tool's development.

1. Introduction

Voice coding has long promised a hands-free future for developers, particularly those with motor or visual impairments or those seeking to mitigate repetitive strain injuries. However, existing tools often suffer from "command rigidity." Most voice assistants rely on static lexicons; if a user deviates from a pre-defined phrase—for example, saying "open sesame" instead of "open file"—the system fails.

Furthermore, developers have unique workflows that require high levels of personalization. A command that is essential for a Java developer may be irrelevant to a web designer. Idiolect addresses this "misalignment" between user intent and bot capabilities by providing a deeply integrated, reconfigurable environment within the IntelliJ Platform.

2. System Architecture

Idiolect is designed to adapt to the user, rather than forcing the user to adapt to the system. Its architecture is built on four primary pillars:

2.1 Offline Automatic Speech Recognition (ASR)

The system utilizes Vosk, a state-of-the-art deep speech recognition system. By running locally, Idiolect ensures low latency and maintains developer privacy, as no audio data is transmitted to the cloud.

2.2 Tiered Intent Recognition

When a user speaks, the system processes the transcription through a prioritized pipeline:

  1. Plugin Control: Immediate commands to enable or disable the tool.
  2. User-Defined Commands: Custom patterns defined via a Domain Specific Language (DSL) or Context-Free Grammar (CFG), allowing for on-the-fly reconfiguration.
  3. LLM (The Recognizer of Last Resort): If no exact match is found, a Large Language Model attempts to map the spoken phrase to the most likely IDE action (e.g., mapping "I want to edit foo.java" to the "open file" action).

2.3 Error Recovery with Tidyparse

To mitigate the frustration of misrecognitions, Idiolect employs Tidyparse. This tool uses SAT solvers to identify the smallest "edit distance" between a misspoken or misrecognized phrase and a valid command, providing a robust mechanism for error correction.

2.4 IDE Integration

Idiolect maps voice commands directly to the 1,000+ internal actions available in the IntelliJ Platform. By using CamelCase identifiers from the IDE’s API, the system can auto-generate natural language descriptions for a wide variety of programming tasks.

3. Evaluation and Results

While the paper does not introduce a new training dataset, the authors conducted a synthetic benchmark to evaluate the system's performance. They synthesized 100 utterances from the default command list using seven different macOS Text-to-Speech (TTS) voices and tested them against three different Vosk models.

3.1 Model Performance

The vosk-model-small-en-us-0.22 model demonstrated the highest effectiveness for specific IDE tasks, suggesting that smaller, optimized models are well-suited for this domain.

3.2 Word Error Rate (WER) and Bias

The evaluation revealed significant variance in Word Error Rates based on the voice used. WER ranged from approximately 0.27 for the best female voice to 0.99 for the worst male voice on certain models. This finding highlights a persistent gender bias in common ASR pipelines, where female-synthesized voices were consistently recognized with higher accuracy.

4. Key Insights

  • The "Idiolect" Philosophy: The tool is named after the concept of an individual's unique use of language. The core insight is that the system must adapt to the user's specific vernacular.
  • Observability: Idiolect provides a real-time visual transcription cue in the IDE. This transparency helps users understand whether a failure was due to background noise or a logic error.
  • LLMs as a Safety Net: Using LLMs as a "last resort" bridges the gap between natural "vernacular" speech and technical API calls, ensuring the system remains functional even when the user forgets exact command names.

5. Limitations and Future Work

The current evaluation relies on synthetic TTS voices, which are cleaner than real human speech; therefore, a comprehensive user study with developers is a priority for future research. Additionally, the authors aim to improve contextual awareness, using LLMs to rerank intents based on the current UI state (e.g., cursor position or active file) and improving support for complex, compound scripted actions.

Summary updated

Abstract

This paper presents Idiolect, an open source (https://github.com/OpenASR/idiolect) IDE plugin for voice coding and a novel approach to building bots that allows for users to define custom commands on-the-fly. Unlike traditional chatbots, Idiolect does not pretend to be an omniscient virtual assistant but rather a reconfigurable voice programming system that empowers users to create their own commands and actions dynamically, without rebuilding or restarting the application. We offer an experience report describing the tool itself, illustrate some example use cases, and reflect on several lessons learned during the tool's development.
BibTeX
@inproceedings{considine2023idiolect,
  author = {Breandan Considine and Nicholas Albion and Xujie Si},
  title = {{Idiolect: A Reconfigurable Voice Coding Assistant}},
  booktitle = {BotSE},
  year = {2023},
  doi = {10.1109/botse59190.2023.00011},
}
Figure