Abstract
Automating web-based customer support tasks remains a significant challenge due to the difficulty of grounding natural language instructions into specific Document Object Model (DOM) elements. We present RUSS (Rapid Universal Support Service), a modular AI agent that translates human-written, step-by-step instructions into a domain-specific language (DSL) called ThingTalk, which is then grounded in a live browser environment. RUSS achieves 76.7% end-to-end accuracy on real-world tasks, significantly outperforming previous end-to-end neural models and demonstrating high user preference in human studies.
Introduction
Most web automation today relies on brittle, site-specific APIs or custom scripts that break when a user interface changes. Meanwhile, the web is populated with human-readable instructions designed for manual execution (e.g., "To reset your password, go to settings and click 'Security'").
The core challenge in automating these tasks is grounding: an AI must accurately map phrases like "the text field under email address" to a specific HTML element with a unique ID in the DOM. This is difficult because instructions are "open-domain"—written in diverse, informal ways—and modern websites often contain thousands of elements. Existing end-to-end neural models frequently fail to map long, complex instructions directly to actions without becoming lost in the noise of the webpage structure.
The RUSS Architecture
RUSS employs a modular architecture that separates intent understanding from target localization. This approach provides a structured "contract" between components, making the system more robust and easier to debug than monolithic end-to-end models.
1. ThingTalk DSL and Semantic Parsing
The team designed ThingTalk, a domain-specific language that serves as an intermediate bridge. It includes primitives like @goto, @click, and @enter, along with a @retrieve function that describes how to find an element (e.g., "the button labeled 'Submit' above the footer").
To translate natural language into ThingTalk, RUSS uses a semantic parser based on a BERT-LSTM architecture. A critical component of this parser is the pointer network, which allows the model to "point" to and copy specific strings from the input—such as URLs or usernames—that it may not have encountered during training.
2. Grounding and Runtime
Once a ThingTalk command is generated, the grounding model searches the live webpage's DOM. It filters elements by type and location and utilizes Sentence-BERT to calculate semantic similarity between the instruction's description and the actual text or attributes of the HTML elements. Finally, a browser-based runtime (using Puppeteer) executes the actions. If an instruction requires missing user data, RUSS can interact with the user via a dialogue interface to request the necessary information.
Datasets and Evaluation
The researchers introduced two significant datasets to facilitate this work:
- RUSS Evaluation Dataset: A high-quality benchmark of 80 real-world customer service tasks (e.g., "Redeem Amazon Gift Card") across 22 popular websites, containing 741 step-by-step instructions mapped to ground-truth actions.
- RUSS Synthetic Dataset: To overcome the data bottleneck of manual labeling, the authors synthesized 1.5 million instruction-to-code pairs using templates and scraped web data. Remarkably, the parser trained on this synthetic data generalized successfully to real-world instructions.
Key Results
- End-to-End Accuracy: RUSS achieved 76.7% accuracy in performing the correct action from a single instruction.
- Grounding Performance: In the grounding sub-task, RUSS achieved 63.6% accuracy, significantly outperforming the state-of-the-art "PhraseNode" baseline (46.5%).
- User Study: In a study with 12 participants, 69% of users preferred using RUSS over manual navigation. Users completed 98% of tasks with RUSS's assistance, compared to 85% when navigating independently.
Conclusion
RUSS demonstrates that a modular approach to web automation—leveraging a DSL and semantic grounding—is far more effective than end-to-end neural mapping. Future work will focus on incorporating visual features to handle cues like "the blue button" and improving dialogue robustness for more complex user interactions.
Summary updated