Ringer: web automation by demonstration

Shaon Barman, Sarah E. Chasins, Rastislav Bodík, Sumit GulwaniOOPSLA 2016 · DOI

TL;DR

This paper introduces Ringer, a novel system that allows non-programmers to automate web tasks by simply demonstrating them. Ringer records user interactions and generates robust scripts that adapt to website changes, significantly outperforming existing tools in reliability and ease of use.

What Problem Does This Paper Address?

The web is awash in data, and users increasingly want to programmatically access it for tasks like scraping information, filling forms, or transferring data between sites. However, creating these automation scripts typically requires expert programming skills and the ability to reverse-engineer complex website structures. Existing tools often break due to frequent website updates, making them unreliable for everyday users. This paper tackles the challenge of making web automation accessible and robust for everyone, not just developers.

Methods

Ringer's core innovation lies in its "record and replay" approach, inspired by how humans interact with web pages.

  • Record and Replay: Users perform a task once, and Ringer records their actions (DOM events) and the nodes they interact with. This demonstration is then used to generate a script.
  • Node Addressing: Instead of relying on fragile, specific selectors (like IDs), Ringer uses a similarity-based approach. It considers hundreds of node attributes to find the corresponding node on a replay-time webpage, making it resilient to changes in underlying implementations.
  • Trigger Inference: To handle the asynchronous nature of modern web pages (where content loads dynamically), Ringer infers "triggers." These are conditions (often based on server responses) that must be met before an action can be executed, ensuring the page is ready. This is learned by observing multiple successful replay attempts.
  • Language Design: Ringer's internal language uses three key constructs: Actions (DOM events), Elements (DOM nodes), and Triggers (conditions for execution).

Datasets

The paper doesn't explicitly mention training datasets for machine learning models in the main text, but it does describe the creation of a benchmark suite.

  • Benchmark Suite: A suite of 34 interaction benchmarks was developed, covering a variety of websites and tasks, including those from Alexa's top sites. These benchmarks were designed to be complex and challenging for web automation tools.

Key Insights and Ideas

  • User-Facing Stability: The core insight is that while website implementations change frequently, the user-facing interface tends to remain relatively stable. Ringer leverages this by focusing on user-level interactions rather than low-level code.
  • Similarity Beats Specificity: The paper demonstrates that a similarity-based approach to identifying nodes is far more robust to website changes than methods relying on a few specific attributes (like IDs or XPath).
  • Triggers are Crucial for Robustness: Dynamic web content and asynchronous operations necessitate triggers. Inferring these triggers from multiple successful replays significantly improves script reliability and can even speed up execution compared to naive user-timing.
  • Non-Programmer Accessibility: By abstracting away the complexities of DOM manipulation and asynchronous programming, Ringer empowers non-programmers to automate tasks they previously couldn't.

Key Results

  • Superior Replay Rate: Ringer successfully replayed 74% (25/34) of benchmarks, compared to CoScripter's 18% (6/34).
  • Robustness to Page Changes: Over a three-week period, 92% (22/24) of Ringer scripts continued to run successfully, demonstrating significant resilience to website evolution.
  • Node Identification Performance: Ringer's similarity-based node addressing algorithm outperformed state-of-the-art methods like iMacros and ATA-QV, achieving an 81.4% success rate on clickable DOM nodes.
  • Speedup from Triggers: Trigger inference provided an average speedup of 2.6x compared to user-timing, with a "no-wait" version achieving 3.6x speedup.

Limitations and Future Work

  • Limitations:
    • Ringer's trigger inference relies on specific assumptions (e.g., "slow is safe," arbitrary waits are rare) and may fail for highly interactive applications like games or those with complex client-side delays not tied to server responses.
  • It cannot handle sites with Flash interfaces.
  • The similarity-based node addressing is "best-effort" and doesn't offer theoretical guarantees.
  • The system is not designed for interactions requiring precise absolute timing.
  • Future Work:
    • Ringer is envisioned as a building block for more expressive end-user programming tools, allowing for parametrization of scripts and generalization to handle complex datasets.
  • The authors suggest further exploration of machine learning for node addressing and more sophisticated trigger inference mechanisms.

Summary updated

BibTeX
@inproceedings{barman2016ringer,
  author = {Shaon Barman and Sarah E. Chasins and Rastislav Bodík and Sumit Gulwani},
  title = {{Ringer: web automation by demonstration}},
  booktitle = {OOPSLA},
  year = {2016},
  doi = {10.1145/2983990.2984020},
}
Figure