LASER: a living analytics experimentation system for large-scale online controlled experiments

Kwan Hui Lim, Ee-Peng Lim, Palakorn Achananuparp, Adrian Vu, Agus Trisnajaya Kwee, Feida ZhuWWW 2014 · DOI

TL;DR

This paper introduces LASER, a unified platform that simplifies and automates the design and execution of large-scale online controlled experiments for evaluating website features and recommendation algorithms. It addresses the complexity and effort typically required for such experiments, making them more accessible to researchers and smaller companies.

What Problem Does This Paper Address?

Conducting rigorous online controlled experiments (like A/B tests) is crucial for understanding how changes to websites or services impact user behavior. However, existing methods often involve significant manual effort, extensive modifications to websites, and can be prone to confounding factors (e.g., promotions). While large tech companies routinely use these experiments, smaller organizations and academic researchers often lack the tools and infrastructure to implement them effectively. This gap limits their ability to gather reliable data on user engagement and the effectiveness of new features or algorithms in real-world scenarios.

Methods

LASER is presented as a comprehensive, web-based system with three core components:

  • Experiment Setup: A user-friendly GUI allows experimenters to configure various aspects of an experiment without coding. This includes defining web pages for tracking, user groups, cookie/session parameters, and experiment batches (with start/end dates). The system supports simple tracking, randomized controlled tests, split tests with custom user selection algorithms, and multi-variate tests.
  • Data Collection: This involves an auto-generated JavaScript tracking code that experimenters embed on their websites. This code handles user grouping (control vs. treatment), tracks user sessions, and logs fine-grained user activities (e.g., clicks, mouse-overs). It communicates with LASER's backend web services asynchronously.
  • Data Visualization: A dashboard component provides real-time and post-experiment insights through pre-defined charts (using Google Charts). These charts visualize aggregated statistics, user activity heatmaps, conversion rates, and user grouping distributions.

The system's novelty lies in its unified, automated approach, minimal modification requirement for experimenters, and support for customized user selection and recommendation algorithms.

Datasets

The paper does not explicitly mention specific datasets used for training or evaluation. Instead, it describes the system's deployment and usage on "more than 30 different websites" between May 2013 and Feb 2014, tracking over 70,000 unique users and 1.5 million user events. The authors mention using results from these actual experiments to demonstrate the system's capabilities.

Key Insights and Ideas

  • Democratizing Controlled Experiments: LASER significantly lowers the barrier to entry for conducting online controlled experiments, making powerful data-driven evaluation accessible beyond large tech giants.
  • Flexibility and Customization: The system's ability to support custom user selection algorithms and integrate with external recommendation engines is a key differentiator, allowing for more nuanced and realistic experimental designs.
  • Real-world Impact: By enabling experiments on live websites with real users, LASER provides more ecologically valid results compared to lab-based user studies.
  • Visualizing Complex Data: The dashboard's real-time visualization capabilities are crucial for understanding experiment progress and results, transforming raw data into actionable insights.

Key Results

The paper focuses on the system's architecture and functionality rather than presenting specific quantitative results from a single experiment. However, it highlights the system's successful deployment:

  • Used to conduct experiments and tracking on over 30 different websites.
  • Successfully tracked over 70,000 unique users.
  • Collected over 1.5 million user events (clicks, mouse-overs, sessions).
  • The system demonstrated its ability to effectively track user activity and provide real-time dashboard summaries for various experiment types.

Limitations and Future Work

Limitations:

  • JavaScript Dependency: The tracking code requires users to have JavaScript enabled in their browsers. The authors note this has minimal impact as only ~2% of users reportedly disable JavaScript.

Future Work:

  • Network Experiments: Extend LASER to support network experiments, which require careful user selection to avoid treatment interference. This suggests a move towards more complex experimental designs involving interconnected users.

Summary updated

BibTeX
@inproceedings{lim2014laser,
  author = {Kwan Hui Lim and Ee-Peng Lim and Palakorn Achananuparp and Adrian Vu and Agus Trisnajaya Kwee and Feida Zhu},
  title = {{LASER: a living analytics experimentation system for large-scale online controlled experiments}},
  booktitle = {The Web Conference},
  year = {2014},
  doi = {10.1145/2567948.2577008},
}
Figure