Zoetrope: interacting with the ephemeral web

Eytan Adar, Mira Dontcheva, James Fogarty, Daniel S. WeldUIST 2008 · DOI

TL;DR

The web is constantly changing, making it hard to access past information. Zoetrope is a system that allows users to interact with historical web pages, enabling them to explore, query, and analyze web content over time through novel "lenses" and visualizations. This opens up new possibilities for understanding the dynamic nature of the web.

What Problem Does This Paper Address?

The World Wide Web is inherently ephemeral. Web pages are frequently updated, links can break, and content that was once readily available can disappear or become difficult to find. This makes it challenging to access historical information, analyze trends, or even simply recall what a page looked like at a specific point in the past. Current web browsers and search engines primarily provide access to the most recent version of a page, treating the web as a static entity. This lack of temporal access hinders our ability to understand the evolution of online content and makes it difficult to perform historical analysis.

Methods

Zoetrope introduces a novel system for interacting with temporal web data, built around a set of core concepts:

  • Content Streams: Zoetrope represents web content over time as a sequence of tuples (Ti, Ci), where Ci is a content item (like a webpage or a part of it) and Ti is the timestamp of when it was sampled.
  • Lenses: These are interactive tools that allow users to select and track specific parts of a webpage over time. There are three main types:
  • Visual Lenses: Allow users to draw a region on a webpage and view that specific visual area across different historical versions.
  • Structural Lenses: Track selected HTML content (DOM elements) regardless of their visual position, making them robust to minor rendering changes.
  • Textual Lenses: Track specific text content, even if its location or surrounding elements change significantly on the page. This is achieved by generalizing XPath queries and using similarity metrics.
  • Transforms and Filters: Lenses are powered by a pipeline of transforms (e.g., cropping, selecting DOM elements) and filters (e.g., time-based, keyword-based, numerical thresholds) that process the content streams.
  • Visualizations: Zoetrope provides various ways to visualize temporal data, including timelines, movies, and cluster visualizations, which can be driven by lenses.
  • Binding and Stacking: Lenses can be bound together (conjunctively or disjunctively) to explore correlated data across multiple lenses or pages. Lenses can also be "stacked" to apply sequential filtering and transformations.
  • Crawler and Indexing: A custom crawler collects web data at regular intervals (e.g., hourly) and stores both the DOM structure (for semantic access) and rendered images (for visual representation). This high-resolution temporal data is crucial for Zoetrope's functionality.

Datasets

The paper describes an experimental dataset consisting of 250 webpages from various domains (news, sports, weather, entertainment) crawled at one-hour intervals for over a month. This dataset was collected using a new crawler design specifically for high-resolution temporal web data.

Key Insights and Ideas

  • The Web as a Temporal Database: The core idea is to treat the web not as a collection of static pages, but as a dynamic, evolving database that can be queried and analyzed over time.
  • Lenses as Powerful Interaction Primitives: The concept of "lenses" provides an intuitive and flexible way for users to define temporal queries and extract specific pieces of information from historical web content.
  • Bridging Visual and Semantic Access: Zoetrope effectively combines visual representations of webpages with their underlying DOM structure, allowing for both visual exploration and semantic querying of temporal data.
  • Handling Web Ephemerality: By focusing on temporal interaction and providing tools to manage content streams, Zoetrope offers a practical solution to the challenge of the ephemeral web.
  • Prototyping Temporal Queries: The visual query interface and the operator-based system make it easier for developers to prototype and test new temporal interactions and applications.

Key Results

The paper demonstrates the functionality of Zoetrope through examples, such as Ed using a traffic page to view historical traffic conditions at 6 PM. The system successfully enables users to:

  • Explore past versions of entire pages or specific regions of interest.
  • Filter temporal data based on time, keywords, numerical values, and logical combinations.
  • Bind multiple lenses to correlate information from different parts of a page or different pages.
  • Visualize temporal data through timelines, movies, and cluster visualizations.
  • Export temporal data for use in external analysis tools like Google Spreadsheets.

The paper also presents findings on the stability of DOM elements, showing that while many elements change or vanish over time, visual placement of key features often remains consistent, suggesting that combining visual and textual features could improve extraction robustness.

Limitations and Future Work

  • DOM Changes and Selection Failures: While Zoetrope handles temporal data, it acknowledges that significant structural changes in webpages can still cause selection failures. Future work aims to augment DOM paths with visual and textual features to improve robustness.
  • Irregular Crawl Rates: The current semantics and assumptions are based on regular sampling. Future work will explore handling irregular crawl rates and interpolating data.
  • Scalability of Storage: The current dataset size is noted as being artificially high due to not compressing files and not eliminating duplicate content. Future work will focus on compression and deduplication.
  • More Sophisticated Queries and Binding: The paper suggests extending the semantics to support more complex temporal logic operations and functions.
  • New Visualizations and Extractors: The system is designed for extensibility, and the authors hope to inspire the development of new visualizations and extraction techniques.

Summary updated

BibTeX
@inproceedings{adar2008zoetrope,
  author = {Eytan Adar and Mira Dontcheva and James Fogarty and Daniel S. Weld},
  title = {{Zoetrope: interacting with the ephemeral web}},
  booktitle = {UIST},
  year = {2008},
  doi = {10.1145/1449715.1449756},
}
Figure