From archive to analysis: accessing web archives at scale through a cloud-based interface

Nick Ruest, Samantha Fritz, Ryan Deschamps, Jimmy Lin, Ian MilliganInternational Journal of Digital Humanities 2021 · DOI

Abstract

This paper introduces the Archives Unleashed Cloud, a web-based interface for working with web archives at scale. Current access paradigms, largely driven by the scope and scale of web archives, generally involve using the command line and writing code. This access gap means that subject-matter experts, as opposed to developers and programmers, have few options to directly work with web archives beyond the page-by-page paradigm of the Wayback Machine. Drawing on first-hand research and analysis of how scholars use web archives, we present the interface design and underpinning architecture of the Archives Unleashed Cloud. We also discuss the sustainability implications of providing a cloud-based service for researchers to analyze their collections at scale.

Introduction: Bridging the Access Gap

We are currently navigating a "digital dark age" paradox. While we possess more historical data than ever before in the form of petabytes of web crawls, this data remains largely inaccessible to non-programmers. Traditionally, researchers have been limited to two extremes: the page-by-page browsing of the Wayback Machine, which is unsuitable for "distant reading" or aggregate analysis, or high-performance computing environments requiring mastery of tools like Apache Spark or Pig.

This "access gap" effectively locks out subject-matter experts in the humanities and social sciences—the very people best equipped to interpret digital history. The Archives Unleashed Cloud (AUC) was developed to bridge this gap, providing a cloud-based "middle layer" that democratizes big data processing through a user-friendly graphical interface (GUI).

The Archives Unleashed Cloud Workflow

The AUC streamlines the transition from raw data to analysis by implementing the FEAV cycle: Filter, Extract, Aggregate, and Visualize.

Backend Architecture

The platform's heavy lifting is performed by the Archives Unleashed Toolkit, a library built on Apache Spark and Scala. To ensure accessibility, the AUC wraps this power within a Ruby on Rails web interface, abstracting the complexity of the command line.

Data Acquisition via WASAPI

To solve the logistical challenge of moving terabytes of data, the AUC utilizes the Web Archiving Systems API (WASAPI). This allows the platform to pull data directly from institutional providers, such as the Internet Archive’s Archive-It service, bypassing the need for users to upload massive files manually.

Generating Scholarly Derivatives

A core philosophy of the AUC is that researchers rarely need raw WARC (Web ARChive) files. Instead, the system generates standardized "scholarly derivatives" that can be used in familiar analytical tools:

  1. Plain Text: Cleaned CSV files of page content, stripped of HTML boilerplate, suitable for text mining.
  2. Hyperlink Networks: GEXF or GraphML files mapping domain-level connections for network analysis in tools like Gephi.
  3. Domain Statistics: Summaries of the most prevalent websites within a collection.

Platform Impact and Sustainability

As of June 2020, the AUC has demonstrated significant scale and utility:

  • User Base: 253 active users.
  • Data Volume: 912 TB of data processed across 1,366 collections.
  • Ecosystem Integration: Approximately 94% of the data originates from the Archive-It ecosystem.

Beyond technical metrics, the paper addresses the critical issue of sustainability. The authors provide a rare cost analysis for digital humanities infrastructure, estimating a processing cost of approximately US$7 per terabyte in the cloud. This figure serves as a vital baseline for future grant-funded research and institutional planning.

Discussion and Future Directions

The AUC project highlights that technical barriers are often social barriers, functioning as a form of unintentional censorship in historical inquiry. By providing browser-based visualizations (via GraphPass and Sigma.js), the platform allows researchers to evaluate collections before committing to large-scale downloads.

However, challenges remain. The platform's current reliance on WASAPI creates a degree of provider lock-in, primarily favoring Archive-It users. Future work aims to expand compatibility with services like Conifer/WebRecorder. Additionally, the "Now What?" problem—where researchers are overwhelmed by even a cleaned 10GB CSV—is being addressed through "Learning Guides" and pre-configured Jupyter Notebooks.

The project is currently transitioning to a permanent home within the Internet Archive’s infrastructure, a move intended to ensure the long-term viability of these essential research tools.

Summary updated

Abstract

Abstract This paper introduces the Archives Unleashed Cloud, a web-based interface for working with web archives at scale. Current access paradigms, largely driven by the scope and scale of web archives, generally involve using the command line and writing code. This access gap means that subject-matter experts, as opposed to developers and programmers, have few options to directly work with web archives beyond the page-by-page paradigm of the Wayback Machine. Drawing on first-hand research and analysis of how scholars use web archives, we present the interface design and underpinning architecture of the Archives Unleashed Cloud. We also discuss the sustainability implications of providing a cloud-based service for researchers to analyze their collections at scale.
BibTeX
@article{ruest2021from,
  author = {Nick Ruest and Samantha Fritz and Ryan Deschamps and Jimmy Lin and Ian Milligan},
  title = {{From archive to analysis: accessing web archives at scale through a cloud-based interface}},
  journal = {International Journal of Digital Humanities},
  year = {2021},
  doi = {10.1007/s42803-020-00029-6},
}

Loading related content...

Figure