Abstract
This paper introduces the Archives Unleashed Cloud, a web-based interface for working with web archives at scale. Current access paradigms, largely driven by the scope and scale of web archives, generally involve using the command line and writing code. This access gap means that subject-matter experts, as opposed to developers and programmers, have few options to directly work with web archives beyond the page-by-page paradigm of the Wayback Machine. Drawing on first-hand research and analysis of how scholars use web archives, we present the interface design and underpinning architecture of the Archives Unleashed Cloud. We also discuss the sustainability implications of providing a cloud-based service for researchers to analyze their collections at scale.
Introduction: Bridging the Access Gap
We are currently navigating a "digital dark age" paradox. While we possess more historical data than ever before in the form of petabytes of web crawls, this data remains largely inaccessible to non-programmers. Traditionally, researchers have been limited to two extremes: the page-by-page browsing of the Wayback Machine, which is unsuitable for "distant reading" or aggregate analysis, or high-performance computing environments requiring mastery of tools like Apache Spark or Pig.
This "access gap" effectively locks out subject-matter experts in the humanities and social sciences—the very people best equipped to interpret digital history. The Archives Unleashed Cloud (AUC) was developed to bridge this gap, providing a cloud-based "middle layer" that democratizes big data processing through a user-friendly graphical interface (GUI).
The Archives Unleashed Cloud Workflow
The AUC streamlines the transition from raw data to analysis by implementing the FEAV cycle: Filter, Extract, Aggregate, and Visualize.
Backend Architecture
The platform's heavy lifting is performed by the Archives Unleashed Toolkit, a library built on Apache Spark and Scala. To ensure accessibility, the AUC wraps this power within a Ruby on Rails web interface, abstracting the complexity of the command line.
Data Acquisition via WASAPI
To solve the logistical challenge of moving terabytes of data, the AUC utilizes the Web Archiving Systems API (WASAPI). This allows the platform to pull data directly from institutional providers, such as the Internet Archive’s Archive-It service, bypassing the need for users to upload massive files manually.
Generating Scholarly Derivatives
A core philosophy of the AUC is that researchers rarely need raw WARC (Web ARChive) files. Instead, the system generates standardized "scholarly derivatives" that can be used in familiar analytical tools:
- Plain Text: Cleaned CSV files of page content, stripped of HTML boilerplate, suitable for text mining.
- Hyperlink Networks: GEXF or GraphML files mapping domain-level connections for network analysis in tools like Gephi.
- Domain Statistics: Summaries of the most prevalent websites within a collection.
Platform Impact and Sustainability
As of June 2020, the AUC has demonstrated significant scale and utility:
- User Base: 253 active users.
- Data Volume: 912 TB of data processed across 1,366 collections.
- Ecosystem Integration: Approximately 94% of the data originates from the Archive-It ecosystem.
Beyond technical metrics, the paper addresses the critical issue of sustainability. The authors provide a rare cost analysis for digital humanities infrastructure, estimating a processing cost of approximately US$7 per terabyte in the cloud. This figure serves as a vital baseline for future grant-funded research and institutional planning.
Discussion and Future Directions
The AUC project highlights that technical barriers are often social barriers, functioning as a form of unintentional censorship in historical inquiry. By providing browser-based visualizations (via GraphPass and Sigma.js), the platform allows researchers to evaluate collections before committing to large-scale downloads.
However, challenges remain. The platform's current reliance on WASAPI creates a degree of provider lock-in, primarily favoring Archive-It users. Future work aims to expand compatibility with services like Conifer/WebRecorder. Additionally, the "Now What?" problem—where researchers are overwhelmed by even a cleaned 10GB CSV—is being addressed through "Learning Guides" and pre-configured Jupyter Notebooks.
The project is currently transitioning to a permanent home within the Internet Archive’s infrastructure, a move intended to ensure the long-term viability of these essential research tools.
Summary updated