Scaling Down Distributed Infrastructure on Wimpy Machines for Personal Web Archiving

Jimmy J. Lin

Abstract

Warcbase is an open-source platform for storing, managing, and analyzing web archives using modern "big data" infrastructure on commodity clusters---specifically, HBase for storage and Hadoop for data analytics. This paper describes an effort to scale "down" Warcbase onto a Raspberry Pi, an inexpensive single-board computer about the size of a deck of playing cards. Apart from an interesting technology demonstration, such a design presents new opportunities for personal web archiving, in enabling a low-cost, low-power, portable device that is able to continuously capture a user's web browsing history---not only the URLs of the pages that a user has visited, but the contents of those pages---and allowing the user to revisit any previously-encountered page, as it appeared at that time. Experiments show that data ingestion throughput and temporal browsing latency are adequate with existing hardware, which means that such capabilities are already feasible today.
BibTeX
@inproceedings{lin2015scaling,
  author = {Jimmy J. Lin},
  title = {{Scaling Down Distributed Infrastructure on Wimpy Machines for Personal Web Archiving}},
  booktitle = {The Web Conference},
  year = {2015},
  doi = {10.1145/2740908.2741695},
}

Loading related content...

Figure