• Energy Research

Consists of a web crawl of unique URLs tweeted with the #YMMFire hashtag. #YMMFire collection took place from May 1-June 25, 2016. Unique URLs were extracted from the dataset, and harvested with Heritrix on August 31-September 2, 2017. This research was supported by a research grant -- 435-2015-0011 -- issued by Social Sciences and Humanities Research Council.

425,227 tweet ids for Wet'suwet'en tweets, collected with Documenting the Now's twarc. Tweets can be “rehydrated” with Documenting the Now’s twarc, or Hydrator. twarc hydrate wetsuweten-20210115-ids.txt > wetsuweten.jsonl Tweets were collected via the Standard Search API on a cron job every five days beginning on February 18, 2020. Collection is ongoing. The account that was used to collect these Tweets failed to collect Tweets for the period from Sun Jul 26 02:00:21 +0000 2020 through Fri Aug 07 20:05:54 +0000 2020.

80,264 tweet ids for Tyendinaga tweets, collected with Documenting the Now's twarc. Tweets can be “rehydrated” with Documenting the Now’s twarc, or Hydrator. twarc hydrate tyendinaga-20210115-ids.txt > tyendinaga.jsonl. Tweets were collected via the Standard Search API on a cron job every five days beginning on February 24, 2020. Collection is ongoing. The account that was used to collect these Tweets failed to collect Tweets for the period from Sun Jul 26 03:46:41 +0000 2020 through Fri Aug 07 20:33:27 +0000 2020.

The value of web archives to support scholarship in the humanities and social sciences is slowly being realized by the increasing availability of scalable tools and platforms. The cost of providing scholarly access is a critical component of developing a long-term sustainability strategy. This paper attempts to answer a straight-forward question: How much does it cost to analyze web archives in the cloud? To make this question more concrete, we examine the creation of three derivatives (extraction of collection statistics, full text, and the webgraph) that serve as the starting points of many scholarly inquiries. Our analysis shows that these typical derivatives costs around US$7 per TB using our Archives Unleashed Toolkit. We describe in detail the methodology and assumptions made to arrive at this figure. To our knowledge, we are the first to quantify the economics of scholarly access to web archives, and we believe that this information is valuable for service planning by archives, libraries, and other institutions.

AbstractThis paper introduces the Archives Unleashed Cloud, a web-based interface for working with web archives at scale. Current access paradigms, largely driven by the scope and scale of web archives, generally involve using the command line and writing code. This access gap means that subject-matter experts, as opposed to developers and programmers, have few options to directly work with web archives beyond the page-by-page paradigm of the Wayback Machine. Drawing on first-hand research and analysis of how scholars use web archives, we present the interface design and underpinning architecture of the Archives Unleashed Cloud. We also discuss the sustainability implications of providing a cloud-based service for researchers to analyze their collections at scale.