Get a copy of the Web

[dlevere]

By Cory Doctorow

Want 80 terabytes of web-crawl? The Internet Archive will give you a copy of (an appreciable slice of) the Web, for research purposes: "we would like to experiment with offering access to one of our crawls from 2011 with about 80 terabytes of WARC files containing captures of about 2.7 billion URI's. The files contain text content and any media that we were able to capture, including images, flash, videos, etc."

[bungholio]

And how are people supposed to download 80TB worth files exactly? I've never seen a computer download faster than 10MB a second, so this would take nearly 100 days. Throwing that idea out the window, who even has the storage capacity for that much? I see external HD's over 1TB are more than $100 each.

[Pyriel]

Someone with a server farm. By the sound of it, they're not offering to give the files to anybody who asks. Presumably they're only going to entertain requests from people who can establish that they have a legitimate use for however much of the data they ask for. Your friend Harold's request to pull the files to the old PC under all the pizza boxes, so he can look for porn he might have missed, will probably go unanswered.