Archiving the Web: LIKE39, the Twitterverse and Everything

The Twitter Archive was the subject of a recent article from the New York Review of Books blog. In ‘Librarians of the Twitterverse’ James Gleick describes the ongoing initiative from the Library of Congress to archive the entire output of Twitter. This is being undertaken in collaboration with Gnip, a social-media data provider in Boulder, Colorado. Characterising the Twitter corpus variously as a “firehose of information”, “an ocean of ephemera” and “a library of Babel”, Gleick commented on the technical challenges of this project, comparing it to the early attempts in the 1850s of telegraph companies in the US and the UK to preserve for posterity every message that passed through their wires.

As with the British Library’s Web Archive, which formed the subject of Peter Webster’s talk at LIKE39 last November, the logistics of this exercise are jaw dropping. The long-term intention is that this perilously ephemeral medium should be made available to future generations of researchers. At LIKE39 Peter, who is the British Library’s Engagement & Liaison Officer for the Web Archive, recounted his team’s task which since 2004 has been to archive websites of cultural and scholarly importance from the UK domain, capturing 11,000 sites so far (16 terabytes worth). Both Virginia Henry and Marja Kingma have already blogged in detail about this event.

For historians, there are ongoing concerns about the “lost web” – those sites that become victims of the disorderly disappearance of organisations and campaigns, and the “orphaned web” – sites that have served their purpose, and are abandoned. It was interesting to note that one question raised was regarding the fate of social media sites, particularly in relation to major events such as the Arab Spring, where Twitter briefly became the focus for information sharing across national and political borders.

Clearly none of these ambitious projects can operate in isolation, and hopefully we will continue to hear about more web archiving initiatives – the Bodleian Library’s futurArch is but one more example of many going on in the background.

‘Archiving the Web’ was just one of last season’s excellent events from the LIKE team.  We’re looking forward to kicking off this year’s programme at the Castle, Farringdon on 31^st January with our very own Suzanne Wheatley’s Elevator Pitches. Follow #LIKE42 on the night. Then try to research it in 50 years time.

- Donald Lickley