Twitter history lost forever?

Lately, I’ve been trying to find a record of a Twitter conversation which I know occurred in March 2009. This has proven to be a much more difficult task than you’d think. Twitter’s standard search functionality can only look back about two weeks, so it’s pretty much useless for any historical research. Taking the tedious approach and paging backwards through my timeline on Twitter’s web site only gets me back to August.

I turned to Google for assistance and I found several site which offer Twitter backup services. Basically, they download all of your tweets as HTML, CSV, XML, etc and let you store that on your hard drive. “Fantastic!” I thought. “I’ll just download my entire history and then search it with a standard text editor.”

Nope.

Apparently, Twitter only allows such services to grab a maximum of 3,200 tweets at once. For a frequent twitterer like me, 3,200 tweets only takes me back until early summer, still far short of March.

So does this mean that most of my twittering is gone, lost forever in the shifting digital dunes of the Internet? It has to be stored somewhere in Twitter’s massive database, right? I assume every tweet ever sent is in there somewhere. But how the hell do I get to mine?

This leads to an even more troubling question: what happens to our digital content when it gets too old? We’re constantly generating content which is entrusted to providers in the cloud: tweets, Facebook status updates, emails, blog entries, photos and more. Some of it we’re pretty careful with. I know where the originals of all my photos are stored. If Flickr died tomorrow, all of my pictures would still be safe.

Some of our content is in the cloud, but we have more control over it. My emails are all entrusted to Google, but I know I can get to any email I’ve saved. My blog posts are all stored on my hosting provider’s server, but I know how to find them if I need to, and I know they’re not going to be lost anywhere.

But transient content like Facebook updates and tweets seem to just disappear when they’re no longer relevant. But sometimes old tweets are relevant. Twitter, take a hint (or maybe some help) from Google and let me search ALL of my tweets when I want to.