8. We have made a list of all the files we have lost, and have become accepting of the fact that we never Backed Shit Up and it was our fault.
9. We have suffered for our loss of such data wherever unavoidable, except when we happened to have Written Shit Down on the off chance as well.
10. We continue to take a filesystem inventory and when we forget to Back Some Shit Up promptly admit it.
Jeffrey Zeldman and Tantek Çelik (amongst other very smart people) had a conversation across several media and sites about owning your data.
Here’s the cliff notes / reading order. It’s totally worth sifting through it all.
Just a few of the many, many highlights.
Glenda-B!@zeldman @t Totally neurotic but big data is 3.0. So much unstructured data & I give my content to the void, but I can't query against it?Jeffrey Zeldman
Twitter needs to do a better job of storing its data. It probably needs to open source and share the data across multiple servers not connected to Twitter.com. LOCKSS: Lots Of Copies Keep Stuff Safe.Tom Sparks
Tom HenrichThis issue becomes more and more important as security tightens in the US and as networked social services become more and more important. I am hoping diaspora or a similar solution becomes viable in the near future.
If smaller organizations can run their own social network platforms and if those platforms can be linked between organisations, that would be a WIN. Groups or super individuals could control their own data and share it in the ways they find most useful.
Tantek ÇelikI backup my tweets not because I'm concerned that the historians of the future will be denied the benefits of access to them, but so that I can more easily refer back to them if there was something I wanted to look up. You noted Twitter's search has some pretty extreme limitations, and that's not acceptable to me.
Simply copying from these shared social services still leaves you vulnerable to their flakiness, poor auto-shortening of links, unscalability, downtime, maintenance, database failures, and acquisitions.
That's why I don't post to other services and copy to my site.
That's not what I'm doing.
Tantek:
I'm not copying from Twitter. I'm syndicating and copying to Twitter. As I said, Twitter is the copy.
There's a big difference. When Twitter goes down, I can keep posting, and my updates still go to the other destinations that take/share updates, e.g. Google Buzz, Identi.ca, Friendfeed, and whatever other service(s) might come along to replace them all. I'm not beholden to Twitter's stability/downtime - the copies there will appear when Twitter returns from such outages.
If your data is vulnerable to some social sharing services' whims or flakiness - you don't own your data - they do (their terms of service even says - they can do as they please with your content, with Flickr as perhaps the only exception).
Tantek’s conclusion:
Your site should be the source and hub for everything you post online. This doesn't exist yet, it's a forward looking vision, and I and others are hard at work building it. It's the future of the indie web.
Jeffrey Zeldman had a thing or two to say about selling your company or product to corporate ownership.
Though bits are forever, our medium is mortal, as all but the most naive among us know. And we accept that some of what we hold digitally dear will perish before our eyes. But it irks most especially when people or companies with more money than judgement purchase a thriving online community only to trash it when they can't figure out how to squeeze a buck out of it.
And with the possible exception of Flickr (better now than the day Yahoo bought it), I can't think of any online community or publication that has improved as a result of being purchased. Whereas we can all instantly call to mind dozens of wonderful web properties that died or crawled up their own asses as a direct result of new corporate ownership.
And finally, a bit of advice.
Stop selling your stuff to corporate jerks. It never works. They always wreck what you've spent years making.
What's the problem?
The main problem is that LifeHacker URLs now don't map to actual content. Well, every URL references the LifeHacker homepage. If you are lucky enough to have the JavaScript running successfully, the homepage then triggers off several Ajax requests to render the page, hopefully with the desired content showing up at some point.
Far more complicated than a simple URL, far more error prone, and far brittler.
Ben Ward pulls no punches about the "#!".
Let's get something very clear: Hash-bang URLs are shit. They're ugly, brittle and a furious hack in the absence of anything else.
Tim Bray (who you might remember as one of the inventors of XML and ATOM) doesn't like the proliferation of hashbangs (#!), either.
There is no piece of dynamic AJAXy magic that requires beating the Web to a bloody pulp with a sharp-edged hashbang. Please stop doing it.
The truth is that if site content doesn't load through curl it's broken.
Please use real URLs. That hashbang nonsense is not good for any of us.
Jeremie Miller writes about The Locker Project's Core Values.
- I own my personal data
- I want my data to be useful to me
- I make the decisions to protect or share my data
Jon Udell of The Elmcity Calendar Curation Project on how to think like the web.
- Be the authoritative source for your own data
- Pass by reference not by value
- Know the difference between structured and unstructured data
- Create and adopt disciplined naming conventions
- Push your data to the widest appropriate scope
- Participate in pub/sub networks as both a publisher and a subscriber
- Reuse components and services
Kyle Neath chimes in on how to design great URLs.
URLs are universal. They work in Firefox, Chrome, Safari, Internet Explorer, cURL, wget, your iPhone, Android and even written down on sticky notes. They are the one universal syntax of the web. Don't take that for granted.
URLs are for humans. Design them for humans.
A URL is an agreement to serve something from a predictable location for as long as possible. Once your first visitor hits a URL you’ve implicitly entered into an agreement that if they bookmark the page or hit refresh, they’ll see the same thing.
Don't change your URLs after they've been publicly launched. If you absolutely must change your URLs, add redirects — it's not that scary.
Spot on.
In wake of the Delicious ”sunsetting” and subsequent hub-bub around the web, Jeremy Keith decided to homestead his bookmarks.
Now I’m hosting the canonical copies of my bookmarks, much like Tantek hosts the canonical copies of his tweets and syndicates them out to Twitter. Delicious gets to have my links as well, and I get to use Delicious as a tool for interacting with my data ...only now I’m not limited to just what Delicious can offer me.
He actually first wrote about this when Magnolia screwed the pooch and lost everyone’s data. He nails the homesteading vision in one sentence.
Really, I should be keeping my links here on adactio.com, maybe pinging Delicious or some other social bookmarking site as a back-up.
Freedom and self-sufficiency always seem to go hand in hand. But specialization is for insects, right? Everyone should know how to make a website and deliver a baby. (I believe that’s how the expression goes.)
I definitely prefer this self-hosting-with-syndication way of doing things. I can use a service like Delicious without worrying about it going tits-up and taking all my data with it. The real challenge is going to be figuring out a way of applying that model to Twitter and Flickr. I’m curious to see which milestone I’ll hit first: 10,000 tweets or 10,000 photos. Either way, that’s a lot of my content on somebody else’s servers.
I’ve been using Delicious since 2005—back when it was del.icio.us. I have over 2,000 bookmarks stored there. I moved to Magnolia for a while but we all know how that ended.
Back then I wrote:
Really, I should be keeping my links here on adactio.com, maybe pinging Delicious or some other social bookmarking site as a back-up.
Recently Delicious updated its bookmarklet-conjured interface, not for the better. I thought that I could get used to the changes, but I found them getting more annoying over time. Once again, I began to toy with the idea of self-hosting my bookmarks. I even exported all my data into a big XML file.
The very next day, some of Yahoo’s shit hit the web’s fan. Delicious, it was revealed, was to be sunsetted.
As someone who doesn’t randomly choose to use meteorological phenomena as verbs, I didn’t know what that meant, but it didn’t sound good.
As the twittersphere erupted in anger and indignation, I was able to share my recently-acquired knowledge:
curl https://{your username}:{your password}@api.del.icio.us/v1/posts/all to get an XML file of your Delicious bookmarks.
A lot of people immediately migrated to Pinboard, which looks like an excellent service (and happens to be the work of Maciej Ceglowski, one of the best bloggers ever to put pixels to screen).
After all that, it turns out that “sunsetting” doesn’t mean “shooting in the head”, it means something more like “flogging off”, as clarified on the Delicious blog. But the damage had been done and, anyway, I had already made up my mind to bring my bookmarks in-house, so I began a fun weekend of hacking.
Setting up a new section of the site for links and importing my Delicious bookmarks was pretty straightforward. Creating a bookmarklet was pretty easy too—I already some experience of that with Huffduffer.
So now I’ll do my bookmarking right here on my own site. All’s well that ends well, right?
Well, not quite. Dom sounded a note of concern:
sigh. There goes the one thing I actually used delicious for, the social network. :(
Paul also pointed to the social aspect as the reason why he’s sticking with Delicious:
Personally, while I’ve always valued the site for its ability to store stuff, what’s always made Delicious most useful to me is its network pages in general, and mine in particular.
But it’s possible to have your Delicious cake and eat it at home. The Delicious API makes it quite easy to post links so I’ve added that into my own bookmarking code. Whenever I post a link here, it will also show up on my Delicious account. If you’re subscribed to my Delicious links, you should notice no change whatsoever.
This is exactly what Steven Pemberton was talking about when I liveblogged his XTech talk two years ago. Another Stephen, the good Mr. Hay, summed up the absurdity of the usual situation:
For a while we’ve posted our data all over the internet on all types of services. These services provide APIs so we can access the data we put into them, so that we can do things with that data. Read that again.
Now I’m hosting the canonical copies of my bookmarks, much like Tantek hosts the canonical copies of his tweets and syndicates them out to Twitter. Delicious gets to have my links as well, and I get to use Delicious as a tool for interacting with my data …only now I’m not limited to just what Delicious can offer me.
Once I had my new links section up and running, I started playing around with the Embedly API (I recently added the excellent oEmbed format to Huffduffer and I was impressed with its power). Whenever I bookmark a page with oEmbed support, I can pull content directly into my site. Take a look at the links I’ve tagged with “sci-fi” to see some examples of embedded Vimeo and Flickr content.
I definitely prefer this self-hosting-with-syndication way of doing things. I can use a service like Delicious without worrying about it going tits-up and taking all my data with it. The real challenge is going to be figuring out a way of applying that model to Twitter and Flickr. I’m curious to see which milestone I’ll hit first: 10,000 tweets or 10,000 photos. Either way, that’s a lot of my content on somebody else’s servers.
Unlike the real world, if we don’t like it, we can kill it
But there is on thing that we should always remember. At the end of the day we, the users, control the success of Facebook just like every other brand or product in the world. Facebook is making money because we are allowing it to become every marketers dream, it is the code from the Matrix wrapped around everything we post, share, comment, or connect with. If tomorrow we stop showing up that valuation would go south very quickly. It happened to Friendster, it happened to Bebo, it happened to MySpace, and it will and should happen to Facebook. Users have expressed concern about the privacy policy and the ownership of content uploaded to their network.
There was a time when “having a website” meant you owned a website that you could do anything you wanted with. Any kind of content. Any kind of structure. Any kind of software. You were truly the Master of Your Domain. But in all fairness, it was sometimes hard to be that (web)master.
If you just wanted to put pictures of your cat on the internet, but didn’t know anything about HTML and FTP, let alone chmod and unix, you were in for a world of hurt.
And then came Blogger (amongst other things).
It was like Twitter without the 140 character ceiling. Just type stuff into the box and press the button. That was it! You just published stuff on the Internet. There weren’t even post titles in the beginning. After that we would see a flood of hosted web services that enabled people to publish stuff on the Internet very easily.
Before we knew it, all of our content was being hosted by these web services. Flickr, Picasa and Photobucket had our photos. Typepad, Wordpress, Blogspot (and a slew of others) had our long form writing… called “blog posts”. Delicious and Magnolia had our bookmarks and read later lists. YouTube, Vimeo and heaps of video sites had our movies. Slideshare had our presentations.
It all seemed like a good idea. Let someone else worry about uptime, backups, redundancy, bandwidth bills, etc. And for a time, things were good.
Then as we published all of our content on other services, we became dependent on them. We became digital sharecroppers. Which maybe wasn’t so bad. But then… Magnolia lost all of its data. Six Apart bought Pownce and closed down the site providing no export option — or even much warning. URL shorteners cropped up, got popular and went away in the shortest of time, taking all of their short to long URL mappings with them.
And of course, there’s Geocities. With all of its neon colors, tiled background, sparkly text and animated gifs, Geocities was a ghetto. But it was a huge ghetto. And now that Yahoo turned it off, it’s gone. Imagine if every ghetto, barrio, favela and shanty town was literally taken away in one moment. That’s a lot of very homeless people (even more homeless than before).
It’s time for something better. It’s time for a web where any person can easily create a website and publish all kinds of content there. It’s time for us to own all of our data, beholden to no one. It’s time that our personal diy rolled websites play nice and integrate closely into external services. It’s time for a real sense of privacy, where not only is our data “protected” from other seeing it, it’s also encrypted at the source so that even if seized by criminal or government alike it’d do them no good. It’s time for easy granular sharing controls allowing to grant access to some content to some people, not all content to all people or to no one.
There will come a time in the not too distant future where having a website will be considered a birthright. It’s time that we start building the tools that will make that a possibility.
Instead of sharecroppers, we must become homesteaders.
There’s surely stuff that I’ve thought about but am not thinking of right now. I’ll write more as it comes to me. It’s also worth noting that while I had a lot of these thoughts independent of talking with others, it turns out that more people are thinking roughly the same stuff. Discussions with Tantek really helped my thoughts coalesce, especially the personal url shortener work that he’s done. He’s using ttk.me with a one letter namespace, 3 character base60 number of days since epoch and one digit nth item of that type on that day. I am too. I jacked that all from him and ported his JavaScript / PHP version to Ruby. Thanks, Tantek. I’ve also talked a fair bit with Brian Ford and Rich Kilmer about all this stuff. Both had the idea of bundling the software package up into a VM instance that one could just throw at some server and hit the ground running. I hadn’t thought of that before. Thanks for that, you two.
Let’s get together and make this thing.