Monday, May 12, 2008

Decentralizing the Web

I have tried to keep this blog free of politics and opinion and centered on implementation and algorithms, but lately, certain things have bugged me, so for the first time, I'm climbing aboard the soapbox, and these opinions are not those of my company.

Isn't the web fundamentally decentralized?

The old days

I've been on the internet/usenet/netnews since the mid 80s. In the early days, my communications mostly occurred in news readers, email clients, Unix talk clients, later MUDs and IRC. In those days, the net was ironically, more federated and decentralized. Servers and Clients were mostly heterogenous flavors of Unix, which practically demanded that most communication server/client code were distributed as source. Servers were typically run from college campuses on modest machines, so the architectures tended to be federated. The lack of a single, uber GUI client, tended to drive people to invent specialized protocols for each application, and the poor portability of clients coupled with eager college students tended to produce multiple server/client implementations. There was no single news server, no single email host, no single chat server. Granted, big sites did exist (UUNET), but there was no virtual monopoly of one server being the sole provider of channel services.

Maybe it's nostalgia, but in the pre-web days, the spirit of the early IETF, it seems to me that engineers more commonly architected for the federated case, architectures were more open for inspection, and people cooperated openly.

The web

The browser ushered in a whole new way of designing applications, with some interesting repercussions. In the pre-web days, if I wanted to create a new messaging platform, I'd have to create a protocol spec, write the server and client, and get tons of people to port it and run servers. Something like RSS wasn't as important either because the protocols already defined machine readable ways to syndicate content.

The browser changed all that. Since the client portion was now handled by a third party, and ported to every piece of hardware known to man, and since it provided a very flexible way of displaying custom user interfaces, focus shifted to building big centralized servers which blossomed when coupled with declining hardware costs, and increased network bandwidth.

With the web, new "channels" sprang up everywhere, AOL/Yahoo/MSN/QQ/ICQ/etc and even today, there's Twitter, there's your Facebook wall and inbox, and tons more. What's disturbing is that these communications channels are often proprietary, and have a single point of failure. If Facebook or Twitter goes down, you're screwed. If my SMTP/IMAP server goes down, I'm screwed, but 50 million other people aren't.

Oh, this is about Twitter crashing again

Yes and no. I was irked into posting this while reading a bunch of blogs where authors simultaneously defend Twitter's downtime, insist that it is virtually a national infrastructure in importance, but also asserted that it should not be decentralized.


First of all, I think it is disgraceful, that in 2008, instant messaging is still balkanized. The IETF standardized XMPP a long time ago and implementations have been proving their worth for a long time. Yet, have AOL, MSFT, Y!, et al adopted it? No, because they don't want to relinquish control. Remember all that talk of a truce in the late 90s? It never panned out.

Blogging and RSS are decentralized, there is no single, universal, host of blogs. Imagine that you could not read, nor write blogs without having a blogger account! So why should microblogging be centralized? The arguments as to why you cannot federate Twitter are pretty weak IMHO. Communication channels are like roads, they are infrastructure, and fundamentally affect the information economy, and it doesn't make sense to me that such fundamental services be centralized behind a single point of failure nor closed in implementation. Besides, federating Twitter would probably produce new business opportunities for aggregators.

Not just about Twitter

If social networks are as important as everyone makes them out to be, why should I have to log into MySpace or Facebook to access them? Why must MySpace or Facebook applications only run on those sites? Shouldn't I be able to access my social network anywhere, on any site, in any application, web or otherwise? On the desktop, I can access my Address Book or corporate LDAP server in many applications, but I can't do that on my own site, without becoming a Facebook application.

Isn't it time to federate social networks?

The OpenID, OAuth, OpenSocial, and Data Portability initiatives are going a long way to address the architecture needed to do this. FOAF/XFN provide possible ways for discovery of social network, but there are pieces missing to the puzzle that need to be addressed before the user experience can approach that of centralized social networks, or give developers easy access to building applications that can access federated data. Google has been doing a good job thus far, but perhaps the IETF and W3C should create working groups to study the issue as well.

Peer to Peer Social Networking

One interesting possibility is using something like Gears to store and replicate your social network. That is, take my social network offline. When I add you as a friend, I could store that information locally on my machine, as well as optionally broadcasting it to several public services which also record the information. Third party websites could embed social applications which use Javascript snippets to ask my permission to run queries against my Gears database, or use OAuth to ask for the data from public services I replicated to. This could be more than a simple contacts list, since authenticated peer-to-peer exchanges (you share your offline social network with me), or social graph crawl/share services, would enable more client side aggregation.

Social networks typically provide enhanced services like profile pages, server-side 'shared' data storage for applications, and activity streams. Designing a federated system for these is an interesting exercise. Federated publish/subscribe message queues are not without precedent, nor is distributed storage. I'm not saying it's a solved problem, but shouldn't we at least try?

In any case, I'd still like to partially solve the problem by taking my social network offline, and enabling social applications which need to probe my graph to do so.

I'm tired of having to log into FB every day, or import my friends into each new site I sign up with, just as surely as I am tired of creating new username/password accounts on every site for login.

Let's return to the good old days and get away from single points of failure, cathedrals, and make the social web, truly a fundament piece of internet architecture, like TCP/IP, DNS, HTTP, and SMTP, and less about Bubble 2.0 startups.

(It's late, I'll probably regret this rant when I read it tommorow). I now return you to your regularly scheduled engineering related posts.


glongman said...

Right on. I want my social profile to be MINE. Indeed I would like to have a web app, call it a Gizmo for lack of a better term, that serves just ME, that is:

My Open ID provider
My Jabber server
My email server
My Blog
My social profile
My OpenSocial container

and that links through all those lovely open api's to the Gizmos for my friends to provide Twtiiter and FriendFeed like functionality.

I want it to be really easy to set this up. In fact I would like my Gizmo to be a container for all of these things and I would like it to be as easy to set up as running an installer on the desktop.

I am a geek. My brother is not. My brother should be able to set up his personal Gizmo as easily as double clicking an installer. Take the desktop experience to the web. No need to know what the words 'apache', 'sql', 'unix' or 'rpm' are to use this thing.

Services like GAE make this more than a mental exercise I think.

As an added benefit I suspect that building 1 person versions of many apps, like a jabber server or a Flickr like server would be much simpler. No need to worry about scaling out to a billion users - this app only serves one.

What would be really nice would be an affordable (like pay low, pay once) SSL certificate that does not identify a traditional multi user server but rather identifies me and my Gizmo. AFAIK no such certificate exists.

Jamie said...

But how would companies like Twitter remain so extremely profitable if they were to be replaced with hippie open standards?

Paying developers for their work without lawyers, venture capitalists, and stockbrokers in the loop would destroy our economy. We need the submersible patents and IPOs to provide jobs for the rest of the working world.

Unknown said...




色情A片,A片下載,色情遊戲,色情影片,色情聊天室,情色電影,免費視訊,免費視訊聊天,免費視訊聊天室,一葉情貼圖片區,情色視訊,免費成人影片,視訊交友,視訊聊天,言情小說,愛情小說,AV片,A漫,av dvd,情色論壇,視訊美女,AV成人網,情色文學,成人交友,成人電影,成人貼圖,成人小說,成人文章,成人圖片區,成人遊戲,愛情公寓,情色貼圖,成人論壇,色情


Anonymous said...