GwtQuery 0.2 released

GwtQuery 0.2 alpha is now available in the repository, both as a jar file, and as a Maven project. Nothing major has changed since the last release except to relayout the directory structure for maven.

Download it at gwtquery.com and have fun!

See the samples/ directory for some example code of how to setup a GWT module to use gwtquery.jar.

-Ray

GwtQuery 0.1 alpha released

I just finished my presentation at Google I/O. It was a real blast, and I met a lot of really cool, smart people.

To celebrate the occassion, I am releasing an alpha of my jQuery-API-clone-in-GWT today. Visit gwtquery.com to download the source. It's a partial implementation at the moment, no deployable, or build script. Expect that to change in the near future. I welcome patches/contributions to flesh out the API and add the missing functionality.

-Ray

Decentralizing the Web

I have tried to keep this blog free of politics and opinion and centered on implementation and algorithms, but lately, certain things have bugged me, so for the first time, I'm climbing aboard the soapbox, and these opinions are not those of my company.

Isn't the web fundamentally decentralized?

The old days

I've been on the internet/usenet/netnews since the mid 80s. In the early days, my communications mostly occurred in news readers, email clients, Unix talk clients, later MUDs and IRC. In those days, the net was ironically, more federated and decentralized. Servers and Clients were mostly heterogenous flavors of Unix, which practically demanded that most communication server/client code were distributed as source. Servers were typically run from college campuses on modest machines, so the architectures tended to be federated. The lack of a single, uber GUI client, tended to drive people to invent specialized protocols for each application, and the poor portability of clients coupled with eager college students tended to produce multiple server/client implementations. There was no single news server, no single email host, no single chat server. Granted, big sites did exist (UUNET), but there was no virtual monopoly of one server being the sole provider of channel services.

Maybe it's nostalgia, but in the pre-web days, the spirit of the early IETF, it seems to me that engineers more commonly architected for the federated case, architectures were more open for inspection, and people cooperated openly.

The web

The browser ushered in a whole new way of designing applications, with some interesting repercussions. In the pre-web days, if I wanted to create a new messaging platform, I'd have to create a protocol spec, write the server and client, and get tons of people to port it and run servers. Something like RSS wasn't as important either because the protocols already defined machine readable ways to syndicate content.

The browser changed all that. Since the client portion was now handled by a third party, and ported to every piece of hardware known to man, and since it provided a very flexible way of displaying custom user interfaces, focus shifted to building big centralized servers which blossomed when coupled with declining hardware costs, and increased network bandwidth.

With the web, new "channels" sprang up everywhere, AOL/Yahoo/MSN/QQ/ICQ/etc and even today, there's Twitter, there's your Facebook wall and inbox, and tons more. What's disturbing is that these communications channels are often proprietary, and have a single point of failure. If Facebook or Twitter goes down, you're screwed. If my SMTP/IMAP server goes down, I'm screwed, but 50 million other people aren't.

Oh, this is about Twitter crashing again

Yes and no. I was irked into posting this while reading a bunch of blogs where authors simultaneously defend Twitter's downtime, insist that it is virtually a national infrastructure in importance, but also asserted that it should not be decentralized.

HUH?

First of all, I think it is disgraceful, that in 2008, instant messaging is still balkanized. The IETF standardized XMPP a long time ago and implementations have been proving their worth for a long time. Yet, have AOL, MSFT, Y!, et al adopted it? No, because they don't want to relinquish control. Remember all that talk of a truce in the late 90s? It never panned out.

Blogging and RSS are decentralized, there is no single, universal, host of blogs. Imagine that you could not read, nor write blogs without having a blogger account! So why should microblogging be centralized? The arguments as to why you cannot federate Twitter are pretty weak IMHO. Communication channels are like roads, they are infrastructure, and fundamentally affect the information economy, and it doesn't make sense to me that such fundamental services be centralized behind a single point of failure nor closed in implementation. Besides, federating Twitter would probably produce new business opportunities for aggregators.

Not just about Twitter

If social networks are as important as everyone makes them out to be, why should I have to log into MySpace or Facebook to access them? Why must MySpace or Facebook applications only run on those sites? Shouldn't I be able to access my social network anywhere, on any site, in any application, web or otherwise? On the desktop, I can access my Address Book or corporate LDAP server in many applications, but I can't do that on my own site, without becoming a Facebook application.

Isn't it time to federate social networks?

The OpenID, OAuth, OpenSocial, and Data Portability initiatives are going a long way to address the architecture needed to do this. FOAF/XFN provide possible ways for discovery of social network, but there are pieces missing to the puzzle that need to be addressed before the user experience can approach that of centralized social networks, or give developers easy access to building applications that can access federated data. Google has been doing a good job thus far, but perhaps the IETF and W3C should create working groups to study the issue as well.

Peer to Peer Social Networking

One interesting possibility is using something like Gears to store and replicate your social network. That is, take my social network offline. When I add you as a friend, I could store that information locally on my machine, as well as optionally broadcasting it to several public services which also record the information. Third party websites could embed social applications which use Javascript snippets to ask my permission to run queries against my Gears database, or use OAuth to ask for the data from public services I replicated to. This could be more than a simple contacts list, since authenticated peer-to-peer exchanges (you share your offline social network with me), or social graph crawl/share services, would enable more client side aggregation.

Social networks typically provide enhanced services like profile pages, server-side 'shared' data storage for applications, and activity streams. Designing a federated system for these is an interesting exercise. Federated publish/subscribe message queues are not without precedent, nor is distributed storage. I'm not saying it's a solved problem, but shouldn't we at least try?

In any case, I'd still like to partially solve the problem by taking my social network offline, and enabling social applications which need to probe my graph to do so.

I'm tired of having to log into FB every day, or import my friends into each new site I sign up with, just as surely as I am tired of creating new username/password accounts on every site for login.

Let's return to the good old days and get away from single points of failure, cathedrals, and make the social web, truly a fundament piece of internet architecture, like TCP/IP, DNS, HTTP, and SMTP, and less about Bubble 2.0 startups.

-Ray
(It's late, I'll probably regret this rant when I read it tommorow). I now return you to your regularly scheduled engineering related posts.

Google I/O and GWT Extreme!

I've been invited to do a session at the upcoming Google I/O conference. The initial idea was to do a presentation on Syndroid, but I felt that was too specific and not generally useful to the audience. What would be useful, would be to cover general purpose techniques in GWT 1.5 to bridge Javascript execution environments as well as high performance tricks. Thus was born: GWT Extreme!.

For those attending Google I/O and want to see this session, I recommend attending Bruce Johnson's presentation on Deferred Binding and Bob Vawter's presentation on Linkers first. In this session, I will go over the following techniques:

• GQuery and compiling CSS selectors at compile time


• The Android Native Interface, packaging GWT applications as Android apps and automatically making calls to native Android Java from within Javascript.


• The ActionScript Native Interface, compiling GWT code to SWF, generating RPC stubs, and making calls between browser GWT and Flash GWT code


• Writing Gears workers in GWT


• If there's time, extreme graphics performance with GWT

As the session is only one hour long, I will not be going in-depth with source examples like I did in my Deferred Binding Presentation, but rather, I will describe the overall architectures and steps needed to achieve them in general terms. I may follow up with more in-depth blog posts later containing source code.

Finally, I am planning to release GQuery (or GWTQuery) at Google I/O, there will be a usable beta-quality build on the project code hosting site.

-Ray

A screenshot of Syndroid

Edit: Ha! I just noticed I spelled Dalvik wrong! There goes my prize!

Work on GQuery progresses, Comment on Slickspeed

GQuery is progressing nicely. I implemented all of CSS3 selectors by following ExtJS's DomQuery implementation, only I added the ability to parse at runtime as well as via a generator at compile time. The compile time generator turns a CSS selector into 100% inlined code. That is, a selector like "#foo", will turn into "return document.getElementById("foo");", no parsing step involved. I've still got a bunch of optimizations to make, add support for XPath and native getElementsByClassName, but even now working with the library in GWT is very cool. I just started looking at DomAssistant as well, to incorporate (i.e. steal) the best algorithms from each.

I've been using SlickSpeed to evaluate compliance of my implementation, as well as check performance, but I noticed an issue with SlickSpeed's benchmarking which is endemic to a lot of microbenchmarks: failure to account for timer resolution.

When you run SlickSpeed on some browsers/operating systems, alot of the results will be 0ms, 4ms, 8ms, or some other multiple, because the Javascript Date.getTime() function has a fixed update resolution on some system, for example, on some systems it seems to be around 16ms or roughly 1 screen update (60Hz = 1/60th second). SlickSpeed runs each selector 4 times, and then divides by 4, which is why you see results like 4ms, 8ms, and the like.

The problem is, there is always the issue of timer jitter and aliasing. For example, if you happen to sample the timer twice before the next update happens, you end up with a 0ms result, even if it took 15ms, whereas if you get unlucky and start the benchmark half way into an update interval, you may record a result rounded up to the next. I need to hack SlickSpeed anyway to allow it to bench GWT, and in the process I'm going to run each selector such that the time exceeds 500ms, plus toss out the min/max values to evaluate the effectiveness of caching on some of those APIs. This will disadvantage my compile time selectors, which avoid a parsing step, but be more fair to comparing the actual implementation of the selector routines. I might make a separate benchmark to time 'setup' speed that these libraries use for parsing the selector, since that is obviously a factor in many end user apps that don't neccessarily run the same query more than once.

-Ray

GWT: The Road to 1.5, Language features and GQuery

Hi folks,
I had to delay the second part of my article on Linkers because the APIs have been changing radically lately and have only just settled down in the trunk. However, to pass the time, I decided to demonstrate some of the huge benefits of 1.5 by implementing a type-safe JQuery clone in GWT.

That's right, for all those folks who don't want to deal with GWT Widgets, but just want to query, wrap, and manipulate plain old DOM elements in GWT, this article is for you. I'm talking to you, you self-hating Java programmer with an inner Javascript Ninja begging to be let out.

Yes this is working code

First, let's just dispense with the details and show you a working example, yes, this is working code:

public void onModuleLoad() {
    $("div").css("color", "red").click(new Function() {
      public void f(Element e) {
        Window.alert("Hello");
        $(e).as(Effects).fadeOut();
      }
    });
  }

What does this do? It makes every DIV element red and clickable, and if you click one, it pops up an alert message, and fades the DIV out.

The API is mostly faithful to jQuery's public API, with the exception of the lack of succinct closures (Hello, Neal Gafter!), but that's ok, we can introduce enough prebuilt composable functors to eliminate alot of the need for inline anonymous functions for common operations.

Small, Fast, Efficient Code, and Other Benefits

Since GWT prunes any code not being used, this example produced a obfuscated + gzipped script size of only 5459. If I change the click handler to simply $(e).setInnerHTML("Boom"), the compiled + gzipped script size is 1400 bytes.

There's a few small differences to take note which are either a benefit, or a hinderance depending on who you ask:
1. Type-Safety, IDE code completion, debuggability, etc
2. Lack of prototype modifications forced slighty change for GQuery plugin usage (see Effects above)
3. Selectors can be compiled at compile time, more on that later.

How 1.5 Enables GQuery

Now, let's talk about the features in GWT 1.5 that make GQuery nice and efficient to implement.

1) Joel Webber of the GWT Team just submitted a patch to GWT 1.5 that implements every DOM2 Core+HTML class as a subclass of the new JavaScriptObject. This not only made the coding of GQuery very easy, but it produces nicely optimized JavaScript.

2) Static Imports. The $ function would not nearly be as nice without static imports.

3) Generics and Covariant return. The Plugin mechanism utilizes this to fake prototype based overrides of the GQuery object.

How to write GQuery plugins? Damn simple

Java doesn't feature metaclasses or prototypes, so there is no way to add methods to an existing instance or class without subclassing, but in order to preserve jQuery style syntax, we can use covariant return, type capture, and other features of JDK 1.5, to fake it.

There are two ways this occurs, first, the $() method can take a final parameter which is a Plugin class literal, or, you can call the as() method on any GQuery and "cast" it to a plugin interface. Here's the implementation of the as() method:

/**
   * Convert to Plugin interface provided by Class literal.
   */
  public <T extends GQuery> T as(Class<T> plugin) {
    return (T) plugins.get(plugin).init(this);
  }

The type of the plugin, which is a subclass of GQuery is captured by the class literal parameter, the Class literal is mapped to a Plugin class factory which creates instances of the plugin, and finally, the current GQuery is passed in, so that the Plugin can access the set of matched elements.

Here is an example plugin implementation:

public class Effects extends GQuery {
  
  static {
    GQuery.registerPlugin(Effects.class, new EffectsPlugin());
  }
  public static final Class<Effects> Effects = Effects.class;
  
  public Effects(Element element) {
    super(element);
  }

  public Effects(Element[] elements) {
    super(elements);
  }

  public Effects(NodeList list) {
    super(list);
  }
  
  public Effects fadeOut() {
    Animation a = new Animation() {

      public void onCancel() {
      }

      public void onComplete() {
        for(Element e : elements) {
          e.getStyle().setProperty("opacity", "1.0");
        }
      }

      public void onStart() {
      }

      public void onUpdate(double progress) {
        for(Element e : elements) {
          e.getStyle().setProperty("opacity", String.valueOf(1.0-progress));
        }
      }
    };  
    a.run(2000);
    return this;
  }
  
  public Effects fadeIn() {
    Animation a = new Animation() {

      public void onCancel() {
      }

      public void onComplete() {
      }

      public void onStart() {
      }

      public void onUpdate(double progress) {
        for(Element e : elements) {
          e.getStyle().setProperty("opacity", String.valueOf(progress));
        }
      }
    };  
    a.run(2000);
    return this;
  }
  
  public static class EffectsPlugin implements Plugin<Effects> {
    public Effects init(GQuery gq) {
      return new Effects(gq.get());
    }
  }
}

That's it. Plugins are nothing more than subclasses of GQuery which have an extra registry step and factory.

I'll put up the source code on http://code.google.com/p/gwtquery after GWT 1.5 milestone 2 is released, which is needed to run this.

You forgot to talk about compile-time selectors

So, yeah, as Generators are a frequent topic of this blog, you really think I wouldn't find a way to sneak them in? GQuery supports two mechanisms for selector evaluation. It can compile and evaluate selectors at runtime (currently just using XPath for prototyping, which isn't available everywhere), or, you can create an interface containing selectors you know about at runtime, and have them precompiled and ready to execute.

As an example:

public interface MySelectors extends GQuery.Selectors {
  @Selector("div")
  public GQuery allDivs();

  @Selector("div.foo")
  public GQuery allFooDivsBelow(Element context);

}

Then to use:

  MySelectors s = (MySelectors)GWT.create(MySelectors.class);
  s.allFooDivsBelow(context).css("backgroundColor", "yellow");

A generator performs a browser-specific compile-time transformation of the selector in the annotation into the most optimal form that the browser supports (XPath, pure-JS, DOM traversal, document.getElementsByClass, etc) Not only does this produce smaller per-browser code, it produces faster code as well.

One last thing, the $$ function

jQuery contains a lot of methods which take essentially property/value pairs in a Javascript object literal. Emulating this with Java syntax would be too tedious (Scala frontend to GWT, please!), however, to ease this use case, I added a global $$ function for manipulation JavaScriptObject/JSON objects easily.

For example, if you want to create a JS object literal, you write:

Properties literal = $$("{ foo: 'bar', baz: 'bam'}");

You can then use this Properties literal in GQuery functions ala jQuery, or, you can access them with JS-like syntax

$("div").attr($$("{ foo: 1, bar: 2 }"); // set every DIV to have a foo attribute = 1, and bar attribute = 2

// And;
$$("{foo: 1, bar : 2}").get("foo") == 1;
$$("{foo: 1, bar: { baz: 3} }").get("bar.baz") == 3;

-Ray