GWT: The Road to 1.5, Linkers

The Google Web Toolkit compiler is a marvelous tool for turning Java code into highly optimized Javascript, but prior to GWT 1.5, there was no mechanism for developers to customize the packaging of the compiler output, or the bootstrap of the application.

C developers are used to the ability to override the entry point glue code ('start' method in crt1.o for example), or to package statically, dynamically, etc, so why not provide this capability for GWT developers as well?

This will allow GWT compiled code to packaged and bootstrapped into environments like Flash/AIR Apollo, Gears, Gadget Containers, and the example I'm going to show today: Android Offline/Online Hybrid applications.

What's a Syndroid

As part of my ongoing prototyping in Syndroid, I stumbled onto the idea of a hybrid Android application. One where some of the UI can be rendered via GWT in the embedded WebKit, and the other part, running as native Java/Dalvik code accessing Android APIs.

For example, you may wish to develop a Gadget with GWT, that runs in several containers, but when it runs on Android, it has additional access to information like Contacts, Location, etc. How can GWT code call native Android functions?

Generators Again? The Android Native Interface

Before showing you the magic sauce that permits this, let's look at an example API of how we might like this to work. This will all seem familiar to those who use GWT RPC. First, let's create an interface that will trigger a GWT Generator.

package org.timepedia.syndroid.client.android;
public interface AndroidNativeInterface {}

and a sample interface exposing a method to retrieve our GPS coordinates

package org.timepedia.syndroid.client.android;
@ImplementedBy(LocationServiceImpl.class)
public interface LocationService extends AndroidNativeInterface {
  String getLocation();
}

next, Android implementation code (*warning, code may not work, I just typed it into the blog without testing)

package org.timepedia.syndroid.android;
public class LocationServiceImpl implements LocationService {
   public Location getLocation() {
       LocationManager lm = (LocationManager)
           Context.getSystemService(Context.LOCATION_SERVICE);
       return locationManager.getCurrentLocation("gps").toString();
   }
}

finally, some GWT code that uses it

LocationService lService = (LocationService)GWT.create(LocationService.class);
Window.alert("My location is: "+lService.getLocation());

Great, but how does it work, and how do Linkers fit in?

There are a few ways that RPC calls between GWT and Android could work. For example, a traditional XHR request could be made to a HTTP service running on Android. But this would require the API to be asynchronous and have a high overhead.

It turns out, that the native WebKit that comes with Android has the capability to extend the native Javascript APIs available via Java. For example,

 WebView wv=new WebView(this);
 wv.getSettings().setJavaScriptEnabled(true);
 wv.addJavascriptInterface(new LocationServiceImpl(), "locationService");

This binds our Android LocationServiceImpl instance to a globally scoped JavaScript object called 'locationService', allowing Javascript code to invoke the locationService.getLocation() method.

We now have two problems to solve, first, generate GWT code in a Generator that invokes this $wnd.locationService.getLocation() global object, and secondly, to generate Android bootstrap code (shown above), an AndroidManifest, and package up everything into a APK file that can be installed on a phone. The latter problem is what Linkers help solve.

The Generator's Job

I won't go into detail about how to implement the Generator, as it's been covered before in this blog, the Generator essentially creates a LocationServiceImpl class, with JSNI method getLocation(), which calls $wnd.locationService.getLocation()

The Linker's Job

The Linker has the bulk of the grunt work here. Here's what it must accomplish:
1. Generate Bootstrap Android application
2. For each generated AndroidNativeInterface, add Javascript bindings
3. Place all generated resources in proper Android asset layout
4. Create Android Manifest file
5. Add extra code to load the initial WebView start page with GWT app
6. Invoke Android tools to compile Dalvik code, and package APK files.


The Linker and LinkerContext interface in GWT 1.5 allow one to specify in a module file, which sets of Linkers are run. GWT 1.5 provides some standard ones like the IFrameLinker and XSLinker, which generate the ordinary selection scripts we see in 1.4. The LinkerContext interface provides a mechanism to discover the outputs from the compiler pass, and to emit new ones.

I'll have to end this installment here, because it is getting long and I need to get back to work, in the next installment, I'll show some actual Linker code for an AndroidGadgetLinker.

GWT:The Road To 1.5, Part 1

The next version of Google Web Toolkit is almost upon us, GWT 1.5, but don't take the minor version bump as an indication of how much it's improved, GWT 1.5 has so many awesome improvements, it would be more proper to call it GWT 2.0. That's one reason why it's been almost a year since the last release.

To celebrate the release, I will be writing a series of brief articles on each of the many improvements of 1.5, hopefully with sample code demonstrations.

Zero Overhead Javascript Interop
One of the cool things about GWT has always been the JSNI concept, or 'Javascript Native Interface', that neatly parallels JNI in ordinary Java, except the 'native' implementation is Javascript.

One of the most common uses of JSNI is to produce wrappers for third party Javascript libraries like Scriptaculous, Dojo, ExtJS, etc. Unfortunately, these wrappers are somewhat expensive prior to 1.5

Common Wrap Patterns
There are two common ways people wrap third party Javascript libraries: Encapsulation of JavaScriptObject ('JSO'), or extension of JavaScriptObject. Prior to 1.5, subclassing of JSO worked, but was not officially supported.

Regardless of which technique is chosen, the wrappers work by providing access to Javascript object properties via JSNI getters and setters, and methods via JSNI methods that delegate to Javascript methods.

Unfortunately, delegation getter methods or other instance methods, were not optimized away or inlined by the GWT compiler. Let's look at an example.

public class TestWrapperEncapsulation {
    private final JavaScriptObject jso;
    public TestWrapperEncapsulation(JavaScriptObject jso) {
        this.jso = jso;
    }
    public native String getFoo() /*-{
      return this.@TestWrapperEncapsulation::jso.foo;
    }-*/;
    public static native JavaScriptObject makeObject() /*-{
      return {foo: 'Hello World'}
    }-*/;
    public static TestWrapperEncapsulation create() {
        return new TestWrapperEncapsulation(makeObject());
    }
}

Given the above wrapper class, if you were to write:

Window.alert(TestWrapperEncapsulation.create().getFoo());

You would not end up with the desired optimal JavaScriptCode, e.g.

$wnd.alert(TestWrapperEncapsulation.create().jso.foo)

rather, this

$alert(this$static, $getFoo(create()));

function $alert(this$static, foo){
 $wnd.alert(foo);
}

function $TestWrapperEncapsulation(this$static, jso){
  this$static.jso = jso;
  return this$static;
}

function $getFoo(this$static){
 return this$static.jso.foo;
}

function create(){
  return $TestWrapperEncapsulation(new TestWrapperEncapsulation(), makeObject());
}

function makeObject(){
 return {'foo':'Hello World'};
}

Now let's take a look at GWT1.5's output:

$wnd.alert($TestWrapperEncapsulation(new TestWrapperEncapsulation(), {foo:'Hello World'}).jso.foo);

function $TestWrapperEncapsulation(this$static, jso){
 this$static.jso = jso;
 return this$static;
}

That's a massive reduction, but still not good enough. There is in fact, no reason to have a wrapper encapsulate the JSO. Instead, with GWT 1.5, we can subclass the JSO and pretend that our Java methods exist on the underlying JSO.

alert(TestWrapper.makeWrapper().getFoo());

public class TestWrapper extends JavaScriptObject {
    protected TestWrapper() { }
    public final native String getFoo() /*-{
       return this.foo;
    }-*/;

    public static native TestWrapper makeWrapper() /*-{
       return { foo: "Hello World" } 
    }-*/;
}

and what does the 1.5 compiler produce?

$wnd.alert({foo:'Hello World'}.foo);

Perfect! Absolutely zero overhead.

Chronoscope 0.8 Released / Upcoming Milestones

Major new features:
1) Internet Explorer/Flash Rendering support
2) Experimental Java2D renderer (servlet/applet)
3) Appendable Datasets / Dataset change listeners

Miscellaneous:
4) many bug fixes
5) code aligned with GWT checkstyle

Internet explorer is supported by using a special Flash9 helper which emulates the WHATWG Canvas rather than using VML/ex-canvas as is traditionally done because performance and compatibility is not as good with VML. You can force flash rendering for all browsers by adding ?_force_flash to the URL of your app.

The Servlet/Applet support isn't transparent in the sense that you have to write different code to instantiate a servlet or applet graph. This will be rectified in the near future by harmonizing the chart factory to produce various types (Flash/Canvas, Servlet, Applet) depending on configuration parameters or browser detection, that is, if you lack both Flash9 and are running on IE, but have Java, then it can fall back to Java, and if you don't want Java, it can fall back to asking a servlet to assist.

Appendable datasets allow you to do streaming charts (finance, network sensors, etc) by appending new points to the end of a dataset. This are efficiently integrated into the internal multiresolution datastructure (log(N) ops), and the chart can now detect when the dataset has been mutated and automatically redraw itself.

Also, we are now committing to weekly releases, here are the planned upcoming milestones.

Next Week, 0.85 release
1) RangeMutableXYDataset - all features of Appendable datasets, plus
supports modifying Y values of already added points
2) Auto-precision/scaling of Axis labels

0.90+ releases (not prioritized yet)
1) IE GSS support (use CSS properties to style chart)
2) Shared Domain charts (two plots, sharing a single X axis,
vertically stacked, like Financial Price/Volume charts)
3) Generalized partition strategy for multiresolution algorithm
4) Restore bar chart to full functionality + control over span intervals (bar width)

Please file any bugs or RFEs in the issue tracker, which is available on the project page where you download the code at http://code.google.com/p/gwt-chronoscope

-Ray

Project Syndroid:Synthesis of GWT and Android for Platform Independent Gadgets

Over the course of the last few months, I've steadily documented my work in attempting to produce portable 'cloud-safe' code that runs anywhere, both in the browser, and in native environments, and now I'd like to take it a step further.

Project Syndroid

My vision is to produce a high level Java Gadget API, that allows the authoring of both OpenSocial and Gears enabled gadgets, that can run in a variety of containers and platforms, the biggest difference being the ability to deploy to Java based phones like Android. The presentation below goes into a little more detail of what I'm proposing and why.

| View | Upload your own

I Need the A-Team (cue music)

Project Syndroid is my submission to the Android Developer Challenge. I can't do it alone, I don't have that much time to dedicate to doing all the platforms. However, I do have a handle on doing the GWT version. What I would like to do is find 2 other expert developers, one in Android, and one in doing Gadgets/Widgets for Sidebar/Dashboard/Google Desktop/Konfabulator, and together, as the syndroid-team, we will split any prize money if we win anything in the developer challenge. I'll take on the task of hashing out the GWT plumbing, while the other two developers work on Android and Widget container code, plus packaging/build/deploy tools to build all versions.

I have set up a group, syndroid@googlegroups.com, for those that would like to discuss the project/idea further, or become one of the core syndroid-team members responsible for major development work.

-Ray

Chronoscope Demo in Flash + WHATWG Canvas on IE

Since it's inception, I've been talking about the design goal of Chronoscope as a scalable visualization platform that runs in any environment. However, until recently, one big hole in that vision remained: Internet Explorer. IE does not support the <CANVAS> element, it does however support a retained-mode/scenegraph-style markup language called VML. Many attempts have been made to emulate the canvas using VML, but they all leave something to be desired, especially when it comes to performance.

This leads naturally to thoughts of using Flash. It just so happens I am mostly finished with a Canvas implementation for IE using Flash, and here is a demo of Chronoscope using Flash. Oh, it runs on Internet Explorer now! (Edit: Flash Version 9 plugin required)

Nobody has ever done this before

"That's why it's going to work"

To be fair, many many people have tried this before. Paul Colton for example with AFLAX, and numerous others have floated the idea or prototyped it. When I started, I did not want to waste time duplicating effort, so I searched for any complete WHATWG Canvas emulations I could find for flash, but found none. And to be sure, there were problems with many of the prototypes that turned up, such as trying to map JS calls to CanvasRenderingContext2D directly to Flash calls via Flash's ExternalInterface, which needless to say would be incredibly slow.

I did find one interesting library that would turn out to help me a lot: AS Canvas by MixMedia, an ActionScript implementation of most of the WHATWG Canvas API, not for the browser, but for Flash developers. I am not a Flash developer nor expert, and I was vaguely familiar with the MovieClip API, however puzzles remained as to how to turn what is a scenegraph style API into an immediate mode one. ASCANVAS achieves this by flushing each stroke()/fill() call into a BitmapData object and then clearing the previous drawing commands. That was the inspiration I needed.

Buckle your seatbelt Dorothy, 'cause CANVAS, is going bye-bye.

Well, not exactly. Rather, Flash will become an option for rendering in Chronoscope.

Chronoscope's CANVAS API was designed to help accelerate performance in drawing when individual drawing commands have a high overhead (such as making RPC calls to a Flash VM or doing lots of DOM operations). The way it achieves this is by super-setting the WHATWG Canvas API with several additional features:

• All drawing happens between beginFrame() and endFrame() calls
• Multiple layers can be created within a Canvas and composited
• OpenGL-style display lists which record a sequence of commands and play them back
• Text rendering and specialized text-layers
• Rotated text
• Fast Clear

The beginFrame()/endFrame() abstraction allows the canvas to defer execution and buffer up multiple commands into a single batch. For Flash, this allows an entire frame of drawing commands to be sent to Actionscript for rasterization in a single call. For DOM-oriented interfaces (SVG/VML/Silverlight/etc), it would allow using innerHTML techniques over DOM operations to render the frame.

The layers API yields the possibility of accelerated hit-detection, since layer creation is cheap, while allowing efficient compositing operations that are non-destructive, which allows fast scrolling and the ability to only update layers which change. It also makes the Java2D version work better.

The Display List abstraction allows one to record a bunch of API calls, and play them back over and over, yielding a more compact buffer of commands, as well as the potential to accelerate parsing of the commands, and cache the results of some drawing operations. For example, a display list with 100 drawing operations, could be executed 100 times for a total of 10,000 drawing operations, while the command buffer in the Canvas only stores and transmits a total of 200 commands.

Text rendering is my biggest complaint about WHATWG Canvas, so it was a no-brainer to include it. Finally, Fast Clear is useful for erasing an entire canvas and/or layer if there is a cheaper way of doing it then clearRect(0,0,width,height) for example.

Unfortunately, no one can be told what VML is, you have to experience it for yourself

And once you have, you'll wish you hadn't. Flash/Actionscript3 with MXMLC however turned out to be a pleasure (mostly).

Here's how the Flash Canvas works. Canvas calls are converted into tokenized commands and pushed into a Javascript array, for example lineTo(10,20) becomes array.push('l', 10, 20) beginFrame() clears this array, and endFrame() uses Array.join() to send the entire stream to the Flash Plugin, which as exported an interface. The Flash code parses this Array, and translates Canvas API calls into semantically equivalent Flash operations.

Needless to say, getting all of the WHATWG Canvas semantics correct is tricky. For example, 'globalCompositeOperation' is tricky to implement because Flash lacks Porter-Duff compositing modes. Path drawing and filling, especially with curves, has subtle differences. And drawing Images from the browser requires a lot of bookkeeping.

My current implementation is about 95% complete. I have a few Porter-Duff modes to implement and CanvasPattern. After completion, I'm looking forward to exporting a pure-JS (non-GWT) version of this that can replace excanvas/iecanvas for high performance (and correct) WHATWG rendering in IE6+

Overall, I'm happy performance seems adequate for my uses. Again, here's a demo of Chronoscope using Flash (Flash Version 9 required)

Timepedia now has a graphics/charting platform thats runs in the browser as Javascript (WHATWG Canvas and Flash), as a Applet or Desktop Java (Java2D version), on the Server (Java2D), and in mobile environments (Android, later J2ME). As a future exploration, I'm looking at modifying the GWT compiler to produce ActionScript and translate the entire Chronoscope codebase into an SWF.

-Ray

Chronoscope in Flash soon

Just a quick Chronoscope status update. I've been working the last few days on a Canvas implementation based on using an embedded Flash component. The primary goal of this is to support Internet Explorer which lacks browser canvas support, but it can be used on any browser. Initial performance is on par or superior to native Canvas (and much much better than SVG/VML) I'll be releasing it in the next 1 or 2 weeks as I clean up some glitches in the rendering.

IE Canvas rendering finally solved (and performant!)
-Ray

Google Gears Image Manipulation API not ambitious enough

Google Gears is a very subversive and disruptive technology IMHO, and I mean that in a good way. Google has the muscle to extend native browser functionality in a cross browser way by sneaking extensions into Gears. Of course, Adobe and Microsoft can do this as well (Flash and Silverlight), but the difference is, Google is offering AJAX-level building block extensions to browser functionality, not an alternative environment-within-the-browser-environment that Flash, Silverlight, and Java applets yielded.

Gears is steadily building momentum, and I'm sure many of Google's properties will soon support offline or enhanced modes using it, which will tend to make it a 'must-have' plugin, hopefully achieving 80-90% penetration in the future. So, before the vast majority of people start using the plugin, let's try to be as ambitious with the extension functionality as we can, so when the rush-to-install happens, people will be getting a version with very rich functionality as the base. We want to avoid the need to check Gears version all over the place and force people to upgrade plugins continually ("what, oh, your Gears version 1.21 doesn't have the image.shear() function, you need Gears 1.25 for that...")

Case in point, the proposed Gears Image Manipulation API. Granted, it's just starting, but I'd like to offer some upfront suggestions before this thing gets finalized.

Don't duplicate Canvas, extend it

The proposed API adds resize() and crop() operations to an image object. And also the ability to turn images back into blobs. The resize() and crop() operations can be done today with JS Canvas, but only WHATWG Canvas allows you to turn an image back into data. I don't think this API goes anywhere near far enough to justify its existence.

Also, the biggest pain with using Canvas today is that every browser but IE supports it, so why not implement a cross-browser offscreen Gears WHATWG Canvas API to start with.

But don't stop there. WHATWG Canvas lacks text rendering, and image rendering that obeys affine transforms, two of the big complaints against the existing Canvas. The Web 2.0 world will love anyone who can get such an extended cross-browser canvas widely deployed.

So start with an off-screen WHATWG Canvas API, add text rendering (and atleast 90 degree rotated text would be nice), plus drawImageWithTransform() that obeys transforms.

Resize, Flip, and Crop aren't enough

Anyone looking to build a client-side photo-manipulation library will want more than just image scaling, cropping, composing, and flipping. They need the ability to run convolution kernels, lookup tables, and rescale/colorspace transforms as well. The most common operations people want to run on photographs, like contrast/brightness enhancement, sharpen/unsharpen, conversion to black and white or sepia, etc use these.

Don't forget RAW and EXIF/image metadata

In addition, the ability to open RAW files, manipulate exposure compensation, and extract image metadata would be a huge boon. An offline Gears photo album would be much cooler if EXIF info could be extracted as images are imported.

My own wish list for Gears Image API

• Implements WHATWG CanvasRenderingContext2D as Base
• Adds text rendering, with minimally 90 degree rotation support 
• Image composition that obeys affine transforms (not just rotate)
• Convolution and Lookup operators (NxN square kernels, N atleast up to 5)
• Support opening RAW images
• Support read (write would be good too!) access to image metadata

Objections?

The first objections that will be raised is bloat in the plugin. Leaving aside the fact that Flash delivered outstanding capability in a slim plugin, the proposed libgd implementation already has many of the core functions needed to satisfy the proposed richer API, and I'm sure it would not be hard for Google engineers to implement the rest. Convolves and Lookups aren't rocket science.

A likely second objection and real hiccup will be achieving cross-platform, antialiased, internationalized text rendering with rotations. I don't have an answer for this one, only that users want it. Almost everyone I've talked to who does client-side rendering wants this.

A third issue is simply complexity and time to market. Resize, Flip, Rotate, and Crop are trivial to implement as simple libgd glue code, whereas full support of WHATWG semantics, would require significantly more time. Although some of this might be mitigated by stealing WebKit or Gecko's implementation and hacking it into Gears.

Fourth, dealing with large photographs, especially RAWs will bring memory issues to bear (unless a smart tile cache oriented system is used), and could be a point of denial-of-service against the Gears plugin if care isn't taken.

Regardless of these objections, I'd still urge Google to go for it. Make client-side image processing and visualization a major part of the next Gears API. Please!

-Ray