Google Gears Image Manipulation API not ambitious enough

Google Gears is a very subversive and disruptive technology IMHO, and I mean that in a good way. Google has the muscle to extend native browser functionality in a cross browser way by sneaking extensions into Gears. Of course, Adobe and Microsoft can do this as well (Flash and Silverlight), but the difference is, Google is offering AJAX-level building block extensions to browser functionality, not an alternative environment-within-the-browser-environment that Flash, Silverlight, and Java applets yielded.

Gears is steadily building momentum, and I'm sure many of Google's properties will soon support offline or enhanced modes using it, which will tend to make it a 'must-have' plugin, hopefully achieving 80-90% penetration in the future. So, before the vast majority of people start using the plugin, let's try to be as ambitious with the extension functionality as we can, so when the rush-to-install happens, people will be getting a version with very rich functionality as the base. We want to avoid the need to check Gears version all over the place and force people to upgrade plugins continually ("what, oh, your Gears version 1.21 doesn't have the image.shear() function, you need Gears 1.25 for that...")

Case in point, the proposed Gears Image Manipulation API. Granted, it's just starting, but I'd like to offer some upfront suggestions before this thing gets finalized.

Don't duplicate Canvas, extend it

The proposed API adds resize() and crop() operations to an image object. And also the ability to turn images back into blobs. The resize() and crop() operations can be done today with JS Canvas, but only WHATWG Canvas allows you to turn an image back into data. I don't think this API goes anywhere near far enough to justify its existence.

Also, the biggest pain with using Canvas today is that every browser but IE supports it, so why not implement a cross-browser offscreen Gears WHATWG Canvas API to start with.

But don't stop there. WHATWG Canvas lacks text rendering, and image rendering that obeys affine transforms, two of the big complaints against the existing Canvas. The Web 2.0 world will love anyone who can get such an extended cross-browser canvas widely deployed.

So start with an off-screen WHATWG Canvas API, add text rendering (and atleast 90 degree rotated text would be nice), plus drawImageWithTransform() that obeys transforms.

Resize, Flip, and Crop aren't enough

Anyone looking to build a client-side photo-manipulation library will want more than just image scaling, cropping, composing, and flipping. They need the ability to run convolution kernels, lookup tables, and rescale/colorspace transforms as well. The most common operations people want to run on photographs, like contrast/brightness enhancement, sharpen/unsharpen, conversion to black and white or sepia, etc use these.

Don't forget RAW and EXIF/image metadata

In addition, the ability to open RAW files, manipulate exposure compensation, and extract image metadata would be a huge boon. An offline Gears photo album would be much cooler if EXIF info could be extracted as images are imported.

My own wish list for Gears Image API

• Implements WHATWG CanvasRenderingContext2D as Base
• Adds text rendering, with minimally 90 degree rotation support 
• Image composition that obeys affine transforms (not just rotate)
• Convolution and Lookup operators (NxN square kernels, N atleast up to 5)
• Support opening RAW images
• Support read (write would be good too!) access to image metadata

Objections?

The first objections that will be raised is bloat in the plugin. Leaving aside the fact that Flash delivered outstanding capability in a slim plugin, the proposed libgd implementation already has many of the core functions needed to satisfy the proposed richer API, and I'm sure it would not be hard for Google engineers to implement the rest. Convolves and Lookups aren't rocket science.

A likely second objection and real hiccup will be achieving cross-platform, antialiased, internationalized text rendering with rotations. I don't have an answer for this one, only that users want it. Almost everyone I've talked to who does client-side rendering wants this.

A third issue is simply complexity and time to market. Resize, Flip, Rotate, and Crop are trivial to implement as simple libgd glue code, whereas full support of WHATWG semantics, would require significantly more time. Although some of this might be mitigated by stealing WebKit or Gecko's implementation and hacking it into Gears.

Fourth, dealing with large photographs, especially RAWs will bring memory issues to bear (unless a smart tile cache oriented system is used), and could be a point of denial-of-service against the Gears plugin if care isn't taken.

Regardless of these objections, I'd still urge Google to go for it. Make client-side image processing and visualization a major part of the next Gears API. Please!

-Ray

Hardcore GWT Hands on Training?

It's the new year, and I've been mulling an idea recently that I'd like to get some feedback on from the community.

The GWT Conference was a blast, and it appears my Deferred Binding presentation was well received. Since then, a number of people have encouraged me to offer some sort of training or mentoring for GWT, either in classrooms, or in the form of corporate on-site training. I hesitate to get too distracted from working on Chronoscope and Timepedia, but after thinking about it some more, I do think it might be worth pursuing, perhaps one 2-day course per month.

What I don't want to do is some token GWT course that glosses over the core of GWT and shows one how to create some cute widgets and stuff them into a web page, without understanding what's going on under the hood. Rather, I'd like to teach, on Day 1, GWT from the ground up exploring in detail what is happening: Java vs JS Output, Deferred Binding, JSNI, Widget attachment, Widget event processing, RPC/Serialization vs On-the-wire representation, the Bootstrap/Selection process, Hosted Mode, debugging, testing, etc.

On Day 2, I'd then delve into more idiomatic patterns: Creating custom widgets, creating your own modules/libraries, Localization, image bundles, integrating with back-end frameworks, RPC and RPC customization, optimizing size and speed, etc

I'd require people to write code in short labs during the class, give personal help in troubleshooting, setting up Eclipse/IDEA/Netbeans environments, and ask that others who have completed labs successfully help participate in helping those having trouble, because I believe a good way to learn something is by in-class participation and trying to help someone else. Who knows, it might be a fun, campfire type atmosphere!

I'd risk losing a lot of newbies doing this, and also risk not getting any advanced users who are self-taught, but hopefully, there are a lot of people in the middle with some GWT experience, who would like a more in depth tutorial hands on. Perhaps I'd do a more marketing oriented/newbie introductory course later.

Ultimately, I'd hope that people who come away from the classes with not only enough knowledge to write code to cover their own use cases, but also have enough knowledge to fix bugs and contribute to GWT itself. The GWT compiler and core libraries should no longer be 'magic' to those taking the course.

Obviously, the course wouldn't be free, especially in offsite training where I'd have to rent the event space and also given the opportunity cost for me, but I would offer discounts if space is donated, charge less for the first 'beta' class, and given some steep student discounts, especially if students would arrive early and help me with class setup.

I'm working out the course material at the moment, and pricing out some meeting spaces. I haven't decided on a per-student price yet, but feel free to email me suggestions, course ideas, and most importantly, whether or not you'd be interested in attending, or your company would. Private replies can be sent to cromwellian / gmail.com

Oh, this would be for the SF Bay Area, hopefully BART/Caltrain accessible.

-Ray

GWT Conference videos now online

Google has quietly put all (most?) of the GWT Conference session videos online now.

Here is my talk on GWT Deferred Binding:

Scala-style Actors in GWT: Architectural details

In the last article, I talked about the targeted API for actors, I'll talk a little about the architecture I am prototyping with.

Store-and-forward messaging

The essential idea for implementing communication between actors is to give them a globally unique address (UUID) first, which is called the ActorEndpoint class. Think of a UUID as the "email address" of an Actor, with Actors themselves being the email client.

Because of limitations in the browser, we can't really use any information about the machine itself in creating these UUIDs, so instead, we create pseudo-version-4 random UUIDs

The Actors.register() registers Actor instances in an internal HashMap so that Actors can be looked up by UUID/Endpoint later.

The only essential difference between a local in-process Actor, and a remote out-of-process Actor, is that the endpoint is a RemoteEndpoint which returns an associated (do nothing) RemoteActor.

So the best way to think of the delivery/receive process is by analog to plain-ole SMTP email queues, except that the "SMTP" server in this case, can only wait for a client to connect, it can't initiate any connections.

Delivery: Local vs Remote

When you invoke Actor.sendTo(anotherActor, msg), it delegates to Actors.addToMailbox(sender, receiver, msg). This code essentially performs a check: if(receiver.getEndpoint() instanceof RemoteEndpoint) it enqueues the message in an envelope addressed to the remote mailbox associated with the UUID of the remote actor.

If the receiver is not a remote actor, and is in the 'started' state, it invokes a method Actor.localReceive() which can process the event immediately. localReceive() delegates to a Generator method (client) or Reflection method (Server) to perform the multiple dispatch.

Transporting the Messages

For outgoing messages, a periodic timer task checks for non-empty outgoing queue, and invokes a GWT RPC call which transmits all enqueued Envelopes, and receives as a result, all messages that had been sitting on the server destined for a local actor in this browser. It then opens the Envelopes received, and uses the UUID of the recipient to lookup the appropriate Actor for local delivery.

For incoming messages, there are two things that can be done: 1) we can poll with RPC for messages. 2) we can use one of the numerous Comet techniques to receive push messages. I'll probably support both in the final library.

The mailbox servlet

All of the communication is orchestrated by a GWT RemoteService that maintains for each UUID, a queue of Envelopes waiting to be delivered. It's currently very simple, transient, and not fail-safe. There is no security for the registrar for globally named actors (e.g. "chatLobby") and no Envelope sequence number to prevent out of order or duplicates, which is sure to happen with Comet + RPC combined channels.

Other cool stuff

Someone has made a JXTA/Scala Actor gateway, which allows messaging across the P2P JXTA network. This could certainly be adapted to GWT Actors. One could also imagine interacting with a JMS Message Queue/EJB MessageBeans, XMPP/Jabber, or Android clients as well. I am trying to take care in making this library 'cloud computing safe' like Chronoscope, so that one day, mobile phone users will be able to play interactive p2p games against Web desktop users. :)

Playing with feedburner options

I hope this doesn't annoy too many people, or break some subscriber's feeds, but I've been toying around with Feedburner (first time) tonight. Let me know if your feed broke.

-Ray

Scala-style Actors in GWT

Another day, another GWT extension.

Recently, on the Chronoscope mailing list, some users have been asking for the ability to stream in updates to the chart, a typical use case being streaming live stock market data. There are issues in Chronoscope blocking this at the moment that require some extensions, but the greater question of how to do live updates of a GWT application got me thinking about how to appropriately model this in GWT.

RPC? Comet? JSON?

So, ordinarily, in the interest of time, I'd take the road most traveled and simply use GWT RPC, some JSNI Comet integration, or some other simple mechanism to receive poll or push data. However, over the last few months, I've really fallen in love with Scala, and I like Scala's abstraction for message passing: Actors.

I like actors for several reasons. First, they are asynchronous by default, hide the underlying implementation (threads, event queues, in-process calls), and well... I just like em because they're cool.

Actors in Java

Unfortunately, Java doesn't support the elegant syntax of Scala, but with a Generator to implement multiple dispatch, it's not too bad. The approach I've taken is to use overloaded receive methods to implement pattern-matching case classes:

public class Pinger extends Actor {
public void receive(Pong p) {
// pong case
sendTo(sender, new Ping());
}

public void receive(Stop s) {
GWT.log("Received a stop message!");
}
}

public class Ponger extends Actor {
int count = 0;

public void receive(Ping p) {
if(count++ < 100) sendTo(sender, new Pong());
else sendTo(sender, new Stop());
}
}

A GWT Generator supplies a compile time dispatch() method which can take dequeued mailbox messages and invoke the appropriate receive() method. Java doesn't have case classes, so you can use Enums or Type-Safe Enum Pattern as messages (as long as they implement org.timepedia.actors.client.Event)

Big deal, Whoop-de-do

What does all this gain us? Well, for one, it abstracts away the RPC mechanism on the client and server. You don't need to deal with RPC interfaces, or AsyncCallback, in fact, RPC can be replaced with Comet for the receive channel if you so desire.

However, the big whoop-de-do is the ability to do peer-to-peer messaging between browsers. Chat? Multiplayer gaming anyone?

Remote Actors

In order to extend actors outside the Javascript environment of your browser, we need to use a server side mailbox server to queue and relay messages between actors in different browsers. In order to publish your actor so that remote actors can retrieve it, all you need to write is:

RemoteActors.register("myId", actor);

and in another peer, you can write:

RemoteActors.select("myId");

public .... extends Actor {
public void receive(ActorEndpoint endpoint) { ... }
}

Wait!? There's no return value? Remote selection is asynchronous (needs to ask mailbox server) so we simply return the result of the select() query as a message using the actor interface!

The returned endpoint represents either a local (in browser), server, or browser peer actor (on someone else's computer).

Chat using Actors

So let's say we want to implement a GTalk chat-like program in GWT, with buddy lists, group chat, and private messages. How would we do this with actors?

public class ChatActor extends Actor {
public void onStart() {
super.onStart();
RemoteActors.select("chatLobby", this);
}

public void receive(ActorEndpoint rae) {
chatLobby = rae.getActor();
sendTo(chatLobby, new JoinLobbyMessage());
}

public void receive(ChatMessage p) {
html.setHTML(html.getHTML()+"
"+p.getMessage());
}


public void onChange(Widget sender) {
sendTo(chatLobby, new ChatMessage(((TextBox)sender).getText()));
}

}

The above code assumes we are using an HTML Widget to store the chat transcript, and a TextBox to handle user input. We start out by selecting the "chatLobby" actor from the server. We don't know where this Actor really resides, it could be someone's PC, but for my test implementation, it resides in a Servlet.

When someone enters text, we send a "ChatMessage" object to the "chatLobby" actor containing the text. What is chatLobby?

public FooServlet extends HttpServlet ... {
public void init() {
ChatLobbyActor lobbyActor = new ChatLobbyActor();
RemoteActors.register("chatLobby", lobbyActor);
}
}

public class ChatLobbyActor extends ReflectionActor {

HashSet<Actor> actors=new HashSet<Actor>;
public void receive(JoinLobbyMessage msg) {
actors.add(sender);
sendTo(sender, new ChatMessage("Welcome to Chat Lobby"));
}

public void receive(ChatMessage msg) {
for(Actor act : actors)
sendTo(act, msg);
}
}

Implementation details

The library isn't ready for release right now, but the mailbox routing servlet uses transient memory arrays to hold per-actor mailboxes (with UUIDs to identify them uniquely). This could be released with transient or persistent JMS queues in some implementations. For communication channels, it uses GWT RPC at the moment. It polls the server periodically for messages, and during the send() operation, it receives all pending messages and sends all pending messages to minimize the number of RPC requests. However, I would like to have a Comet implementation before release that gives you the option of receiving messages pushed on a comet channel.

Holy Grail

I have an itch, a real bad, unproductive itch, to implement a multiplayer graphics game in GWT using Chronoscope's Canvas, Fred Sauer's GWT-Sound library, and this GWT Actors library. And I'm not talking turn based action, but real, predictive physics simulation, such as a 2D space spacewar/omega-race style shooter.

But if I start messing around with this, I'll never finish this blog series, nor fix the Chronoscope bugs, nor launch Timepedia. Argh!

In an upcoming article, I'll talk more about the implementation under the hood, and hopefully release the code.

-Ray

Deferred Binding slides online

My GWT Conference presentation on Deferred Binding was a little rushed due to time, so here are the slides for those who wish to review the material:Deferred Binding@Google Docs.

I'll post up a PDF version later.

-Ray