Small update to GWT Exporter

I recently discovered a pretty critical bug in GWT Exporter that can cause an infinite loop doing export. This was fixed in version 2.06 which available in the trunk and maven repository.

-Ray

Google AppEngine and GWT now a marriage made in heaven

The announcement that Google AppEngine now supports Java is incredible news. Not just because it opens the doors to running arbitrary JVM languages like Scala, JRuby, PHP, etc on AppEngine, but because of the ability to wire up Java on the client, and Java on the server, through Google Web Toolkit. You can use all of your familiar Java tools for editing, debugging, testing, and packaging.

With the new system, you can write a POJO, add JPA or JDO annotations, and write server-side logic to persist these POJOs in either a RDBMS like MySQL, or in BigTable/AppEngine. Moreover, you can export your DAO or logic interfaces through GWT RPC, and call them directly from the client, seamlessly, and painlessly.

Almost Painlessly

The one hitch you'll encounter as a GWT developer is trying to serialize or deserialize persistence capable types. This is nothing new for GWT developers who have tried this with Hibernate before, and there are workarounds such as Hibernate4GWT. This problem occurs because the persistence classes are enhanced with an extra field to hold state which enables them to work when detached from the persistence context. GWT RPC computes its own CRC based on the fields of a class in order to ensure compatibility between server and client and the extra field causes problems.

In general, when it comes to sending serialized ORM POJOs down the wire, I think it's a risky practice, because you're likely to pull in a lot more of the reachable object tree than you bargained for unless you're careful. A better approach might be to use DTOs based on ProtocolBuffers.

However, it is sometimes nice to do it if your POJOs are relatively flat and you want to rapidly prototype. If your insist on using your ORM'ed POJOs over RPC, there is a trick to making it work.

Making JDO/JPA enhanced classes work over GWT RPC

The first step to making things work is to disable detachable objects and tag your class as serializable.

@PersistenceCapable(identityType = IdentityType.APPLICATION, detachable = "false")
public class MyPojo implements Serializable {

}

This does two things. First, it tells the persistence engine that you'll be managing object identity, usually through a primary key, and secondly, when your POJO is accessed outside of a transaction/session, you want it to be transient not detached. A detached object remembers where it came from, so that after modifications, it can be reattached and merged back into the datastore. A transient object forgets that it once came from the datastore, and thus if you try to repersist it, you'll end up inserting a copy.

This has a major downside in terms of ease of use, but it does prevent the enhancer from injecting hidden fields into your class to manage detached state, and it is these hidden fields which break GWT RPC compatibility.

But I don't want copies!

In using transient objects, you'll break a desired design pattern, which is to fetch an object through RPC, modify its properties, and send the same object back through RPC to be merged into the datastore. A quick and dirty work around is to use reflection to lookup an attached object on the server, copy all of the persistent fields from the transient object, and then merge the persistent object. Here's a prototype class that does this (but not recursively, so it doesn't handle anything but primitive fields):

public class PersistenceHelper {
  public static  Object findPrimaryKey(T tInstance) {
    if (tInstance == null) {
      return null;
    }
    for (Field l : tInstance.getClass().getDeclaredFields()) {
      if (l.getAnnotation(PrimaryKey.class) != null
          || l.getAnnotation(Id.class) != null) {
        l.setAccessible(true);
        try {
          return l.get(tInstance);
        } catch (IllegalArgumentException e) {
          e.printStackTrace();
          return null;
        } catch (IllegalAccessException e) {
          e.printStackTrace();
          return null;
        }
      }
    }
    return new IllegalArgumentException(
        "Class " + tInstance.getClass().getName()
            + " does not have a method called getId()");
  }

  public static  void copyPersistentFields(Object entity, T tInstance)
      throws IllegalAccessException, NoSuchMethodException,
      InvocationTargetException {
    for (Method f : tInstance.getClass().getMethods()) {
      if (f.getName().startsWith("set") && Character
          .isUpperCase(f.getName().charAt(3))) {
       f.setAccessible(true);
        Method getter = tInstance.getClass()
            .getMethod("get" + f.getName().substring(3));
        getter.setAccessible(true);
        f.invoke(entity, getter.invoke(tInstance));
      }
    }
  }
}

The way you'd typically use this is as follows:

public  T mergeTransient(T tInstance) {
    EntityManager e = em.get();
    if(e.contains(tInstance)) {
      e.persist(tInstance);
      return tInstance;
    } else {
      Object primaryKey = PersistenceHelper.findPrimaryKey(tInstance);
      if(primaryKey != null) {
        Object entity = e.find(tInstance.getClass(), primaryKey);
        if(entity == null) {
          e.persist(tInstance);
          return tInstance;
        }
        else {
          try {
            PersistenceHelper.copyPersistentFields(entity, tInstance);
          } catch (IllegalAccessException e1) {
            e1.printStackTrace();
            throw new IllegalArgumentException("Can't copy fields from transient class to persistent class.");
          } catch (NoSuchMethodException e1) {
            throw new IllegalArgumentException("Can't copy fields from transient class to persistent class.");
          } catch (InvocationTargetException e1) {
            throw new IllegalArgumentException("Can't copy fields from transient class to persistent class.");
          }
          e.persist(entity);
          return (T) entity;
        }
      } else {
        // primary key may be null, assume insert
        e.persist(tInstance);
        return tInstance;
      }
    }
  }

Less than ideal

After experimenting with this pattern, I've come to the conclusion that although it works, I don't feel warm and cozy serializing instances out of the datastore, I like to have full control over what I'm sending down to the client so I can optimize for size and speed. However, I don't want to write boilerplate sychronization code for DTOs. In a later article, I'll detail a pattern for using ProtocolBuffers with GWT and a DSL for terse/concise manipulation of them.

It's still awesome

Even though there are some issue surrounding using RPC seamlessly with the Datastore ORM later, it completely trumps the time saved not having to do ANY configuration AT ALL to deploy an application. Words cannot describe how much of a time saver this is. No messing around with apt-get. No editing Apache configs. No Setting up log rotation and archiving. No dealing with backups. No bother with figuring out the right way to shard your db for your expected growth. No need to harden your own machines and firewalls. No need to provision anything but the application ID.

To be sure, there are still things you can't do on AppEngine. I can't run JNI. I can't launch threads. I can't use Java2D/JAI/JavaSound. And probably most relevant, I can't host long-running comet sessions. And if you really really need to do those, you can rent a server somewhere to do it.

However, the majority of applications don't use these capabilies, and for these developers, AppEngine is an epic win.

GWT's type system is more powerful than Java

This may come as a shock to some people. Isn't GWT just Java syntax you say? Yes, it is, and it does not extend the Java grammar in anyway. Yet, it is nonetheless true that GWT is more powerful than Java.

The Overlay Type

The reason why GWT is more powerful is because it is actually a unification of two type systems, Java and Javascript. And while it seems that these are relatively walled off from one another, there is a bridge that unites the two, and that is the GWT Overlay Type system.

The overlay system in essence, permits the 'overlay' or attachment of Java types to Javascript objects. And since you can pass Java objects into Javascript and back, this means you can in fact, overlay types on Java as well.

Categories and Extension Methods

Some languages have a facility called Categories or Extension Methods, modern examples include Objective-C, Groovy, and C#. A category allows you to pretend as if an existing reference implements additional methods, when it actually doesn't.

In reality, they are syntax sugar for invocation of static utility functions. That is:

SomeType x = new SomeType();
SomeTypeUtils.doSomething(x);

becomes

SomeType x = new SomeType();
x.doSomething();

Transparently, behind the scenes, the compiler rewrites the invocation of x.doSomething() into SomeTypeUtils.doSomething(x). GWT's JavaScriptObject overlays are essentially Categories, as the compiler transparently rewrites methods that appear to exist on an Overlay into static method calls on the underlying JavaScriptObject. This is one reason why JSO methods have to be effectively final, as there is no polymorphism allowed.

This all sounds very interesting and theoretical, but what's the practical benefit?

Type-Safe Enumerations with Zero Overhead

Let's say you want to write a method that can set the display property of an element to one of the legal values, and only the legal values. Traditional approaches would include using a Java enum and writing a setter method that accepts only this type:

enum DisplayType {
   BLOCK, INLINE, TABLE, NONE;
}

public static void setDisplay(Element e, DisplayType t) {
   e.getStyle().setProperty("display", t.name().toLowerCase());
}

Unfortunately, this will generate a lot of bloat, since each enum value is a class, the class must be initialized by a static initializer, and the ultimate CSS property value string has to be obtained through method calls that might not inline because of polymorphic dispatch.

Think about what we want here. Don't we really just want to create a subclass of java.lang.String, create a bunch of this String subclass constants, and write a method to accept that type? Unfortunately, java.lang.String is final. You can't subclass it in Java.

But you can in GWT, and I'll show you how!

Turn a String into a JSO

public class DisplayType extends JavaScriptObject {
   protected DisplayType() {}
   public native static DisplayType make(String str) /*-{
      return str;
   }-*/;

   public native String value() /*-{
      return this;
   }-*/;
}

For brevity, I left out the extra code to support Hosted Mode. In hosted mode, you must wrap the 'str' argument in an array, e.g. [str] and fetch it using return this[0], but that's not important. What this code is effectively doing is casting a String into a DisplayType and imposing an additional categorical method on this reference, which is value(). Keep in mind, the value() is never actually attached to the prototype of the underlying Javascript object.

To use,

DisplayType NONE = DisplayType.make("none");
DisplayType BLOCK = DisplayType.make("block");
DisplayType INLINE = DisplayType.make("inline");
DisplayType TABLE = DisplayType.make("TABLE");

Now, what is the output of the GWT compiler when compiling element.getStyle().setProperty("display", BLOCK.value())? Here it is:

element.style['display']='block';

In fact, even calling the setDisplay() method with various DisplayType enums results in inlined assignments to the element.style property with no method calls!

Adding methods to Numbers

In Groovy, Scala, and some languages, you can even add methods to primitive integers. Using GWT overlay types, you can even do this!

 public class Int extends JavaScriptObject {
    protected Int() {}
    public static native Int make(int x) /*-{
      return x;
    }-*/;
    
    final public native int value() /*-{
      return this;
    }-*/;
    
    final public Int square() {
      return make(value() * value());
    }
  }

 int val = (int) Duration.currentTimeMillis();
 Int x = Int.make(val);
 Int sq = x.square();
 Window.alert(String.valueOf(sq.value()));

And what do you think the generated code looks like? Try this:

val = (new Date()).getTime();
x = val;
sq = x * x;
$wnd.alert('' + sq);

There you have it. I successfully added a method to a primitive 'int' called square() without using a wrapper, and with no overhead whatsoever. This opens the doors to implementing primitive 'wrappers' for GWT for int, double, float, etc, that are not wrappers at all and have no overhead, which would be very useful in many circumstances where Integer, Double, Float, in java.lang are too heavy.

So, can we conclude from this that GWT is more powerful than Java? ;-)

-Ray

Relaxing constraints on GWT.create()

Recently I've been playing around with some GWT ideas that really cry out for a more liberal deferred binding system. Currently, GWT imposes the restriction that deferred binding can only happen through the GWT.create() method. There's a couple of problems with this:

1. Can't narrow type signature for custom library


2. Can't create a method to decorate the returned result


3. Can't parameterized or override the bindings at callee site

To illustrate this, I've compiled some motivation examples.

RPC requests signed by OAuth

interface MyService extends RemoteService<MyServiceAsync> { ... }
// example call
OAuth.withSignature(MyService.class).method1(arg1, arg2, asyncCallback);

public class OAuth {
  @GwtCreate  // method must be static, class parameter must be final
  public static <S, T extends RemoteService<S>> S withSignature(final Class<T> service) {
        // GWT.create with non-literal ONLY allowed 
        // if enclosing method is @GwtCreate and 
       // variable statically resolvable to literal param
        S async = GWT.create(service);  
        ((ServiceDefTarget)async).setRequestBuilderCallback(new OAuthRequestBuilderSigner());
        return async;
  }
}

Currently today, it would look like this:

MyServiceAsync foo = GWT.create(MyService.class);
RequestBuilder rb = foo.method1(arg1, arg2, asyncCallback);
OAuthRequestBuilderSigner.sign(rb);
rb.send();

The tight coupling at the callsite also makes it difficult to swap out implementations easy (like using AuthSub, or one of the other 4 Google Friend Connect auth techniques). An alternative is to make a separate subtype of each with a custom generator, e.g.

MyServiceOAuth extends MyService, GWT.create(MyServiceOAuth.class)

I'd argue that the above gets cumbersome with multiple services and multiple authentication types. I've taken some liberties above by adding a type parameter to RemoteService to make the return type statically resolvable, as well as allowing a global callback mechanism on ServiceDefTarget for RequestBuilder override. Note that type-safety is ensured, one can't call this method with something that is not a RemoteService, and it will be flagged at edit-time in your IDE.

An EasyMock library for Hosted/Web Mode

Subscriber mock = GMock.mock(Subscriber.class);
publisher.add(subscriber);
//...
GMock.replay(mock);

public class GMock {
  // selectively override module binding rules ONLY for this method
  @GwtCreate(generator=com.gmock.rebind.GMockGenerator.class)
  public static <T> T mock(Class<T> toMock) {
        return GWT.create(toMock);
  }
}

In this method, the mock() method acts like GWT.create() except that it overrides the current binding rules, forcing the specified generator.

GWT Exporter

Exporter.export(Foo.class);

public class Exporter {
   @GwtCreate
   public static <T extends Exportable> void export(Class<T> exportable) {
       ExporterImpl ximpl = GWT.create(exportable);
       ximpl.export();
   }
}

Note, the type safety, one can't try to Export a non-exportable class. This will be flagged at edit time in the IDE.

GIN/Guice dependency injection

Processor pimpl = Gin.inject(Processor.class, TestProcessorModule.class);

public class Gin {
  @GwtCreate(generator = com.google.gin.rebind.GInjectorGenerator)
  public static <T, S extends AbstractGinModule> 
      T inject(Class<T> interf, @GwtCreateParam Class<S> module) {
         return GWT.create(interf, module);
  }
}

This one is more controversial, but allows GWT.create() to be parameterized by literal metadata that is available to the generator. This is semantically equivalent to what we have today:

@GinModule(TestProcessorModule.class)
interface MyTestInjector extends GInjector {
   Processor getProcessor();
}

but without the need to actually write the interface.

Combing compile-time and run-time parameters

Final example, mix-and-match both compile-time variables and run-time variables:

OAuth.withSignature(MyService.class, debugMode ? "/debugService" : "/MyService").method1(arg1, arg2, asyncCallback);

public class OAuth {
  @GwtCreate  // method must be static, class parameter must be final
  public static <S, T extends RemoteService<S>> 
        S withSignature(final Class<T> service, String endPoint) {
        // GWT.create with non-literal ONLY allowed if enclosing method is 
        // @GwtCreate and variable statically resolvable to literal param
        S async = GWT.create(service);  
        ((ServiceDefTarget)async).setRequestBuilderCallback(new OAuthRequestBuilderSigner());
        ((ServiceDefTarget)async).setServiceEntryPoint(endPoint);
        return async;
  }
}

Spring cleaning and the blog

To try and improve readability and reach, I made a few changes today:

1) Re-enabled SyntaxHighlighter for code snippets
2) Widened the width of the main content to 600px (for code blocks)
3) removed Twitter and added Google Friend Connect
4) enabled CAPTCHA to reduce comment span
5) tweak some colors so the subsections are easier to read.

If you have any other advice for improving the Look and Feel, leave a comment.

-Ray

Structural Typing for GWT and Javascript

In a previous post I described a type of impedance mismatch between Javascript and Java idioms that makes the GWT Exporter still less than ideal for supporting Javascript programmers. A short illustrative (but contrived) example:

public class Customer implements Exportable {
   private String firstName, lastName;
   @Export String getFirstName() { 
      return firstName; 
    }

   @Export void setFirstName(String fn) {
      firstName = fn;
   }

   @Export String getLastName() { 
      return lastName; 
    }

   @Export void setLastName(String ln) {
      lastName = fn;
   }
}

Today, you may use this export in Javascript like this:

var cust = new Customer();
cust.setFirstName("Ray");
cust.setLastName("Cromwell");
Processor.doSomething(cust);

The example is contrived because you could use a constructor, but with more complex objects with nested types, you wouldn't use a constructor, but either a builder pattern, or inject the types after construction.

Javascript developers however don't work in the world of Javabean interfaces, they prefer easy construction of configuration/builder information via object literals:

Processor.doSomething({firstName: "Ray", lastName: "Cromwell"})

Moreover, when passing in say, a bind of event callbacks, they'd prefer to write:

foo.addEvents({
    click: function(e) { ... }
    move: function(e) { ... }
    drop: function(e) { ... }
});

The challenge is to seemlessly bridge this idiomatic mismatch between JS and Java GWT code without actually having to write much bridging code or adapters.

Structural Typing

Java is a manifestly typed language. All types have to be declared, and type checking is done by explicit hierarchy. Javascript is a dynamically typed language, with essentially no type checking at all. Orthogonal to this is the concept of Structural Typing. Haskell, ML, and Scala are all examples of languages which support structural typing. Structural typing was also planned for Javascript 2 before it got killed.

So what's a structural type? Recall the Customer example from the previous section. It was a class with two fields, firstName, and lastName, both Strings. If Java supported a structural type system, I could declare a method in two ways:

public void process(Customer cust) { ... }

public void process({firstName: String, lastName: String} cust) { ... } 

In this invented syntax, the cust parameter to the second process() function is an anonymous type, we don't know its real name. However, we are stating that as long as it consists of two fields named "firstName", and "lastName", and the types are both Strings, then we can access these fields and treat it like a Customer (although it may not be one)

Hmm....I smell an idea...

Structural Type Exports

What if I rewrite the Customer POJO class with a @StructuralType annotation:

@Export
@StructuralType
public class Customer implements Exportable { ... }

The dispatch code for an exported process() function could then look like this:

$wnd.Processor.prototype.process = function(cust) {
  if(cust instanceof $wnd.Customer && isExportedInstance(cust)) {
    // JSNI dispatch to @Processor::process(LCustomer;)(unwrap(cust));
   }
  else if(typeof(cust) == 'object') {
      // cust is not an instance of the Customer POJO, but an object
      if(cust.firstName != undefined && cust.lastName != undefined) {
          var scust = new Customer();
          scust.setFirstName(cust.firstName);
          scust.setLastName(cust.lastName); 
          @Processor::process(unwrap(scust));
      }
   }
}

The GWT compiler would auto-inject this structural type check and initialization code simply by annotating a parameter or return type involved with @StructuralType

Refinements to the idea

Taking this a step further, one could override the expected type literal field names to be checked

@SType("fn")
public void setFirstName(String firstName) { ... }
@SType("ln")
public void setLastName(String lastName) { ... }

which would allow the object literal to be specified as {fn: "Ray",ln: "Cromwell"}. Another extension would allow partial matches to succeed with default values supplied:

@Optional("Ray")
public void setFirstName(String firstName) { ... }

which would reduce the structural type to just a lastName field for matching purposes, but would allow the specification of firstName to be supplied and injected into the setter as "Ray" if it wasn't present.

Structurally Typed Interfaces

GWT Exporter already supports closure conversion for single-method Java interfaces. That is, if the following interface:

public interface MyCallback {
     public void go(Object arg);
}

occurs as a type parameter in an exported method, the Javascript developer may supply function(arg) { ... } and the GWT generated bridge code will automatically convert this into a GWT object instance that implements the MyCallback interface type. What it cannot support is the example given earlier:

foo.addEvents({
    click: function(e) { ... }
    move: function(e) { ... }
    drop: function(e) { ... }
});

which would represent something like:

public interface MyEventHandler {
    public void onClick(Event e);
    public void onMove(Event e);
    public void onDrop(Event e);
}

However, using structural typing conventions, and an @StructuralType annotation on an interface, we can auto-convert object literals containing multiple function closures into a Java interface for GWT.

I am still working out the details of the full overload resolution algorithm for overloaded function types, but some version of this proposal will make it into GWT Exporter 3.0.

-Ray

GWT Exporter 2.05 released

The release adds two new features:

1) arrays are return types or parameters supported, that is:

@Export public double[] getFoo() { }
@Export public SomeExportable[] getBar() {}
@Export public void setFoo(double[] foo) { }
@Export public void setBar(SomeExportable[] bar) { }

now works, in both Web mode and Hosted Mode.

2) Export Overlays implemented. This means you can now Export classes you don't own.

@ExportPackage("gwt")
public interface WindowExport extends ExportOverlay<Window> {
   @Export void alert(String s);
}

will export the GWT Window class into the namespace $wnd.gwt.Window, and export the alert() method of this class.

The JARs are downloadable from: http://timefire-repository.googlecode.com/svn/mavenrepo/org/timepedia/exporter/gwtexporter/2.05/