Five good reasons to port your code generator to Xtend

If you happen to have an existing code generator written in Xpand or some other (template) language, here are five good reasons why you should migrate to Xtend.

Xtend is fast

Code generation adds to the development turn around time. We all know it : tools can slow us down. They take their share of CPU cycles during the daily development and all that automation is not worth it, if you have to wait for your machine all day long. Many template languages are interpreted and are therefore rather slow as they cannot benefit from optimizations done during compilation. Code generators written in these languages suffer from that of course, and if you don't care you can easily multiply the time your build takes with a slow code generator.

Xtend code is translated to readable and idomatic Java code. It runs as fast as Java, which is *very* fast and there's no additional overhead nor dependencies at runtime. Rough measures have shown that code generators implemented with Xtend are up to ten times faster then the equivalent Xpand version.

Xtend is debuggable

Code generators, just like any other software system, have a tendency to get a bit complex over time. It's not so funny to find and fix a bug in a real world code generator without any debugging support. Unfortunately that's the situation with most template languages : They don't have any good debugging support.

As Xtend code is translated to readable Java code you can always use the advanced Java debugging tools you are used to. Additionally the Eclipse debugger lets you choose whether you want to debug through the Xtend source code or the generated Java code. You can even switch back and forth during the same debug session (see the screenshot).

Better IDE Support

Xtend's Eclipse support is propably the most advanced after Java. The integration between Xtend and Java is seamless : Navigation, Call Hierarchies and even cross language refactorings just work. And Xtend features the whole Java type system and gives you errors and warnings as you type.

Template Expressions and Grey-Space Support

In contrast to typical template languages such as Xpand, templates in Xtend are expressions which yield some value. This allows for composing multiple templates and lets you pass the results around and process them.

A unique feature to Xtend is the greyspace support. It solves the conflict between pretty printing a template and pretty printing the to-be-generated output. Common template languages cannot distinguish between indentation meant for the template and indentation meant for the output. Xtend can. Here's an example (note the different coloring of whitespace):

Better Extendability

In order to reuse a code generator and adapt it to the needs of a particular project, you need to be able to hook into it and change certain implementation details. In Xpand this was supported through AOP: You could hook into any template invocation and do what seems fit.

Although very flexible, this approach is also very fragile. The original code generator cannot be evolved, since clients might hook into any template call and you break them easily. Even worse you don't get any compile-time feedback about the broken extension hook.

With Xtend you can use dependency injection together with extension providers. No AOP anymore and the solutions is even faster and statically typed. If you need to exchange some implementation, just reconfigure your dependency injection configuration and your are good.

Summary

There are many more advantages and cool features in Xtend, but I hope the five listed here are convincing enough. If you don't have the time to do it yourself, but can afford some money, itemis is always there to help.

Codegeneration With Xtend from Sven Efftinge

Xtend's Extension Providers

Xtend supports extension methods very similar to how they are supported by C#. The basic idea is to make static methods available as instance method on the first argument's type. Example :

import static extension java.util.Collections.*

...

// we can use static methods from Collections like this 
val maxValue = myCollection.max // calls Collections.max(myCollection)
val minValue = myCollection.min // calls Collections.min(myCollection)
...
 

This makes code much more readable as you don't have to read inside out, but can read from left to right. Also the discoverabilty of available features via content assist works much better.

But of course using static methods all over the place is problematic, as you bind your code to the implementation which makes it hard to test and to reconfigure for different situations (e.g. use a different database system).

Enter Extension Providers!

In Xtend you can put the extension keyword to a local field or a parameter. This will make its instance methods available on the first parameter's type.

I'll explain that with an example using JPA and Java EE.

 // Java code
@PersistenceContext
EntityManager em

public LineItem createLineItem(Order order, Product product, int quantity) {
    LineItem li = new LineItem(order, product, quantity);
    order.getLineItems().add(li);
    em.persist(li);
    return li;
}

public void removeOrder(Integer orderId) {
    Order order = em.find(Order.class, orderId);
    em.remove(order);
}
 

If you add the keyword extension to the field declaration, the instance methods of the PersistenceManager get projected onto the entities. So you no longer have to write em.persist(li) but can just write li.persist:

@PersistenceContext extension EntityManager

def createLineItem(Order order, Product product, int quantity) {
    val li = new LineItem(order, product, quantity)
    order.lineItems += li
    li.persist
    return li
}

def removeOrder(Integer orderId) {
    val order = Order.find(orderId)
    order.remove
}
 

Extension provider allow for adding layer specific functionality to any classes in a non-invasive way. And you don't have to use static methods for that.

Better APIs with Extension Providers

When designing an API, you can make it very easy for the clients to have the right extension providers on the scope.

One possibility is to provide an abstract base class which contains visibly extension fields:

abstract class AbstractDao {
    @PersistenceContext
    protected extension EntityManager em; 
}

class Concrete extends AbstractDao {
 // use extension methods from EntityManager
}
 

If you don't like inheritance that much, another approach is to mark a parameter of an abstract method with the extension keyword.

interface JPACallBack {
    def void doStuff(extension EntityManager em)
}
 

When implementing that method, the IDE will automatically add the extension keword for you. The active annotations API uses that idiom.

Note that in case you define the abstract class or the interface in Java, you can add the @Extension annotation instead of the keyword.

IDE Support

Sometimes the reader might be a bit unsure where a certain member has been declared. Of course the hover as well as navigation always shows the correct declaration. In addition the semantic coloring highlights extension methods ...

...and you can even inspect the desugared version of an expression in the hover :

Working With JSON Data From Java

UPDATE : I have uploaded the example to github : JSONized@Github.

I am preparing excercises for the Xtend tutorial Sebastian and I will be giving on Monday at EclipseCon.  For one excercise we parse an itunes music library in JSON format in order to execute different queries on that data. Of course the data is well structured, so why not adding a thin typed facade over the generic JSON objects?   I've built a little active annotation which let's you do the following:

Step 1: Find some Data

Let’s take data from Yahoo as an example. The data contains the most popular new album releases. Here's the first album:

{
 "query": {
  "count": 24,
  "created": "2013-03-21T20:13:42Z",
  "lang": "en-US",
  "results": {
   "Release": [
    {
     "UPC": "602527291567",
     "explicit": "0",
     "flags": "2",
     "id": "218641405",
     "label": "Streamline/Interscope/Kon Live",
     "rating": "-1",
     "releaseDate": "2009-11-23T08:00:00Z",
     "releaseYear": "2009",
     "rights": "160",
     "title": "The Fame Monster",
     "typeID": "2",
     "url": "http://new.music.yahoo.com/lady-gaga/albums/fame-monster--218641405",
     "Artist": {
      "catzillaID": "0",
      "flags": "115202",
      "hotzillaID": "1810013384",
      "id": "58959115",
      "name": "Lady Gaga",
      "rating": "-1",
      "trackCount": "172",
      "url": "http://new.music.yahoo.com/lady-gaga/",
      "website": "http://www.ladygaga.com/"
     },
     "ItemInfo": {
      "ChartPosition": {
       "last": "1",
       "this": "1"
      }
     }
    },
.... 
}

Step 2: Copy the example data into a @Jsonized annotation:

@Jsonized('{
 "query": {
  "count": 24,
  "created": "2013-03-21T20:13:42Z",
  "lang": "en-US",
  "results": {
   "Release": [
    {
     "UPC": "602527291567",
     "explicit": "0",
     "flags": "2",
     "id": "218641405",
     "label": "Streamline/Interscope/Kon Live",
     "rating": "-1",
     "releaseDate": "2009-11-23T08:00:00Z",
     "releaseYear": "2009",
     "rights": "160",
     "title": "The Fame Monster",
     "typeID": "2",
     "url": "http://new.music.yahoo.com/lady-gaga/albums/fame-monster--218641405",
     "Artist": {
      "catzillaID": "0",
      "flags": "115202",
      "hotzillaID": "1810013384",
      "id": "58959115",
      "name": "Lady Gaga",
      "rating": "-1",
      "trackCount": "172",
      "url": "http://new.music.yahoo.com/lady-gaga/",
      "website": "http://www.ladygaga.com/"
     },
     "ItemInfo": {
      "ChartPosition": {
       "last": "1",
       "now": "1"
      }
     }
    }]
   }
  }
}')
class MusicReleases {
  // empty
}

Step 3: Enjoy statically-typed accessor methods!

The annotation will create getters and setters derived from the example data. It's just a super thin layer and you have full access to the underlying JsonObject if you want to go more dynamic. If you want to learn how to do such things, you should come to our tutorial. It's going to be fun!

Xtend 2.4 now available!

The committer team behind Eclipse Xtend is proud to announce the availability of Version 2.4. The new release has over 300 bug fixes. The most important enhancements are:

Active Annotations

They are Lisp-like macros for Java (or Xtend). You declare an annotation and explain to the compiler how annotated elements should be translated to Java source code. The compiler will find those annotations on the classpath and execute them during compilation. The IDE and the compiler are fully aware of how annotations contribute to the compilation process, such that content assist, navigation, linking and error messages all reflect the changes appropriately.
You have to see it in action, it is code generation like it should be! Read More

New Type System

Xtend 2.4 has gotten a whole new type system implementation, which not only improves the compilation performance but fixes many many outstanding typing issues. Also the type inference algorithm has become much smarter and even takes the control flow into account for type inference. Xtend can even infer types where other languages such as Scala fail.

Android Support

Xtend is a perfect match for Android development as it relies on the JDK and comes with very little extra dependencies. The new version has special support for debugging Android emulators and some other goodies.

New Language Features

Xtend 2.4 allows to turn local variables and parameters into extension providers, comes with a neat literal syntax for immutable collections, many new operators and some other enhancements.

New IDE Features

The Eclipse plug-in now supports formatting, more quick fixes, more refactorings, better content assist and more little gems which let you get into a smoeeth development flow.

Read the full release notes and get the release

Many Thanks

... to all the committers and contributors who have put this together: Sebastian Zarnekow, Jan Köhnlein, Dennis Hübner, Moritz Eysholdt, Holger Schill, Sebastian Benz, Knut Wannheden, Natalia Ossipova, Anton Kosyakov, Christian Hülsmeier, Stefan Oehme, Ed Merks, Benjamin Schwertfeger, Lieven Lemiengre, Hendrik Eeckhaut, Christoph Kulla, and many more!

Fun With Active Annotations: My Little REST-Server API

These days I've been playing around a lot with Xtend's new language feature addition, called Active Annotations. It's so exciting want kind of things you can do with them. Today I prototyped a nice little REST-API (code can be found at github):

@HttpHandler class HelloXtend {

  @Get('/sayHello/:name') def sayHello() '''
    <html>
      <title>Hello «name»!</title>
      <body>
        <h1>Hello «name»!</h1>
      </body>
    </html>
  '''
 
  @Get('/sayHello/:firstName/:LastName') def sayHello() '''
    <html>
      <title>Hello «firstName» «LastName»!</title>
      <body>
        <h1>Hello «firstName» «LastName»!</h1>
      </body>
    </html>
  '''

  def static void main(String[] args) throws Exception {
    new Server(4711) => [
      handler = new HelloXtend
      start 
      join
    ]
  } 
}

This is a single class with no further framework directly running an embedded Jetty server. The interesting part here is, that you don't have to redeclare the parameters, as the active HttpHandler annotation will automatically add them to the method. See the method is overloaded but both declare zero parameters? That's usually a compiler error, but here they actually have five resp. six parameters, because my annotation adds the parameters from the pattern in the @Get annotation as well as the original parameters from Jetty's handle method signature. Just so you can use them when needed.

Not only the compiler is aware of the changes caused by the annotation, but so is the IDE. Content assist, navigation, outline views etc. just work as expected.

This is how it works

The HttpHandler implements org.eclipse.jetty.server.handler.AbstractHandler and adds a synthetic implementation of the method handle(). The effective code it adds to the class we've seen above is :

// generated, synthetic Java code
public void handle(final String target
                 , final Request baseRequest
                 , final HttpServletRequest request
                 , final HttpServletResponse response) throws IOException, ServletException {
  {
    Matcher sayHelloMatcher = Pattern.compile("\\/sayHello\\/(\\w+)").matcher(target);
    if (sayHelloMatcher.matches()) {
      String name = sayHelloMatcher.group(1);
      response.setContentType("text/html;charset=utf-8");
      response.setStatus(HttpServletResponse.SC_OK);
      response.getWriter().println(sayHello(name, target, baseRequest, request, response));
      baseRequest.setHandled(true);
    }
  }
  {
    Matcher sayHelloMatcher = Pattern.compile("\\/sayHello\\/(\\w+)\\/(\\w+)").matcher(target);
    if (sayHelloMatcher.matches()) {
      String firstName = sayHelloMatcher.group(1);
      String LastName = sayHelloMatcher.group(2);
      response.setContentType("text/html;charset=utf-8");
      response.setStatus(HttpServletResponse.SC_OK);
      response.getWriter().println(
          sayHello(firstName, LastName, target, baseRequest, request, response));
      baseRequest.setHandled(true);
    }
  }
}

Of course there's some room for optimizations :-) but I hope you get the idea. The implementation of the @HttpHandler and its processor is added below :

@Active(typeof(HttpHandlerProcessor))
annotation HttpHandler {
}

annotation Get {
 String value
}

class HttpHandlerProcessor implements TransformationParticipant<MutableClassDeclaration> {
 
  override doTransform(List<? extends MutableClassDeclaration> annotatedTargetElements, 
                       extension TransformationContext context) {
    val httpGet = typeof(Get).findTypeGlobally
    for (clazz : annotatedTargetElements) {
      clazz.extendedClass = typeof(AbstractHandler).newTypeReference
      val annotatedMethods = 
          clazz.declaredMethods.filter[findAnnotation(httpGet)?.getValue('value')!=null]
  
      // add and implement the Jetty's handle method
      clazz.addMethod('handle') [
        returnType = primitiveVoid
        addParameter('target', string)
        addParameter('baseRequest', typeof(Request).newTypeReference) 
        addParameter('request', typeof(HttpServletRequest).newTypeReference) 
        addParameter('response', typeof(HttpServletResponse).newTypeReference)
    
        setExceptions(typeof(IOException).newTypeReference
                    , typeof(ServletException).newTypeReference)
    
        body = ['''
          «FOR m : annotatedMethods»
            {
            «toJavaCode(typeof(Matcher).newTypeReference)» matcher = 
                «toJavaCode(typeof(Pattern).newTypeReference)»
                .compile("«m.getPattern(httpGet)»").matcher(target);
              if (matcher.matches()) {
                «var i = 0»
                «FOR v : m.getVariables(httpGet)»
                  String «v» = matcher.group(«i=i+1»);
                «ENDFOR»
                response.setContentType("text/html;charset=utf-8");
                response.setStatus(HttpServletResponse.SC_OK);
                response.getWriter().println(
                    «m.simpleName»(«m.getVariables(httpGet).map[it+', ']
                    .join»target, baseRequest, request, response));
  baseRequest.setHandled(true);
              }
            }
          «ENDFOR»
        ''']
      ]
  
      // enhance the handler methods
      for (m : annotatedMethods) {
        for (variable : m.getVariables(httpGet)) {
          m.addParameter(variable, string)
        }
        m.addParameter('target', string)
        m.addParameter('baseRequest', typeof(Request).newTypeReference) 
        m.addParameter('request', typeof(HttpServletRequest).newTypeReference) 
        m.addParameter('response', typeof(HttpServletResponse).newTypeReference)
      }
    }
  }

  private def getVariables(MutableMethodDeclaration m, Type annotationType) {
    // parses the pattern in @Get and returns the variable names
  }
 
  private def getPattern(MutableMethodDeclaration m, Type annotationType) {
    // replaces the variables in the pattern with (\\w+) groups, so we can use it as a regex pattern.
  }
}

Xtend's New Type System

"We are almost at a stage where only double PhDs in math & comp.sc can qualify for Java compiler writers !"

(Srikanth S. Adayapalam (from Eclipse JDT) on the complexity of type inference in Java 8)

As Xtend implements Java's typesystem, we have to deal with exactly the same amount of complexity ... oh wait it's even better as Xtend has

• advanced type inference,
• sugared feature calls for properties,
• operator overloading,
• an implicit 'it' and
• extension methods (which are not only static as in C#)
• etc.

So you better add a couple more PhDs to the mix.

Write once, run ...

We are used to writing key parts of our software twice (at least!). The old type system did its job in the beginning but as we added more and more corner cases and complex type inference scenarios with lambdas and such it started to become too slow and too complex to be further enhanced. Good we have a solid test suite so we actually noticed the point at which even harmless looking changes broke existing tests... To make a long story short: Last year in April we decided to rewrite it from scratch.

The Lonesome Hero

Sebastian had some very promising ideas around a completely different architecture which would not only improve performance but also would allow everyone to better understand and debug the code again. We couldn't afford stopping all other developments and maintenance so one team member had to take the journey while the others were working on cool new features. Almost a year ago Sebastian started work on a complete rewrite. We discussed language semantics from time to time and such things but I rarely looked at the actual code. Most of the semantics were clear as it is based on the Java Language Specification and could be proved with the different Java compilers. So he was climbing this huge mountain all alone.
I know it was super risky and that it's not a good idea to rely such an important part of a project on just one person. If he had failed we had wasted a year. But I knew he would do it (And if he couldn't I don't know who else could).

It has been a long time and we even had to postpone the release which was initially planned for December. Finally by the end of January the new type system implementation was in a state that it could be activated and replace the old one. The switch was super smooth. It just worked.

Here is the corresponding bugzilla, which has been closed just yesterday. It blocked 95 downstream bugs! Most of them are now fixed, we just had to go through them and test, verify and close them. Not only that, the performance was already as good as the old one in the beginning without any further profiling and improvements. With just a little optimisation we were already able to improve the performance by 25%.

Also worth noting is, that the whole team was able to start working in the new code base after just two days of introduction. The code definitely is much easier to grasp and to debug yet covers much more of the essential complexity. Also we have an even more comprehensive test suite now. With over 20.000 unit tests you can confident that you don't accidentally break something. All tests run against the lower version bounds as well as the upper version bounds of our dependencies. There's no way software like this can be maintained and evolved without such a test suite.

The new type system is an incredible piece of engineering and a huge success. It's the basis for an awesome upcoming release of Xtend with many cool new features and enhancements.
Sebastian, I owe you one and so does the growing Xtend community!

Java 8 vs. Xtend

As you might know Java 8 will finally bring an important new feature to the Java programming language :
Lambda Expressions, a feature already supported by many other languages, such as Xtend

Xtend is a statically typed JVM language that translates to readable Java source code. It's very different to other JVM languages in the sense that aims to be a better Java by better supporting existing Java idioms and fully supporting the existing ecoystem. Xtend doesn't force you to rewrite your existing Java apps or drop your beloved and well-proven frameworks, but instead allows for using existing Java APIs in a much nicer way without any interoperability issues. In contrast to e.g. Scala or Clojure it is more like an extension to Java than a replacement.

Identifying and supporting existing Java idioms is not too hard given the huge existing code base but what about the future? Will Xtend also be a better match for the upcoming Java 8 libraries which were explicitly defined for the new Java 8 lambda syntax?

To find out I simply looked at Brian Goetz’s latest version of State of the Lambda : Libraries Edition and translated the contained Java code examples to Xtend.

In the following you see the code for Java as written by Brian and the translation to Xtend. Both are using the exact same Java 8 APIs and contain the same amount of static type information!

Example 1

Java 8:	shapes.forEach(s -> { s.setColor(RED); });
Xtend:	shapes.forEach[color = RED]

This first example already reveals most of the important differences.

First the general syntax for a lambda expression in Java is using an arrow (->), while Xtend uses squared brackets (like smalltalk). Also in Xtend a lambda expression doesn't need to be passed using braces. Although you could write shapes.forEach([color = RED]) you don't need to.

In Java 8 the body of a lambda is either a block or a single expression. So whenever you need to use a statement you have to add the curly braces and semicolons. In Xtend everything is an expression and the lambda itself is already a block expression.

In Xtend you can omit the declaration of a parameter name. In that case the parameter is called 'it' and is implicit (similarly to 'this'). If you want to give it a name you do it like this : shapes.forEach[ s| s.color = RED]

Unrelated to Lambdas but interesting in this case: Xtend allows to use assignments to call setter methods.

Example 2

Java 8:	shapes.stream() .filter(s -> s.getColor() == BLUE) .forEach(s -> { s.setColor(RED); });
Xtend:	shapes.stream .filter[color == BLUE] .forEach[color = RED]

Not too much new stuff in here. Xtend lets you omit empty parenthesis and you can call a getter using the property's name.

Example 3

Java 8:	List<Shape> blue = shapes.stream() .filter(s -> s.getColor() == BLUE) .into(new ArrayList<>());
Xtend:	val blue = shapes.stream .filter[color == BLUE] .into(new ArrayList)

Here we see type inference in action. While you can use the diamond operator in Java, you can leave it out in Xtend. Also the variable doesn't need to be explicitly typed as the type can be fully inferred from the right hand side.

Example 4

Java 8:	Set<Box> hasBlueShape = shapes.stream() .filter(s -> s.getColor() == BLUE) .map(s -> s.getContainingBox()) .into(new HashSet<>());
Xtend:	val hasBlueShape = shapes.stream .filter[color == BLUE] .map[containingBox] .into(new HashSet)

Note how readable the lambdas get when you can leave out all the cryptic clutter. Let's compare a last example from a later section of the document:

Example 5

Java 8:	List<Album> sortedFavs = albums.stream() .filter(a -> a.tracks.anyMatch(t -> (t.rating >= 4))) .sorted(comparing(a -> a.name)) .into(new ArrayList<>());
Xtend:	val sortedFavs = albums.stream .filter[tracks.anyMatch[rating >= 4]] .sorted(comparing[name]) .into(new ArrayList)

Although the libraries as well as the examples have been written for Java 8, the Xtend code is still much less cluttered with symbols and therefore significantly more readable. Yet the type information is exactly the same! Xtend is just a bit smarter with type inference and is syntactically less rigid.

Besides that and the fact the Xtend compiles to readable Java 5 code, it has many other important features to offer.