Showing posts with label java. Show all posts
Showing posts with label java. Show all posts

Wednesday, September 04, 2013

Better I18n in Java

Internationalization is not the most loved topic for Java developers. This is merely because it makes code less readable and introduces the need to work with property files and constants. As a result there is no static typing in that area and we are in need to get supported by tools, such as IDEs. In this blog post I want to show how this can be elegantly solved with Xtend.

I18n - The JDK Way

The JDK offers well working facilities for externalizing messages and formatting values such as dates and currencies in a proper localized way. For externalizing messages the class ResourceBundle is your friend, as it not only reads and parses property files but also supports flexible way of how property values are looked up based on Locales. The other interesting bit is the MessageFormat hierarchy which contains strategies to turn data types into localized strings.

The provided functionality is very powerful but enforces a cumbersome programming model. The standard way is to have a bunch of property files and whenever you need to get the contained values you use a ResourceBundle which has to be created like this:

ResourceBundle messages = ResourceBundle.getBundle("MyMessages", localeToUse);

Now you obtain the messages using the property keys, which are just strings. To minimize very likely spelling problems and it is good practice to have a declaring all the keys as constants. Then you only have to declare them once in the code and once for each property file.

public class MyMessagesKeys {
  public final static GREETING = "greeting";
  ...
}

Clients can access the values using the constants :

messages.getString(MyMessagesKeys.MY_KEY);

A common requirement and feature is to have variables in the messages which should provided by the program. These arguments can even be of different types, where the MessageFormat types come in. But again the Java compiler doesn't know if there are any arguments, nor how many and of which types they are. So you are back in untyped land and will learn about your errors only at runtime.

Let's see how we can improve the API for the JDK classes, such that we get rid of all the duplication and weakly typed access.

DRY With Xtend

Xtend is not a language which is meant to be used instead of Java, but is an extension to Java. You should use it in places where Java just doesn't cut it, like in this example.

In the following we're going to built a statically typed API to be used from Java by means of an active annotation. The basic idea is, that we make the compiler turn a constant containing a message into a Java facade and a corresponding properties file. Consider the following declaration of messages:

@Externalized class MyMessages {
  val GREETING = "Hello {0}!"
  val DATE_AND_LOCATION = "Today is {0,date} and you are in {1}"
}

This is all the developer should need to declare. Now we implement an active annotation, that turns the field declarations into static methods. It even adds typed arguments for the placeholders in the messages. That is the generated Java provides the following sigatures:

// Generated Java signatures

/**
 * Hello {0}!
 */
public static String GREETING(String arg0) { ... }

/**
 * Today is {0,date} and you are in {1}
 */
public static String DATE_AND_LOCATION(Date arg0, String arg1) { ... }

The actual values are written to a *.properties file and the implementation of the static methods use a ResourceBundler to look them up. So in case you need a locale-specific translation, you only need to add an additional *.properties file. Even that could be easily validated for miss-spelled or non declared keys. But I'll leave this feature as an exercise for other people :-).

Java clients can now access the messages in a statically typed way, and if you are working within Eclipse you even get the default message displayed when hovering over a property.

Building The @Externalized Annotation

Developing the active annotation is relatively easy. First you need to create a separate Java project. Although an active annotation is just library, it has to sit in an upstream project or jar, i.e. cannot be used within the same project. We would run into chicken 'n egg problems during compilation if that was allowed.

In that project create a new Xtend file and name it Externalized.xtend. We first declare the annotation and annotate it with @Active so the compiler is aware of it being an active annotation. We also have to provide a processor class which is executed by the compiler:

@Active(ExternalizedProcessor) annotation Externalized {}

class ExternalizedProcessor extends AbstractClassProcessor {

  override doTransform(MutableClassDeclaration clazz, extension TransformationContext ctx) {
    // To be implemented ...
  }

  override doGenerateCode(ClassDeclaration clazz, extension CodeGenerationContext ctx) {
    // To be implemented ...
  }

}

As you can see two callback methods have been overridden, which correspond to certain phases in the compiler. The first phase we participate in is doTransform in which the annotated classes can be mutated. The second phase doGenerateCode allows us to write to the file system. We will later write the properties files during this phase. The doGenerateCode phase has been introduced in version 2.4.3, so make sure you have updated accordingly if you want to use this.

Step 1: Transforming The Fields Into Methods

We want to turn all field declarations into static methods, so we first iterate over the declared fields of the given class.

override doTransform(MutableClassDeclaration clazz, extension TransformationContext ctx) {

  for (field : clazz.declaredFields) {

    // get the actual value of the field
    val initializer = field.initializerAsString

    // create a message format object for that value
    val msgFormat = try {
      new MessageFormat(initializer)
    } catch(IllegalArgumentException e) {
      field.initializer.addError("invalid format : " + e.message)
      new MessageFormat("")
    }

    // check the syntax and report back in case of problems
    val formats = msgFormat.formatsByArgumentIndex
    if(msgFormat.formats.length != formats.length) {
      field.initializer.addWarning('Unused placeholders. They should start at index 0.')
    }

    // add a method using the field's name
    clazz.addMethod(field.simpleName) [

      // return type is always string and the method is static
      returnType = string
      static = true

      // add parameters for the given arguments in the message format object
      formats.forEach [ format, idx |
        addParameter("arg" + idx,
          switch format {
            NumberFormat: primitiveInt
            DateFormat: Date.newTypeReference()
            default: string
          })
      ]
      
      // add the value as a comment, for documentation purpose
      docComment = initializer
      
      // add the actual body. It's generated Java code.
      val params = parameters
      body = [
        '''
          try {
            String msg = RESOURCE_BUNDLE.getString("«field.simpleName»");
            «IF formats.length > 0»
              msg = «toJavaCode(MessageFormat.newTypeReference)»
                    .format(msg,«params.map[simpleName].join(",")»);
            «ENDIF»
            return msg;
          } catch («toJavaCode(MissingResourceException.newTypeReference)» e) {
            // TODO error logging
            return "«initializer»";
          }
        ''']
    ]
  }

  // now that we have the methods we can just remove the fields, 
  // as they are no longer needed
  clazz.declaredFields.forEach[remove]

  // add a static ResourceBundle to be used from the methods we just created.
  clazz.addField("RESOURCE_BUNDLE") [
    static = true
    final = true
    type = ResourceBundle.newTypeReference
    initializer = ['''ResourceBundle.getBundle("«clazz.qualifiedName»")''']
  ]
}

Now we have successfully transformed an @Externalized class written in Xtend into a Java class containing corresponding static methods.

Step 2: Generating The Properties File

The last thing we need to do now is to write the values of the fields into the properties file. For this we participate in the code generation phase by overriding doGenerateCode. While during doTransform we got a mutable Java representation of the original sources, in this phase we get the unmodifiable original source passed in.

override doGenerateCode(ClassDeclaration clazz, extension CodeGenerationContext ctx) {
    
  // obtain the target folder for the given compilation unit
  val targetFolder = clazz.compilationUnit.filePath.targetFolder
    
  // compute the path for the properties file
  val file = targetFolder.append(clazz.qualifiedName.replace('.', '/') + ".properties")

  // write the contents to the file
  file.contents = '''
    «FOR field : clazz.declaredFields»
      «field.simpleName» = «field.initializerAsString»
    «ENDFOR»
  '''
}

Check Out The New Release

The described example is included in the "active annotations examples" coming with latest release.

Monday, December 17, 2012

Java 8 vs. Xtend

As you might know Java 8 will finally bring an important new feature to the Java programming language :
Lambda Expressions, a feature already supported by many other languages, such as Xtend

Xtend is a statically typed JVM language that translates to readable Java source code. It's very different to other JVM languages in the sense that aims to be a better Java by better supporting existing Java idioms and fully supporting the existing ecoystem. Xtend doesn't force you to rewrite your existing Java apps or drop your beloved and well-proven frameworks, but instead allows for using existing Java APIs in a much nicer way without any interoperability issues. In contrast to e.g. Scala or Clojure it is more like an extension to Java than a replacement.

Identifying and supporting existing Java idioms is not too hard given the huge existing code base but what about the future? Will Xtend also be a better match for the upcoming Java 8 libraries which were explicitly defined for the new Java 8 lambda syntax?

To find out I simply looked at Brian Goetz’s latest version of State of the Lambda : Libraries Edition and translated the contained Java code examples to Xtend.

In the following you see the code for Java as written by Brian and the translation to Xtend. Both are using the exact same Java 8 APIs and contain the same amount of static type information!

Example 1

Java 8:
shapes.forEach(s -> { s.setColor(RED); });
Xtend:
shapes.forEach[color = RED]

This first example already reveals most of the important differences.

First the general syntax for a lambda expression in Java is using an arrow (->), while Xtend uses squared brackets (like smalltalk). Also in Xtend a lambda expression doesn't need to be passed using braces. Although you could write shapes.forEach([color = RED]) you don't need to.

In Java 8 the body of a lambda is either a block or a single expression. So whenever you need to use a statement you have to add the curly braces and semicolons. In Xtend everything is an expression and the lambda itself is already a block expression.

In Xtend you can omit the declaration of a parameter name. In that case the parameter is called 'it' and is implicit (similarly to 'this'). If you want to give it a name you do it like this : shapes.forEach[ s| s.color = RED]

Unrelated to Lambdas but interesting in this case: Xtend allows to use assignments to call setter methods.

Example 2

Java 8:
shapes.stream()
      .filter(s -> s.getColor() == BLUE)
      .forEach(s -> { s.setColor(RED); });
Xtend:
shapes.stream
      .filter[color == BLUE]
      .forEach[color = RED]

Not too much new stuff in here. Xtend lets you omit empty parenthesis and you can call a getter using the property's name.

Example 3

Java 8:
List<Shape> blue = 
    shapes.stream()
          .filter(s -> s.getColor() == BLUE)
          .into(new ArrayList<>());
Xtend:
val blue = 
    shapes.stream
          .filter[color == BLUE]
          .into(new ArrayList)

Here we see type inference in action. While you can use the diamond operator in Java, you can leave it out in Xtend. Also the variable doesn't need to be explicitly typed as the type can be fully inferred from the right hand side.

Example 4

Java 8:
Set<Box> hasBlueShape = 
    shapes.stream()
          .filter(s -> s.getColor() == BLUE)
          .map(s -> s.getContainingBox())
          .into(new HashSet<>());
Xtend:
val hasBlueShape = 
    shapes.stream
          .filter[color == BLUE]
          .map[containingBox]
          .into(new HashSet)

Note how readable the lambdas get when you can leave out all the cryptic clutter. Let's compare a last example from a later section of the document:

Example 5

Java 8:
List<Album> sortedFavs =
    albums.stream()
          .filter(a -> a.tracks.anyMatch(t -> (t.rating >= 4)))
          .sorted(comparing(a -> a.name))
          .into(new ArrayList<>());
Xtend:
val sortedFavs =
    albums.stream
          .filter[tracks.anyMatch[rating >= 4]]
          .sorted(comparing[name])
          .into(new ArrayList)

Although the libraries as well as the examples have been written for Java 8, the Xtend code is still much less cluttered with symbols and therefore significantly more readable. Yet the type information is exactly the same! Xtend is just a bit smarter with type inference and is syntactically less rigid.

Besides that and the fact the Xtend compiles to readable Java 5 code, it has many other important features to offer.

Tuesday, October 23, 2012

Introducing : Active Annotations

In this article I'd like to introduce you to a new important language feature for Xtend. Xtend is a statically-typed language which compiles down to readable Java source code.

Active Annotations in a Nutshell

Active Annotations allow for participating during the translation of Xtend code to Java source code. You basically declare an annotation and with it provide the processing instructions where you can
  • issue errors and warnings,
  • provide corresponding quickfixes,
  • apply all kind of changes and enhancements to the target Java classes,
  • introduce new Java types or
  • even update or create ordinary text files (e.g. web.xml).

To use such an annotation you just have to have it on your class path like any ordinary Java annotation.
You could for instance want to have an annotation called @Entity, which turns simple classes with fields into data classes tailored for your project-specific context. You'd define and maintain the definition of what an entity is right in your project and would version and distribute it just like any other API. The IDE will be fully aware of the processing changes. Not only will errors, warnings and quickfixes be available in the IDE but also the scopes, type computation and content assistance will behave according to your processing instructions.

General Use Cases

Let's have a look at some use cases. Xtend already comes with two annotations, whose processing for now is hard coded into the compiler. We did so, to gain some experience with the general idea and will of course replace the hard coded version with 100% library-based annotations in a future release.
One of these annotations is the @Property annotation, which translates a simple field into a Java-Bean property. That is given the following Xtend code
   // Xtend
   @Property String name
you'll get this Java code :
   // Java
   private String name;
   public String getName() {
      return this.name;
   }
   public void setName(String name) {
      this.name = name;
   }
The other annotation Xtend already has is @Data which translates a class with fields into a value object รก la domain-driven design, i.e. it creates an immutable class with a value-based equals and hashcode implementation and a nice toString implementation. We will likely support other generic use cases such as @Delegate, which generated delegate methods for an annotated field and @Builder which will automatically derive a builder API for a set of classes.

Framework-specific Use Cases

You might have seen pre-built annotations which do participate in the compilation project with other technologies. The real coolness about active annotations in Xtend is that everybody can develop their own project specific ones easily. If you want to change the behavior, just navigate to the annotation and do your changes. You will instantly see the result, since every referencing Xtend file will be re compiled on change. And it is fully transparent since everything is translated to readable Java source code!
With this simplicity to develop and deploy them people will be able to use them to remove tedious structural boilerplate in many Java APIs. We for instance built two active annotations for GWT. One for remote services, which automatically derives the required interfaces and adds the required super class. Another which automatically adds fields declared in a UI-Binder XML file. You can find some slides here and the source code is on github.

Outlook

Today we have a working prototype, which you can play around with if you are brave enough. But there's no documentation and the API changes on a daily bases, so better wait a bit. We will release a beta version of this new feature with the next release (planned for December/January), but this will still lack some important features and the API will be subject to change.

I am very excited and think active annotations really solve a huge class of problems in a very nice way and am really curious with what kind of annotations people will come up in the future.

Alan Kay once said about Lisp that it is not a language but a building material. Active annotations bring the power of Lisp Macros to the statically typed Java world.

Monday, June 18, 2012

Vert.x and Xtend

Vert.x is a framework for asynchonous, scalable, concurrent applications. It’s conceptually very similar to node.js but runs on the JVM leveraging its native support for multiple threads.

Its programming model is based on a tell don't ask paradigm, that is you never wait for responses (and thereby block the current thread), but tell what to do as soon as a response is available by passing a handler. It also isolates state (even static one) so no locking is required.

Vert.x comes with special APIs for Java, JavaScript, Ruby and Groovy. They also plan to support Scala and Python.

So what about Eclipse's Xtend?


Xtend is not just another JVM language but an alternative way to write Java applications. It translates to comprehensible Java source code and unlike other JVM languages Xtend is 100% interoperable with Java and is designed to work great with existing Java APIs. That’s why no special API for Xtend is required.

The language is statically typed and features very advanced IDE support which tightly integrates with Java projects. A release of Xtend with lots of cool new features (e.g. debugging support) is just a couple of days away (June 27).

Some Examples


I have converted the Java examples shipped with vert.x to Xtend. The code is on github and is a fully working Eclipse project including working launch configurations to run and debug all the examples. If you are used to working in Eclipse, this is an easy way to try Vert.x : Just clone & import (using the "Import existing Projects into workspace" wizard) and you are done (Thanks to Doug for explaining the setup).

In the following I want to compare a couple of Java code snippets with their equivalent Xtend code, to demonstrate the expressiveness.

Https Example


The https example consists of two classes one for the client and one for the server. The client simple executed a get request on port 4443 and the server 'localhost' and prints the received data to System.out. Here's the Java code:
// Java
@Override
public void start() {
  vertx.createHttpClient()
    .setSSL(true)
    .setTrustAll(true)
    .setPort(4443)
    .setHost("localhost")
    .getNow("/", new Handler<HttpClientResponse>() {
      public void handle(HttpClientResponse response) {
        response.dataHandler(new Handler<Buffer>() {
          public void handle(Buffer data) {
            System.out.println(data);
          }
        });
      }
    });
  }
The Java API allows to configure the HttpClient using chained calls to setter methods. The last call is the get-request (getNow) where the path as well as a response handler is provided. Instead of blocking the current thread and waiting for the response you just tell what to do when the response is available. The Java API expects an instance of Handler<HttpClientResponse> for that.

Xtend allows to use the very same API with the following code:
// Xtend
override start() {
  vertx.createHttpClient => [
    SSL = true
    trustAll = true
    port = 4443
    host = "localhost"
    getNow("/") [
      dataHandler [ data |
        println( data )
      ]
    ]
  ]
}
First an http client is created, instead of using the chained setters (we could), we use the with-operator (=>) which allows us to use and initialize the http client from the left hand side within a lambda expression (the block in squared brackets). Within the lambda the HttpClient is bound to the implicit variable 'it', which like the self reference 'this' can be omitted when used as a receiver.
Next we call the setters using assignments, that is the expression
    SSL = true
is translated and equivalent to
    it.setSSL(true)
The interesting part is how we pass the response handler in Xtend :
    getNow("/") [
      dataHandler [ data |
        println( data )
      ]
    ]
The call to getNow(String, Handler<HttpClientResponse>) is done by passing just the string in the parentheses and a lambda expression right after the call. You need to understand two things here:
  1. If the last argument of a feature call is a lambda expression it can be passed after the method call.
  2. A lambda expression automatically coerces to the expected target type if it's an interface with just one method (which is a common idiom not only in vert.x).
Within the response handling a data handler is registered by calling setDataHandler(Handler<Buffer>) on the response object. Note that since this method gets just one lambda expression passed you don't have to write the parenthesis (in Xtend empty parenthesis are optional). Also note, that this time we gave the parameter a name (data) since I found it sightly more readable than :
    getNow("/") [
      dataHandler [
        println( it )
      ]
    ]
The Java implementation of the server prints out the header keys to the console and returns a tiny document:
// JAVA
public void start() {
  vertx.createHttpServer()
    .setSSL(true)
    .setKeyStorePath("server-keystore.jks")
    .setKeyStorePassword("wibble")
    .requestHandler(new Handler<HttpServerRequest>() {
      public void handle(HttpServerRequest req) {
        System.out.println("Got request: " + req.uri);
        System.out.println("Headers are: ");
        for (String key : req.headers().keySet()) {
          System.out.println(key + ":" + req.headers().get(key));
        }
        req.response.headers().put("Content-Type", "text/html; charset=UTF-8");
        req.response.setChunked(true);
        req.response.write("<html><body><h1>Hello from vert.x!</h1></body></html>", "UTF-8").end();
      }
    }).listen(4443);
  }
Translated to Xtend and applying the same patterns we used for the client this looks like the following:
// Xtend
override start() {
  vertx.createHttpServer => [
    SSL = true
    keyStorePath = "server-keystore.jks"
    keyStorePassword = "wibble"
    requestHandler [
      println("Got request: " + uri)
      println("Headers are: ")
      for (it : headers.entrySet) {
        println(key + ":" + value)
      }
      response.headers.put("Content-Type", "text/html; charset=UTF-8")
      response.chunked = true
      response.write('''
        <html>
          <body>
            <h1>Hello from vert.x!</h1>
          </body>
        </html>
        ''', "UTF-8").end
      ]
    listen(4443) 
  ]
}
There are two things worth mentioning in addition to what we've discussed in the first example:
  • You can use the variable name it every where. For instance in a for loop like in the code snippet above.
  • Xtend supports multiline string literals. The common literals using single quote or double quote can be multiline as well, but by using triple single quotes you get smart whitespace handling. That is the indentation before the tag will be pruned for all lines, i.e. the result will be well formatted. Also the triple quotes allow for having interpolation expressions, which are not used in this example.

Can we do even more?


Although the standard Java API already works very well there's room for special Xtend API. Vert.x for instance communicates with JSON objects a lot, which you might want to declare easily. Xtend doesn't support native JSON syntax, but you can design powerful APIs using a combination of extension methods and operator overloading. I added just three methods to be able to construct instances of vert.x's JsonObjects like this:
class JsonExample extends Verticle {
  
  override start() {
    val eb = vertx.eventBus
    val pa = 'vertx.mongopersistor'
    val albums = _(
      _(
        'artist'- 'The Wurzels',
        'genre'- 'Scrumpy and Western',
        'title'- 'I Am A Cider Drinker',
        'price'- 0.99,
        'categories'- _('action', 'comedy')
      ),
      _(
        'artist'- 'Vanilla Ice',
        'genre'- 'Hip Hop',
        'title'- 'Ice Ice Baby',
        'price'- 0.01
      ),
      _(
        'artist'- 'Ena Baga',
        'genre'- 'Easy Listening',
        'title'- 'The Happy Hammond',
        'price'- 0.50
      ),
      _(
        'artist'- 'The Tweets',
        'genre'- 'Bird related songs',
        'title'- 'The Birdy Song',
        'price'- 1.20
      )
    )
    
    // First delete everything
    eb.send(pa, _('action'- 'delete', 'collection'- 'albums', 'matcher'- _()))
    eb.send(pa, _('action'- 'delete', 'collection'- 'users', 'matcher'- _()))
    
    // Insert albums - in real life price would probably be 
    // stored in a different collection, but, hey, this is a demo.
    
    for (album : albums) {
      eb.send(pa, _(
        'action'- 'save',
        'collection'- 'albums',
        'document'- album
      ))
    }
    
    // And a user
    eb.send(pa, _(
      'action'- 'save',
      'collection'- 'users',
      'document'- _(
        'firstname'- 'Tim',
        'lastname'- 'Fox',
        'email'- 'tim@localhost.com',
        'username'- 'tim',
        'password'- 'password'
      )
    ))
  }  
}
These are the three extension methods:
  def static <T> Pair<String,T> operator_minus(String key, T value) {
    new Pair(key, value)
  }
  
  def static JsonArray _(Object ... entries) {
    val result = new JsonArray
    for (e : entries) {
      switch e {
        String : result.addString(e)
        Number : result.addNumber(e)
        Boolean : result.addBoolean(e)
        JsonObject : result.addObject(e)
        JsonArray : result.addArray(e)
      }
    }
    return result
  }
  
  def static JsonObject _(Pair<String, ?> ... entries) {
    val result = new JsonObject
    for (e : entries) {
      switch value : e.value {
        String : result.putString(e.key, value)
        Number : result.putNumber(e.key, value)
        Boolean : result.putBoolean(e.key, value)
        JsonObject : result.putObject(e.key, value)
        JsonArray : result.putArray(e.key, value)
      }
    }
    return result
  }

Sunday, June 10, 2012

Xtend - The Movies Example

The upcoming release of the new version of Xtend is just a couple of days away (June 27). Although the technical (OSGi) version is 2.3, for me it's really more like a 1.0 release. It now has everything you need to write beautiful Java programs, like the little movies example we have been using in some workshops we did recently.

The movies example is also included in the example project that will be shipped with the Eclipse plug-in and is about reading a file of movie data in and doing some queries on it.

The Data

The movie database is a plain text file (data.csv) with data sets describing movies. Here's an example data set:

Naked Lunch  1991  6.9  16578  Biography  Comedy  Drama  Fantasy

The values are separated by two spaces. The columns are :

  • title
  • year
  • rating
  • numberOfVotes
  • categories (where any number of categories is allowed)

Let's start by declaring a data type Movie reflecting the data set:

@Data class Movie {
  String title
  int year
  double rating
  long numberOfVotes
  Set categories 
}

The @Data annotation will turn this class into a value object, that is the compiler will create

  • a getter-method for each field,
  • a hashCode()/equals() implementation,
  • implementation of Object.toString() and
  • a constructor accepting values for all fields in the declared order.

Parsing The Data

Let's now define another class which reads the text file into a list of movies so that we can do some analysis on the data. We will access the data from within a JUnit test, so simply initializing a field is appropriate:

import java.io.FileReader
import java.util.Set
import static extension com.google.common.io.CharStreams.*

class Movies {

  val movies = new FileReader('data.csv').readLines.map[ line |
    val segments = line.split('  ').iterator
    return new Movie(
      segments.next, 
      Integer::parseInt(segments.next), 
      Double::parseDouble(segments.next), 
      Long::parseLong(segments.next), 
      segments.toSet
    )
  ]
}

The field's type (List) is inferred from the expression on the right hand-side and we want the field to be final, so we declare it as a value using the keyword code val.

The initialization on the right hand side first creates a fresh instance of java.io.FileReader. Then the method readLines() is invoked on it. But if you have a look at FileReader you won't find such a method. It's in fact a static method coming from Google Guava's CharStream and is imported as an extension :

import static extension com.google.common.io.CharStreams.*

CharStream.readLines(Reader) returns a List on which we call another extension method called map. That one is defined in Xtend's runtime and is always imported and therefore automatically available on all lists. The map-method expects a function as the parameter. It invokes that function for each value in the list and returns a list containing the results of the function invocations.

Function objects are created using lambda expression (the code in squared brackets). Within the lambda we process a single line from the text file and turn it into a movie by splitting the string using the separator and calling iterator() on the result. As you might know java.lang.String.split(String) returns a string array (String[]). But as Xtend auto-converts arrays to lists when needed, we can call iterator() on it.

val segments = line.split('  ').iterator

Now we use the iterator to create an instance of Movie:

return new Movie (
  segments.next, 
  Integer::parseInt(segments.next), 
  Double::parseDouble(segments.next), 
  Long::parseLong(segments.next), 
  segments.toSet
)

Answering Some Questions

Now that we've the text file turned into a List, we are ready to do some queries on it. We use JUnit to make the individual expressions executable.

Question 1: How Many Action Movies Are Contained?:

@Test def void numberOfActionMovies() {
  assertEquals(828, 
    movies.filter[categories.contains('Action')].size)
}

It's using the extension method filter to filter the movies. The lambda expression checks whether the current movie's categories contains the entry 'Action'. Note that unlike the lambda we used to turn the lines in the file into movies, we haven't declared a parameter name this time. We could have given the parameter an explicit name 'movie' by writing the following:

assertEquals(828, movies.filter[movie | movie.categories.contains('Action')].size)

But if we leave out the name and the vertical bar the variable is automatically named 'it' which (like this) is an implicit variable. That's why we can either write

assertEquals(828, movies.filter[it.categories.contains('Action')].size)

or just

assertEquals(828, movies.filter[categories.contains('Action')].size)

Lastly we call size on the resulting iterable, which again is an extension method (java.lang.Iterable doesn't define such a method).

Question 2: What's The Year The Best Movie From The 80ies Was Released.

@Test def void yearOfBestMovieFrom80ies() {
  assertEquals(1989, 
    movies.filter[(1980..1989).contains(year)].sortBy[rating].last.year)
}

Here we filter out all movies where the year is not included in the range from 1980 to 1989 (the 80ies). The range-operator (..) again is an extension defined for two ints and returns an instance of org.eclipse.xtext.xbase.lib.IntegerRange.

The resulting iterable then is sorted by the rating of the movies. Since it's sorted in ascending order, we take the last movie from the list and return its year.

We could have sorted descending and take the head of the list as well (note the minus sign):

movies.filter[(1980..1989).contains(year)].sortBy[-rating].head.year

Btw. the calls to movie.year as well as movie.categories in the previous example of course access the corresponding getter methods, which were generated because of the @Data annotation.

Question 3: The Sum Of All Votes Of The Top Two Movies

@Test def void sumOfVotesOfTop2() {
  assertEquals(47_229, 
    movies.sortBy[-rating].take(2).map[numberOfVotes].reduce[a, b| a + b])
}

First the movies are sorted by rating, then we take the best two. Next the list of movies is turned into a list of their numberOfVotes using the map function. Now we have a List which can be reduced to a single Integer by adding the values.

Thursday, May 31, 2012

Distance / Time = Speed

I blogged about introducing a Distance datatype using Xtend some time ago, which allows you to write the following:


@Test def distances() {
  assertEquals(15.km, 13.km + 2_000.m)
  assertEquals(30.km, (13.km + 2_000.m) * 2)
}


We've enhanced this example a bit and put it into the Xtend tutorial project, which you can materialize into your workspace using Eclipse's example wizard.

We added two more datatypes Time and Speed, so that you now can write things like this:


@Test def speed() {
  assertEquals(42.km/h, (40_000.m + 2.km) / 60.min)
}


The datatype Time is defined similarly to the Distance type. It defines operators and some static methods, which when imported as extension method allows you to write things like 60.min etc.
In order to be able to divide some Distance by some Time (i.e. create Speed), you need to define a static operator like so :


@Data class Speed {
  BigDecimal mmPerMsec

  def static operator_divide(Distance d, Time t) {
    new Speed(d.mm / t.msec)
  }
}


Just import this as an extension method and you are able to write :
  (40_000.m + 2.km) / 60.min

Finally we also want to write Speed literals like 42.km/h.

Can you guess how this is done?

(If not, it's in the example :-)
Eclipse updatesite: http://download.eclipse.org/modeling/tmf/xtext/updates/composite/latest/ )

Monday, May 07, 2012

Martin Fowler's State Machine DSL with Xtext 2.3


In his book on domain-specific languages, Martin Fowler introduces a small example along which he shows different techniques to implement a domain-specific language. The example goes like this:

Imagine you work for a company specialized on developing and installing systems for secret compartments and you have many customers with very different mechanisms. Mrs. H for instance wants to have a secret panel in her bedroom, which can only be opened after the door has been closed, the second drawer in her chest has been opened and the bedside light was turned on. Also the panel should be closed and locked immediately after someone opens the door - no matter in what state the system is in that case.

Martin proposes the following script as an appropriate definition of Mrs. H's secret compartment system:

events
  doorClosed
  drawOpened
  lightOn   
  reset doorOpened
  panelClosed
end

commands
  unlockPanel
  lockPanel
  lockDoor
  unlockDoor
end

state idle
  actions {unlockDoor lockPanel}
  doorClosed => active
end

state active
  drawOpened => waitingForLight
  lightOn    => waitingForDraw
end

state waitingForLight
  lightOn => unlockedPanel
end

state waitingForDraw
  drawOpened => unlockedPanel
end

state unlockedPanel
  actions {unlockPanel lockDoor}
  panelClosed => idle
end

The script starts with two sections which declare the events and the commands. After that the different states are declared. Some of them execute declared commands as a side effect (the actions block). Also they contain declarations of transitions, i.e. to what state the system switches when a certain event is fired.

As I have already implemented this language with previous versions of Xtext, I'd like to make it a bit more interesting this time. Let's replace the more or less useless declaration of commands with the possibility to write and call real code!

events
  doorClosed
  drawOpened
  lightOn   
  reset doorOpened
  panelClosed
end

state idle
  do { 
    println("opened the door.")
    println("locked the panel.")
  }
  doorClosed => active
end

state active
  drawOpened => waitingForLight
  lightOn    => waitingForDraw
end

state waitingForLight
  lightOn => unlockedPanel
end

state waitingForDraw
  drawOpened => unlockedPanel
end

state unlockedPanel
  do { 
    println("opened the panel.")
    println("locked the door.")
  }
  panelClosed => idle
end

As you can see the main change is that you are now able to call Java libraries right from within your state machine. I want these expressions to be statically typed (incl. full support for Java generics) and to keep the code as readable and dense as the rest of the DSL we need to have type inference like in Scala. Feature-wise everything should be supported, from for-loops to conditional logic, from the typical literals to more advanced concepts like lambda expressions (it's 2012 after all). 

But how would we talk to a door resp. panel service in order to open and close them? We could use static methods, but that's bad design since then the application is hardly testable and you cannot easily switch the concrete implementations behind these services. So let's use dependency injection. 
To support that I added a services block to the language where you list all required services. Now we can use these services within the action code :

events
  doorClosed
  drawOpened
  lightOn   
  reset doorOpened
  panelClosed
end

services
  DoorService door
  PanelService panel
end

state idle
  do { 
    door.open
    panel.close
  }
  doorClosed => active
end

state active
  drawOpened => waitingForLight
  lightOn    => waitingForDraw
end

state waitingForLight
  lightOn => unlockedPanel
end

state waitingForDraw
  drawOpened => unlockedPanel
end

state unlockedPanel
  do { 
    door.close
    panel.open
  }
  panelClosed => idle
end

Our language would now be very testable and should integrate nicely with any Java project. 

How Do I Implement Such A Language In Xtext?

To get a full implementation of this DSL, including not only a parser, linker, unparser, etc. but also a compiler generating readable and executable Java code as well as having nice integration in Eclipse, you need to create a fresh Xtext project, using the wizard and define the following grammar:

grammar org.xtext.example.mydsl.MyDsl with org.eclipse.xtext.xbase.Xbase

generate myDsl "http://www.xtext.org/example/mydsl/MyDsl"

Statemachine :
  {Statemachine}
  ('events' 
    events+=Event+ 
  'end')?
  ('services'
    services+=Service*
  'end')?
  states+=State*;

Service :
  type=JvmTypeReference name=ID;

Event:
  resetEvent?='reset'name=ID;

State:
  'state' name=ID
    ('do' action=XBlockExpression)?
    transitions+=Transition*
  'end';

Transition:
  event=[Event] '=>' state=[State];

I don't want to go into a detailed explanation of the grammar language, since that is explained in the documentation. But note, that in order to be able to write full Java types in the service declaration we refer to a library grammar rule (JvmTypeReference). The same applies for the expressions: The library grammar 'org.eclipse.xtext.xbase.Xbase' predefines the full expressions and all we have to do here is to import the grammar in the first line and call the rule XBlockExpression within the rule State.

Now that we have defined the syntax of our language, we need to tell how it is translated to Java. For that matter we write some code, which creates a Java DOM out of the state machine DOM produced by the parser. To make this very readable and concise Xtext comes with an internal DSL implemented in the programming language Xtend. The actual code looks like this:

 def dispatch void infer(Statemachine stm, 
                         IJvmDeclaredTypeAcceptor acceptor, 
                         boolean isPreIndexingPhase) {
   
   // create exactly one Java class per state machine
   acceptor.accept(stm.toClass(stm.className)).initializeLater [
     
     // add a field for each service annotated with @Inject
     members += stm.services.map[service|
       service.toField(service.name, service.type) [
         annotations += service.toAnnotation(typeof(Inject))
       ]
     ]
     
     // generate a method for each state having an action block
     members += stm.states.filter[action!=null].map[state|
       state.toMethod('do'+state.name.toFirstUpper, state.newTypeRef(Void::TYPE)) [
         visibility = PROTECTED
         
         // Associate the expression with the body of this method.
         body = state.action
       ]
     ]
     
     // generate a method containing the actual state machine code
     members += stm.toMethod("run"newTypeRef(Void::TYPE)) [
       
       // the run method has one parameter : an event source of type Provider 
       val eventProvider = stm.newTypeRef(typeof(Provider), stm.newTypeRef(typeof(String)))
       parameters += stm.toParameter("eventSource", eventProvider)
       
       // generate the body
       body = [append('''
         boolean executeActions = true;
         String currentState = "«stm.states.head.name»";
         String lastEvent = null;
         while (true) {
           «FOR state : stm.states»
             if (currentState.equals("«state.name»")) {
               «IF state.action != null»
                 if (executeActions) {
                   do«state.name.toFirstUpper»();
                   executeActions = false;
                 }
               «ENDIF»
               System.out.println("Your are now in state '«state.name»'. Waiting for [«
                 state.transitions.map[event.name].join(', '].");
               lastEvent = eventSource.get();
               «FOR t : state.transitions»
                 if ("«t.event.name»".equals(lastEvent)) {
                   currentState = "«t.state.name»";
                   executeActions = true;
                 }
               «ENDFOR»
             }
           «ENDFOR»
           «FOR resetEvent : stm.events.filter[resetEvent]»
             if ("«resetEvent.name»".equals(lastEvent)) {
               System.out.println("Resetting state machine.");
               currentState = "«stm.states.head.name»";
               executeActions = true;
             }
           «ENDFOR»
           
         }
       ''')]
     ]
   ]
 }


That's all you need. With just the grammar and the Xtend code above you have implemented the full language! I've pushed the full project to github.
This is what the Java code generated for Mrs. H's controller looks like.

What Do You Get?

As already mentioned the expressions are feature complete and statically typed. The compiler can be run from any Java process (e.g. ant, maven, gradle, command line). Also the tooling is just like you would expect: content assist, coloring, hovers, refactorings, dead code analysis and even debugging work out of the box:


Content Assist For Expressions

Call-Hierarchy Integrated in JDT
Content Assist For Types

Dead Code Analysis