Sven Efftinge's Blog

I like free beer!

2012-01-11T09:05:00.003+01:00

Did you know about http://99-bottles-of-beer.net?
They list programs generating lyrics for a song about beer in 1440 different programming languages (it will be 1441 very soon ;-)). These are the lyrics to generate.

So here is a version written in Xtend:

It uses a couple of interesting features:

Seriously, I think this is one of the most concise and readable versions. Compare yourself.

Collection Literals in Java (or a hidden gem in Eclipse JDT)

2012-01-04T09:57:00.002+01:00

Yesterday I watched a devoxx presentation by Joshua Bloch about the Evolution of Java" which I enojoyed.
In that talk he discusses the various language features which have been added to Java in terms of how important resp. problematic their addition was. I agree with most of his reasoning but not with the following two:

static imports
He said they are ok, but not too useful and that he makes use of them only about once a month.
diamond operator
He claimed, that this was an important addition, since now creating new collections is less noisy.

I disagree for the following reasons

Joshua Bloch actually also helped designing a very cool collection library which is part of Google Guava. If you don't know it, have a look! Besides other useful stuff that library contains factory methods for the common collection types.
For instance to create an ArrayList you could write the following:

List<String> names = Lists.newArrayList();

Making use of a static import one can write something like this:

import static com.google.common.collect.Lists.*;
...
List<String> stuff = newArrayList("Foo","Bar","Baz");

But the inconvenience of adding a static import every time you want to create a new collection is a problem. JDT to the rescue! There is an almost hidden gem in JDT, that allows you to have static import favorites for content assist. This is my configuration:

Having your static imports configured here, will make their methods available through code completion without having an import first. Instead the import will automatically be added as you apply the proposal.

I only create collections that way, because it's so easy and readable, and I don't have much use cases for the diamond operator (in fact inference on the left hand side would have been much more interesting). On the other hand I use static imports in almost any class file.

Btw, those static imports not only make the factory methods available but also all the nice higher-order functions like, collect, filter, etc..
A welcome productivity gain!

Upcoming Presentations on Xtext / Xtend

2012-01-03T09:25:00.002+01:00

There will be a couple of opportunities to learn about the Xtext / Xtend technology stack in the next couple of months. Here are the dates:

Hope to see you there!

Groovy, Scala, Java, Xtend - an UPDATED stupid comparison

2011-12-21T10:19:00.002+01:00

This is an update to yesterday's post, based on feedback I got so far:

Measurement

Henning suggested that the number of iterations is too small:

10000 iterations are probably not enough for a comparison: If I go from 10k to 100k iterations, the Scala version needs 5x time -- the Java version only 2x.

Indeed. Here's what I got when using 1M iterations:

Java (the pretty version):

Took : 245 ms, number of elements : 3000000

Xtend :

Took : 278 ms, number of elements : 3000000

Xtend (using switch - see below) :

Took : 298 ms, number of elements : 3000000

Scala (using for comprehensions) :

Took : 5140 ms, number of elements : 3000000

Scala (using while loops) :

Took : 931 ms, number of elements : 3000000

Scala (using while loops and java.util.collections - see below) :
```
Took : 315 ms, number of elements : 3000000
```

Groovy (using << operator - see below):

Took : 1683 ms, number of elements : 3000000

Groovy

Thorsten and Jochen pointed out that I used the wrong operator for adding new elements to a list. I should use '<<' instead of '+=' since the latter would create a new list for the single elements before adding them. I've fixed this and the Groovy program is much faster now.

Jochen also pointed out that the comparison is unfair since, because of the dynamic dispatch Groovy does heavy caching assuming that a method is invoked for the same type of argument most of the time.

My initial idea was to do a comparison where Groovy has a strength (i.e. dynamic dispatch). After all the other examples also do dispatching at runtime. The Java and Scala version don't call another method, but since Xtend does so and is as fast as Java I'd say this extra call doesn't change much.

Jochen also states:

Imagine a class with those three foo methods written in Java. Then in Groovy you can still run your test. I dare to say that in Xtend you cannot make this work without a dispatcher method somewhere - and your dispatch keyword is implying that - thus I assume you cannot make this then work using a Java class.

I think this is actually a big problem with Groovy. Although it's syntax might look a bit like Java it behaves totally different.
In contrast Xtend binds statically just like Java and in fact a multi method in Xtend behaves always the same no matter if you call it from Java or Xtend.

He further says

But let us not stop here... let us assume that you write the base class in Java, where it knows foo(String) foo(Object) and foo(Boolean). Using this class the Integer variant would call the foo(Object) method in your code. Now extend that class in Java with a foo(Integer) method and Groovy would no longer call foo(Object) but foo(Integer).
I doubt you can easily simulate that case.

I can indeed not do that in Java. But I can do that with Xtend and it still behaves as expected no matter form what language you'd call that code.

Scala

Jan said:

I rewrote your little program to use while loops and the java utils collections and the scala version was as fast as java - but nearly as ugly to :-D

Here's the updated Scala version with while and java.util.collections, which is now almost as fast as the Java version:

object Scala extends Application {
  val absoluteResult = new ArrayList[Any]()
  val before = System.currentTimeMillis()
  var i =0
  while (i<1000000) {
    i = i+1
    val result = List("foo", 23, true).iterator
    while (result.hasNext) {
      absoluteResult.add(foo(result.next))
    }
  }
  println("Took : "+(System.currentTimeMillis() - before)
      +" ms, number of elements : "+absoluteResult.size)
  
  def foo(obj : Any) =
    obj match {
          case _:String => "String"
          case _:Boolean => "Boolean"
          case _:Integer => "Integer"
          case _ => throw new IllegalArgumentException()
    }
}

Clojure

Stefan did a nice little example in Clojure. It doesn't use any mutable collections but uses a pure functional style:

(defn foo [x]
(cond
(number? x) "Number"
(string? x) "String"
(= (class x) Boolean) "Boolean"))

(time (println (count 
(map foo 
(apply concat 
(repeat 10000 ["foo" 23 true]))))))

On his machine the Java example takes '31ms' and the Clojure program prints

Prints: 30000 "Elapsed time: 194.046477 msecs"

Xtend

An alternative way to do the dispatch in Xtend is using type guards in switch,
which doesn't make a huge difference performance-wise :

class XtendWithSwitch {
  def static void main(String[] args) {
    val absoluteResult = newArrayList()
    val before = System::currentTimeMillis()
    for (times : 0..1000000) {
      val result = newArrayList('foo', 23, true)
      for (y : result) {
        absoluteResult += foo(y)
      }
    }
    println("Took : "+(System::currentTimeMillis() as int - before)
      +" ms, number of elements : "+absoluteResult.size)
  }

  def static foo(Comparable comp) {
    switch(comp) {
      String : 'String'
      Boolean : 'Boolean'
      Integer : 'Integer'
      default : throw new IllegalArgumentException()
    }  
  }
}

Java

Finally based on Henning's suggestion I beautified the Java version a bit using Guava and static imports:

public static void main(String[] args) {
  List<Object> absoluteResult = newArrayList();
  long before = System.currentTimeMillis();
  for (int i=0; i < 1000000; i++) {
    for (Object y : newArrayList("foo", 23, true)) {
      absoluteResult.add(foo(y));
    }
  }
  System.out.println("Took : "+(System.currentTimeMillis() - before)
    +" ms, number of elements : "+absoluteResult.size());
}

static String foo(Object s) {
  if (s instanceof String) {
    return "String";
  } else if (s instanceof Boolean) {
    return "Boolean";
  } else if (s instanceof Integer) {
    return "Integer";
  } else {
    throw new IllegalArgumentException();
  }
}

Groovy, Scala, Java, Xtend - a stupid comparison

2011-12-20T19:29:00.005+01:00

Disclaimer: This is probably the worst possible way to compare execution performance. I'm posting it nonetheless since the result was significant. Judge for yourself.

Today I took some time to check out the latest Eclipse plugins for both Groovy and Scala. I mainly wanted to know how nice the tooling is and played around with it a bit. I had been doing some Scala a couple of years ago, but since the Eclipse plugin not even existed those days it was a bit too painful for me to get used to it.

Groovy's tooling looks very good. There are just a few oddities but mainly I think it's well done... in contrast to the Scala plugin (I used 2.0.0 RC4). It holds a lot of surprises. For instance, with the Scala-IDE installed you'll get Scala-related template proposals in the Java and the Groovy editor. Still Scala the language would be my personal choice if I had to decide between the two. Anyway, this post is not about language features or Eclipse-plugin quality, but about what I experienced regarding runtime performance.

I knew Groovy's performance is bad. But I expected the performance problems to diminish, if I use a scenario using dynamic dispatch and mostly untyped objects. So I wrote the following brain dead code:

class Groovy {

  public static void main(String[] args) {
    def absoluteResult = []
    def before = System.currentTimeMillis()
    for (times in 1..10000) {
      def result = ['foo', 23, true]
      for (y in result) {
        absoluteResult += foo(y)
      }
    }
    println("Took : "+(System.currentTimeMillis() - before)
      +" ms, number of elements : "+absoluteResult.size);
  }

  static String foo(String s) {
    'String'
  }
  
  static String foo(Boolean s) {
    'Boolean'
  }
  
  static String foo(Integer s) {
    'Integer'
  }

}

It doesn't do anything interesting but leverages Groovy's built-in way of resolving overloaded methods using the runtime type. I thought it's fair to use this feature in a comparison, also because Xtend's dispatch methods have the same behavior. The result of the code above would be a list containing the string 'String', 'Boolean', and 'Integer' each 10.000 times.

When I run this code on my machine, it prints something like the following:

Took : 2025 ms, number of elements : 30000

A possible Xtend version

This is how I could achieve the same using Xtend:

class Xtend {
  def static void main(String[] args) {
    val absoluteResult = newArrayList()
    val before = System::currentTimeMillis()
    for (times : 1..10000) {
      val result = newArrayList('foo', 23, true)
      for (y : result) {
        absoluteResult += foo(y)
      }
    }
    println("Took : "+(System::currentTimeMillis() - before)
           +" ms, number of elements : "+absoluteResult.size)
  }

  def static dispatch foo(String s) {
    'String'
  }
  
  def static dispatch foo(Boolean s) {
    'Boolean'
  }

  def static dispatch foo(Integer s) {
    'Integer'
  }
}

On a first glance it looks very similar. Note, the dispatch keyword, which turns a set of overloaded methods into dynamically dispatched methods (i.e. the default Groovy behavior).
In contrast to the Groovy version this one is fully statically typed and as it turned out much faster. (Again as you see my measurement technique is everything but professional, but given such a big difference I dare to say it's faster).

It prints something like:

Took : 43 ms, number of elements : 30000

I also did a Scala version:

object Scala extends Application {
  val absoluteResult = MutableList[Any]()
  val before = System.currentTimeMillis()
  for (times <- 0 until 10000) {
    val result = List("foo", 23, true)
    for (y <- result) {
      absoluteResult += foo(y)
    }
  }
  println("Took : "+(System.currentTimeMillis() - before)
      +" ms, number of elements : "+absoluteResult.size)

  def foo(obj : Any) =
    obj match {
          case _:String => "String"
          case _:Boolean => "Boolean"
          case _:Integer => "Integer"
          case _ => throw new IllegalArgumentException()
    }
}

I got this kind of result:

Took : 130 ms, number of elements : 30000

And finally a Java version:

public class Java {
  public static void main(String[] args) {
    List<Object> absoluteResult = new ArrayList<Object>();
    long before = System.currentTimeMillis();
    for (int i=0; i < 10000; i++) {
      List<Object> result = new ArrayList<Object>();
      result.add("foo");
      result.add( 23);
      result.add(true);
      for (Object y : result) {
        absoluteResult.add(foo(y));
      }
    }
    System.out.println("Took : "+(System.currentTimeMillis() - before)
      +" ms, number of elements : "+absoluteResult.size());
  }

  static String foo(Object s) {
    if (s instanceof String) {
      return "String";
    } else if (s instanceof Boolean) {
      return "Boolean";
    } else if (s instanceof Integer) {
      return "Integer";
    } else {
      throw new IllegalArgumentException();
    }
  }
}

Which prints:

Took : 40 ms, number of elements : 30000

Writing Android UIs with Xtend

2011-12-12T14:46:00.014+01:00

(Update: You can find the project on github : https://github.com/svenefftinge/xtend-android-experiments )

Xtend is a great fit for Android development. Since Xtend translates to Java source code, it's very easy to use it. After you've installed the Xtend SDK and the Android Development Tools you only need to do two things to get started.

In Eclipse's preferences configure Xtend's compiler to generated to the gen/ folder.
Add the Xtend lib and Google Guava to your project's classpath.

Why is it useful?
Using a combination of Xtend's powerful extension methods and closures, it's possible to come up with a very declarative API to define UI models. It takes a couple of minutes to write an API which allows for defining a UI like the following:

class AppActivity extends Activity {
 
  override void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
  
    contentView = this.linearLayout [
   
      orientation = VERTICAL
  
      view = this.textView ("Hello Android!")
  
      view = this.button ("Click Me!") [
        onClickListener = [ 
          this.textMessage('Hello you clicked me!').show
        ]
      ]
    ]
  } 
}

The API is made available by means of a static extension import. To outline the idea, here's how the button method is defined:

def static button(Context context, String text,
      (Button)=>void initializer) {
  val result = new Button(context)
  result.text = text
  initializer.apply(result)
  return result
}

Used as an extension method button can be invoked on any instance of Context. Since the last argument is a function type it can be passed after the actual method invocation:

view = this.button ("Click Me!") [
        onClickListener = [ 
          this.textMessage('Hello you clicked me!').show
        ]
      ]

Also note how you can simply assign a closure to onClickListener (which is btw. invoking setOnClickListener()). The compiler will automatically convert it to an abstract class of View.OnClickListener.
And everything is 100% statically typed!

What’s So Special About Xtend’s Extension Methods?

2011-11-22T09:39:00.004+01:00

You should read this, if you are :

interested in new programming language concepts
understood how great dependency injection is
want to learn about some uniqueness in Eclipse Xtend

Extension Methods

Extension methods enable you to "add" methods to existing types without creating a new derived type, recompiling, or otherwise modifying the original type. In C# extension methods are a special kind of static method, but they are called as if they were instance methods on the extended type. (Wikipedia November 2011)

In Xtend you could, for instance, add a new method to java.util.List like this:

def static <T> T head(List<T> list) {
   for (element : list)
     return element 
  return null 
}

Which then can be used as an extension method on any instance of Iterable<T>:

newArrayList(“foo”,”bar”).head()

Using the prefix notation is much closer to how we think, read, and write. From left to right that is. Also the IDE can now provide useful proposals because the receiver is known. The readability gets even better if you use chained method invocations. For instance, let's assume you want to filter the list before obtaining the head. Using the regular infix syntax known from Java the code would like like that :

head(filter(newArrayList(“foo”,”bar”) , [e | e != “foo”]))

But reading inside out doesn’t seem to be the best way to understand what’s going on. So better use extension methods:

newArrayList(“foo”,”bar”).filter(e | e != “foo”).head

Much better, isn't it?

Static Methods Are Bad!

But wait, don’t extension methods advocate the heavy use of static methods? And isn’t that bad coding style?

Yes, often it is.

With static methods you’ll bind your code not only to the signature of a certain method but to the actual implementation. There’s no way (aside from byte code manipulation) to change the implementation without touching the client code. That’s probably not a big deal with the kind of extension methods I just showed. But what if you want to have DAO-like methods available on your domain model types? Accessing them in a static way seems totally uncool, since we wouldn’t be able to run the code with mocks or exchange the database layer easily or just fix or change something later on. That's why static methods are a bad choice most of the time.

Typically you would want to use a dependency injection container to have such services injected. This is where Xtend’s extension methods are different from the one you find in C#.

Xtend allows to use methods from local fields as extension methods.

Let’s assume we have the following Java interface:

interface SaveSupport { 
  void save(Entity entity);
 }

And given we have some domain model type Person which implements Entity, you can write the following code:

class Controller {  
  @Inject extension SaveSupport
    def testStuff() {
     val p = new Person() 
    p.name = “Fred Flintstone” 
    p.save // translated to ‘this._saveSupport.save(p)’
   }
 }

That’s a big deal, because now you can use an object oriented coding style but keep your domain model free from layer specific code at the same time! And everything is easily testable and can be run in different scenarios using different DI configuration. There are other unique language features in Xtend, like the template expression, but the combination of extension methods and a good dependency injection framework (Guice) really changes how you structure your software system and reason about it.
 

New Xtend Website

2011-11-04T09:42:00.003+01:00

With the recent release of Xtext 2.1, also the language Xtend got improved in various ways.

Today we announce a new landing page for Xtend:

xtend-lang.org

So go find out about this little Java-enhancer and if you happen to be at EclipseCon Europe today

make sure to attend Sebastian's talk on Xtend.

Xtend - A Powerful Tool For Everyday Programming

Theater Stage at 2 p.m.

Build Your Own JVM Language!

2011-11-01T23:21:00.000+01:00

Xtext has already simplified development of domain-specific languages a lot. From a simple grammar definition you not only get a full language infrastructure but also powerful Eclipse-based tool support.
Now it's getting even better.

Today we release Xtext 2.1, which comes with surprisingly easy to use support for defining little languages targeting the JVM (Java Virtual Machine). You only need to define two files:

the grammar
a mapping to Java concepts

The first one is not new. It's still the same concise and readable grammar language we always had in Xtext.

The second script is the big deal: it allows you to map your language concepts to any number of Java types, fields and methods. By doing that you implicitly tell the framework what it needs to know to provide you with a fully working IDE including an incremental compiler!

We have written a nice little tutorial explaining the five steps needed to get a language like the following:

import java.util.List

package my.model {

    entity Person {
        name: String
        firstName: String
        friends: List<Person>
        address : Address

        op getFullName() : String {
            return firstName + " " + name;
        }
        
        op getFriendsSortedByFullName() : List<Person> {
            return friends.sortBy( f | f.fullName);
        }
    }
    
    entity Address {
        street: String
        zip: String
        city: String
    }
}

As you can see, it supports all kinds of advanced features such as Java generics and full expressions even including closures. But you not only get a working parser, linker and compiler but a fully-working high-end Eclipse plug-in on top! The screencast demos the resulting language and IDE and briefly shows how that can be built. Be creative and have fun!

This Is Not Your Father's Java

2011-10-25T19:30:00.003+01:00

The Java language and so its community has traditionally been relatively conservative when it comes to API design and language features. After all Java seems to be the very last language on earth without closures! Also a lot of other language features are highly demanded by many people.

With limits come creativity!

To compensate for the verbose language and missing convenience, the Java community created the three single best IDEs in the world. Also the popularity and maturity of code generation and language add-ons like AspectJ or even XML stems from the limitations in Java as a language. With the rise of alternative languages for the JVM, Java folks started to think out of the box. Modern API design is pushing Java to its limits and with some creativity you can even emulate missing language features.

Next week at EclipseCon Europe I'll be giving a talk on creative API design in Java.

The talk will be structured like a quiz: I show a working piece of Java code to the audience and explain what it does. Then you tell me how it does so. So it's a slightly similar structure to the popular Java Puzzlers talks but in this case the puzzlers are actually unconventional but useful idioms. I expect a lively discussion about the pros and cons of the various idioms and I'm almost sure there is something new for everybody. Feel challenged? Come to the session! :-)

This Is Not Your Father's Java - Wednesday November 2nd, 17:00 - 17:30

Xtend 2.1 - Type-safe Groovy-style Builder API

2011-10-20T10:20:00.006+01:00

The upcoming release of Xtend (Version 2.1, November 2nd) is going to be a huge leap forward. Besides the many new editor features, we managed to improve the performance of linking and type inference by many times. The editing experience is now very close to what you are used to from Eclipse's Java tooling. We are working on a comprehensive New & Noteworthy page where all the cool new features will be listed. Everything will be ready for EclipseCon Europe on November 2nd.

In this post I want to focus on the new language features we added. There are mainly two of them, which are both interesting for themselves but even more so in combination.

The Implicit Variable 'it'

If you name your variable 'it' all its members are accessible without qualifying them with the variable reference (just like 'this').

Example:


def appendFoo(StringBuilder it) {
  append('foo')
}

Also note that variables with the name 'it' are the only ones which can be overridden in lexical scopes. So you could for instance write the following, where the closure shadows the outer variable it.


def foo(List<String> it) {
  foreach( it | toUpperCase )
}

You should understand the other new language features to see how this is meant to be used.

Sugar For Closures With One Argument

Many languages have special syntactic sugar for closures with just one argument. So does Xtend 2.1. The general syntax for a closure with one argument is :


[Type arg | expression-using-arg ]

With zero arguments it still is


[| expression-using-only-outer-context ]

And the new sugared version for closures with one argument is:


[expression-using-it]

And of course the name of the implicit argument is 'it'.

Closures As Last Arguments

We added another feature, which is common in many other popular languages: If you invoke a method where the last argument is a closure you can place the closure after the
actual method invocation.

Example:


newArrayList('a','b','c').foreach [ toUpperCase ]

Groovy-style Builder Syntax

Using these three language enhancements in combination allows for defining statically-typed groovy-style builder APIs. Check the screencast to see it in action.

If you are attending EclipseCon Europe and would like to know more about the possibilities of Xtend, make sure to go to Sebastian's session.

Riding the Waterfall

2011-08-15T11:00:00.001+01:00

Did you know we are working on a research project called “Xbase”? It’s a two year project which is going to end in May 2012. So far we are absolutely in time with everything we had written down in the project plan two years ago. We've developed Xbase itself and Xtend on top of it. Next up in the plan is a textual language for EMF Ecore files as well as a validation language. Both are based on and leveraging Xbase. Also in the plan is support for debugging for Xbase-based languages, evaluating the usefulness of Xbase in the context of web-development and extensive documentation. So here are the details for the next couple of months:

Xcore

Ed already blogged about it: It’s going to be the primary syntax to design Ecore models and it will not only come with decent tooling to define the structural elements of an Ecore model. It will allow you to express behavior of EOperations and derived EStructuralFeatures using Xbase. It’s still in a very early state but we’ll have something cool to show at EclipseCon Europe (as long as the talk gets accepted that is ;-))

Validation language

The validation language is developed together with our friends from Paranor. It will focus on a declarative description of compiler issues, including the messages and severities and will automatically allow language users to change the severities on a per project level.

Debugging

Developing with DSLs includes debugging them. We want to allow debugging any Xbase-based DSL right within the Java debugger. The idea is that you can debug through the underlying Java but at the same time (i.e. during the same debugging session) you can also step over the original DSL code. This might be best explained with the small prototype I created a couple of weeks ago.

Evaluation in the context of web development

To proof the relevance of Xbase and Xtext we have put this evaluation task into the project plan. The idea is to write a small web application framework, which might not be as general as something like Spring Roo or the Play! Framework but is much more concrete than the domain model example we ship. Think of it as a comprehensive real world application of Xtext 2.0. The project plan states that this is going to be open-sourced :-)

Extensive Documentation (The Xtext book)

The Xbase project plan says we are going to work a couple of months on extensive documentation. The best form would be a printed book. So that’s what we want to do.

Maintenance

We have just added Xbase to the already relatively large Xtext project. We’ve also added Xtend, a statically typed programming language, and plan to have a couple of other new components as well. But we cannot just throw stuff in all the time. We need to make sure that the overall quality is improved steadily and this gets harder as the framework grows. The good thing is, that we won’t do (we can’t since we published API) big changes to the core framework as we did in previous years. But still there’s enough to do and even the further improvement of the runtime and the editor parts of Xbase, Xtend and Xtext itself is written down in the project plan. Isn't waterfall planning a great thing? ;-)

BTW: Xtext 2.0.1 and Xtext 2.1.0

A service release has been available for two weeks now and the 2.1 release is planned for mid October. It will contain a lot of bugfixes, a more stable version of the reimplemented serializer, and more performance improvements (we are still working on that). It will also come with one or the other enhancement. It’s best to check bugzilla to find out about the details. We use the SR2 flag for it.

Nordic Coding Part III

2011-08-04T13:25:00.000+01:00

The third Nordic Coding event is taking place on August 19 in Kiel, Germany.

I think, we again managed to come up with a very nice program. Here it is:

JavaScript - The World's Most Misunderstood Programming Language (Thomas Bahn)
JavaScript can be a very nice dynamically typed, functional language if you use it properly. Thomas is going to show us what the language's cool core features like closures and prototypes look like and how to use them.

Scala rocks! An Introduction to Scala (Arno Haase)
Everybody is talking about Scala these days. So it's time to have some introduction to this nice functional language at Nordic Coding. I know Arno loves the language and he can be very passionate about things he likes :-)

Java SE - The road forward, 2011 edition (Dalibor Topic)
The State of the Union of the Oracle Java SE platform group. In this presentation you'll learn more about where we are now, how we got there and where we are going in the future.

So I hope you like such a language-oriented program as much as I do.
Like always DiWiSH will serve some food and drinks after the last talk.
Hope to see a lot of you there!
Here is the link to the official announcement (in German)

Upcoming Eclipse Democamps Featuring Xtext

2011-06-06T08:01:00.001+01:00

Have you yet checked whether there is an Eclipse DemoCamp for Indigo near your place?

There is a lot of Xtext coverage in DemoCamps in Europe and one in the US this month. So if you want to learn about Xtext 2.0 and its new features, you should consider to come to one of the following events.

June 7th	Frankfurt, Germany	Xtext 2.0 (with me)
June 11th	Krakow, Poland	Xtext Demo (Marcin Lewandowski)
June 14th	Bonn, Germany	Xtext 2.0 (Jan Koehnlein)
June 17th	Vienna, Austria	Xtext 2.0 (Sebastian Zarnekow)
June 20th	Munich, Germany	Xtext 2.0 (Sebastian and me)
		APPlause (built with Xtext) (Peter Friese)
June 20th	San Jose, USA	Xtext 2.0 (Moritz Eysholdt)
June 21st	Dresden, Germany	Xtext 2.0 (Jan Koehnlein)
June 21st	Darmstadt, Germany	APPlause (built with Xtext) (Marcus Ficner)
June 28th	Hamburg, Germany	Xtext 2.0 (me)
June 28th	Amsterdam, Netherlands	Xtext 2.0 (Christophe Bouhier)
June 29th	Berlin, Germany	APPlause (built with Xtext) (Heiko Behrens)

In addition to that Sebastian and I are giving a talk at Jazoon in Zurich, Switzerland on June 21st. I'm looking forward to see many of you at one of these events!

5 simple steps to Fowler's DSL with Xtext 2.0

2011-06-03T08:13:00.002+01:00

You might have read (about) the book Domain-Specific Languages written by Martin Fowler. As the name suggests it is about these little useful programming languages you can built very easily with Xtext. Unfortunately Martin decided to use more low level technology in order to explain how to implement such a language. So if you don't want to spend hours of hours dealing with complex low level technology just to end up with a parser but nothing else, here are the five steps needed to get a fully working implementation for the DSL he uses throughout the book (Mrs Grant's Controller). It doesn't get simpler.

Below you can find a detailed description. For those of you who prefer to just sit and watch, here are three screencasts I've recorded around this example.

Creating the project and defining the language

Defining the code generator and see its integration

Features, Features, Features

1) Download Eclipse and Install Xtext

For many people this is the hardest part. But believe me things have improved and it is now quite easy to get a working Eclipse distribution including the latest milestone of Xtext 2.0. :-)

Download an Eclipse Classic distribution for your platform from here :
http://www.eclipse.org/downloads/index-developer.php

Unzip and start Eclipse, choose Help -> Install New Software... and use the update site
http://download.itemis.com/updates/milestones
in order to install the Xtext SDK feature

2) Use the Xtext project wizard

Select New -> Project..., and within the dialog choose Xtext Project. You can stick with the defaults on the next page, but make sure you choose the Experimental Features configuration in order to get the latest and greatest features, such as rename refactoring.

You should use the experimental features

3) Define the grammar

The Xtext grammar file is automatically opened. Just copy and paste the following grammar definition for Martin's statemachine DSL in.

grammar org.xtext.example.mydsl.MyDsl with org.eclipse.xtext.common.Terminals

generate statemachine "http://www.xtext.org/example/secretcompartments/Statemachine"

Statemachine :

{Statemachine}

('events'

events+=Event+

'end')?

('resetEvents'

resetEvents+=[Event]+

'end')?

('commands'

commands+=Command+

'end')?

states+=State*

;

Event:

name=ID code=ID

;

Command:

name=ID code=ID

;

State:

'state' name=ID

('actions' '{' actions+=[Command]+ '}')?

transitions+=Transition*

'end'

;

Transition:

event=[Event] '=>' state=[State]

;

4) Run the generator

Now you need to run the generator which creates the parser and a couple of other parts of the language infrastructure. To do so right click into the editor and choose

Run As -> Generate Xtext Artifacts

5) Define the code generator

A DSL isn't worth much if you are not able to execute it somehow. Xtext has already provided you with a stub for a code generator written in our cool new template language Xtend (Xtend as well as the grammar language are also implemented with Xtext of course).

To get a working implementation which produces a Java implementation of a defined statemachine, just copy the following into the file MyDslGenerator.xtend.

package org.xtext.example.mydsl.generator

import org.eclipse.emf.ecore.resource.Resource

import org.eclipse.xtext.generator.IGenerator

import org.eclipse.xtext.generator.IFileSystemAccess

import org.xtext.example.mydsl.statemachine.Statemachine

import org.xtext.example.mydsl.statemachine.Event

import org.xtext.example.mydsl.statemachine.Command

import org.xtext.example.mydsl.statemachine.State

class MyDslGenerator implements IGenerator {

override void doGenerate(Resource resource, IFileSystemAccess fsa) {

fsa.generateFile(resource.className+".java", toJavaCode(resource.contents.head as Statemachine))

}

def className(Resource res) {

var name = res.URI.lastSegment

name.substring(0, name.indexOf('.'))

}

def toJavaCode(Statemachine sm) '''

import java.io.BufferedReader;

import java.io.IOException;

import java.io.InputStreamReader;

public class «sm.eResource.className» {

public static void main(String[] args) {

new «sm.eResource.className»().run();

}

«FOR c : sm.commands»

«c.declareCommand»

«ENDFOR»

protected void run() {

boolean executeActions = true;

String currentState = "«sm.states.head.name»";

String lastEvent = null;

while (true) {

«FOR state : sm.states»

«state.generateCode»

«ENDFOR»

«FOR resetEvent : sm.resetEvents»

if ("«resetEvent.name»".equals(lastEvent)) {

System.out.println("Resetting state machine.");

currentState = "«sm.states.head.name»";

executeActions = true;

}

«ENDFOR»

}

private String receiveEvent() {

System.out.flush();

BufferedReader br = new BufferedReader(new InputStreamReader(System.in));

try {

return br.readLine();

} catch (IOException ioe) {

System.out.println("Problem reading input");

return "";

}

'''

def declareCommand(Command command) '''

protected void do«command.name.toFirstUpper»() {

System.out.println("Executing command «command.name» («command.code»)");

}

'''

def generateCode(State state) '''

if (currentState.equals("«state.name»")) {

if (executeActions) {

«FOR c : state.actions»

do«c.name.toFirstUpper»();

«ENDFOR»

executeActions = false;

}

System.out.println("Your are now in state '«state.name»'. Possible events are [«

state.transitions.map(t | t.event.name).join(', ')»].");

lastEvent = receiveEvent();

«FOR t : state.transitions»

if ("«t.event.name»".equals(lastEvent)) {

currentState = "«t.state.name»";

executeActions = true;

}

«ENDFOR»

}

'''

}

Code Generation 2011

2011-05-16T07:28:00.000+01:00

Have you heard of Xtend, yet? It's the successor to the template language Xpand. It's implemented with Xtext and it's based on Xtext's new embeddable expression language Xbase.
Xtend's main purpose is to be the most concise and most productive way to write code generators.

Of course, the best conference to present such a language is "Code Generation 2011" in Cambridge, UK. A nice familial conference, taking place next week, with lots of very friendly people all interested in one topic. This year's keynotes are given by the authors of two of the main libraries Xtext is based on: my friend Ed Merks, the original author and lead of EMF, and Terence Parr, the author of the Antlr Parser Generator.

Terence has also developed a new version of his template language (StringTemplate) which he is hopefully able to demo next week. Our friends from Obeo will have a talk about Acceleo which is an OMG standards based template language. They do a pretty good job in compensating the little freedom in language design they have by adding rich tooling. I especially like the tracing stuff they have.

There are also various other interesting presentations like the hands-on sessions on MPS and MetaEdit+. Also Heiko and Peter are going to give their very entertaining talk about cross-mobile platform development. I have yet to see a more practical and at the same time entertaining presentation on how to benefit from code generation and DSLs.

Unfortunately I cannot go to all the mentioned sessions because some of them take place in parallel. Anyway, the two sessions I definitely will attend are starting on Tuesday at 11am. They are both classified as hands-on sessions and will be giving them together with Sebastian and Jan.

During the Xtend hands-on you will learn everything you need to get started with this nice little language. Learning Xtend is quite easy if you are familiar with Eclipse and Java, because the integration of the tooling is tight and the syntax is only different at a few places. Mostly Xtend just leaves superfluous stuff out (e.g. type inference). This session will have a huge hands-on part and doesn't require any previous knowledge.

The other session will be about Xtext 2.0. We will cover the basic principles as well as advanced topics. Also Xbase, the embeddable expression language, will be covered. In this session the hands-on parts will likely not be as excessive as in the Xtend session, because we have to expect a mixed audience with respect to prior knowledge of Xtext. The session shall be interesting to both experienced and not so experienced listeners.

Punting tour during CG 2009

Both sessions only require you to bring your own laptop (or a friend with one) and a JVM with SDK (1.5 or higher).

I'm looking forward to see you next week in nice Cambridge. Hopefully we will have great weather so we can go punting and / or enjoy the evenings otherwise in this beautiful town. Apart from the great program the evenings are usually the best thing at most conferences and this is supposed to be especially true if you are in Cambridge in May. Well at least if the weather is as nice as as it was in 2009 when I took the picture on the right hand side.

Eclipse Xtend - beta available

2011-04-18T15:30:00.000+01:00

I proudly announce that a beta release of the upcoming Xtend language is available for download now. Xtend is a modern statically-typed programming language which tightly integrates with and runs on the Java Virtual Machine. It has its roots in the Java programming language but improves on a couple of concepts:

Advanced Type Inference - You rarely need to write down type signatures
Full support for Java Generics - Including all conformance and conversion rules
Closures - concise syntax for anonymous function literals
Injected extension methods - Enhance closed types with new functionality injected via JSR-330
Multiple dispatch aka polymorphic method invocation
Operator overloading - make your libraries even more expressive
Powerful switch expression - type based switching with implicit casts
No statements - Everything is an expression
Template syntax with intelligent whitespace handling
Translates to Java not bytecode - understand what's going on and use your code for platforms such as Android or GWT

It is not aiming at replacing Java but meant to be a better fit for implementing dependency-injected components and is especially well suited for doing code generation. Its library is just a thin layer over the Java Development Kit (JDK) and it interacts with Java exactly the same as it interacts with Xtend code. Also Java can call Xtend functions in a completely transparent way. And as it is implemented with Xtext, a modern Eclipse-based IDE closely integrated with the Java Development Tools (JDT) is available.

Xtend and Java vis-a-vis

Downloads

There are prepackaged Eclipse distributions available for download. They are shipped with the latest stable build and the nightly update sites are registered. So make sure you do "Help -> Check For Updates" on a regular basis to get the latest improvements.

Here are the download links :

	eclipse-SDK-3.7M6-Xtext-2-nightly-HEAD-linux-gtk-x86_64.tar.gz
	eclipse-SDK-3.7M6-Xtext-2-nightly-HEAD-linux-gtk.tar.gz
	eclipse-SDK-3.7M6-Xtext-2-nightly-HEAD-macosx-cocoa-x86_64.tar.gz
	eclipse-SDK-3.7M6-Xtext-2-nightly-HEAD-macosx-cocoa.tar.gz
	eclipse-SDK-3.7M6-Xtext-2-nightly-HEAD-win32-x86_64.zip
	eclipse-SDK-3.7M6-Xtext-2-nightly-HEAD-win32.zip

Getting Started

Xtend is shipped with a reference documentation which can be accessed through Eclipse's Help system.
Use the menu action [Help -> Help Contents] and you will find it.

Next up you might want to start playing around. The easiest thing to do is to materialize the Xtend tutorial project, which explains all the concepts by example. To do so use the project wizard and choose [File -> New... -> Example ...] and choose [Xtext Examples / Xtend tutorial].

Getting Feedback

It is important for us to get as much feedback as possible. Please use the Xtext Forum / Newsgroup to ask questions and bugs.eclipse.org to file bugzillas (product : TMF / component : Xtext).

The official release will be June 22nd. The language and its library is mostly finished and will only slightly be changed until then.

Acknowledgement

The language has been designed and implemented based on Xtext 2.0 and more important together with my friends and colleagues Sebastian and Jan (Unfortunately Moritz was busy with Xtext's serializer :-)). Our work is funded by itemis and the Federal Ministry of Education and Research.

Dear Java killers,

2011-04-15T13:36:00.006+01:00

I love to see all the languages for the JVM popping up on almost a daily basis. Language design is big fun and it's the ultimate thing for a developer to write programs in a language you've built your own. :-)

I especially like the languages which try out new ideas and don't take themselves too seriously, because thinking out of the box is the best you can do in order to push things forward.

However there are also a couple of languages who try hard to attract a certain audience: The Java community. The number of Java developers out there is huge (it had been about 9 million two years ago) so if you could make only a part of this crowd love and use your language it would be a huge success.
Languages like Groovy, Scala, Clojure, Gosu, Fantom, or lately (but not yet existent) Ceylon and Kotlin more or less try to attract this group. Actually I shouldn't really count Clojure in, because it is so different (I like it though). But all the other languages seem to try to be a better Java for Java developers.

Here are my seven most important Dos and Don'ts you should consider when developing a language for the Java community:

1) Don't make unimportant changes

People are reluctant to change. You should leverage existing knowledge where possible. Changing keywords because you think they sound better might be good for your ego but not for the acceptance of your language. Most languages fall into this trap somehow. The best example is the recently announced Ceylon, which renames all kinds of Java keywords with little reason. For instance the keywords extends and implements are different. In an interview on InfoQ Gavin King explains why he made that decision:

"Java is irregular here. In a class definition, the list of one or more interfaces follows the keyword "implements". In an interface the list of one or more interfaces follows the keyword "extends", which by coincidence may also appear in a class definition where it means something completely different (it is followed by a single class). Ceylon is regular. Lists of types always follow "satisfies". The keyword "extends" is always followed by an expression that instantiates the superclass."

Given that over nine million people are used to it and absolutely understand the semantics, I'd consider this an unfortunate decision.

2) Static Typing

The Java folks are used to top-notch tooling based on static type information. I don't think that errors found by static typing are actually that important, but the tooling you can build on top of this information is. And again it is a matter of the audience you are trying to attract. I wouldn't try to come up with a new statically-typed Ruby, since that doesn't make much sense. I know Mirah and it's not bad. My point is that the Ruby community is not interested in static typing.
Static typing is an important part of the Java culture and the problems it causes can be reduced (see my other advices). Besides that Groovy generally looks a bit overgrown, I'd say the lack of static typing has been a problem for the success of Groovy. Obviously the Groovy folks have understood that and try to fix it with Groovy++.

3) Don't touch generics

Static type systems are complicated. I don't think that more than 1% of all Java developers have really understood generics. All this co- and contra variance, wildcards, raw types, upper and lower bounds and the involved conversion rules are super complicated.
However If asked most Java developers think they know generics and feel quite comfortable with it.
I'm glad that languages like Scala do important work in this area by trying different approaches and having more sophisticated type systems. That's important for the industry but it scares off and there are nine million people out there who feel comfortable with an existing, quite sound type system (aside from very few practically irrelevant problems).

After all generics in Java are rather sound, it's just that the problem they solve is essentially complex.

Also note that reified generics are a bad idea. Type arguments are static type information. They are lying at runtime and the situations where they are helpful are rare. This also breaks interoperability with other JVM languages btw.

4) Use Type Inference

The biggest problem with static typing and generics is that people think that this is the reason why Java code is so cluttered with redundant type signatures. That's not true! It is just a language design decision that one has to be very explicit (or ceremonial) about everything he/she writes down.

A modern statically-typed language should make heavy use of type inference. That is in most situations you give the developer the choice whether to put a type signature or not. Code needs to be concise and readable.

Ceremonial languages like Java don't esteem their users, they think they are stupid and therefore force them to do all kinds of repetitive things. I'm glad the way language or library users are treated has changed in recent years such that developers are given more responsibility for what they do. Only then you can expect quality. The ceremonial style is the biggest problem with Java which urgently needs to be fixed.

5) Care about Tool Support (IDE)

Your language won't be successful within the Java community if it lacks decent tool support. Scala ignored this for quite some time and are now trying to catch up a bit.
My advice is to start working on the IDE while designing the language, because the IDE is an important part of the overall user experience people have with your language. Many syntactical decisions can have a great influence on how the tooling works. A simple example:

In Java when declaring a local variable, Eclipse first proposes you with a type and then afterwards with a name proposal using the simple name of the field's type :

Name proposals in Eclipse

This is simply not possible if you switch the order of type and name like in Scala ( name : Type ), because you first have to type the name.

This is just a simple example, there are other problems hiding in the complexity of a language implementation. Make sure you have an experienced IDE developer in your team and care about the holistic user experience of your language. For Java folks the IDE is an important part of it.

6) Closures

There is no way your language can be successful without closures. Besides the ceremonial verbosity and inflexible syntax of Java the lack of closures is where Java suffers most.
Also for closures type inference and a concise syntax are essential.

7) Get rid of old unused concepts

Removing unnecessary concepts is also a good idea. Java was designed for the C/C++ crowd and was meant to be used in embedded devices and the web. It turned out that it is used quite differently today and most Java developers really don't care about bit shifting, break and continue statements and fall-through switches anymore. Remove them!

APPENDIX

A personal note to the Gavin Kings in this world:

Eclipse Xtext is a language development framework, which not only helps implementing parsers, linkers, compilers and interpreters but also a top-notch Eclipse IDE. You should use it for any kind of language development, because it boosts your productivity and is fun to use.

A general note on Xtext:

Of course there is no mass-market for programming languages. Although Xtext handles them quite well and is used for commercial programming language IDEs, the main focus is to allow people to easily define small, focussed domain-specific languages. The good news is that we have developed a concise, statically typed Java like expression language library supporting everything I mentioned before. And *you* can easily embed it into your DSL! See what you can do with it:

Nordic Coding with OSGi, Xtext and Git (April 15th)

2011-04-04T18:14:00.001+01:00

Kiel

In two weeks on April 15th we are having the next Nordic Coding event in Kiel (link is german). There will be three talks each 40 mins.

First Martin Lippert explains the core principals behind OSGi before he shares some entertaining real-life experiences he had with OSGi.
Next Jan Köhnlein tells us about and demos Xtext 2.0. Sorry for the self advertisement but we did some very exciting stuff during the last months and just *have* to show it. :-)
The last talk is given by Ralf Ebert, who taught me to use and love Git. Git is absolute fantastic and Ralf is a very good teacher.

The event is again for free (incl. soft drinks and food), but you have to register in advance: To do so just send an e-mail to mail@diwish.de.

A rough performance comparison between Xpand and Xtend 2

2011-03-21T23:22:00.000+01:00

Xtend 2 is still in an early state. It works quite well, but there are still some issues and we haven't even started measuring and improving performance yet. Although I was pretty sure that Xtend 2 would easily outperform Xpand, I was curious how much.

Today, I wrote a small code generator (the traditional domain model to Java Beans stuff) in Xpand and Xtend 2 and compared the execution time with different numbers of model elements, just to get a first impression.
Disclaimer: Doing benchmarks is hard and I don't claim that do a fair comparison here, although I tried.

The Xpand version

The Xtend 2 version

Generating files for 1000 entities (1 file each) containing 20 properties and 5 operations each took (on average exec. the generator 10 times) :

Xtend 2 : 663 ms

Xpand : 4254 ms

If I remove the actual file writing and only do the in memory string concatenation the result was (again on average exec. the generator 10 times) :

Xtend 2 : 247 ms

Xpand : 3302 ms

So it seems as if the computation and string concatenation in Xtend2 is roughly 10 times as fast as in Xpand. As we haven't yet profiled Xtend 2, there should be room for improvements.

On a side note, I have also implemented the quicksort benchmark you can find here with Xtend 2, and found that the Java version was only twice as fast as Xtend. Compared to Groovy which was 20 times slower than Java it's quite good. It is mostly slower than Java, because we do a lot of conversion between ints and Integer (auto boxing) when using operators, so some special treatment in this area should make that even faster. Anyway, sorting int arrays is not the main focus of Xtend ;-).

If you are doing code generation or are interested in it and you are visiting EclipseCon, you should definitely come to our tutorial on Thursday. There is much more to Xtend 2 than just the improved execution performance.
And don't be afraid about missing the lunch, attendees get served a delicious, warm, and freshly prepared lunch after the tutorial :-).

A Guice Modules DSL

2011-03-21T02:45:00.001+01:00

[Update] The code is available at github

Yesterday Sebastian and I were attending GuiceCon. A nice little get-together with the Guice developers and some very smart Guice users. As Guice is key in Xtext's architecture we took the opportunity to show the people what we do with it. The first idea was to just explain how we use Guice, but while we travelled across California last week a better idea came up : We could develop a Guice Modules DSL using Xtext.

So we did and we did it from scratch right in front of the audience without any tricks or shortcuts. The good thing was that everyone in the room knew the domain very well, so we didn't need to explain anything about the domain of the language.
First I defined the grammar and a first shot on the code generator and then Sebastian added type conformance checks and improved content assist. Today I've spent another two hours on it. The screencast below demos the status quo.

Note that there will be an extended tutorial at EclipseCon this Thursday, where you can learn how to create such things yourself in no time. Not to forget the short talk on "What's new in Xtext 2.0", Tuesday 2 pm.

Have fun!

Xtend Screencast Part 3 - Rich Strings (i.e. Codegeneration)

2011-03-07T10:06:00.001+01:00

Xtend is a programming language with special support for code generation and traversing typed object graphs (like EMF models). It is developed with Xtext 2.0. So the question is what is so special about code generation, that we need special support in a language? Well it's mostly string concatenation, and the most readable way of doing string concatenation is using a template syntax.

Today's screencast covers rich string expressions. It resembles the basic template syntax known from Xpand but in Xtend they are expressions, so you can use them everywhere a normal expression is expected. The new and unique aspect about Rich Strings is their intelligent whitespace handling system, including nice tool support. See it yourself:

You can get your hands on this at EclipseCon in just two weeks and/or at JAX in two months.
The final release of Xtext 2 and the new Xtend language is June 22nd (Eclipse Indigo).

Xtend Screencast Part 2 - Switch Expression

2011-03-04T09:17:00.000+01:00

In this part I show and explain the switch expression, which is defined in Xbase, so you can also have that in you Xtext DSL if you want :-)
In the tutorial at EclipseCon on Thursday we will show how to embed Xbase expressions in a simple DSL.

Here again I use Xtend for the demo:

Xtend Screencast Part 1 - Basics

2011-03-02T00:04:00.000+01:00

Only three weeks till EclipseCon!
I can't tell how excited I am. Sebastian and I are going to Los Angeles in just two weeks in order to travel up the Highway 1 to EclipseCon for one week. I know the weather will be great :-)
At EclipseCon we finally can show people what we have been working on the last two months and what kind magic can be done with Xtext 2.0 in such a short time frame.

In the meantime we (Jan, Sebastian and I) want to help you get into a similar mood :-). Sebastian already blogged about it and Jan will definitely do so very soon. I decided to record a series of screen cast.
The screen casts are not edited at all, because I thought it would be more authentic and trustworthy that way.

The first screen cast covers the basic concepts. I hope you like it:

[1] The Xbase development is sponsored by itemis and the Federal Ministry of Education and Research.

REPL for Xbase

2010-12-13T09:53:00.000+01:00

Sebastian Benz blogged about a REPL (Read-Eval-Print loop) he implemented for Xbase.
He also wrote a very good introduction to explain what Xbase is and why it is such an important addition to the Xtext framework.

His blog isn't yet included in Eclipse Planet, so I wanted to makes sure that you won't miss it.
I think the REPL is a great way to explore the power of a language. Maybe we should use it in the proposed EclipseCon tutorial if it's getting accepted. :-)

Xtend 2 - the Successor to Xpand

2010-12-03T15:41:00.000+01:00

M2T Xpand is one of the widest used template languages for code generation. It is statically typed, supports polymorphic dispatching of template definitions and provides a very convenient way to navigate through models. The navigation convenience mostly comes from the built-in higher order functions for collection types, the possibility to have extension functions using Xtend and some syntactic sugar for common expressions.
Also Xpand is known to provide a very open architecture. There are several hooks in the interpreter to integrate profilers, pretty printers or whatever comes to mind. That's all very nice and has led to quite some success.

But there are also some issues with Xpand and Xtend (Just in case you don't know I am not blaming other people's work, because I'm the author of Xpand, Xtend and Check)

Performance - Xpand and its siblings are interpreted and the interpreter was written without much performance considerations, i.e. is slow.
Tooling - The tooling is hand-made with JFace and Co, the code is old, not well tested and unfortunately a bit buggy.
Ugly conceptual short cuts - built-in higher order functions instead of closures, built-in collection types instead of generics, and a couple of other unsound semantics which pop up occasionally and bite the user frequently.

Xtend 2

The reason I started work on Xtext in the first place was, because it had been so painful to write the IDE support for Xpand, Xtend and Check. I thought that there must be some higher abstraction to derive functionality like syntax coloring, content assist, outline views, etc. from a language description. So I wrote the first prototype of Xtext in 2006 and luckily found some friends and a sponsor to create what was chosen to be the most innovative project at Eclipse this year.

Today Xtext is ready to be the basis for a rewritten version of Xpand. We call it Xtend 2 as it is more like a successor to Xtend which allows to have Xpand's template syntax as an expression. After having designed Xpand and Xtend and having seen so many different applications of it with different requirements and problems to solve, we know what features are helpful and which are problematic. Also during the last years with Xtext I have learned so much about language and IDE design and so did my team. All the languages we built for customers and all the languages we saw people developing with Xtext have helped to get a good feeling of what is good language design and what is not. We are now in a promising position to design a very, very nice language.

And indeed we are all thrilled about the concepts we put together so far. I'm writing up the language specification these days and will post it as soon as it is more consumable state. Today I wanted to share my excitement and give you an idea of what it is going to look like.

What are the main differences?

Based on Xbase

Xtend 2 is of course using the expressions defined in Xbase. That basically means it is tightly integrated with Java, has closures, full generics and a couple of very neat features like operator overloading or the super cool switch statement.

Compiles to Java
Xtend2 files are directly compiled to Java classes. In fact they are compiled to Java source code, which means that you can read (and debug) what the compiler makes out of your Xtend 2 coding. We will put much effort in making the generated code very readable and performant at the same time.

Contains successful features from Xpand and Xtend...
The cool stuff from Xpand and Xtend will be there such as polymorphic dispatch, extension methods or the template syntax.

... but they are even better now
For instance extension methods can be any static Java function and more important any methods from injected objects. This basically allows you to exchange the implementation of any extension methods only by changing the DI configuration. This in turn replaces the whole AOP story we have in Xpand and Xtend.

"Simplicity is the Ultimate Sophistication"
You'll see a lot of other small improvements, but the coolness of the language doesn't come from the amount of features but from the short and carefully chosen list of concepts working consistenly together. It is the overall composition of concepts and the holistic experience (tooling, etc.) which gives a language its "feel". And although I don't have the language in my hands right now, it already feels fantastic. :-)

We want to show this at EclipseCon

There is a submission for a tutorial on Xtend 2 we submitted for EclipseCon 2011. My favorite conference at my favorite location. It is extremely important to us to be able to share with you guys what we have been working on for the last months (actually years if you count the development of Xtext in).
Chris asked people to use the comment system to provide feedback, in order to help the committee composing a great program. So please, if you are interested in attending a hands on tutorial for Xtend 2 at EclipseCon 2011, tell them. :-)

Extreme Nordic Coding

2010-12-03T11:23:00.000+01:00

Yesterday the first Nordic Coding mini conference took place in my hometown Kiel. In the beginning I was a bit skeptical whether it was possible to motivate enough people to come to such a code-centric and Java-centric event. Kiel is quite small and I wasn't sure if the local IT industry was big enough. But after we got 100 registrations in advance there was only the snow to spoil the party.

Yesterday we've had the total snow chaos in Kiel and a lot of people couldn't even manage to get to work or school or elsewhere. So I was a bit afraid that nobody was going to come to the first Nordic Coding event. But I was wrong and eventually more than half of the registered people made it to the event.

The first presentation was held by Karsten Lentzsch of JGoodies fame. He lives in Kiel and shared some very enlightening and entertaining stories about his experience with customers and how they tackle UI-programming. He is no doubt an absolute expert in that field and also a great presenter.

After that Sebastian Zarnekow and I talked about how we push Java to its limits in order to make code more readable and expressive. It was fun giving the talk as both Sebastian and I really love programming and could put a lot of passion into that topic.

Afterwards we had some interesting conversations and delicious finger food. Thanks to DiWiSH for sponsoring and organizing the event with us.

The feedback was great and I'm looking forward to the next event which will be in about four months. I asked everybody to submit proposals for presentations and already got three interesting responses. It's my desire to have one presenter from Kiel minimum at each event.

See you next time!

Nordic Coding

2010-11-08T09:21:00.001+01:00

On December 2nd the first nordic coding event is happening in Kiel. It's a mini, mini conference (two to three presentations) around software development on the Java platform. It is planned to have such an event at least twice a year. For the first event we will have two presentations:

1) Karsten Lentzsch (JGoodies.com) will talk about UI-Design and why so many smart people regularly fail in writing good user interfaces. Karsten is known to be a great presenter and I am really looking forward to his talk.

2) Sebastian Zarnekow (Xtext/itemis) and me are talking about modern Java programming. We're going to cover how to use the more complex capabilities of the Java language such as reflection, annotations and generics to design readable and convenient Java APIs.

DiWiSH will serve free snacks and drinks and we surely will have a lot of interesting discussions and maybe a beer in the irish pub afterwards. The event is free of charge and I hope to see a lot of Kiel's (and Hamburg's and inbetween's) software developers there.

You have to register in advance. A printable announcement (in German) can be found here.

Xbase - A new programming language?

2010-09-30T09:47:00.015+01:00

No!

It's the basis for a plethora of new programming languages and domain-specific languages!

What is Xbase?

Xbase is a partial programming language implemented in Xtext and is meant to be embedded and extended within other programming languages and domain-specific languages (DSL) written in Xtext.

Why Xbase?

Developing textual modeling languages (aka DSLs) has become incredibly easy with Xtext. Structural languages which introduce new coarse-grained concepts, such as services, entities, value objects or statemachines can be developed in minutes. However, software systems do not consist of structure only. At some point a system needs to do something, hence we want to specify some behavior which is usually done using so called expressions. Expressions are the heart of every programming language and are not so easy to get right. That is why most people do not add support for expressions in their DSL, but try to solve this differently.

The most often used workaround is to only define the structural information in the DSL and add behavior in a second step by modifying or extending the generated code. It is not only unpleasant to write, read and maintain closely related information in two different places, on two different levels of abstraction and in two different languages, this also only works for compilers (i.e. code generators) but not for interpreters. (Additionally they are a lot of other reasons why mixing generated and hand written code is problematic, which is not the topic of this blog post.)

But still as of today this is the preferred solution since adding support for expressions (and a corresponding compiler) for your language is hard - even with Xtext.

Actually being able to call out to the host language is one big advantage internal DSLs have over external DSLs. With Xbase it will be possible to explicitly allow more complex programming at certain places within your DSL, while still have full control over the syntax and semantics of your language. And you neither have to reinvent the wheel by implementing a full-blown programming language nor do your language's users have a hard time to understand the expression language, since it is closely related to Java and well specified.

Also the more Xbase-based languages we see the more commonly known it will be.

Main Decisions

We want Xbase to be expressive and convenient to use, but at the same time easy to understand and easy to adapt. Understanding not only means learning how to use it but also understanding the language infrastructure, i.e. the parser, compiler, type checkers, etc. Because people shall be able to reuse and adapt that stuff easily.

The main target audience for Xbase are Java developers. That is why an Xbase expression looks like a Java expression (or statement) at a first glance. This means the most commonly used Java statements and expressions (e.g. string literals, if statement, foreach loop, method invocation, constructor call) are also available as is in Xbase. On the other hand Java is a very complicated language, especially when it comes to the details. After all the spec counts over 600 pages and while it is very precise most of the text deals with exceptional conditions often involving the special handling of built-in types, etc.

Xbase shall be significantly simpler, so we have to make some decisions.

Runs on the JVM

The JVM is a great, popular platform. In order to ship a compiler, interpreter as well as static typing, Xbase needs to bind to some target platform.
Other platforms such as C/C++, ObjectiveC or JavaScript are also very interesting target platforms for Xtext languages, but for now the main focus of Xbase is the JVM. This seems to be natural decision, since Xtext itself runs on the JVM. Also we know a lot about this platform and the community.

Compiles to Java

The compiler will translate to Java instead of byte code directly. This is for the following reasons:

Anybody should be able to integrate the expressions compiler with any Java code generator
The output as well as the implementation of the compiler shall be as readable / understandable as possible
The code can be used with non-JVM platforms like GWT or Android
We want to leverage the optimizations coming with proven Java compilers

Another pragmatic reason is, that while we plan to have a debugger for Xbase based languages, it won't be part of next year's release. Therefore people will have to debug on the Java code level, which wouldn't be possible if we were generating byte code directly.

Interpreter

We also want to ship an interpreter in order to allow interpreted DSLs using Xbase.

Statically Typed

Xbase is statically typed. This means that there is a type checker and also that the compiler will use static type information to do it's job. Most important to users, might be the rich tooling we can and plan to provide based on Xtext and Eclipse in general.

However, it should be possible to remove the type checking phase and change the compiler to do dynamic method invocations, etc.

Full Java Generics

Xbase uses fully-fledged Java generics and doesn't change anything here. While Java Generics are not perfect they have been understood (or at least people think they have ;-)) by a lot of people.
Introducing a different type model would hurt adoption. Under the hood this is backed-up by the JVM-Types we introduces with Xtext 1.0.

No built-in types

While the Jvm-Types support every Java type, Xbase will automatically convert any references to built-in types and array types to their corresponding wrapper types resp. lists.
This means you can use built-in types in your languages if you want to (and you should be able to extend Xbase in a way that it can, too), but you don't have to.
The compiler might use built-in types in the generated Java code, but statically and conceptually everything is a subtype of java.lang.Object (i.e. pure OO).

Closures

The main addition in Xbase is the concept of closures. While it looks like Java will have them one day, the lack of them is a major problem with Java.
Xbase comes with a small runtime library, where interfaces for Functions are part of. Closures in Xbase are just sugar for anonymous classes of one of these Function types.

For instance the following Java expression:

new Function1<String,String>() {
  public String apply(String s) {
    return s.toUpperCase();
  }
}

can be written like this in Xbase:

String s | s.toUpperCase()

Xbase also provides sugar for the types of functions. That is

(String)=>String

is a shorthand for

Function1<String,String>

Type Inference

Type inference is another important feature of any modern statically typed language. Type inference basically means that the compiler doesn't force you to write redundant information about types. In Java for example the type of a local variable needs to be specified although it could be inferred from the initialization expression:

Map<String,Person> namesToPerson = new HashMap<String,Person>();

In Xbase you don't have to write the type signature twice, but can write the following instead:

val namesToPerson =  new HashMap<String,Person>();

Of course namesToPerson would be of type HashMap<..> here. If you want to be explicit, you can add the type information optionally:

val Map<String,Person> namesToPerson =  new HashMap<String,Person>();

Xbase does type inference for type arguments in closures as well. That is the argument types don't need to be specified if they can be inferred from the current context.

Also note, that the typing service of Xbase can be used in your language in order to do type inference (for instance for return types in method signatures).

Operator Overloading

Xtext comes with a fixed set of operators, with a fixed precedence and associativity. The difference to Java is, that those operators are not bound to certain built-operations on built.in types but are just shorthands (or sugar) for certain method invocations.

That is if some type T has a method plus(T2), you can either write

myT.plus(myT2)

myT + myT2

This concept is known from Groovy (although it's slightly different there).

Simplicity over Syntactical Flexibility

With operator overloading we could have gone a step further as done in Scala. In Scala the operators aren't fixed keywords but words with certain characteristics (usually starting with a certain letter). That would allow to have operators which are not predefined in the language. However, this would have introduced a couple of additional lexer rules, which had limited the available syntactical space dramatically. This had made extending the language much harder (and even impossible in many cases).

In general we decided to prefer simplicity over syntactic flexibility. This is because with Xbase you already have the largest syntactic freedom. You just create a sublanguage and add or remove anything you want.

Languages like Scala really need to have all this flexibility, because they are designed to add new language features as a library. These special rules about identifiers and operators and other syntactic flexibility like newlines as expression separators (and the situations when this doesn't work) as well as the different ways to invoke functions is what makes Scala syntactically flexible but complicated at the same time.

Xbase is designed to let you easily add new language features on the language level. If you need a certain syntax you can just have it. The base language remains simple.

Everything is an Expression

There's just no good reason to separate between expressions and statements. Although most statements are inherently imperative (i.e. about side effects), there's no reason to have this separation (which is a limitation) built into the language.
Instead in Xbase everything is an expression, that is everything returns something (and has a type at compile-time). This allows to use the typical imperative statement constructs deeply nested like in the following expression:

this.setFoo(if (isFoo) "foo" else "bar")

In Java we have the ternary operator to do branches within expressions. In Xbase you can use the if expression, but you can also have for and while loops, try-catch clauses or even the nice switch expression deeply nested.

Powerful Switch Expression

This is one of the new features we added. I like pattern matching, but think it is way too complex for many people to use and most people to integrate in their language.
Also I like polymorphic dispatching, like we always had in Xpand and use a lot in Xtext.

On the other hand the switch expression in Java is just stupid. It is complex (fall through) and limited (finally switch over strings in Java 7 ?).

So what we do in Xbase is

we remove fall through (first match wins)

we allow to switch over anything (based on equals)

we introduce so called type guards (which automatically applies down casts)

Example:

val p = getMeSomeObject();

switch ( p ) {
 Foo case p.isSpecialFoo() : "SpecialFoo";
 Foo : "OrdinaryFoo";
 Bar : "It's a "+p.barKind()+" bar";
  default : "don't know";
}

I hope this is intuitive and readable. You can find the details in the Xbase language specification.

Current state

The development of Xbase has just begun. We have a first draft of a language specification and grammars as well as some infrastructure, but we are still in a very early state.

I hope this post made you interested in Xbase. Feedback is very welcome.

Xtext in Indigo

2010-09-29T13:11:00.008+01:00

We've started development on the next release of Xtext just four weeks ago. Prior to that we did the service release and some internal prototyping and evaluation for upcoming features. We also already have had our yearly kick-off meeting in order to define the main goals for the next release.

With this blog post I'ld like to share what we are up to. Feedback is welcome.

Xtext Version 2.0

Yes, that's right. We are aiming at a new major version. This means lots of

great new features and improvements as well as incompatible changes. While our API lifecycle contract is, that nothing remains compatible between two major versions, we of course are aware of what API most users use. So we reconsider any changes and don't do them easily.

The differences might turn out to be comparable to those we had between the last two releases, which were considered easy to migrate to.

What we did so far

During M2 we did a lot of clean up. We removed any deprecated code and incremented the bundle versions for those with incompatible changes to 2.0.0.
Also the whole outline view infrastructure as well as the auto editing has been rewritten. The new outline view is cleaner, faster and simpler. In addition Xtext now leverages JFace partitions. That is one can register different Jface service implementations, such as content assist, per partition. By default there are distinct partitions for comments and string literals as well as the default partition type for everything else. See the full list of solved bugzillas here.

Milestone builds

Yesterday we had our M2 of the current indigo stream. We don't have a promoted milestone build, as we had some issues with the build servers. Please use the nightly builds in case you want to give the latest changes a spin.

Expect the first promoted milestone build (M3) of the current development stream of Xtext in six weeks (November 9th).

What's next? (Plan for Indigo)

The development plan for Indigo is laid out around one big topic, which is called Xbase. Xbase is an expression language bound to the JVM, which will be shipped with Xtext and can be embedded and extended in other Xtext-based languages. It is basically a simplified version of Java's expressions but also adds some new concepts. I will post a separate blog entry on Xbase tomorrow. Note, that Xbase is a research project we are doing together with Christian-Albrecht University of Kiel which is sponsored by the Federal Ministry of Education and Research.

Refactorings & Clean ups

As you might know, we usually implement most functionality twice (or more often), because we are simply too stupid to get it right in the first place :-).
Even with Xtext 1.0.0 this is not different, so we have identified the following parts of the framework which we want to clean up or even re-implment:

Outline view (done)

Auto Editing (done)

Bracket Matching (done)

Serializer

Formatter

some smaller things

New Features

But hey, we know most people are interested in new features. So this is what we want to work on in addition to Xbase:

Generic Rename Refactoring

Additional Import Namespace Functionality (organize imports, add import, warnings and errors)

Default Implementation and Convenience API for Text Hovers

Lexer and Parser Fragments

View Framework (Convenient way to define tree views, such as call or type hierarchies)

More Languages

We of course eat our own dog food and therefore will implement a couple of new languages based on Xbase.

One is a textual notation to define Ecore elements similar to EMFatic, but where one can specify the behavior in derived EStructuralFeatures and EOperations in the language.

Another is a successor to Xpand and Xtend, called Xtend 2. That one comes with a couple of very interesting features I'll talk about in subsequent blog posts.

We are also thinking about having DSLs for several different view points in Xtext itself, such as scoping. Don't miss tomorrow's blog post on Xbase. :-)

Disclaimer: As usual this is what we would like to do and defines the general direction. But as we don't have any fortuneteller in our team, we might (will) adjust the plan slightly as we go.

Will talk about Xtext in Antwerp and Stockholm soon.

2010-08-26T21:40:00.003+01:00

I have the pleasure to present Xtext in two of the most beautiful european cities within the next two weeks.

Belgian Eclipse Community Meeting (Antwerp)

I'll be in Antwerp next Tuesday and meet up with Belgian's Eclipse Community.

The event is free of charge and the details can be found here. There's also a presentation of EGit, which I'm looking forward to see (I'll have lots of questions :-)).

"Developing Domain-Specific Languages with Xtext" (Stockholm)

A week later on Thursday, the 9th, I'll be in Stockholm. The event is sponsored by Jayway and there will be two presentations. First I'll talk about Xtext and afterwards Patrik Nordwall and Andreas Källberg will present Sculptor, which is an open-source framework/code generator based on textual DSLs implemented in Xtext.

The event is also free of charge. Register here.

Xtext 1.0.1 (SR1) release and git migration on Wednesday

2010-08-23T15:50:00.011+01:00

In the last two months Xtext got a great number of performance and memory footprint improvements, bug fixes and extensive documentation enhancements (most of them done by Sebastian while us others were on summer vacation :-)). We also fixed a couple of annoying bugs in Xpand's editor.

You can find a detailed list of all 89 fixes in this bugzilla query.

The upcoming Xtext 1.0.1 (Helios SR1) release will be built on Wednesday.

Migration to git
We are also happy to announce that this will be our last CVS based release, since we will switch to git on the very same day. I also want to thank Ralf Ebert for his excellent git training he gave us last week in exchange to an Xtext training. Git is cool! :-)

Parsing Expressions with Xtext

2010-08-09T09:17:00.021+01:00

Parsing simple XML-like, structural languages with Xtext is a no-brainer. However, parsing nested expressions is often considered a bit more complicated. This is because they are more complicated due to their recursive nature and also because with Xtext you have to avoid left recursive parser rules. As the underlying parser (generated by Antlr) uses a top-down approach it would recurse endlessly if you had a left recursive grammar.

Let's have a look at parsing a simple arithmetic expression:

2 + 20 * 2

If you know EBNF a bit and wouldn't think about avoiding left recursion, operator precedence or associativity, you'ld probably write a grammar like this:

Expression :
 Expression '+' Expression |
 Expression '-' Expression |
 INT;

This grammar would be left recursive because the parser reads the grammar top down and left to right and would endlessly call the Expression rule without consuming any characters, i.e. altering the underlying state of the parser. While this kind of grammars can be written for bottom-up parsers, you'ld still have to deal with operator precedence in addition. That is define that a multiplication has higher precedence than an addition for example.

In Xtext you define the precedence implicitly when left-factoring such a grammar. Left-factoring means you get rid of left recursion by applying a certain technique, which I will show in the following.

So here is a left-factored grammar (not yet working with Xtext) for the expression language above :

Addition :
 Multiplication ('+' Multiplication)*;

Multiplication:
 NumberLiteral ('*' NumberLiteral)*;

NumberLiteral:
 INT;

As you can see the main difference is that we have three rules instead of one and if you look a bit closer you see, that there's a certain delegation pattern involved. The rule Addition doesn't call itself but calls Multiplication instead. The operator precedence is defined by the order of delegation. The later the rule is called the higher is its precedence. This is at least the case for the first two rules which are of a left recursive nature (but we've left-factored them now). The last rule is not left recursive which is why you can write them down without applying this pattern.

We should allow users to explicitly adjust precedence by adding parenthesis, e.g. write something like (2 + 20) * 2.

So let's add support for that (note that the grammar is still not working with Xtext):

Addition :
 Multiplication ('+' Multiplication)*;

Multiplication:
 Primary ('*' Primary)*;

Primary :
 NumberLiteral |
 '(' Addition ')';

NumberLiteral:
 INT;

So once again: if you have some construct that recurses on the left hand side, you need to put it into the delegation chain according to their operator precedence. The pattern is always the same, the thing that recurses delegates to the rule with the next higher precedence.

Construction of an AST

Now that we know how to avoid left recursion, let's have a look at what the parser produces. In Xtext each rule returns some value. Parser rules return AST nodes (i.e. EObject instances), enum rules return enum literals and datatype rules as well as terminal rules return simple values like strings and the like (EDatatype in EMF jargon).
Xtext can automatically infer whether some rule is a parser rule, i.e. constructs and returns an AST node or if it is a datatype rule. Above's grammars only consisted of datatype rules so all they would produce is a string.
In order to construct an AST we need to add Assignments and Actions. But before we do that we need to talk about return types.

The return type of a rule can be specified explicitly using the 'returns' keyword but can be inferred if the type's name is the same as the rule's name.
That is

NumberLiteral : … ;

is a short form of

NumberLiteral returns NumberLiteral : …. ;

However in the case of the expressions grammar above, the rules all need to return the same type since they are recursive. So in order to make the grammar functional we need to add a common return type explicitly (but the grammar is still missing some bits):

Addition returns Expression:
 Multiplication ('+' Multiplication)*;

Multiplication returns Expression:
 Primary ('*' Primary)*;

Primary returns Expression:
 NumberLiteral |
 '(' Addition ')';

NumberLiteral:
 INT;

The AST type inference mechanism of Xtext will infer two types: Expression and NumberLiteral. Now we need to add assignments and Actions in order to store all the important information in the AST and to create reasonable subtypes for the two operations.

In the following you see the final fully working Xtext grammar:


Addition returns Expression:
 Multiplication ({Addition.left=current} '+' right=Multiplication)*;

Multiplication returns Expression:
 Primary ({Multiplication.left=current} '*' right=Primary)*;

Primary returns Expression:
 NumberLiteral |
 '(' Addition ')';

NumberLiteral:
 value=INT;

Let's go through the grammar as the parser would do it for the expression

( 1 + 20 ) * 2
(I'm sure it's pretty hard to follow what's going just by reading this text. Therefore I have prepared a small video where I visualize and explain what is going on. You can find it at the end of this section.)

The parser always starts with the first rule (Addition). Therein the first element is an unassigned rule call to Multiplication which in turn calls Primary. Primary now has two alternatives. The first on is calling NumberLiteral which consists only of one assignment to a feature called 'value' of what the INT rule returns.

But as the first token in the expression is an opening parenthesis '(' the parser will take the second alternative in Primary, consume the '(' and call the rue Addition. Now the value '1' is the look ahead token and again Addition calls Multiplication and Multiplication calls Primary. This time the parser takes the first alternative because '1' was consumed by the INT rule (which btw. is a reused library terminal rule).

As soon as the parser hits an assignment it checks whether an AST node for the current rule was already created. If not it will create one based on the return type, which is NumberLiteral. The Xtext generator will have created an EClass 'NumberLiteral' before which can now be instantiated. That type will also have a property called value of type Integer, which will get the value '1' set. This is what the Java equivalent would look like:

// value=INT
if (current == null)
 current = new NumberLiteral();
current.setValue(ruleINT());
...

Now that the rule has been completed the created EObject is returned to the calling rule Primary, which in turn returns the object unchanged to its own caller. Within Multiplication the call to Primary has been successfully parsed and returned an instance of NumberLiteral. The second part of the rule is a so called group (everything within the parenthesis). The asterisk behind the closing parenthesis states that this part can be consumed zero or more times. The first token to consume in this part is the multiplication operator '*'. Unfortunately in the current situation the next token to consume is the plus operator '+', so the group is not consumed at all and the rule returns what they got from the unassigned rule call (the NumberLiteral) .

In rule Addition there's a similar group but this time it expects the correct operator so the parser goes into the group.
The first element in the group is a so called action. As Xtext grammars are highly declarative and bi-directional it is not a good idea to allow arbitrary expression within actions as it is usually the case with other parser generators. Instead we only support two kinds of actions. This one will create a new instance of type Addition and assign what was the to be returned object to the feature left. In Java this would have been something like:

// Multiplication rule call
current = ruleMultiplication();

// {Addition.left=current}
Addition temp = new Addition();
temp.setLeft(current);
current = temp;
...

As a result the rule would now return an instance of Addition which has a NumberLiteral set to its property left. Next up the parser consumes the '+' operator. We do not store the operator in the AST because we have an explicit Addition type, which implicitly contains this information.
The assignment ('right=Multiplication') calls Multiplication another time and assigns the returned object (a NumberLiteral of value=20) to the property named right.

If we now had an additional plus operation '+' (e.g. 1 + 2 + 3) the group would match another time and create another instance of Addition. But we don't and therefore the rule is completed and returns the created instance of Addition to its caller which was the second alternative in Primary. Now the closing parenthesis is matched and consumed and the stack is reduced once more.

We are now in rule Multiplication and have the multiplication operator '*' on the look ahead. The parser goes into the group and applies the action. Finally it calls the Primary rule gets another instance of NumberLiteral (value=2), assigns it as the 'right' operand of the Multiplication and returns the Multiplication to Addition which in turn returns the very same object as there's nothing left to parse.

The resulting AST looks like this:

The following video tries to explain what you just have read. The grammar is slightly different, but I'm sure this won't be problem for you ;-).

Associativity

There is still one topic I should mention, which is associativity. There is left and right associativity as well as none associativity. In the example we have seen left associativity. Associativity tells the parser how to construct the AST when there are two infix operations with the same precedence. The following example is taken from the corresponding wikipedia entry:

Consider the expression a ~ b ~ c. If the operator ~ has left associativity, this expression would be interpreted as (a ~ b) ~ c and evaluated left-to-right. If the operator has right associativity, the expression would be interpreted as a ~ (b ~ c) and evaluated right-to-left. If the operator is non-associative, the expression might be a syntax error, or it might have some special meaning.

We already know the most important form which is left associativity:

Addition returns Expression:
 Multiplication ({Addition.left=current} '+' right=Multiplication)*;

Right associativity is done using the following pattern (note the quantity operator and the call to the rule itself at the end):

Addition returns Expression:
 Multiplication ({Addition.left=current} '+' right=Addition)?;

And if you don't want to allow multiple usages of the same expression in a row (hence non-associativity) you write:

Addition returns Expression:
 Multiplication ({Addition.left=current} '+' right=Multiplication)?;

Note that sometimes it's better to allow associativity on parser level, but forbid it later using validation, because you can come up with a better error message. Also the whole parsing process won't be interrupted, so your tooling will generally be more forgiving.

(This post has also been included into the Xtext User Guide since version 1.0.1)

Xtext does musical scores

2010-07-02T13:45:00.009+01:00

Dennis Harmath has built a LilyPond IDE for Eclipse. He sent a very nice testimonial to our newsgroup:

I'm happy to announce Elysium, a LilyPond IDE for Eclipse - proudly brewed with Xtext!

I must say that I really enjoyed developing tooling support for LilyPond, the definitive textual DSL of music notation using Xtext, it was great fun to see the grammar evolve with the help of automatic abstract syntax inference, and the IDE features I got "for free" from Xtext really impressed me. Hooking into Xtext's build process to implement incremental build was also much easier than writing an own builder. In the future, developers might reuse LilyPond's abstract syntax as a metamodel for music notation e.g. to write converters from/to LilyPond, hence bringing more music notation support to Eclipse.

There's also a screenast showing Elysium in action:

Xtext Webinar is online

2010-06-09T07:55:00.003+01:00

Yesterday's webinar on Xtext 1.0 (to be released June 23) was recorded and can be watched at Eclipse Live : http://live.eclipse.org/node/886

I think it went very well, at least Jan and me had a lot of fun talking about our beloved framework. :-)

We had quite a good number of attendees:

199 registered
153 attended live

And the poll showed that there were as many Xtext users as there were people who are new to Xtext. So that the focus of giving an overview of the framework and how it works but also demo the new features seemed to have been appropriate.

Are you using Xtext?
35% (31) – Yes, I use it frequently
14% (12) – Yes, I use it sometimes
47% (40) – No, but I'm interested in using it
2% (2) – No, I don’t have plans to use it

We hope the webinar was and is helpful for you. Thank you!

5 Good Reasons To Watch Our Xtext Webinar

2010-06-03T08:41:00.010+01:00

Next Tuesday (June 8th) there will be an interactive webinar on Xtext at Eclipse Live. Here are the best 5 reasons for you to show up:

Reason I : You want to learn how easy Domain-Specific Language (DSL) development can be

People state that developing external DSLs is much more complicated than internal DSLs. With Xtext the opposite is the case. Xtext itself provides a concise language to describe DSLs, so that it takes only minutes to create a first draft of your language. You don't have to deal with complex meta programming or multiple syntax alternatives as you do in internal DSL development. With Xtext your language definition is concise and declarative and you get language specific IDE support.

Reason II: You want to see the new Helios features in action

In the Helios release Xtext graduated to version 1.0. It has grown up from a tiny little editor generator to a mature language development framework. Still simple things are simple, but you now get much more out of the box. Features like namespace-based scoping, a workspace index, a builder infrastructure, validation and linking against dirty editor state, quick fixes, linking to any Java elements and tight integration with JDT, enhanced serialization and formatting support and much more.

Reason III: You'ld like to see how "Modeling 2.0" looks like

The days of heavy weight, dogmatic modeling approaches are over. In 2010 Modeling technology is mature and a pragmatic solution to many problems. Xtext and EMF are a dream team! Models are now also code and integrate seamless with common development environments such as version control.

Reason IV: You need a decent IDE for a particular programming language

It's astounding how many programming languages are used in the different industries. There are so many languages I'm sure you haven't heard of. Most of them have one thing in common: They lack decent tool support. I'm not talking about syntax coloring for Vim, but something close to what modern Java IDEs offer. We at itemis have already successfully implemented a couple of IDEs for various programming languages. We will show something in the webinar.

Reason V: The webinar is free and interactive

As usual with Eclipse Live webinars, the event is free of charge and the integrated chat application allows us to communicate. The Xtext committers will attend and be happy to answer any questions. Unfortunately Sebastian won't be able to join (for personal reasons). But Jan was so kind to jump in and do the presentation with me. We are looking forward to meet you online!

Have fun.

Screencast: Xtext in the automotive industry

2010-05-31T09:52:00.006+01:00

A couple of month ago I had already blogged about how BMW Car IT use Xtext to develop a full-blown IDE for the Autosar standard . Now Sebastian Benz has published a video demonstrating ARText. See what you can do with Xtext.

ARText in Action! from Sebastian Benz on Vimeo.

WYSIWYG with Xtext

2010-04-15T16:07:00.004+01:00

Yesterday I came across a post in the EMF newsgroup which got me to do this little prototype of two languages. One mark up document language and one where you can declare the text styles to use in that mark up language. See the screencast:

The very rough prototype code can be downloaded from here.
It is based on Xtext 1.0.0 >= M6.

Google's Protocol Buffers implemented in Xtext

2010-04-14T08:23:00.002+01:00

Eat your own dog food is an excellent advice. So do I and therefore I use Xtext to implement languages from time to time. Usually these are languages we either implement for one of our open source projects (examples or the recently added MWE2 language I developed together with Sebastian) or for customer projects.

A couple of weeks ago I implemented the DSL of Google's Protocol Buffers using Xtext. I don't want to add it as an example to Xtext, because it doesn't add any specific value over the existing examples nor do I have the time to start a new project or maintain it in any way. However, I thought there might be some people who want to have a look as an example or want to take it over and extend/maintain it further.

I didn't know Protocol Buffers so I had to learn the language using the documentation and the unit tests they have. The concepts of the language (scoping, imports) are very close to Xtext's defaults so that it took me only 3 hours to implement it. Most of the time was spent to get the lexing part right, which was not so close to the defaults :-).

If you want to have a look, here are the two Eclipse projects. Feel free to do whatever you want with that code. I of course would love to see someone creating a new open source project for it or adding it to an existing. In any case I'll be glad to help you getting started with the code.

The code should work with every post-M6-milestone as well as the final Xtext 1.0.0 release which will be published in June.

I also recorded a short screen cast demoing the result:

Modeling Day @ JAX 2010

2010-04-13T09:19:00.003+01:00

The JAX conference is coming closer. This year there's a "Modeling Day" consisting of nine 45 min talks. We not only have well-known speakers but also in-depth practical presentations. If you want to learn about Eclipse Modeling Technologies and see how they are applied in industry the JAX Modeling Day is for you.
The fishbowl discussion we have at the end of the day should be a lot of fun as well.
All sessions will be held in English.

Looking forward to see you there!

Xtext won Eclipse Community Award!

2010-03-27T16:25:00.012+01:00

What a great community you are! Thank you so much for giving us this kind of support. We are all incredibly proud about the fact that Xtext has won the Eclipse Community Award 2010.

I also want to take the opportunity to say thank you to the team once more. Especially to Dennis Huebner who couldn't make it to this year's EclipseCon. Itemis and in particular our bosses Wolfgang Neuhaus and Jens Wagener also deserve a big thank you: If they hadn't believed in my vision two years ago and still do, there were no Xtext and we were all working on different things and hadn't had the chance to combine our forces to create our favorite programming language and IDE development framework.

I already put the award logo Lynn sent me on our website. See how nice it fits into the Xtext-Nova theme.

EclipseCon 2010 was so much fun. I'm still in San Francisco but am already looking forward to next year's issue of my favorite conference. Thank you for all the support!

EclipseCon 2010

2010-03-22T14:03:00.005+01:00

I'm sitting in my room at the Hyatt hotel in Santa Clara, California. It's 6 am and I'm glad I could sleep til 5 am at least. Seems that the 7h bike trip we did yesterday did its job (we hoped that being outside most of the day would get us in tune with the local time zone). I only wished the saddle hadn't been so hard....

Today, the conference starts and I'm currently looking at what sessions I'ld like to attend.

Monday
Monday starts with the introduction of the e4-Rover Mars Challenge. I bet you've already heard of it. Right after that Jan, Moritz and Heiko give a 3 hour tutorial called "Xtext meets e4". They will show how one can model a workbench in a textual way. They have worked hard on preparing the workshop material and I'm curious to see how the audience like what they have. After lunch the shorter presentations begin. I'm going to see what Stephan and Kenn think the twenty modeling things are and am looking forward to the b3 talk. After that Peter has his session on Xpand just before Heiko presents on how one can apply MDSD to develop IPhone apps. Also I want to note Jan's talk on combining GMF and Xtext.
In the evening I have a modeling panel and after that there's the community awards ceremony. Xtext is finalist in the category "most innovative project". How cool is that? :-)

Tuesday
In the morning I'll be out on another bicycle trip. After that I'll have to practice my presentations for the afternoon (yes I practice and I always hope nobody watches me while I'm doing so :-)).
After lunch there are a couple of talks I find interesting. Kai presents on CSS in e4 and Berni does his penalty shoot out of textual modeling tools (didn't know there are other than Xtext ;-)).
I'm also interested in what's new in CDT. Not because I'm a big fan of C/C++, but I'm interested in Eclipse-based IDE development. At 2 pm it's showtime for Sebastian and me. We'll demo the coolest IDE features you get when developing a language in Xtext. And right after that there's the Modeling Runway, where a couple of modeling projects present their projects (5 min each).

Later that day I'll likely attend the Graphiti talk. Let's see if it is any better than GMF. I think I'll have a break after that in order to read through Wayne's revised Eclipse Development Process in order to be prepared for the Development at Eclipse panel I am attending.

Wednesday
I am not yet decided whether to go to "Modular Architecture from Top to Bottom" or "API Design and Evolution" in the morning. Both seem very interesting, but I guess I'll have more fun asking controversial questions in the API talk :-). In the afternoon I probably will visit the Groovy IDE talk. The new version of the Eclipse plugin looks great. Especially the tight integration with JDT is pretty cool. But I also might go out for another trip, since the weather is forecast to be great. If not I'll go to see what cool features Eike and his team brought to CDO lately. In the evening Sebastian will present the new version of MWE2. It's implemented in Xtext and we're very proud of the result and how easy it was to implement it using Xtext.

Thursday
For me the day certainly starts with "Understanding and Using Git at Eclipse ". Actually we're still using CVS and I personally am ok with it because of the great tooling Eclipse provides. The only problem we have is its slowness. Synchronizing my development workspace sometimes takes 15 min. So we plan to switch to Git right after Helios.
In the afternoon the talk about Dependency Injection got my attention. I love DI, though I am also a bit concerned about how DI is used in e4. I'll make sure to have some discussions with the e4 folks about my concerns. I'll have to bring my bicycle back to the store in the early evening, so I won't be able to attend any other talks on Thursday.

These are just my personal selection of a great variety of interesting talks you can attend at this year's EclipseCon. All in all it looks like a very diverse program and I'm looking forward to four interesting days full of great talks and discussions. I especially like the trend towards shorter presentations because I expect them to be better prepared and more focused.
....ups, it's already 8:17 am (I've had a great breakfast in the meantime) and I am missing the e4-Rover Mars Challenge. I'll have to go now. See you!

Multiple dispatch (and poor men's patter matching) in Java

2010-03-18T07:03:00.015+01:00

There are times where you do not want to or simply cannot add a certain aspect of behavior to your domain model's class hierarchy.
Most applications are layered but share the same domain model (the times where people translate state into layer-specific representations are hopefully gone). Sharing the same domain model classes between different layers is convenient and effective. However if you do that, your domain model tends to become a so called anemic domain model, that is you do not add the behavior to your classes but hold it externally in order to keep the domain model layer independent. This ensures that your domain model only contains code which belongs to the domain and not to a technical layer. That's ok because you do not want to have UI-specific or database-specific code in your domain model if you pass it through all layers.

The problem is that you cannot leverage Java's polymorphism anymore. As a result you either write lengthy if-else cascades in order to dispatch between different types or you use the visitor pattern, which is some kind of external polymorphism support. Well actually the old visitor pattern is not that much external but very invasive. You have to prepare your domain model to be able to accept visitors. That's not always possible and also a lot of work.

Other languages support features to solve this problem. One is pattern matching as known from functional languages which also has made its way to object oriented languages. It's nice and powerful but not available in Java. :-(
Another feature is "multiple dispatch". Languages supporting multiple dispatch (aka "multimethod") decide which method to invoke based on the actual runtime type of the passed arguments. Java in contrast links the method at compile time based on the static types.

This is why in Java the following would print "Hello Fruit":

print((Fruit) new Banana());

static void print(Fruit f) {
 sysout("Hello Fruit");
}

static void print(Banana b) {
 sysout("Hello Banana");
}

If Java would support multiple dispatch it would print "Hello Banana", because at runtime the passed argument is an instance of Banana.

In Xtext we have "added" multiple dispatch to Java (we call it polymorphic dispatch). For instance a label provider is implemented like this:


public class MyLabelProvider extends AbstractDeclarativeLabelProvider {

String text(Banana b) {
return "yellow fruit";
}

String text(Apple b) {
return "keeps the doctor away";
}
}

The framework just calls ILabelProvider.getText(Object element) and the implementation of this method dispatches to the best matching label(Object) method. So if you are interested how it works, have a look at the PolymorphicDispatcher class. Other examples where we use this technique are validation, scoping, qualified name provider, content assist, outline view, etc.

This stuff is very useful for defining such kinds of APIs because it allows us to define nice defaults while the user can add specific behavior just by adding the corresponding methods.

But what if you are in the middle of some implementation and only need to make some decisions based on the type? Write code like this?


Fruit fruit = new Banana();
String result=null;

if (fruit instanceof Banana) {
  Banana b = (Banana) fruit;
  result =  // do stuff with b;

} else if (fruit instanceof Apple) {
  Apple a = (Apple) fruit;
  result = // do stuff with a;

} else if (fruit instanceof Peach) {
  Peach p = (Peach) fruit;
  result = // do stuff with p;
}

System.out.println(result);

You might get into trouble with these guys.
How about this one?:


System.out.println(
new Match() {

  String is(Banana b) {
    return // do stuff with b;
  }  

  String is(Apple a) {
    return // do stuff with a;
  }

  String is(Peach p) {
    return // do stuff with p;
  }

}.apply(new Banana()));

It's only a bit shorter, but I like it much better because it is functional and more declarative. That is I don't have to declare variables and do side-effects within the different if statements. Also this works for multiple parameters as well.

P.S.: If your domain model is implemented in EMF you get a so called Switch-class generated, which acts similar. So if you only dispatch within that hierarchy and you only need one argument, this works great as well.

Xtext Helios M5 is out!

2010-02-09T08:33:00.003+01:00

Last Friday we (eclipse.org) officially released the M5 of the Helios release train. Xtext, being part of the train, got a couple of new & noteworthy features we (Xtext team) are very happy (and sort of proud) with.

I'm personally most excited about the new builder infrastructure, which allows to revalidate, index and build any kind of EMF resource. Some of the new features we added in M5 demonstrate the huge potential of that builder infrastructure. My favorite is the open element dialog, which I use frequently. Besides that we also added a couple of other features like, auto editing, bracket matching and styled label providers. Our new & noteworthy document shows the most important feature additions using screenshots and small screencasts (we hope you like it).

During M6 we will add some features as well as doing the regular bug fixes. Also we will do a major rename refactoring (actually it is already done), so you might face some migration efforts when you upgrade. For M7 we are going to focus on performance improvements and further bug fixing. Documentation, tutorials and migration guide will be worked on during the RC phase. In parallel we're working on a new general purpose language developed in Xtext. It is meant to be some kind of library. Everyone should be able to reuse, adapt and extend that language within her own Xtext languages. That language won't be released with Helios, but as it is open-source you might be able to check it out by that time.

Btw.:
Have you already voted for the Eclipse Community Awards?
No? Go here.

Xtext's new builder infrastructure

2010-01-05T10:00:00.009+01:00

The M4 of the current development stream of Xtext (Helios) finally contains the long awaited builder infrastructure. This topic turned out to be a very complex beast, and although we already took two attempts the last one took us about six weeks of very intensive work. But we love the result :-)

What is this builder infrastructure for?
I'd already blogged about the idea of name space based linking.
The builder infrastructure collects lightweight descriptions of all Xtext resources in the workspace. The linking can then use this information. In addition we use this information to compute the transitive hull of affected resources.

Also the builder takes care of validation and will support an extension point where users can hook in code generators and the like. Other features where we'll make use of the new information includes: more sophisticated searches, call graphs, and of course find reference actions. A generic rename refactoring is also something we want to start working on as soon as possible.

The documentation will be updated during the RC phase.

The New & Noteworthy page contains a short screen casts showing the builder in action:

Dirty State aware editing

Note, that state in the builder is shadowed by any dirty editor state:

What about the EMF Index?
Wasn't the EMF Index intended to do this?
It took us some time to find out what Xtext really needs, and in the end we learned that there are a couple of important specialties, like the lazy linking or the linking by name. We didn't want to add those concepts to the EMF index. Therefore we started out to solve the actual problem in Xtext first and later see if the requirements fit and can be handled by the EMF Index project. As a result we came up with a very small interface which can be implemented to use the EMF Index as soon as it supports the needed use cases and is mature enough. Most of the stuff we did is Xtext related, the code which might be replaced by the EMF Index is just a couple of lines.

RE: The Hidden Costs of Domain-Specific Languages

2009-11-25T11:08:00.004+01:00

Today, Johan Den Haan pointed me to a post about "Hidden Costs" of DSLs via twitter.
The post starts with the following statement:

While I am a big fan of Domain-Specific Modeling, I am openly not in favor of Domain-Specific Languages.

... which leaves me puzzled. I'm pretty sure that the DSM guys call their abstractions DSLs as well.
Anyway, the post itself is not particularly interesting, but at the end of it there are a couple of questions about DSLs. This sort of questions which I've heard often and which contain a lot of miss- and preconceptions. Thought it was a good opportunity to answer them.

So here are the questions and my answers:

If we take so much time to master a general purpose language, should we invest a comparable amount of time in limited-use languages (i.e. DSL)?

No, you shouldn't and you won't. DSLs are much easier to learn, since they are limited and focused. The important thing is that you have to understand the domain. But that is necessary in any case, no matter if you translate everything to a general purpose language or write it down like it is meant to be (i.e. using a DSL). If you know the domain very well, but are not able to grasp the DSL, than the DSL might not be that good.

How can we get support for a DSL, apart from its own creators?

How can you get support for an API, apart from its creator? Well, by looking at the code or any existing documentation, or by asking any other users. In many aspects DSLs are comparable to libraries or frameworks.

Where’s community support?

It depends on whether it is a widely used DSL (i.e. there is a community) or one you use and have defined in your one-man project. For the DSLs used in Ruby on Rails there's much community support, for instance. Same for the DSLs we have in Xtext.
Where's the community support for your homegrown libraries?

What happens after the departure of the language’s creator?

It is like with any other artifact in your software system: A DSL definition needs to be clean, understandable and maintainable. If you know that an important team member is leaving the project, make sure she's not taking any exclusive knowledge with her. That is she should coach other people before she lefts.

What’s BNF? Do I need it?

see http://en.wikipedia.org/wiki/Backus%E2%80%93Naur_Form :-)

DSL critics say really useful DSLs are hard and expensive to create. DSL supporters answer that DSLs are not designed, they evolve. Well, won’t any of those “evolutionary steps” risk breaking the entire development based on that DSL, much like a broken app build?

Useful DSLs can be very simple and at the same time a very complex DSL can be absolutely useless. It's not the complexity of a DSL, which makes it useful, but whether how good it is to solve the problem at hand. KISS and YAGNI are important principles here as well.
When defining concepts and abstractions it is key to find a good compromise between simplicity (abstraction) and re-usability. The simpler a DSL and the higher its potential for reuse the better. You're hitting a sweet spot, then. You start leaving that sweet spot as soon as you add concepts to the DSL which are only used by a small subset of the users. Often it is better to have two simple DSLs in two projects, than putting everything into one big DSL.

Back to the question: I don't think that evolution and design is mutual exclusive. Don't everybody design software systems incrementally these days.
Changing a textual DSL is not a big problem in practice. At least not as long as you have a decent test suite (which you should have no matter if you use DSLs or not). However, managing compatibility and versioning can become quite complicated, but again it's the same as with APIs.

Will the evolution in the language be severe enough to trash what has been done so far?

You can shoot your foot, if you want. But you don't need a DSL to do so.

Can you imagine yourself developing a complex C++ software system while C++ itself was still being designed and developed?

Can you imagine writing a complex C++ software system while the standard libraries are still being designed and developed?
I can imagine, but I am not keen on doing that. That is why we have things like versioning and that is also why people think about compatibility when enhancing public programming interfaces. Such interfaces can be a method signature, but it can also be a syntax of a language. It doesn't matter. If it breaks it breaks. And the designer has to be explicit about the contract of that interface as well as its life cycle.

Xtext - Using existing Ecore models

2009-11-24T10:19:00.011+01:00

Maybe it's because it was not possible in the old oAW Xtext or because the wizard suggests a grammar which makes use of the "generated Ecore model"-feature. But somehow some people tend to think that you cannot use existing Ecore models with Xtext. YOU CAN!

The feature is described in the documentation, but maybe it got too few words (Which is not because we don't use it and don't want you to use it, but because it's so simple).

In the following I want to show you how to do this. You'll see that it is relatively straight forward.
Let's assume you have an existing Ecore model, describing a library of books (sounds familiar?;-)):

In order to create a textual syntax with Xtext you only need to create an Xtext project and change the proposed grammar to something like this :

So if you want to reuse an existing ecore model, it is just a matter of switching from a "generated Ecore"-statement to an "import"-statement. Note that as soon as you do this, the grammar is statically analyzed and you'll get all kinds of compiler errors and warnings for missing attributes and classes in the imported Ecore models (you can import multiple ecore models, of course).

Next, you run Xtext's generator and you'll get an EMF Resource writing and reading instances of your Ecore model to the textual syntax defined in aboves Xtext grammar.
In addition you'll get an editor for your new language :

You can, of course, still use any EMF-based editor, since it's just an EMF model:

Note, that you cannot only import multiple Ecore models but can even mix generated and imported Ecore models in the same grammar.

To sum it up: The feature we have in Xtext to derive an Ecore model from the grammar description, is a very convenient option to get you started. If you have one or more existing Ecore models, you won't want and need to use it. But if you start developing a new language this feature speeds up your turn-arounds dramatically.

Xtext - Tour Dates

2009-11-13T15:00:00.014+01:00

I just realized that there's an invasion of Xtext presentations and demos about to overrun the world during the next couple of weeks. And all of them are free of charge!

Date	Country	City	Title	Presenter
2009/11/16	USA	New York	Modeling Day: Building DSLs with Xtext	Heiko Behrens
2009/11/16	USA	New York	Modeling Day: Converging Textual and Graphical Editors	Moritz Eysholdt
2009/11/17	France	Paris	Demo Camp: Building DSLs with Xtext	Sebastian Zarnekow
2009/11/17	Germany	Leipzig	Demo Camp: Und heute generier' ich	Alexander Nittka
2009/11/18	Germany	Hamburg	Java User Group: Building DSLs with Xtext	Sven Efftinge, Sebastian Zarnekow
2009/11/18	Canada	Toronto	Modeling Day: Building DSLs with Xtext	Heiko Behrens
2009/11/18	Canada	Toronto	Modeling Day: Converging Textual and Graphical Editors	Moritz Eysholdt
2009/11/23	Germany	Berlin	Demo Camp: Building DSLs with Xtext	Peter Friese
2009/11/26	Germany	Frankfurt	Demo Camp: Using Xtext in a large scale modeling and implementation project	Nicolai Busse
2009/11/26	Germany	Stuttgart	Demo Camp: Textual DSLs with Eclipse Xtext (and a little bit code code generation with Xpand)	Markus Völter
2009/11/30	Austria	Vienna	Demo Camp: Xtext	Michael Clay
2009/12/04	Germany	Hamburg	Demo Camp: Combining graphics and text in model editors	Jan Köhnlein
2009/12/10	Italy	Florence	Demo Camp: Xtext, a Textual Modeling Framework	Francesco Guidieri

Xtext in the automotive industry

2009-11-12T16:22:00.014+01:00

At this year's Eclipse Summit Europe there was a very nice Xtext-related talk given by Sebastian Benz who is working at BMW Car IT.
The talk was about how they used Xtext in order to develop a textual syntax and corresponding IDE for the AUTOSAR standard. The name of the project is ARText and it is freely available to all AUTOSAR members as part of the Artop project.

One of the biggest challenges when implementing the AUTOSAR standard is to create scalable solutions. AUTOSAR projects might become very, very big and in order to work with such huge projects the tools need to perform very well. In the modeling world people usually strife for repository-based (i.e. database-based) solutions as soon as projects get really big. However, the folks at BMW Car IT wanted to develop AUTOSAR projects in a traditional text-based manner and given all the good experiences with tools like JDT or IntelliJ, it's clear that text-based IDEs can scale very well.

In order to implement the language they first had tried the old Xtext version from oAW, which was way too slow. Then, later, when the new TMF Xtext came around they gave it another try and saw that the performance had been improved significantly. Sebastian showed the following slide in his talk:
After that he explained how they have solved a couple of other problems, such as supporting different releases of the standard or making the language extendable. All the solutions looked very nice, they must have some very skilled people at BMW Car IT.

He compared working with the state-of-the-art commercial graphical modeling tool (What's the name of it?) and the ARText IDE and found that the use of ARText reduces development time by about 40%.

In the end there was this nice summary slide I don't want to withhold:

It was a very nice talk and of course a pleasure to see that Xtext is used by such smart people in such an interesting environment. And even better it seems that they've had as much fun using Xtext as we had and still have when developing (and using) it. :-)

Btw.: If you have other interesting applications of Xtext or Eclipse Modeling in general, please contact me. (Even if you don't want me to blog about it ;-))

Edited: Sebastian has recently posted a screencast showing ARText in action:

ARText in Action! from Sebastian Benz on Vimeo.

Eclipse Summit Europe 2009

2009-10-29T08:03:00.004+01:00

This year's Eclipse Summit Europe is again a very nice happening. It's so important to meet all the people you're chatting and mailing with in person.
Yesterday, Sebastian and I had the chance to demo some of the new Xtext features we've been working on in the Helios train. It was much fun and we got very positive feedback for both the talk and (more important) the features. We'll post the demos to our blogs very soon.

I'm especially excited about how well Xtext is accepted within the Eclipse Community. Yesterday there were at least three talks about other projects which make use of Xtext:

The b3 guys are working on a DSL to describe builds
In the new EMF Search implementation the query language is implemented with Xtext
The Relax NG tools are as well implemented in Xtext

How about e4? Designing UIs works good in a WYSIWYG style, but there are scenarios where you want to program parts of it. In addition storing the UI descriptions in XMI is not necessarily the best option. You cannot read it and it's very hard to diff and merge them. I can imagine that a programmer friendly textual syntax might please the modeling skeptics a bit. We can still use a WYSIWYS editor on top.

Implementing the CSS functionality with Xtext could be another opportunity to get more for less. To find out, I'll have to attend Kai's talk about the CSS customization in e4 this morning.
There are lots of other interesting sessions to attend today. Unfortunately I have to decide between listening to

Sebastian Benz (BMW) who talks about how Xtext is applied in the automotive industry (AUTOSAR) and
Jan Koehnlein and Jos Warmer who talk about how to combine Xtext and GMF

because they are both scheduled for 14:40.

Helios M2 of Xtext, Xpand and MWE

2009-10-05T09:15:00.010+01:00

The M2 builds are promoted (with Athena!). And here's the promised update on what we've already done for the M2 of Helios.

You can find the details in the bugzilla links. Here I'll only mention new big features.

Xtext (fixed bugzillas)

Quickfix Support
It is now possible to register quickfixes for validation rules. The feature is documented as part of the validation section.

JVM - EMF bridge
This one allows referencing Java elements from EMF models, as it is desired in most DSLs which are used in a JVM context. We already wrote a small MWE prototype (without XML syntax), which makes use of this new component. It took us a day to come up with a neat MWE editor. Our upcoming base language will be based on top of this type system.
Sebastian is going to blog about the details soon.

EMF Index integration
We made some good progress with regards to the index integration. However, as it was not finished by M2 and the changes were too big we decided to hold it back. It will be shipped with M3 and I will write a separate blog about it soon.

Xpand (fixed bugzillas)

Incremental Generation
There's support for incremental generation now. It leverages EMFCompare to find out what has changed and only triggers those parts in a generator which could possibly be affected by the changes. In typical scenarios where one has big models and only very little changes between the builds this can yield a significant performance boost. The implementation is there and the unit tests should be a good starting point (org.eclipse.xpand.incremental.tests). Documentation and an example workflow will follow.

Xpand Profiler
If you want to improve the overall execution performance of your generator, the new Xpand profiler comes in handy. Heiko already blogged about it. There's also documentation available.

MWE (fixed bugzillas)

For MWE only some bugs were fixed. The new MWE syntax we're working on has not yet been checked in. It is primarily meant as a test language for the Java EMF bridge. In order to replace the current parser and infrastructure we need to implement an XML based reader (for backward compatibility) and of course the interpreter needs to work on EMF models. This is not planned for M3 as we have little resources and want to focus on the EMF Index integration as well as the Java EMF bridge. But I'm pretty sure we'll do this during one of the next milestones.

Disclaimer: We encourage everybody to use and test our milestones. It really helps to get as much feedback as possible. However, note that every API we introduce between M1 to M5 might be changed or even be removed completely in future milestones. Also milestones are usually not as stable as releases.

If you want to test the milestones just take the composite updates site from itemis:
http://download.itemis.com/updates/milestones/
Check the Antlr feature ("Xtext Antlr Support 0.8.0 M2" -> "Xtext Antlr Feature") and p2 will do the rest.

Just in case you're interested in what is planned for M3. Here are the bugzilla queries:

Xtext in wind energy

2009-09-15T13:21:00.005+01:00

I've visited the Fraunhofer IWES in Bremerhaven this week. Fraunhofer is a famous german research institute and the IWES department focusses on wind energy. In their current research project they make heavy use of Xtext and other Eclipse Modeling technologies.

The project is about specifying and connecting the different components a wind turbine is made of and simulating real time scenarios using such models.
The problem they want to solve is that today's computers are much too slow to compute structural, fluid, or thermo dynamics of detailed models in detailed surroundings in a reasonable time. Also the components are usually tested and simulated in isolation only.
The general approach to face this is to reduce the level of detail in order to speed up computation. This has of course the drawback that the resulting data gets more imprecise. The goal of the research project is to dynamically change the level of detail for the different components during the simulation. This is modeled using a state machine.
For instance if you want to simulate a storm in order to find out at what wind force a blade breaks you have to start with low wind force and increase it constantly. The system would allow to replace a lower detailed blade with a higher detailed one in certain scenarios. The engineer specifies in what circumstances detailed information is interesting and the system can use fast models with low details for the rest of the time. For instance one would want to replace a low-detailed blade with a high-detailed blade when the wind gets so strong that it is likely that the blade breaks.

The other problem they have to solve is that there are many different tools (e.g. Modelica, NASTRAN, ANSYS) used to model and simulate the different components. Each of them have their strengths and weaknesses so the research team needs to find a universal way to reuse and interconnect all of these tools.

To solve these two issues they have developed a DSL used to model the different components. From that description they generate representations for the different tools in different levels of detail. In addition they have developed Xtext implementations of the most interesting languages they use, that is Modelica (an object-oriented, equation based language), NASTRAN, and ANSYS.

I want to mention that the Fraunhofer IWES are looking for smart software engineers. So if you're interested in wind energy and Eclipse Modeling it might be a good fit. I haven't spent much time in Bremerhaven but on a first glance it looked much nicer than I thought.

Xtext - Road to Helios

2009-08-27T10:43:00.004+01:00

After the Galileo release of Xtext we are now focussing on the next iteration. We've had a face-to-face meeting in Kiel and discussed the different ideas, our process and what the "themes" for our Helios development are.

In this post I want to share our vision of how the road to Helios could! look like. I say 'could' because although we have a rough idea of where we want to go, we know from the past, that the actual road we go will slightly differ. That is things we find useful today might have become less important in six months. "Round And Round The Earth Is Turning" :-).
I'll outline our visions and priorities along the four themes we identified for the Helios development period.

Theme I : Clean Code

Our code base is already comparatively clean but we want it to be even cleaner. Cleanness in code means the following to us:

DRY - Don't repeat yourself

Every information should be kept in one place only. This is as everything I write not meant dogmatic. So, there are of course situations where it is more pragmatic to duplicate a piece of information.

KISS - Keep it small and simple

Solutions should only be as complicated as they need to be. Sometimes it is hard to understand for developers that they have solved a problem but we do not want to check it in in that way. That is of course not because we do not appreciate the effort. It is because we think it is too complex, and we do not want to add superfluous complexity to the code base. Of course opinions of what is superfluous and what is not might as well differ :-).

Keep it testable and have tests

Code needs to be testable. If it is not testable unit testing is either not possible or it is too complicated and therefore is fragile and complex. We want to have sufficient unit tests. Sufficient is again a weak statement. Usually one has too little tests, so having too many tests is very seldom the case. Mature and published code can't have too many tests, as long as each test covers a different case. Note that the code cleanness of unit tests is as important as the code quality of the production code. When we fix bugs we usually do test first. That is we write a failing test reproducing the reported miss behavior and fix it afterwards.
In Xtext we have more LOC in the tests than in production code.

Comments

Any comments need to reflect the code! Wrong comments are far more problematic than no comments.

Inlined comments are a smell. They tell you that your code is not readable. Most of the time the simplest thing to do in order to avoid such comments is extracting a method and naming that method like the comment you wanted to put in front of that code snippet.

JavaDoc comments are important. This is an aspect where we should improve our code base.
JavaDocs should be as small as possible and as verbose as needed in order to understand the contract behind the API at hand. We only add JavaDocs to primary hooks. Internal stuff is seldom commented because we can do a quick view at the code in order to understand what's going on. At least as long as the code fulfills the next requirement.

Readability

Code is read far more often than it is written or edited. So it's especially important to write the code so that your intentions are clear. Small functions and expressive names are good starting points.

Theme II : Usability (UI Quality & Features, API Quality, Documentation Quality)

UI Quality and Features

In Galileo we mainly focused on putting the main abstractions in place and creating a solid framework. Some UI features were therefore postponed.
One of the first things we want to do in Helios is adding the EMF Index and making the usage of it and the corresponding scope provider the default. Based on that we have information about all cross file references, which enables us to not only automatically trigger validation of referencing models, but also provide advanced navigation features and things like rename refactoring.

API Quality

It is our desire that people have fun when working with the abstractions we provide.
The mantra for last year's development was 'We aim to make simple things simple and complex things possible.' (by Alan Kay). Which means to us that what the 80% of things we want to do with a framework needs to be as simple as possible, but at the same time we need to make sure that the other 20% are still possible (and ideally also not too hard to accomplish). Of course there are use cases which simply do not fall into the scope of a framework but that is a different story.

Documentation Quality

Our documentation is in a relatively good shape, but still can be improved. We are convinced that good documentation is key for the overall acceptance of a framework.
So keeping the documentation current and keeping a good quality is part of our development process. That is adding or changing code is always paired with adding and changing tests and updating the documentation.

Theme III : Performance & Scalability

This has been a major theme for last year's development as well.
Given user stories like the talk about Xtext and AUTOSAR proposed for ESE 2009, it seems that we met this non-functional requirement fair enough.
But that does not mean that we don't need to check it from time to time, or that users neverface performance issues. We take this aspect very serious.
But of course we do not write optimized code as along as we haven't measured that optimization is worth the extra unreadability, complexity or what ever.
What we do is having regular profiling sessions and thinking about different use cases when we design concepts.

Theme IV : Increase Applicability (Base Language, Grammar Features)

So far all the themes are more or less about, improving the quality. I haven't talked about many new features (despite the mentioning of EMF Index and what it means for some UI features we want to implement). You will of course see many improvements and new features I haven't mentioned in this post. We decide which bugzillas to work on from milestone to milestone (I'll write about what is coming in M2 in a separate post.)
But there is one huge "feature" we want to do and I want to mention it here:

Base Language

We want to provide a base language, which can be extended and customized by users. The idea is not new: The Intentional Workbench as well as MPS both have similar things. Despite that these frameworks are of a different nature (they are not text-based) we also don't want to implement Java or C# (that is what they do), but come up with a much simpler and nicer language. Actually I like Scala very much, because it has very little concepts in it and allows to define new concepts out of them. But with an Xtext-based language you don't need the syntactical flexibility of Scala or Ruby, because the parser and the compiler are open and you can change the language as it seems fit. Imagine state machines with action implementations, entity models with implementation for operations and derived attributes, validation languages etc. Looking at the current land scape of external DSLs I'ld say that about 90% of all languages would benefit from support for embedding expressions. Some people already have implemented their own expression language others reuse the language of the target platform, i.e. they write the behavior in Java and mix it into the generated code. Sebastian and I gave a talk about this at Code Generation 2009 in June.

In order to show and prove how great such a language is, we plan to implement an EMFatic version that supports adding implementation for EOperations and derived EStructuralFeatures.
With that you'll no longer need to change the generated code (at least most of the time) and you'ld have everything in one place.
In addition DSLs for common viewpoints in Xtext, such as scoping, validation, quick fixes, formatting, etc. would be nice.

The base language is a huge effort and will likely not be finished by next year. As of now I'm not sure if and how much it will be included in the Helios release.

Java adaption

Another really nice thing we're already working on is, having a thin EMF adapter layer to JDT's Java model. This allows for referencing Java elements from within a DSL, which is very often desired. Reimplementing MWE with a nicer syntax will be a matter of a couple of days when we have this feature. We want this to be tightly integrated with EMF Index.
This is also a bigger topic but will definitely be part of Helios (actually we plan to have it finished by M4).

Grammar Features

There are a lot of ideas of additional features for the grammar language. We haven't decided on them yet but things like multiple grammar inheritance might get more important when we ship a base language. But what exactly such a feature would mean and what the consequences are still has to grow on us.

DISCLAIMER

Please don't take any of these ideas as guaranteed. This is what we currently have in mind but our view of the world will change. Also note that developing the base language we have in mind is a big effort, so it might take us more than just one release cycle.
I'll write a post for each milestone in order to tell what we exactly we did and do.

Xtext 0.7.1 available

2009-07-17T21:23:00.003+01:00

We've just published a bugfix update to the Galileo release. Actually it is not only an update of Xtext but also of MWE and Xpand. All three projects are now shipped in version 0.7.1.

The update contains a lot of nice bugfixes (see Sebastian's blog for details). Getting it should be pretty easy for everyone this time. It's all available from our update site at itemis :

http://download.itemis.com/updates

Moritz made it a P2 composite site pointing to the corresponding update sites at Eclipse as well as to the highly recommended Antlr generator fragment. So it is just a matter of pressing 'update' if you already had Xtext installed and had added the updatesite above. If you want to install Xtext for the first time, just add the update site above and install the Xtext SDK. This will install everything you need (including MWE UI and the like, which was previously not automatically added). Have fun!

Thank you Moritz, Dennis and Sebastian. It works great! :-)

RT @Xtext #webinar #bugfix-release #dsl-contest

2009-07-15T07:56:00.003+01:00

Just in case you're not sure what exciting things to do within the next couple of days, here are some suggestions:

First you could attend the Xtext webinar Heiko and I are giving today (at 5:00 pm GMT). There you'll learn something about the general principles and ideas behind this new language framework. We'll also show some examples.
You have to register in advance!

Second you grab the bugfix release (0.7.1) of Xtext which is due this Friday (July 17th). It includes a lot of nice bug fixes. Sebastian already blogged about it.

With that on your box you'll be well prepared to develop a nice, witty language using Xtext and submit it for the DSL contest itemis is running. Creativity wins!

Xtext scopes and EMF index in action

2009-07-02T13:23:00.014+01:00

This is a post about scoping and how to use the EMF index for that. It is in some sense a practical follow up on another blog post about the general idea behind indexing and scoping in Xtext. The topic is somewhat advanced and bleeding edge. This post describes the needed steps to get the current index based default scoping up and running. I've prepared a small screencast demonstrating the result in action. The example language can be downloaded from here.

Today, the common way to do cross resource references in Xtext is to do it via resource URIs. That is if you want to reference a model element (EObject) from another resource, you typically put the whole resource on the scope by adding a corresponding import. Example:

import "platform:/resource/my.project/src/othermodel.dsl"

//.. refer to elements from othermodel.dsl

The corresponding default scoping is very simplistic. Every object in the current resource and in the referenced resources can be referenced by its simple name (as long as it has a 'name').

Although this is very easy to understand, it has it's limitation when it comes to more sophisticated design. If you for instance want to hide some elements or have duplicate simple names in different packages (this can be the case if you use elements, which are developed by others).

In many programming languages we have the notion of namespaces, which are much more flexible and powerful. Java, for instance, is file system agnostic. Although it forces you to put the files into folders which correspond the packages, it ultimately is just based on namespaces (packages, types).
That said Java's namespace mechanism is also a bit limited. For instance I cannot have imports in nested namespaces but only per file. And I cannot nest packages but only classes and interfaces.

Scala and C# both allow to have multiple nested packages within one file and you can put imports per namespace, so that imported names are only visible within that namespace.

In order to demonstrate how to use the index together with Xtext, I've implemented a DefaultIndexBasedScopeProvider which implements a similar semantic. There's a small example I've prepared, where you can see how it can be used. It is mainly a matter of configuring the different implementations with Guice. Programming is not needed as long as you're happy with the defaults.

Here's how it works

The index registeres a builder, which is invoked on resource changes. In order to make your model elements visible, you'll have to contribute a so called Indexer using an extension point.

<extension point="org.eclipse.emf.index.indexer">
<indexer class="org.eclipse.xtext.example.DomainmodelExecutableExtensionFactory
                  :org.eclipse.xtext.index.DefaultDeclarativeResourceIndexer"
            fileextensions="dmodel"/>
</extension>

Please ignore the ExecutableExtensionFactory, which is declared in order to make any executable extension Guice aware, that is you can use dependency injection. This is a different topic and might be covered by another blog post.

The actual class to be instantiated is the one after the colon (':'): The DefaultDeclarativeResourceIndexer, which delegates to an instance of IQualifiedNameProvider, which itself is injected. This means that its implementation can be arbitrarily changed.
The contract of a name provider is very simple: it computes a qualified name for an element, if it returns null, the element is not indexed and hence not referable.

By default we use a DefaultDeclarativeQualifiedNameProvider, which if not otherwise specified looks up a simple name (if there's an attribute 'name') and concatenates it to the qualified name of its parent. It's named 'declarative' because you're able to change the described default behavior per type by just adding a method like this:

String qualifiedName(MyType foo) {
  // compute different qualified name for MyTypes
  // ...
}

It will automatically dispatch to this method as soon as it has to compute a qualified name for an instance of MyType.

With this in place we'll have our elements automatically indexed as long as they are in a project, which have the index nature enabled. Being indexed means that they are globally visible by their qualified name, which is comparable to how public Java elements are globally visible as soon as they are on the classpath.

What's next?

In order to use the index and have it injected into your components (e.g. your scope provider) you'll have to configure the singleton instance from the index bundle into your Guice module. In the example the corresponding binding goes into the UI module and looks like this:

public IndexStore bindIndexStore() {
 return EmfIndexUIPlugin.getDefault().getIndexStore();
}

With that in place you can inject the index store by just adding a dependency in your code:

@Inject
private IndexStore store;

Guice will automatically put the instance into such declared dependencies.
Now that we have a binding for IndexStore we can add the index based scoping to the runtime module:

public IndexStore bindIndexStore() {
 return new PersistableIndexStore();
}
  
@Override
public Class bindIScopeProvider() {
 return DefaultIndexBasedScopeProvider.class;
}

Note the additional IndexStore binding, which is overridden by the binding we previously added to the UI module, but is needed in order to use this stuff at runtime (i.e. without running within exquinox). So it gets active as soon as you run without UI.

How the DefaultIndexBasedScopeProvider works

The DefaultIndexBasedScopeProvider
- looks up EAttributes with name 'importNamespace'
- and translates the globally unique qualified name into shorter ones using those import statements.

By default qualified names with or without a wildcard at the end are supported. For an import of a qualified name the simple name is made available as we know from e.g. Java, where

import java.util.Set;

makes it possible to refer to 'java.util.Set' by its simple name 'Set'.
Contrary to Java the import is not active for the whole file but only for the namespace it is declared in and its child namespaces. That is why you can write the following in the example DSL:

package foo {
  import bar.Foo
  entity Bar extends Foo {
  }
}

package bar {
  entity Foo {}
}

Of course the declared elements within a package are as well referable by their simple name:

package bar {
  entity Bar extends Foo {}
  entity Foo {}
}

Of course the following would as well be ok:

package bar {
  entity Bar extends bar.Foo {}
  entity Foo {}
}

Disclaimer
All this is in a very early stage. The index is not finished and its architecture is not settled down yet. Also the scope provider implementation might be changed in future (I'm sure it will).
Additionally, there are other things around this which we have to work on before considering this mature.

But as I know that there are a lot of bleeding edge users out there, I wanted to share the current state, so you might find a starting point to play with it. The index is an enabler for more advanced functionality in Xtext and in EMF based development in general. So expect it to become an important part in Eclipse Modeling.

Feedback is highly appreciated and should either go to the newsgroup for Xtext (the scoping part) or to the EMFT newsgroup (the index part), because the index project is a component under EMFT.

Galileo! Thank you team!

2009-06-24T10:47:00.012+01:00

Today is the day. The framework we've been working on for over a year has been released.
And before I fall into a big dark hole (because I forgot to think about what is next), I wanted to use this medium in order to thank my team. I never ever had the pleasure to work with such nice and talented people. Without them there would be nothing to release.

Usually I prefer to stick talking about the team as a whole as opposed to talking about the individuals, because the synergy is so important and needs to be emphasized. But on the other hand it's important for me that everybody knows who was involved and made this possible.
So I will make an exception this time and want thank them in alphabetical order:

Heiko (Behrens), our drummer, came in one year ago. The first thing he did was developing the Ecore model generation from Xtext grammars. He's an excellent software developer and at the same time (and that is rare) a highly skilled communicator and team player. He developed a profiler for Xpand and improved the performance of the static analysis of Xpand templates about 300%. Also he designed and implemented our nice new website.
Thank you, Heiko. I have learned a lot from you.

Michael (Clay) is one of the few people who were already active contributors to the previous version of Xtext. He worked a lot on the code completion functionality as well as implemented a textual format for Ecore models as one of the examples. I'm sorry that especially in the beginning, I was not able to integrate him enough, since we had a lot of offline discussions, which he couldn't join. Thank you for contributing (and for your patience).

Moritz (Eysholdt) joined our team in April 2008 in order to write his master thesis on co-evolution of meta models and models (he finished with 100 out of 100 points). In parallel he not only managed the support for the XSD adapter he wrote for Xpand, but wrote the serialization algorithm for Xtext. That is definitely not an easy one and he solved it without too much worry. Good to have you!

Peter (Friese), our tweeting bird, always has an eye on how the project is viewed from outside. He is not only one of the kindest people I know but is so active in the Eclipse Community, that some people even have thought that he was the founder of itemis. Thank you, for doing all that community work and spreading the word. Also thank you for coming up with the WikiText infrastructure, which we used to write the documentation.

I've been working with Dennis (Huebner) for about 6 years now. He's always been a great person to work with. Last year we had serious problems with our build infrastructure. Every now and then strange errors came up and we didn't know what to do. So at some point Dennis took this thing over and managed it. It was great to see how he cared about the whole project, he fixed broken builds even at night or at the weekends. Without him there would be no release today.

Jan (Koehnlein) joined the team in February, 2008. Among all the cool stuff he did, the most noteworthy things may be co-authoring the initial design of the grammar language as well as implementing the partial parsing algorithm. Also he's the most experienced with EMF. Jan has really become a friend of mine over the past year and I especially enjoyed visiting all the conferences with him. But, Jan, please do not talk me into drinking whisky again. Thanks!

Patrick (Schoenbach) is the good fairy behind the scenes. He's responsible at any day time and when ever there was something wich needed urgent action, Patrick was there. Thank you, Patrick, I hope we can meet some day.

Knut (Wannheden) joined the team very late, but had provided a lot of useful features before. Knut is one of these seldom seen developers who look deep into the code and understands what's going on in an incredible short amount of time. Whatever he contributes is of highest quality. Good to have you.

Sebastian (Zarnekow). He is unbelievable. He joined the team late 2008. If you look at the commit statistics, you'll see what kind of productivity gain Sebastian brought to the project. From day one he cared about every single line, he reviewed every bit which was committed. On the other hand he contributed masses of code, which is between the best I've seen so far. Sebastian is propably the one with the most knowledge about the whole system and has become something like a co-lead to me. Besides other things he tought me to improve keeping discipline (still learning ;-)) and why it is important. Sebastian, you made my year.

We've just been out for lunch where I got the chance to take this nice photo (Note the name of that restaurant!). Hopefully we can convince someone to open a new restaurant or rename an existent to 'Helios' soon.

Have fun (while downloading the complete Xtext distro from http://xtext.itemis.com/)!

Xtext : New Website

2009-06-19T08:29:00.006+01:00

It has been online for a couple of days and also been mentioned on twitter as well as in the blogosphere. But we haven't announced it "officially" yet. So here we go:

We have a new website :-)

I think Heiko really did an excellent job. Given the limited time he had (as it states in the screenshot there are only five days left...) the result is just fantastic.

How do you like it?

Code Generation Conference 2009

2009-06-18T08:40:00.005+01:00

This year's Code Generation Conference in Cambridge UK is again a great event. So far I listened to a lot of interesting and entertaining talks and had many discussions on code generation and language development. It's organization is just perfect. They even came up with great weather and fireworks (each night!)!

Thanks to Mark Dalgarno and his team, who again did an excellent job.

I've uploaded the slides of the presentation I did together with Sebastian on Tuesday.

Today is the last day, and there'll be a workshop on a new technology called Xtext.
Sounds exciting ;-)

Xtext : From oAW to TMF

2009-06-05T19:41:00.003+01:00

Only two weeks until the release of Xtext 0.7.0 (simultaneous with Galileo). If you've been using the Xtext version previously shipped with openArchitectureWare (oAW Xtext) you might wonder what has changed and how to migrate to the new version from TMF.

We now have a migration document describing the most important decisions, changes, and the motivation behind them. We hope it will not only help oAW Xtext users to get their projects migrated but also provide them a good start for learning about the new ideas and concepts. Any feedback is highly appreciated!

We are confident that migrating to TMF Xtext is a good idea and pays off soon. Your DSLs will not only run on faster and better code (we e.g. have lots of unit tests), but you'll automatically benefit from cool new features like serialization, pretty printing, template proposals, and semantic highlighting (just to name a few).

Also we, the committers, love the project and are keen on helping you in the newsgroup :-).
Note, that itemis offers migration services and trainings, just in case you're busy or don't want to do it yourself. Just drop me a note in that case.

Code Generation 2009

2009-05-14T07:32:00.005+01:00

Some of my team mates and I will be attending this years Code Generation conference in Cambridge.
Last year's Code Generation conference has been one of the most interesting of the year, so I'm really looking forward to it.

We'll be doing
- a workshop on TMF Xtext
- a talk on general challenges in DSL Design
- and a talk about practical experience at Deutsche Boerse

I especially enjoyed the different gold fish bowls and panels. Looking at this year's topics they should become pretty interesting again.
So if you're at all into code generation, language design or modeling Code Generation 2009 is a must attend event.

Xtext M7 - New & Noteworthy

2009-05-11T19:06:00.004+01:00

Last week was the last milestone (M7) for this year's release train (Galileo). We're pretty happy with the result. It's in a much better shape than what was available after M6 as we cleaned up a number of things and fixed many bugs.

Besides that Xtext now supports lazy linking, the content assist framework has been overhauled and finally is in a shape we're very happy with. In addition M7 enables template proposals (including corresponding preference page) for all languages without the need for any extra programming or configuration. Also the preference page for syntax coloring now works as expected.
Finally we made some progress regarding the EMF Index.

See the New & Noteworthy document for the whole list of changes in M7.

Xtext now contains all features and concepts we wanted it to have in the final release.
There will be five release candidates (RC) before the final build will be made available:

RC1	05/19/2009
RC2	05/26/2009
RC3	06/02/2009
RC4	06/09/2009
RC5	06/16/2009
0.7.0	06/28/2009

We'll spend the next weeks on improving the overall user experience. This includes things like

Documentation
Website
Wizards
UI polishing
minor code clean ups
Bug fixing

In order to allow you helping us ;-), we've prepared a set of Galileo M7 distros containing everything necessary to get started with Xtext. The zips can be downloaded from itemis' website:

Any feedback is as always highly appreciated. Have fun!

EclipseCon: Slides and other stupid things

2009-03-25T19:39:00.003+01:00

So far EclipseCon was much fun and I'm pretty confident that we will have a good time for the rest.

One thing which might be especially entertaining is the talk Peter is giving this afternoon (15:30 / Theater).
Now that he's been living in America for some time, it seems that he like many other American finally is convinced about the stupidity of modeling. So if you also think modeling is stupid, let's meet in the talk to gossip about it. See you there ;-)

Btw.: I just uploaded the slides for yesterday's talks (xtext , generic editor). Unfortunately slideshare is not able to show the embedded videos and transitions Jan and I put into the slides :-(

TMF Xtext M6

2009-03-19T11:06:00.002+01:00

The M6 builds are published and ready for download. We've prepared a distro containing everything you need and have written a short getting started tutorial which should give you a feel of how Xtext works. M6 Xtext is using MWE and Xpand, so oAW Xtext users should feel comfortable. Note that the M6 release is not intended to be used in production. We'll still change things and if you don't get support by one of the committers (see below), you might need to have good code archeologists on your site ;-). The production-ready release will be available in June (Galileo).

During M6 we've focused on a large number of things (see New and Noteworthy). For instance, TMF Xtext now generates EMF classes for the derived EPackages (using existing EPackages is of course supported as well). The generated EClasses make use of latest improvements in EMF, namely the MinimalEObject. This has reduced memory footprint by about 30% and at the same time made the runtime even faster.

Another noteworthy thing is the new composable code generator API. Xtext is now shipped with lots of so called generator fragments, each adding functionality for a certain aspect. For instance, the generation of EMF classes is implemented as such a fragment. You can choose from those provided fragments, configure them individually and even add your own fragments in order to have a generator, generating exactly what you need. A fragment can contribute, to Manifest.MF, plugin.xml and the Guice modules. Sebastian has written about it.

Looking at our project plan, one can see that M6 is also the API freeze milestone. Have a look at Heiko's post and the corresponding wiki page to see how we handle API contracts.

Jan and I will present TMF Xtext M6 at EclipseCon next week.

There are a couple of things we want to do in M7 (enum rules, lazy linking, etc.). In addition it's planned to put the EMF Index into a shape, where it can become a solid foundation for linking and navigation in Xtext. Jan will give a short talk about the EMF Index at EclipseCon.

We do consulting

Although mentioned, that M6 is not yet production-ready, we at itemis already help a couple of customers using TMF Xtext in industry projects. It's so important to use new technology to solve real problems and gather feedback which is based on real projects as soon as possible. Using TMF Xtext in many different projects gives us a pretty good feel for what's important and what is not. And so far the feedback from those projects is very positive.

It's also good to see that our business model actually works well (even in these times). This is especially important since doing open-source development which is only funded by doing consulting, is not that common at Eclipse.

So, don't forget that we provide professional support! This does not mean that we won't do regular unpaid support through the newsgroup. It's just that if you need things like prototypes, reliable response times, onsite trainings, etc. you can get it.

itemis on TV

2009-02-02T08:00:00.003+01:00

There was a competition in western Germany comparing medium-sized companies and how good they are as employer. And itemis won first price :-)

See my colleagues on TV : http://tinyurl.com/aez2bd
And in the Bild-Zeitung : http://tinyurl.com/b9rpgc

Xtext - new parser backend

2009-01-19T16:13:00.005+01:00

So far, Xtext's parsing was based on Antlr 3.0, a very popular Java-based parser generator.
Unfortunately we've had an IP-issue with Antlr. As Xtext is an Eclipse project, all used libraries need to get IP-approved by the EMO. This has been done for the runtime of Antlr 3.0 but not for the generator. The generator is implemented in StringTemplate, Terence Parr's template language. The problem is that StringTemplate itself is implemented in Antlr 2.x, which won't get an approvement, because it is not clear where all it's code orginally comes from.
As Terence won't be able to migrate StringTemplate to Antlr 3.0 before this summer, we had to solve this somehow. :-(

The good news are, that we found a very nice solution:
We're implementing our own parser generator :-)

Actually Sebastian already did so and is still working on things like error recovery and error reporting. The parser backend is based on a packrat parsing algorithm, which means it is scanner-less. We've read all kinds of related papers (Actually, I was surprised how good the quality of those papers are) and played around with prototypes.

The new implementation is three times faster. This does not mean that Antlr is three times slower, but that we've used it in a rather expensive way :-). The replacement of a lexer-based parser with a packrat algorithm also allows for much simpler composition of grammars (a.k.a. grammar modularization). And we can provide nice hooks for error recovery and handling.

We now develop both solutions in parallel, so that the dependency to Antlr's generator is only 'works-with' as oppposed to 'pre-req'. But we hope that we're soon able to remove the Antlr implementation completely.

Stay tuned.

Xtext Scopes and EMF Index

2009-01-08T10:32:00.012+01:00

There is a new proposal for a so called EMF Index. At ESE I got the impression that a lot of people are looking for such a thing or have already built their own. To make clear what we expect from such a project, I'll try to explain why and how TMF Xtext needs such an "Index".

The main difference between Xtext and Oslo's MGrammar or other parser generators, is that Xtext not only provides abstractions (mostly DSLs) to describe the syntax of a language, but also for implementing other aspects. One is linking. So where other frameworks create a tree, Xtext also takes care of the cross-links, hence creates a graph (a.k.a model).

How does this work?

Let me explain this by example.
Assume you want to parse the following model:

entity Animal
entity Dog extends Animal

That is, two declarations of something we call 'entity' one 'extending' the other. The extend declaration 'extends Animal' cross-links to the actual declaration 'entity Animal'.
So that we're able to write something like this when working on the parsed model later:

myDog.getExtends().getName().equals("Animal")

What do we need to do, to get this working?

First of all, one has to specify the syntax of the language including the syntax for the cross link. With Xtext one not only specifies the syntax but also writes down how a model is created during parsing:


MyModel : (entities+=Entity)*;
Entity : 'entity' name=ID ('extends' extends=[Entity|ID])?;

This would result in an ecore model of the following structure:


EPackage {
   EClass MyModel {
        containment entities : Entity[]
   }
   EClass Entity {
        name : EString
        extends : Entity // the crosslink
   }
}

Naturally a parser is only able to create a tree, so parsing an instance of the DSL defined above would result in an unlinked model, which has to be linked in a second phase using the provided ID (which was 'Animal' in the introductory example).

So how do I find an Entity which is 'identifiable' by the text 'Animal'?
By default Xtext assumes that the name of an EObject (if it's EClass has such an EAttribute) is the identifier. All the named elements within the same file are visible (as long as they have a unique name). We also have a very simple import mechanism:
If you have an EObject, containing a string in an EAttribute called 'importURI', Xtext automatically creates an outer scope containing the content of the referred EMF Resource. "Outer scope" what's that?

Scoping
In Xtext scopes (IScope) are nested. Each scope makes EObjects visible by an identifier (String).
Assume we have added the import feature described above:


MyModel :
(imports+=Import)*
(entities+=Entity)*;

Import :
'import' importURI=STRING;

Entity :
'entity' name=ID ('extends' extends=[Entity|ID])?;

... we would be able to have two files:

myModel1.dsl


 entity Animal

and otherModel.dsl

 import "myModel1.dsl"
entity Dog extends Animal

The scope used to do the linking in the declaration of entity 'Dog' would have an outer scope containing the definitions from the imported file ('->' means outer):

 scope (elements from otherModel.dsl) -> scope(elements from myModels1.dsl)

If we would add additional import statements, we would get additional outer scopes in the order of declaration:


import "myModel1.dsl"
import "myModel2.dsl"
import "myModel3.dsl"
entity Dog extends Animal

results in


scope(elements from local resource) ->
scope(elements from myModels1.dsl) ->
scope(elements from myModels2.dsl) ->
scope(elements from myModels3.dsl)

So the linker would ask the most inner scope for an element called 'Animal'. If it contains such an element it returns it if not it asks it's outer scope.
This means that an inner scope overlays elements from the outer scope. So it would be ok to have a declaration of 'Animal' in the local file, but the one imported from 'myModel1.dsl' wouldn't be referenceable anymore.


import "myModel1.dsl"
entity Dog extends Animal
entity Animal // this one overlays the definition imported from myModel.dsl

If you don't want to allow overwriting things, you'll have to add constraints, which is of course possible but is a different topic.
Ok, I hope you have an idea of how linking in TMF Xtext basically works.

Although the described default semantics might be sufficient in many cases, sometimes scoping and linking is a bit more sophisticated. We won't need (and currently have) something like an Index, but it might speed things up, if one wouldn't need to load referenced resources while linking but just ask something like an Index, what's in a resource. The Index could provide a normalized EMF URI, which can then be set into a proxy.
Also there are IDE things like "Find Model Element" or code completion for available resources, which would be easy to implement on top of an EMF Index.

Advanced Scoping and Linking
Anyway, if you want to have something more file-system independent like Java's class path, where one imports name spaces instead of actual URIs, you would need some kind of repository (similar to the class path) containing all referenceable elements. This is because it would far two expensive to "scan the world" each time you want to satisfy a link.

In fact I think that leveraging the Java class path is a very good idea, since it is well understood by Xtext users and is well supported in the development phase (Eclipse JDT, or even the OSGi support from PDT) and at runtime. That's why Xtext has a URIConverter introducing a class path scheme for EMF resources. So what we want to do most of the time is to scan the class path for EMF resources and index them.
We would need to index them per container (jar, class-folder, etc.), because the class path is also scoped hierarchically.
Such a hierarchy could look like so:

 classpathScope{stuff from bin/} ->
classpathScope{stuff from foo.jar/} ->
... ->
classpathScope{stuff from JRE System Library}

And of course, we would like to have these global scopes backed up by the EMF Index transparently integrated into our scoping hierarchy. This turns out to be very natural if we look into a final example, showing how we would implement the scoping for Java:

// file contents scope
import static my.Constants.STATIC;

public class ScopeExample { // class body scope
 private Object field = null;

 private void method(String param) { // method body scope
  String localVar = null;
  innerBlock: { // block scope
   String innerScopeVar = null;
   Object field = null;
   // ?SCOPE?
  }
 }
}

The object scope created in the inner block (//?SCOPE?) would look like so:

 blockScope{field,innerScopeVar}->
methodScope{localVar,param}->
classScope{field}->
fileScope{STATIC}-> //the static import
classpathScope{static fields from bin/} -> // (e.g. my.Constants.STATIC)
classpathScope{static fields from foo.jar/} ->
... ->
classpathScope{static fields JRE System Library}

For performance reasons it would be useful to have some kind of database (EMF Index) backing up the class path scopes. Especially during development (modeling) , because it would be necessary to re-index changed models.

EMF Index
So mainly we want to have something which tells us what elements are available in a given 'world'. Such a 'world' like a Java class path includes EObjects (from several EMF resources). It should be possible to define and configure arbitrary implementations of 'worlds' (databases, web, workspace, etc.). Elements contained in a world, need to be selectable using an identifier (unique within a world). It also should be possible to add arbitrary additional information to such entry.

As mentioned, IMHO such an Index is important to track changes during development (i.e. modeling). Also we want to have code completion for globally available elements, look model elements up by name, etc.. At runtime we need to load all the models anyway, so the need for an index is not that important.

This has been a lengthy post (sorry). But if you made it to this point, it would be very helpful to hear what you think about this. Would the scope abstraction work for the languages you have in mind? What do you expect from an EMF Index? Maybe answers to the latter question better go to the EMFT news group :-)

EMFatic implemented in TMF Xtext (Screencast)

2009-01-06T22:04:00.009+01:00

Michael Clay is working on a TMF Xtext implementation of EMFatic (see this screencast).
He's also prepared a screencast showing template proposals on Martin Fowler's DSL example.

We'll show more of TMF Xtext at EclipseCon in march.

W-JAX slides online

2008-11-10T14:12:00.003+01:00

I've just uploaded the slides of the presentations I gave at last weeks W-JAX conference in munich.

Scala - (It's an introduction to a programming language called Scala. Most of the slides contain code snippets only, so it may not be that usable without explanation.)

Vermisste Sprachfeatures in Java (it's in german and is about some IMHO important language features Java lacks.)

Multi-line String Literals in Java

2008-10-17T12:28:00.008+01:00

Having multi-line string literals is a common and useful thing in most modern languages. It is for instance very convenient to use multi-line strings to specify test data in-line as opposed to referring to external files containing the data. As Java lackes multi-line string literals, one observes the definition of data in external files very often.

I've implemented multi-line strings in Java using a library approach, so that it's possible to write the following:


    /**
  * @param args
  */
 public static void main(String[] args) {
  System.out.println(S(/* 
     Wow, we finally have
      multiline strings in
       Java! HOOO!
          */));
 }

which will print the following to standard out:


                                 Wow, we finally have
                                 multiline strings in
                                 Java! HOOO!

How does this work?
Well, first of all you have to make sure that your source is on the class path, then the following code does the job:


   public static String S() {
       StackTraceElement element = new RuntimeException().getStackTrace()[1];
       String name = element.getClassName().replace('.', '/') + ".java";
       InputStream in = getClassLoader().getResourceAsStream(name);
       String s = convertStreamToString(in, element.getLineNumber());
       return s.substring(s.indexOf("/*")+2, s.indexOf("*/"));
   }

Obviously this don't perform that well, but for unit testing it's sufficient.
Maybe it would be cool to add some support for interpolation functionality to it (using e.g. groovy or Xpand).

Article on Xtext and oAW

2008-09-24T08:13:00.002+01:00

I just found a really nice article/tutorial on Xtext and oAW on the web.

It walks you through the process of
1) designing a dsl with xtext
2) adding constraints
3) enhancing the generated text editor
4) generating some code out of it
in great detail (about 30) and very well written.

The author comes to this nice conclusion :

Finally, consider the effort we had to invest. It took a couple of pages to describe the process, but when you count the actual lines of code we had to write and the number of clicks we had to perform, you may agree that this does not amount to anything worth worrying about. The framework kept most of the complexities away from us.
It is the power of frameworks like oAW that let us leverage the potential of DSLs and related concepts into our daily projects.

:-)

Language feature supporting internal DSLs

2008-09-18T20:49:00.013+01:00

There's a lot of discussion going on about internal or embedded DSLs and what programming languages are best suited to develop them.
Mainly an internal DSL is about pushing the syntactic flexibility of a language to it's limits in order to write against an API in a domain-specific manner. What that means depends on the DSL but in most cases it means simplification of client code by implying context, allowing declarativity and avoiding DRY-violation.

Many people think, that it is the dynamic nature of a language which makes it suitable for internal DSLs. That is only partly true. More important than the possibility to do meta programming is to have a flexible syntax allowing to invoke functions with different syntaxes (omitting paranthesis, semicolons, etc.). Ruby and Scala are both very flexible in this sense.

Working on the new version of Xtext, I had an idea for what I think would be a cool language feature.

Introducing new syntax via libraries
When declaring a function, one typically only declares a name and what kind of parameters need to be passed in. Programming languages themselfs then have defined a (more or less flexible) generic way of how such functions can be invoked.

Example:

def foo(String x) : "foo"+x;

which is invoked like this :

foo("bar")

Imagine a language which alternatively allows to specify the concrete syntax of how such a function is invoked.:

def  myFunc : 'foo' x=ID : "foo"+x;

Where the part between the first and the second colon introduces a new syntax (I've reused the Xtext syntax here) which can be used to invoke the function, so one could simply write:

foo bar

Where 'foo' is a newly introduced keyword (limited to the static scope where the definition is visible of course) and bar is an identifier which is a builtin lexer token of type String.

Another example:

def Person :
    ('girl'|male?='boy') name=ID (lastName=ID)? ('from' city=ID)? :
    new Person(male,name,lastName,city);

def greet :
    p1=Person 'greets' 'the'? p2=Person :
    p2.greetedBy = p1;

which would allow for expressions like:

boy Sven from Kiel greets the girl Scarlett Johansson

and would effectively construct two instances of person and link them. :-)

Obviously this would be a tough thing to implement (parsers which dynamically change their behaviour) and it would also be problematic to avoid ambiguities with syntaxes imported by other libraries or introduced by the host language. I haven't spent much time on thinking it out yet. So maybe there are obvious show stoppers I overlooked or one of your neighbors has already developed such a language a decade ago?

Product-line engineering with UML

2008-09-05T07:23:00.005+01:00

Some time ago one of our current customers, an international tractor manufacturer, wanted us to come up with a domain-specific language to describe his products (i.e. tractors) in order to manage the different variants, etc.

I always was really skeptical about using UML to define DSLs, but after investigating several month in tweaking it, we came up with an IMHO very pragmatic and concise solution.
With this a farmer is now able to simulate how it feels to work with the different tractors.
The DSL provides the typical variability people need, when defining a new product:
- number of wheels
- number of exhaust pipes
- whether the farmer wants to sit at the front or behind
- etc.

Here's an example of how such model looks like. Isn't that declarative and intentional?

With Ed on board and his new contacts at NASA, we hope we can reuse this DSL in order to model space ships, etc. at NASA.

Testing Processing Time on a JVM

2008-09-03T12:34:00.006+01:00

I think most people would agree that having some unit tests, which ensures that non-functional requirements are met, is a good idea. With processing time people usually use time in order to measure whether a particular function runs within the requirements. However, everybody knows that measuring time is inaccurate, since it varies depending on the machine the tests are running and whether there are other process consuming resources in parallel.

Wouldn't it be cool to just count the atomic operations per thread executed by e.g. the Java VM?
We could than have a reference system, which is used to give each atomic operation a value (e.g. nanoseconds).

Then I would like to write tests like this:

@ExecutesIn(23000)
public void testFoo() {
doStuff("Foo "+" bar");
}

Where the number 23000 is the time it would take to run the code on the reference system. The test would always take the same time, no matter it was run on an x386 machine or a modern multi GHz box.

Questions:
Does anyone know whether such a tool/library/framework already exists?
And if not, do you have any idea of how this could be implemented?

DSLs in the Netherlands

2008-09-01T10:35:00.008+01:00

Last friday I was in the Netherlands near Amsterdam to give a talk about Xtext at profict summer camp 2008. This was really a nice event in an impressing location (see picture - people work there!). They'd invited thee speakers:

First Neal Ford gave a very entertaining introduction on DSLs. After that he talked about implementing internal DSLs with Java, Groovy and Ruby. He also mentioned external DSLs very briefly and recommended Antlr for this very much.

His talk gave a nice foundation for my introduction of Xtext. Which I did by leveraging the terminalogy Neal introduced (As Neal is a collegue of Martin Fowler and has done a lot talks on DSLs with him, I assumed that he will stick to Martin's terminology and I was right) . I also showed the implementation of the example from Martin Fowler's upcoming Book on DSLs, which should help to compare the different technologies (internal DSL in Ruby, Antlr, Xtext).
I've been asked to put my presentation slides online. So here they are.

The last presenter, Zef Hemel from TU Delft, showed us a web framework called WebDSL, which uses a lot of external DSLs to make development of database driven web applications (CRUD and more) easier. It is implemented in strategoXT, which is crazy stuff.

After that there was a panel while outdoor a great smelling barbecue was prepared. Unfortunately I had to catch my flight back, so I had to leave without any barbecue :-(.
Anyway the event was really nice and I've met a lot of very nice people there. It seems that DSLs are a hot topic in the Netherlands.

Ed Merks now working for itemis

2008-09-01T09:51:00.004+01:00

Last Friday, Ed Merks (Eclipse Modeling PMC Lead, EMF Lead, Board Member... you know him) has officially announced that he's now working for itemis AG.
I'm especially happy with this because when I first heard that Ed has quitted IBM, I was a bit worried whether he would be able to do the great job he's doing at Eclipse in the future.

I'm pretty sure that this was a good step, not only for itemis and Ed but also for Eclipse and Eclipse Modeling, since this ensures that Ed can follow up on helping Eclipse Modeling becoming the most open, flexible and powerful DSL toolkit.

Why? Because the business model of itemis is not to develop another commercial product on top of Eclipse but to help people using the open-source technology at Eclipse directly (we're doing consulting, trainings, support, etc. then).

And we don't want to do consulting which could be eliminated by better tools and better documentation. We think that DSLs and generators/interpreters will and should become mainstream. Therefore we need to lower the barriers and make the tools and languages as good as possible. We want to make software development better. AFAIK this doesn't conflict with Ed's goals :-)

Refactorings for oAW 4.3

2008-08-08T08:47:00.005+01:00

Steven Reinisch, now studying at hasso-plattner-institute for software systems engineering, has added some cool refactorings to Xpand and Xtend languages of openArchitectureWare 4.3 during this year's Google Summer Of Code.

During the development he lived in Kiel in order to work in our office at itemis. He managed to understand the code base of Xpand more or less independently. It was a pleasure to work with him and I'm looking forward to future contributions from Steven (He said he's willing to do so :-)).

It's not sure where and how this work will be made available. If we don't put it into oAW 4.3.1, itemis will include it in the distros available at oaw.itemis.eu when it's finished.

Here's a screencast steven made.

MDSD and DSLs

2008-07-15T11:21:00.005+01:00

I'm regularly reading Martin Fowler's blog and am especially interested in his writings about DSLs. I usually tend to agree with and like what he says about that topic.
In his recent posts he started to talk about MDSD and how it's related to DSLs. As I'm somewhat involved in the MDSD community, I wanted to clarify what MDSD means to me.

Basically, MDSD is about
1) defining domain-specific abstraction (no matter how) and
2) make them executable (no matter how).

Defining abstractions

Sometimes the general-purpose programming language (GPL), e.g. Java or Ruby, can be used to define such abstractions (we call these abstractions APIs or internal DSLs). If the GPL is not flexible enough or we need static typing or better tooling, etc. we design an external DSL.

Most people think that developing an external DSL is complicated, because if it's a textual DSL you need to develop a parser, if it's graphical you need an editor, etc.. Therefore people customize/extend existing languages like XML or UML a lot.

However, nowadays it's not complicated to define an external DSL anymore. Frameworks like Xtext and the Eclipse Modeling Project in general offer a variety of tools and libraries to make the design of external DSLs easy and more important flexible, in terms that the DSL can grow and evolve over time. DSLs are a lively part of your project and therefore need to evolve like the rest of your software.

Make DSLs executable

"A difference is that most MDSD people assume that you'll want to generate code from that model rather than executing the model directly."

I don't assume code generation. Interpretation is of course a common option. Both approaches have their pros and cons. Generally speaking, one could say that code generation has strength when the DSL is of a structural nature (like a domain or component model), the target language is statically typed and you don't need to have dynamic execution of DSL expressions.
But it's also perfectly ok to generate code out of behavioural languages (such as statemachines) or interpret structural information like hibernate or spring do with their XML-based configuration DSLs.

Other misconceptions about MDSD

"In order to use model-oriented DSLs you need a different, RepositoryBasedCode, approach to tooling."

There are a lot of modeling tools using this approach. However we (Eclipse Modeling) use plain text files to store our models, because we think it's important to have all relevant information to build the software system in the same version control system. Also we want to reuse common, proven tools such as Diff and Merge.

"DSLs seem to have a proportionately higher emphasis in the MDSD world than they do in the mainstream programming world."

MDSD is not a holistic approach to develop software or a replacement to "mainstream programming", it's just the "how can I use external DSLs"-technique. And DSLs are just one useful thing among many others. MDSD emphasizes DSLs like AOP emphasizes Aspects.

"A particularly visible sub-community of MDSD is centered around ModelDrivenArchitecture (MDA). I'm not much of a fan of MDA in particular ..."

MDA indeed is a heavyweight thing which is based on overly complex standards and faces problems which are not relevant in most situations (like being platform independent). Also the suggestion of using UML and it's profile mechanism to define DSLs and layering code generators (make them target platform independent) with multiple model transformations is questionable.
To me MDSD has not much in common with MDA.

"Model Driven Software Development (MDSD) ... considers itself as an alternative to the traditional style of programming"

To me it's more like an extension. Again: MDSD is only about using DSLs, interpreters and code generators and integrate them in your project. To be a good software developer you need to have much more than this in your mental tool box.

"Most fans of MDSD base their enthusiasm on the basis that models are ipso facto a higher level abstraction than programming languages."

I also have met people (mostly with a CASE or MDA background) who haven't yet understood that models are just information like code is information and DSLs are just about specifying information in an abstract, concise, intentional and non-redundant manner. Models are code!
Sometimes those people even talk about "hand-written code" like it was something dirty. They argue that with models and code generation everything gets automated and forget that especially modeling in a graphical tool takes more time than writing the same information into a text document. IMHO such statements are one of the reasons why people like Martin think of MDSD being a holistic, tool-centric, heavy-weight approach (this is what I read between the lines).

MDSD Best Practices

2008-06-26T09:31:00.003+01:00

InfoQ just published a new article written by Jan, Peter and me.
It's called "Best Practices for Model-Driven Software Development".

I hope you enjoy it!

Next Monday: Eclipse Stammtisch in Dortmund, Germany!

2008-06-17T16:32:00.003+01:00

Just a short reminder for those who're interested in meeting up with Ralph and other Eclipse enthusiasts.
On next monday (June 23rd) an Eclipse Stammtisch is taking place at 7 p.m in the Strandbar SOLENDO (http://www.solendo.de) near Dortmund harbor (didn't know they even have water there ;-)). Anyway they have BEER!
Cheers!

Fowler's DSL example with Xtext

2008-06-10T14:42:00.010+01:00

In his recent DSL related blog entries (ParseFear and Syntactic Noise), Martin Fowler mentioned a DSL example he's using in his upcoming DSL book. Here you can see how it is implemented using Antlr and a lot of Java.

This is how it can be done using Xtext and openArchitectureWare.
1) Download the oAW distro from itemis
2) Create a new Xtext Project
3) Paste the following Xtext grammar into the opened editor:


Statemachine :
  'events'
     (events+=Event)*
  'end'
  'commands'
     (commands+=Command)*
  'end'
  (states+=State)*;
 
Event :
  (resetting?='resetting')? name=ID code=ID;
 
Command :
  name=ID code=ID;
 
State :
  'state' name=ID
     ('actions' '{' (actions+=[Command])+ '}')?
     (transitions+=Transition)*
  'end';
 
Transition :
  event=[Event] '=>' state=[State];

4) start the generator (right click on generate.oaw Run->oaw workflow)
your're done.

You not only get an Antlr based parser but also get an EMF based AST (SemanticModel), and a fully fledged eclipse editor.

Of course it's up to you whether you want to interpret models or generate some code out of them. I choosed to write a small Xpand template file, generating a "controller".
Here it is.

Codegeneration 2008

2008-06-10T10:53:00.003+01:00

Codegeneration 2008 is coming! And there'll be a lot of friends giving talks.
Peter and Frank are talking about model-driven Lego.
Markus will give two talks (Implementation Techniques for Domain-Specific Languages and Building Interpreters with EMF, Xtext and Scala).
I'll do two tutorials together with Arno:
One is called Concrete Syntaxes of DSLs and the other will give an Overview of Eclipse Modeling.
Finally on Friday, Karsten, Jan and me are holding a workshop about Using openArchitectureWare for M2M and M2T.
As this conference is specialized on code generation I'm looking forward to some interesting discussions. See you there!

Current Development at itemis Kiel

2008-05-16T21:15:00.004+01:00

Just to give you a short update.
We've been working on a complete rework of Xtext for the last two weeks. While Jan and me have been developing the grammar language and witten the code generator, Peter and Dennis started to develop Xtext's new IDE framework.

Started with the new grammar language written in the new grammar language, we are now able to generate the parser, meta model and some other cool stuff from that description. Actually it took us some time to understand and manage the whole bootstrapping process. It 's never been so easy to shoot yourself in the foot... :-)

Currently we're working on a generic serialization mechanism. With this, one can modify instantiated models and write them back without loosing original layout information or comments.

Hopefully it won't be long until we can start reimplementing Xpand using the new framework (we yet have to add syntactic and semantic predicates to the grammar language in order to describe Xpand).

The current state can be checked out from eclipse's cvs : :pserver:anonymous@dev.eclipse.org:/cvsroot/modeling/org.eclipse.tmf/org.eclipse.xtext
Milestone planning can be found here : http://wiki.eclipse.org/Xtext_Project_Plan
Feedback or feature requests are of course welcome. Please use the developer mailing list for that : xtext-dev@eclipse.org

Game development with oAW

2008-04-16T10:30:00.006+01:00

Last week Jens Wagener posted a small riddle in his blog. He stated that it is claimed that 98% of all people are incapable of solving it.
To put it short: I didn't solve it. (Thank God! I'm not that freaky ;-))

But hey, as we're currently testing our next release of openArchitectureWare, I thought I should give it a try and do something which helps solving it (our at least help to understand that it is really hard to solve ;-)) using oAW.

So first I wrote a DSL implemented in Xtext and then I specified the constraints outlined in the riddle using Check. (At this point you should have read Jens' post so you can understand what I'm talking about)

Using the DSL one writes the attributes (Color, Role, Pet, Language and Magazine) for the five houses down, one house per line. As I said the constraints outlined in the riddle are implemented using Check. So if there are any of theses constraints violated the editor provides the appropriate feedback.
The puzzle is solved when all markers have gone and there are no more wildcards 'X' in it.

This is a screenshot, of the "game" implemented in oAW:

So if you want to play around with it, just drop me a mail, and please don't come up with boring rule engine solutions ;-)

Initializers in Java

2008-03-28T10:11:00.005+01:00

I'm working on slides about language features I'ld like to see in Java. One of them are initializers.
IMHO the lack of this features is the main reason why everybody uses XML. It's just ugly to construct object graphs in an imperative way. Usually such code looks like:


Customer c = new Customer();
c.setName("foobar");
c.setId(4711);

Address address = new Address();
address.setStreet("Schauenburgerstr. 116");
address.setZip("24118");
address.setCity("Kiel");

c.setAddress(address);

Order o1 = new Order();
o1.setArticleId(0815);
o1.setAmount(2);

c.addOrder(o1);

Would be much better if we could specify such a data structure like this:


Customer c = new Customer {
name     = "foobar",
id       = 4711,
address = new Address {
 street = "Schauenburgerstr. 116",
 zip     = "24118",
 city    = "Kiel",
},
orders = [new Order {
 articleId = 0815;
 amount = 2;
}]
};

Actually Java already supports this in some way (not ideal but IMHO better than doing it in a procedural manner). The following code is working Java code:

Customer c = new Customer() {{
name    = "foobar";
id      = 4711;
address = new Address {{
 street = "Schauenburgerstr. 116";
 zip     = "24118";
 city    = "Kiel";
}};
addOrder(new Order {{
 articleId = 0815;
 amount = 2;
}});
}};

The code creates anonymous subclasses with initializers in it. However, as there are no such thing as properties for now you may want to replace the field access by getters and setters. Note that you don't have to make the fields public. So with a combination of getters and protected fields you get immutable types.
Btw.: the same mechanism forks for collections of course:


List l = new ArrayList() {{
           add(new Order() {{
                   setName("stuff");
           }});
           add(new Order() {{
                   setName("foo");
                   setAmount(34);
           }});
   }};

I'ld still prefer "real" initializers and "real" collection literals, but all in all I think this seems to be a pretty useful idiom.

DSL development with MagicDraw

2008-03-19T22:56:00.003+01:00

Ekkehard Gentz just showed me how to customize MagicDraw so that diagrams are really domain specific (no UML anymore). And it looks pretty cool. You can (and it seemed not to be too hard to do so) hide everything you don't want, extend and customize existing UML2 diagrams, etc.
In the end the modeling language ekkehard showed me didn't look like UML anymore. It was very simple and just showed the domain-specific concepts.
Unfortunately, if you go and process those models you are back in UML-hell again. But as all the customization information is modeled within the profile, one could transform a real domain-specific meta model (based on ecore of course) and a corresponing transformation from it. This would encapsulate the UML stuff completely.
Ekkehard will present his DSLs at this year's JAX during the DSL day.

Mega modeling at EclipseCon

2008-03-19T16:24:00.005+01:00

Yesterday, at EclipseCon there was a BOF called "Mega Modeling Mania", which was pretty interesting. It started with a discussion whether UML is a good starting point for domain specific languages or not. Some people argued that with profiles you can do a lot of cool stuff, but the tools have failed. But in the end I think (and hope ;-)) most people were convinced that there is no point in starting with such a huge meta model with hundreds of concepts in it just to design a language having couple of them.

After that we were discussing whether Ecore is good enough for meta modeling. Some people would like to see CMOF concepts like "Package Merge" or "Associations". Actually, the absence of associations was one of things I really liked when I started to do meta modeling in Ecore. And until today I never missed them. I got the impression that it is more a matter of taste, and that people who meta model using graphical syntax and often use bidirectional relationships want to design them at once. So to me this sounds more like a tooling thing...
Package merge seems to be an application of model-to-model transformations. And because it's so specific it makes sense to have a DSL to describe those transformations. Markus did this when he developed XWeave. Actually Achim Demelt told me that XWeave is even more powerful than Package Merge. So why should we want to have that in the core, i.e. EMF? It's already there and you can use it if you want.
To me everything seemed unecessary complicated.

EclipseCon 2008

2008-03-02T13:58:00.003+01:00

Originally I didn't plan to attend this years' EclipseCon. But as we have big plans wrt our contributions to Eclipse Modeling, I decided to register myself in order to meet up with the other "Eclipse Modelers". Wolfgang will join as well.
Too bad I didn't submit some talks myself. Anyway, there are many interesting presentations to be visited. :-)

oAW at EclipseCon and QCon

2008-02-21T22:01:00.002+01:00

Markus Voelter and Bernd Kolb will present oAW (and Eclipse Modeling) at EclipseCon. There are several talks regarding Eclipse Modeling. Strange thing that some of them take place at the same time in different rooms... ???
Anyway, if you're there make sure to visit their talks, because they will present some cool new features :-)
In addition Markus will talk about Xtext at this year's QCon. You have to go there if you 're at all interested in DSLs!
Markus and Bernd: Thank you for traveling around the world and presenting it. :-)

itemis labs kiel - Team blog

2008-02-18T12:54:00.004+01:00

There's a rumor about Itemis AG in Kiel working on great ;-) open-source software hosted at eclipse.org as well as openarchitectureware.org.

If you want to keep yourself informed about their latest activities go here:

http://apps.itemis.de/roller/itemislabkiel/

Sven Efftinge's Blog

I like free beer!

Collection Literals in Java (or a hidden gem in Eclipse JDT)

Upcoming Presentations on Xtext / Xtend

Groovy, Scala, Java, Xtend - an UPDATED stupid comparison

Measurement

Groovy

Scala

Clojure

Xtend

Java

Groovy, Scala, Java, Xtend - a stupid comparison

A possible Xtend version

I also did a Scala version:

And finally a Java version:

Writing Android UIs with Xtend

What’s So Special About Xtend’s Extension Methods?

Extension Methods

Static Methods Are Bad!

New Xtend Website

Build Your Own JVM Language!

This Is Not Your Father's Java

Xtend 2.1 - Type-safe Groovy-style Builder API

Riding the Waterfall

Nordic Coding Part III

Upcoming Eclipse Democamps Featuring Xtext

5 simple steps to Fowler's DSL with Xtext 2.0

Code Generation 2011

Eclipse Xtend - beta available

Dear Java killers,

Nordic Coding with OSGi, Xtext and Git (April 15th)

A rough performance comparison between Xpand and Xtend 2

A Guice Modules DSL

Xtend Screencast Part 3 - Rich Strings (i.e. Codegeneration)

Xtend Screencast Part 2 - Switch Expression

Xtend Screencast Part 1 - Basics

REPL for Xbase

Xtend 2 - the Successor to Xpand

*Extreme* Nordic Coding

Nordic Coding

Xbase - A new programming language?

What is Xbase?

Why Xbase?

Main Decisions

Runs on the JVM

Compiles to Java

Interpreter

Statically Typed

Full Java Generics

No built-in types

Closures

Type Inference

Operator Overloading

Simplicity over Syntactical Flexibility

Everything is an Expression

Powerful Switch Expression

Current state

Xtext in Indigo

Xtext Version 2.0

What we did so far

Milestone builds

What's next? (Plan for Indigo)

More Languages

Will talk about Xtext in Antwerp and Stockholm soon.

Xtext 1.0.1 (SR1) release and git migration on Wednesday

Parsing Expressions with Xtext

Xtext does musical scores

Xtext Webinar is online

5 Good Reasons To Watch Our Xtext Webinar

Screencast: Xtext in the automotive industry

WYSIWYG with Xtext

Google's Protocol Buffers implemented in Xtext

Modeling Day @ JAX 2010

Xtext won Eclipse Community Award!

EclipseCon 2010

Multiple dispatch (and poor men's patter matching) in Java

Xtext Helios M5 is out!

Xtext's new builder infrastructure

RE: The Hidden Costs of Domain-Specific Languages

Xtext - Using existing Ecore models

Extreme Nordic Coding