Friday, December 03, 2010

Xtend 2 - the Successor to Xpand

M2T Xpand is one of the widest used template languages for code generation. It is statically typed, supports polymorphic dispatching of template definitions and provides a very convenient way to navigate through models. The navigation convenience mostly comes from the built-in higher order functions for collection types, the possibility to have extension functions using Xtend and some syntactic sugar for common expressions.
Also Xpand is known to provide a very open architecture. There are several hooks in the interpreter to integrate profilers, pretty printers or whatever comes to mind. That's all very nice and has led to quite some success.

But there are also some issues with Xpand and Xtend (Just in case you don't know I am not blaming other people's work, because I'm the author of Xpand, Xtend and Check)

  • Performance - Xpand and its siblings are interpreted and the interpreter was written without much performance considerations, i.e. is slow.
  • Tooling - The tooling is hand-made with JFace and Co, the code is old, not well tested and unfortunately a bit buggy. 
  • Ugly conceptual short cuts - built-in higher order functions instead of closures, built-in collection types instead of generics, and a couple of other unsound semantics which pop up occasionally and bite the user frequently.
Xtend 2
The reason I started work on Xtext in the first place was, because it had been so painful to write the IDE support for Xpand, Xtend and Check. I thought that there must be some higher abstraction to derive functionality like syntax coloring, content assist, outline views, etc. from a language description. So I wrote the first prototype of Xtext in 2006 and luckily found some friends and a sponsor to create what was chosen to be the most innovative project at Eclipse this year.

Today Xtext is ready to be the basis for a rewritten version of Xpand. We call it Xtend 2 as it is more like a successor to Xtend which allows to have Xpand's template syntax as an expression. After having designed Xpand and Xtend and having seen so many different applications of it with different requirements and problems to solve, we know what features are helpful and which are problematic. Also during the last years with Xtext I have learned so much about language and IDE design and so did my team. All the languages we built for customers and all the languages we saw people developing with Xtext have helped to get a good feeling of what is good language design and what is not. We are now in a promising position to design a very, very nice language.

And indeed we are all thrilled about the concepts we put together so far. I'm writing up the language specification these days and will post it as soon as it is more consumable state. Today I wanted to share my excitement and give you an idea of what it is going to look like.

What are the main differences?

Based on Xbase

Xtend 2 is of course using the expressions defined in Xbase. That basically means it is tightly integrated with Java, has closures, full generics and a couple of very neat features like operator overloading or the super cool switch statement.

Compiles to Java
Xtend2 files are directly compiled to Java classes. In fact they are compiled to Java source code, which means that you can read (and debug) what the compiler makes out of your Xtend 2 coding. We will put much effort in making the generated code very readable and performant at the same time.

Contains successful features from Xpand and Xtend...
The cool stuff from Xpand and Xtend will be there such as polymorphic dispatch, extension methods or the template syntax.

... but they are even better now
For instance extension methods can be any static Java function and more important any methods from injected objects. This basically allows you to exchange the implementation of any extension methods only by changing the DI configuration. This in turn replaces the whole AOP story we have in Xpand and Xtend.

"Simplicity is the Ultimate Sophistication"
You'll see a lot of other small improvements, but the coolness of the language doesn't come from the amount of features but from the short and carefully chosen list of concepts working consistenly together. It is the overall composition of concepts and the holistic experience (tooling, etc.) which gives a language its "feel". And although I don't have the language in my hands right now, it already feels fantastic. :-)

We want to show this at EclipseCon

There is a submission for a tutorial on Xtend 2 we submitted for EclipseCon 2011. My favorite conference at my favorite location. It is extremely important to us to be able to share with you guys what we have been working on for the last months (actually years if you count the development of Xtext in).
Chris asked people to use the comment system to provide feedback, in order to help the committee composing a great program. So please, if you are interested in attending a hands on tutorial for Xtend 2 at EclipseCon 2011, tell them. :-)

8 comments:

  1. If I nicely understood, Xtend 2 is a prolongation for Xtext.

    I love Xtext, but for now, I find there is a big drawback : no eclipse plugin can be created for custom DSL processing & code generation (not supported in DWE2 for now).
    I maybe missed something, but I find this unpleasant :(

    If there is a workaround, I'll take it happily and change my mind :)

    ReplyDelete
  2. Cool! :) After all: a template is "just" a function which happens to produce a String (into some Stream, obviously).

    One question, though: does the compiled aspect mean that Xtend2 (and Xbase, ultimately) will not have an interpreter? I've use interpreted Xpand and Xtend several times to be able to do some heavy lifting where performance wasn't really an issue and I'd love to still have that possibility without having to incorporate a generation pipeline (however performant as I'm expecting Xtend2 to be used for that ;)) into my runtime.

    ReplyDelete
  3. @Frédéric :
    Just because an MWE2 file cannot be executed within OSGi currently doesn't mean you can't create a compiler / code generator. There is an explicit hook IBuilderParticipant to do code generation within the incremental project builder of Eclipse.
    Xtend2 is going to be the perfect fit here :-)

    @Meinte : We will focus on the compiler first. An interpreter might come later (Xbase will have an interpreter).

    ReplyDelete
  4. I have only one statement about what you are doing here: "it's exactly what I've dreamed in the last years by working with Xtext, Xpand and Xtend!".

    Best
    Serano

    ReplyDelete
  5. Hi Sven,

    really nice to see the Xpand language to evolve.

    We currently incorporate Xpand in our transformation sequence to transform models at a very high level of abstraction into compilable code. M2M transformations translate the domain specific model into a software model at a very low level of abstraction. M2T transformations transform such models into code (often 1:1). As such, we are using QVTO OCL everywhere (model verifications, M2M), except for the last stage of our transformation sequence where we are unable to use OCL queries (from libraries) in our Xpand templates.

    This is very inconvenient for us as we need to duplicate the queries shared between transformations (e.g. model interfacing between M2M and M2T).

    So for us it would be an important requirement for the language to allow invoking QVTO OCL queries within our templates.

    Furthermore, we agree with Meinte as he says the templates are just there to create a string stream. The workflow (transformation sequence) should direct each string stream to the other units (possibly files). Separation of concerns ...

    So make us really happy ... ;-)

    ReplyDelete
  6. @Niels:
    Xtend2 is tightly bound to the Java type system.
    So you have to have a Java facade in order to invoke OCL.

    ReplyDelete
  7. I try to doing 15 minutes -extended from xtext documentation.
    My grammar is Domainmodel.xtext above:
    grammar org.example.domainmodel.Domainmodel with
    org.eclipse.xtext.common.Terminals

    generate domainmodel "http://www.example.org/domainmodel/Domainmodel"

    Domainmodel:
    (elements += AbstractElement)*
    ;

    PackageDeclaration:
    'package' name = QualifiedName '{'
    (elements += AbstractElement)*
    '}'
    ;

    AbstractElement:
    PackageDeclaration | Type | Import
    ;

    QualifiedName:
    ID ('.' ID)*
    ;

    Import:
    'import' importedNamespace = QualifiedNameWithWildcard
    ;

    QualifiedNameWithWildcard:
    QualifiedName '.*'?
    ;

    Type:
    DataType | Entity
    ;

    DataType:
    'datatype' name=ID
    ;

    Entity:
    'entity' name = ID
    ('extends' superType = [Entity | QualifiedName])?
    '{'
    (features += Feature)*
    '}'
    ;

    Feature:
    (many ?= 'many')? name = ID ':' type = [Type | QualifiedName]
    ;
    -------------------------------------------------------------------------------------------------
    DomainmodelGenerator.xtend file below but ı take an error these lines.Error:Couldn't resolve reference to JvmType 'org.eclipse.xtext.example.domainmodel.Entity'.If you are help me ,ı am very happy.I think, add package org.exlipse.xtext.xtend2.lip but ı dont found this package
    importorg.eclipse.xtext.example.domainmodel.Entity
    import org.eclipse.xtext.example.domainmodel.Feature
    ----------------------------------------------------------------------------
    package org.eclipse.xtext.example.generator

    import org.eclipse.emf.ecore.resource.Resource
    import org.eclipse.xtext.generator.IGenerator
    import org.eclipse.xtext.generator.IFileSystemAccess
    import org.eclipse.xtext.example.domainmodel.Entity
    import org.eclipse.xtext.example.domainmodel.Feature
    import org.eclipse.xtext.naming.IQualifiedNameProvider

    import com.google.inject.Inject

    class DomainmodelGenerator implements IGenerator {

    @Inject extension IQualifiedNameProvider

    override void doGenerate(Resource resource, IFileSystemAccess fsa) {
    for(e: resource.allContents.toIterable.filter(typeof(Entity))) {
    fsa.generateFile(
    e.fullyQualifiedName.toString("/") + ".java",
    e.compile)
    }
    }

    def compile(Entity e) '''
    «IF e.eContainer != null»
    package «e.eContainer.fullyQualifiedName»;
    «ENDIF»

    public class «e.name» «IF e.superType != null
    »extends «e.superType.fullyQualifiedName» «ENDIF»{
    «FOR f:e.features»
    «f.compile»
    «ENDFOR»
    }
    '''

    def compile(Feature f) '''
    private «f.type.fullyQualifiedName» «f.name»;

    public «f.type.fullyQualifiedName» get«f.name.toFirstUpper»() {
    return «f.name»;
    }

    public void set«f.name.toFirstUpper»(«f.type.fullyQualifiedName» «f.name») {
    this.«f.name» = «f.name»;
    }
    '''
    }

    ReplyDelete
    Replies
    1. You need to run the egenrator on the *.xtext file (right click, run as->)

      Delete