Monday, October 01, 2012

Current Development and Future Plans for Xtext and Xtend

It has been a couple of weeks now since we released Xtext and Xtend with Eclipse Juno. Time for an update on what we are doing and where we want to go. In the following I describe the main new topics the team is working on. In addition to what is described below there is of course also a lot of maintenance work we are doing, i.e. fixing bugs and improving the core framework. Also we are doing some awesome work for customers, which I’m not allowed to talk about (But, yes, you can hire us. Even the whole team!).

Editors For Other Platforms

We are experimenting with support for other IDE-platforms than Eclipse. The first platform we are looking at right now is the web, more concrete Orion. Other platforms we want to look at are Netbeans, Xcode, Visual Studio and IntelliJ. Also basic configuration settings for TextMate and the like could be something we will be looking at.
Which one do you think is most interesting?
This part of our work is funded by Zukunftsprogramm Wirtschaft


Improved Type System

The type system and linking for Xbase-based languages (Xtend falls into this class) is currently redeveloped. Although the existing one does a decent job it has some architectural flaws which makes it too slow when working with bigger files. The new one is fast and it can do even more and cooler things. Type inference, for instance, now uses control flow analysis. That is you can write the following:
val myMap = newHashMap
myMap.put(23, new StringBuilder)
myMap.put(42, "some string")
// myMap is of type Map<Integer, CharSequence> by now
There are some other new features, which are possible with the new typesystem and even more importantly it will fix the outstanding typing bugs and be a solid foundation for future improvements. I'm sure there will be a blog post about the details in the next couple of weeks.

Formatting

Xtend gets a new formatter. I’m already using it on a daily basis and must say that it works very well. Especially with a language like Xtend which is syntactically much more flexible than Java, you need to look into language idioms to make sure a formatter does what you expect. There will also be a preference page to adjust formatting settings, but it won’t have as many options as the Java formatter in the beginning. Would be nice to have some more beta testers and feedback. What kind of options do you want to see?
Although we are implementing the formatting for Xtend now, we plan to have a new or at least an improved API for all Xtext languages. And of course the formatting for the expression goes into Xbase.

Active Annotations

Active Annotations allow you to participate in the translation step from Xtend to Java source code. It’s a bit like Java’s annotation processing but much more flexible and less complicated to integrate at the same time. You basically declare an annotation and implement a processing method, where you define how to translate annotated Xtend code to Java. I'll write a separate blog post on this.
Also Sebastian is giving a sneak-preview on Active Annotations tomorrow at JavaOne and the feature will play an important role in the session Web Development with GWT and Xtend Olli and I are giving at EclipseCon in a couple of weeks.

Release Plans

Currently the overhauled type system has highest priority and we are targeting a release as soon as it is done. We hope to have it in December this year. That release will contain the formatting and might have an @Beta-flagged prototype for Active Annotations included. We plan to have a prototype for Xtext & Orion later this year as well, maybe the team can even give a so demo during their EclipseCon talk.

Additional Topics

I want to make contributing simpler. The idea is to provide a one click download, which materialises a working IDE, workspace and target platform on your box. The git repository should already be connected with our gerrit instance. Also we want to start writing breaking and ignored unit tests for defects so potential contributors can easily reproduce a problem and see how our test framework works. Let's see if we can make contributing and fixing bugs simple and fun.
Another topic is to provide an alternative for debugging Xbase-languages. Currently they work through JSR-045 which is great as long as you run your code in a standard JVM. Unfortunately JSR-045 isn't supported by Android's Dalvik-VM or GWT's new SuperDevMode. We need to come up with a solution for this as well.
Finally I want to mention that there is now a package for "Java and DSL Developers" on the Eclipse download page. It doesn't have a neat welcome page so far but it's a one-stop-shopping for people who want to get started with Xtend or Xtext (or both).

22 comments:

  1. The active annotations sound very cool. But shouldn't they (at least optionally) support specifying (in the method) the translation to Xtend which is then translated to Java? You wrote that the method directly specifies the translation to Java, which, as we know, is "more cumbersome" than Xtend.

    Markus

    ReplyDelete
    Replies
    1. You'll translate to a typed Java AST and it will be possible to use typed Xbase expressions to specify the body as well as string literals (in that case it is Java syntax). The implementation is done in a special interpreted dialect of Xtend.

      Delete
  2. Active annotations sounds a bit like the compiler plugin you can write in .Net land. They use it to implement LINQ. Am I correct?

    Anyway: Sound really interesting.

    ReplyDelete
  3. I might be able to help with the one-click materialization of the ide/workspace. I've been doing some work in this area as a background task for my current consulting engagement. I've been using Maven/Tycho, but I am also familiar with Buckminster, which I believe is what you are using for the Xtext build and in fact would probably be much easier in this case. Who do I contact to start collaborating with you guys?

    ReplyDelete
    Replies
    1. Awesome! Yes buckminster would be the preferred tool. How about discussing the details in a bugzilla, I've filed one : https://bugs.eclipse.org/bugs/show_bug.cgi?id=390852

      Delete
  4. Perhaps consider Sublime Text 2 as an alternative to TextMate.

    ReplyDelete
  5. I would love to see Xtext/Xtend2 support in IntelliJ. ;-)

    ReplyDelete
  6. Hi,

    I got interested in the new formatting api. Could you post some example ?

    Btw, wouldn't possible to release it before other feature, since you are already using it ? ;)

    ReplyDelete
    Replies
    1. It's mostly a reusable formatter implementation of Xbase (and Xtend in particular). Just because I use it doesn't mean it's release-ready :-)

      Delete
  7. This might seem as a silly thing to mention when there are more important things to work on, but rather than go towards other IDEs I would personally like more customization in Eclipse. Namely, the ability to disable automatic parenthesis closing and more syntax highlight options (especially one for classes and types).

    ReplyDelete
    Replies
    1. It's not silly to ask for enhancements. Please file bugzillas, we need to get this kind of feedback in order to learn about your workflows and how to help improving it. Thanks!

      Delete
  8. I would be very interested in the Orion integration. Would this be a native port of the xtext framework to javascript (this sounds like an enormous task) or would it be via REST services hosted on the server.

    The usecase I would be interested in is embedding the orion editor in a web application used by business analysts/power users to maintain sources conforming to a DSL.

    I guess the grammar development would still happen in the existing xtext eclipse based plugin but that an additional output would be an orion grammar:

    http://wiki.eclipse.org/Orion/Documentation/Developer_Guide/Plugging_into_the_editor#Example_of_a_.27grammar.27_provider

    In this way you could have browser based parsing/validation/syntax highlighting, but the execution of the DSL would presumably remain on the server.

    As I say the next would be to port the model representation to javascript facilitating a javascript based interpreted runtime for the DSL.

    Anyhow, sounds very exciting and XText is a phenomenal piece of work!

    Peter.

    ReplyDelete
    Replies
    1. Hi Peter,

      we don'T plan to rewrite everything in JavaScript but rather refer to server-side functionality for most services. The only thing that currently runs in JavaScript natively is syntax coloring.

      Delete
    2. Yes I think bringing xtext to the web would be awesome!! I think we would find many interesting uses.

      Delete
  9. +1 Sublime Text support. Even basic Syntax support and Build Support would be awesome.

    ReplyDelete
  10. I think Xtext on Netbeans platform would be the most interesting choice. If you cover Eclipse + Netbeans you are getting to most developers market, and we could write DSL knowing that Ide's preferences will not be an reason to not use it.

    ReplyDelete
  11. Bring to visual studio! We have no textual DSL tools!

    ReplyDelete
  12. I believe the support for Web is the most interesting option ... Any developer who knows how to work with a rich IDE (NetBeans/IntelliJ etc ) can easily work on Eclipse with XText. Let's don't forget what DSL is for, it is for Domain Specific Languages and the best hope is for Domain Users to use the DSL Editors created with XText. Or most of the Domain Users are not necessary Developers. As soon as you tell them to install a rich-IDE like Eclipse/NetBeans/IntelliJ/etc you already lost their trust. Give them a link to a Web Page where they can use a DSL Editor created with XText and there is a better chance that the Domain Users will accept to work with the DSL Editor.

    Cheers,

    Ioan

    ReplyDelete
    Replies
    1. I really agree with Loan

      Delete
  13. In this post I think bringing xtext to the web would be awesome and useful for every people!!

    ReplyDelete
  14. I really agree that the next roadmap for Xtext is to bring the DSL editor to the web. One way for that may be to think of Xtext/RAP integration...

    ReplyDelete
  15. Hello, is there any examples of integrations of Xtext and Orion? It is very interesting to experiment by myself

    ReplyDelete