Thursday, May 28, 2015

The Future of Xtext

this is a write up of a presentation I gave together with Sebastian last week at XtextCON. It is about our future plans of Xtext. But before we talk about the future I would like to start with a quick summary of ...

The Status Quo

Xtext is in development for seven years now. We have always been between 6 and 10 active developers, currently being 9. With a total LOC of 5.747.523 our code base is huge. However you have to take into account that we not only check in generated code (it sometimes helps us to spot problems) but have many, many tests. All in all there is 27.539 usages of @Test which are executed in many different scenarios and against different platforms. All together we count over 180k unit tests.

This kind of test coverage together with a passion for code quality is key keep a software system maintainable and extensible after years.

Xtext 2.8.x

In the coming week we will release 2.8.3, a bugfix version of the 2.8.x stream that introduced many cool new features. With version 2.8.3 we have fixed over 120 bugs and improved quality furthermore. This is also the release that will be part of Eclipse Mars.

Error Reporting

Not sure if you have noticed, but with Mars also comes the new error reporting tool, an initiative by the developers behind Eclipse Code Recommenders. It basically asks you to automatically send in error reports for all kinds of exceptions and UI freezes you experience when working with Eclipse. Although it is not yet included in the Luna SR2 packages we already got many helpful tickets through that feature from the milestone builds. Please, turn this feature on and provide as much information as possible. It really helps!

What about future releases?

To figure our what the community demands, we did a survey. Thanks for everyone who attended! The survey was targeted at Xtext users, and besides some more demographic questions, the most important question was:

"What should be changed or improved?"

The answers included a lot of valuable feedback. Most of them could be assigned to one of the following four themes. I put some of the original answers and comments from the survey on the slides.

People asked for more documentation, specifically around Xbase and in the code (i.e. JavaDoc). Also many asked for an overview chapter where a bird's eye view of the architecture and life cycles is explained. We agree that these parts should be added and enhanced and have put it on our agenda. Fortunately the documentation is now written in markdown and change is only one pull-request away (for everyone ;-)).

Performance has always been high on our list and for every release we spend a lot of time on profiling and optimization. If you suffer from performance problems, first make sure, that you are on the latest version. Performance makes huge steps from release to release.

Here is some of the improvements we have been working on lately:

  • Do nothing in UI thread - computations of any UI services have been moved to background jobs, such that nothing blocks the user.
  • Cancel everything - all UI services, such as outline, mark occurences, semantic highlighting, etc. get canceled on change (i.e. as you type). Since the state is outdated we can stop waste CPU cycles on it.
  • Cancelable builds - also incremental builds now get automatically canceled, when another change comes in.
  • Storaged Resources - Xtext can write fully resolved and computed resources to disk in a binary format, so it can later be loaded again quickly
  • Incremental standalone compilation (new in 2.9) - The eclipse builder has always been incremental. Now Xtext also has an incremental standalone compiler, that together with a running demon ensures super slick incremental compilation of changed files.

Of course, improving performance is never done!

Continuous integration with Xtext and Xtext-based languages has traditionally been a bit of a pain. This is mostly because we started out as an Eclipse only project, and building Eclipse plug-ins is "different". The Xtext project has had a continuous integration build for years, but our build infrastructure is not exactly simple. Targeting different platforms requires a bit of a zoo of build technologies on our side, which should not be necessary for clients. Since almost two years now, there is Maven support, including a plug-in to run your language in an arbitrary maven build. The same has been added for Gradle as well.

The next step is to allow for different project layouts, i.e. Eclipse vs. Maven/Gradle layout and allow for developing Xtext languages without any kind of Eclipse dependency. Which brings us to the last theme.

Xtext is a language framework that has significantly lowered the barrier to develop programming languages and external domain-specific languages. One of the sweet things is the cool Eclipse support you get. However even if you only need a compiler for a language without any Eclipse tooling the Xtext framework provides you with rich abstractions for scoping, indexing, incremental compilation and more. With the next release (version 2.9) it should be possible to create a runtime project only, without any Eclipse dependencies and without the necessity to deal with OSGi or eclipse plug-in mechanism. Our goal is to allow for vanilla Java projects with a normal Gradle or Maven build.

For the IDE part, we are working on supporting web editors and IntelliJ IDEA specifically at the moment. But maybe even more important in the long run, is the general extraction of the IDE services from Eclipse dependencies, such that they can be reused on other platforms easily. Tom Schindl for instance already uses Xtext in JavaFX applications and we are in contact with a group that has ported Xtext to Visual Studio through IKVM.

Just to be clear: The Eclipse support stays important and will be a first class citizen for the Xtext framework in the future, too!


As you can see, the direction is a mixture of improving quality and performance, making accessibility easier but also reaching out to new platforms. I hope you find this direction as exciting as we do. We plan to release Xtext 2.9 in the last quarter of 2015. In case you want to try the current state of the web or intelliJ IDEA support, go ahead and download the recently released beta version. More betas will be released in the coming weeks / months.
So long, have fun and remember your feedback is helpful and very welcome!

Monday, March 16, 2015

Xtext moved to Github

I'm just back from EclipseCon and XtextDay in San Francisco, where we announced the new 2.8.0 release of Xtext and Xtend among some other interesting developments. One of those is our...

Move to Github

In order to simplify the contributor's story, Xtext is taking part in the "social coding initiative" of the eclipse foundation, and the primary git repository has been moved over to github. As a consequence we will no longer use gerrit for code reviews but github pull requests. Also it now makes more sense to fork, watch and star our repository.

Xtext is of course still an Eclipse project!

Changing documentation in two clicks

The website has been reimplemented in jekyll and the entire documentation is now written in github-flavored markdown. A link on every page that says 'edit on github' will open up a markdown editor on github, where anybody can do changes and propose them in form of a pull-request with a second click. This allows for wiki-style turnarounds but with a review and CLA check in it.
Just go try it out :-)

Automated setup

While documentation and website improvements can be made in the browser, you usually want to run some of our over 30.000 unit tests when you did code changes, or be able to compile and debug code. With a big project like Xtext that targets multiple platforms, the setup of a workspace and the corresponding tools is unfortunately not exactly simple. But fear not, this is all automated, thanks to an oomph setup. So even here it is only a couple of clicks until you have a fully working setup, that includes an IDE, the workspace, correct target platform, the team's workspace preferences and so on. Just everything you need and everything like we committer have it.

Code contributions

The team has many active committers that are very happy to review your high-quality pull-requests. If you want to start working on something it would be best if you pick up one of the bugzillas flagged with 'v2.9' or whatever the upcoming release is in case you read this post at a later point in time. This is how we schedule the things we agreed on working on.

Thursday, January 29, 2015

Auto-GWT - Boilerplate Free GWT Programming

Maybe not especially new for everybody, but: GWT is NOT dead!

I've been at the GWT.create conference yesterday, a two-day conference on GWT only that took place in San Jose last week and this week in Munich. With over 400 attendees it was sold out. If that is not enough evidence of the liveliness of the project, you may have missed that Google runs huge projects on GWT. The latest one is Google Inbox. Add all the nice improvements (like increased compile speed) the latest releases have to offer and you will conclude that GWT is the best way to write browser apps in Java or any JVM language. The generated JavaScript code is highly optimized and runs even much faster than typical hand-written JavaScript code.

I had the pleasure to present the new Auto-GWT library we have built for GWT together with my colleague Anton. Thanks to all the attendees for the great questions and the overwhelmingly positive feedback. As a kind of wrap up after the conference I have uploaded the slides, the demo and the Auto-GWT project.

You can find all necessary information on the project's website :
Forks and pull requests are very welcome!

Auto-GWT in action:

Slides from GWT.create:

Friday, January 23, 2015

XtextDay and EclipseCon San Francisco 2015

In case you've missed it, we are organizing a so called XtextDay co-located with EclipseCon 2015. The program is already online for a couple of weeks:

Sebastian and I will start the day talking about the newest developments (e.g. Intellij IDEA support, web editor support, incremental standalone builders, etc.). After that we will hear how Xtext languages are used to design REST APIs. In the afternoon we have in-depth sessions on Xbase, performance and scoping, and after the coffee break we will learn how a silicon valley company use Xtext to build a commercial product for designing systems on a chip (IoT). The full program details can be found here. But that's not all..

Additional Xtext content at EclipseCon

Although it's possible to register only for the XtextDay, I recommend to book the whole EclipseCon conference as it is not only an awesome community event where you will find many new friends, but in addition to the XtextDay, there is some Xtext-related content there as well. It starts with a beginner's tutorial on Monday, which is a good preparation for the technically deeper talks following. In addition to the XtextDay and the tutorial we have:

So quite some content in addition to XtextDay. If you consider using Xtext for something or you are already using it, then you should definitely come and join the sessions, discussions and party! :-)

Thursday, October 09, 2014

Eclipse Xtext goes IntelliJ IDEA

The Xtext framework has been in development for six years now. With a team size between five and nine people (we are now at nine) plus many contributors working on it, it has become a successful framework for development of domain specific languages and programming languages in general. From parsers, linkers and compilers to interpreters, type-checkers and full IDE-support, the framework covers everything a grown up language infrastructure needs. So far however, the IDE part has been Eclipse-only. This will change now.

The Eclipse based IDE support of Xtext is very mature. We will further improve robustness and performance and might add one or the other feature in the future, but the Eclipse-support really has come to a state where not much is missing. Personally I like Eclipse as a code base as well as a tool, but the reduced activity in the platform itself has resulted in a noticeable decline of excitement and we see more and more devs preferring IntelliJ IDEA for their Java development. Also there are many shops where the IDE is not set, so developers can choose. Those teams with mixed IDEs suffer from lacking DSL tool support for their non-Eclipse IDEs, of course.

Long story short, this summer we started working on support for IntelliJ IDEA. It shall work without requiring duplicate implementation for the main aspects. So as an Xtext user you can still work against the core Xtext APIs and any existing Xtext language is going to work in IDEA without further ado. A thin layer translates the information to IntelliJ’s concepts (PSI, etc.). Of course everything is composed with dependency injection, such that IntelliJ-specific optimisations and customisations are still possible. The work is coming along nicely and we hope that a first public beta version can be released early next year. Of course this includes IntelliJ IDEA support for Xtend as well!

Here’s a short screencast showing a bit what’s already working.

Xtext on IntelliJ IDEA from Xtext Team on Vimeo.

More platforms to come

Although the current focus is IntelliJ (and Eclipse of course) we think in the long term it is important for the Xtext framework to support other platforms as well. Next up will be serious support for web browsers, which is especially interesting for the many DSLs targeting non-developers (i.e. “business DSLs”).

Tuesday, April 15, 2014

XtextCON Update

XtextCON 2014 is still 40 days away, but I have to announce that ...

We are sold out

Initially I planned for 80 attendees. It turned out that was much too small, so we have added 30 more tickets as the event and venue can handle 110 people without problems. Today we have 112 registrations and since I want to make sure that everyone has an excellent time at this first XtextCON we closed registration today. I'm really sorry if you haven't booked yet but wanted to do so. We'll likely do an XtextCON next year again. 

New Sessions

We have added some really cool new sessions. All in all XtextCON will feature 18 Speakers and 28 sessions in two tracks. The added sessions are:
 - Xtext + Sirius : <3 (Cedric Brun)
 - Xtext + CDO - Does it blend? (Stefan Winkler)
 - Oomph - Automatically Provision a Project-Specifc IDE (Eike Stepper, Ed Merks)
 - 3D Modeling with Xtext (Martin Nilsson, Esa Ryh√§nen)
 - Handle Based Models for Xtext with Handly (Vladimir Piskarev)

Checkout the updated program.

Thursday, February 13, 2014

Announcing XtextCON 2014

I'm super stoked to announce the first edition of XtextCON, an annual conference on Eclipse Xtext.

With close to 6000 newsgroup items per year the Xtext community is one of the largest and most active at Although conferences such as EclipseCON feature many talks on Xtext, they usually cannot go into much detail as those conferences are targeting many different technologies. XtextCON should be an event where we can dive deeper. It is a community event and is meant to help users solving technical problems, get feedback on design ideas and learn about advanced topics.
XextCON is targeting beginners as well as advanced users. Beginners get the opportunity to attend a one-day workshop before the conference. So they get up to speed for the deeper talks at the main conference while those who are experienced with Xtext but feel like there knowledge is a bit outdated will get a quicker heads-up in the morning of day one.
The program covers many different aspects such as scoping, linking, how the index works, how to build type system, and many more. Also we have talks on graphical editing or generally how to improve the performance of your languages, code generators and IDEs. Btw: Thanks to the many attendees of the survey, it helped a lot to put the program together.

Xtext Clinic

In addition to the two main tracks, we will run a special Xtext-clinic on both days, where attendees get the opportunity to discuss their individual Xtext-related issues with committers and experts.

The Location

The event takes place in Kiel at the ATLANTIC hotel, which sits directly at the beautiful harbour of Kiel face to face with our office. Kiel is a small city in northern Germany at the baltic sea and May is the best time to visit it. The area around it is one of the most beautiful places at this time.
It could be a good idea to bring your family and spend the weekend here as well.
So, check out more details on the website and make sure to make a reservation early. Tickets are limited! Looking forward to see you there!