Friday, March 01, 2013

Xtend's New Type System

"We are almost at a stage where only double PhDs in math & comp.sc can qualify for Java compiler writers !"

(Srikanth S. Adayapalam (from Eclipse JDT) on the complexity of type inference in Java 8)

As Xtend implements Java's typesystem, we have to deal with exactly the same amount of complexity ... oh wait it's even better as Xtend has

  • advanced type inference,
  • sugared feature calls for properties,
  • operator overloading,
  • an implicit 'it' and
  • extension methods (which are not only static as in C#)
  • etc.
So you better add a couple more PhDs to the mix.

Write once, run ...

We are used to writing key parts of our software twice (at least!). The old type system did its job in the beginning but as we added more and more corner cases and complex type inference scenarios with lambdas and such it started to become too slow and too complex to be further enhanced. Good we have a solid test suite so we actually noticed the point at which even harmless looking changes broke existing tests... To make a long story short: Last year in April we decided to rewrite it from scratch.

The Lonesome Hero

Sebastian had some very promising ideas around a completely different architecture which would not only improve performance but also would allow everyone to better understand and debug the code again. We couldn't afford stopping all other developments and maintenance so one team member had to take the journey while the others were working on cool new features. Almost a year ago Sebastian started work on a complete rewrite. We discussed language semantics from time to time and such things but I rarely looked at the actual code. Most of the semantics were clear as it is based on the Java Language Specification and could be proved with the different Java compilers. So he was climbing this huge mountain all alone.
I know it was super risky and that it's not a good idea to rely such an important part of a project on just one person. If he had failed we had wasted a year. But I knew he would do it (And if he couldn't I don't know who else could).

It has been a long time and we even had to postpone the release which was initially planned for December. Finally by the end of January the new type system implementation was in a state that it could be activated and replace the old one. The switch was super smooth. It just worked.

Here is the corresponding bugzilla, which has been closed just yesterday. It blocked 95 downstream bugs! Most of them are now fixed, we just had to go through them and test, verify and close them. Not only that, the performance was already as good as the old one in the beginning without any further profiling and improvements. With just a little optimisation we were already able to improve the performance by 25%.

Also worth noting is, that the whole team was able to start working in the new code base after just two days of introduction. The code definitely is much easier to grasp and to debug yet covers much more of the essential complexity. Also we have an even more comprehensive test suite now. With over 20.000 unit tests you can confident that you don't accidentally break something. All tests run against the lower version bounds as well as the upper version bounds of our dependencies. There's no way software like this can be maintained and evolved without such a test suite.

The new type system is an incredible piece of engineering and a huge success. It's the basis for an awesome upcoming release of Xtend with many cool new features and enhancements.
Sebastian, I owe you one and so does the growing Xtend community!

8 comments:

  1. +1, looking forward to the upcoming Xtend release, thank you so much Sebastian!

    ReplyDelete
  2. As both an Xtend and Xbase user, I've been eagerly waiting for the new type system implementation. Thanks for the great work!

    I'm current working on a DSL which integrates Xbase and JvmTypes in a dataflow-like language. In that context I have a couple of questions:
    - Will the DSL implementation concerned with Jvm types, deriving a Jvm model and compiling it need to be rewritten?
    - In the DSL, relations (edges) between actors (nodes) implicitly represents type constraints that may be used by inter-element type inference, a bit like Xtend supports type inference across methods. Is the new implementation suitable for such DSL-custom type inference?

    ReplyDelete
    Replies
    1. Some code has changed of course but for instance the JvmModelInferrer is not affected too much. If you have customized the TypeProvider and ScopeProvider you'll have to do some migration but we'll provide a guide.

      The type inference is based on where in a JvmModel the expressions live (are associated). So if you put your expression into a method with a specific return type it will be checked. You can also say that the return type (or for instance the type of a field) is derived from the expression.

      Delete
  3. Hi Guys,
    that sounds really great! I'm looking forward to the new release...
    Regards
    Serano

    ReplyDelete
  4. Great post and congratulations with this achievement.

    ReplyDelete
  5. Hi,

    Does/will Xtend support the following:
    - default (aka package) access
    - nested/inner classes
    - declaring interfaces?

    The manual is, well, silent about those.

    ReplyDelete
    Replies
    1. 1) default (aka package) visibility is supported in 2.4. Like in Java 8 'package' is used as the modifier keyword.
      2) nested/inner classes are planned but don't make it into 2.4.
      3) declaring interfaces is supported in 2.4 as well as enums and annotations
      4) We'll update and enhance the documentation in the next days.

      Delete
  6. This comment has been removed by a blog administrator.

    ReplyDelete