Friday, June 03, 2011

5 simple steps to Fowler's DSL with Xtext 2.0

You might have read (about) the book Domain-Specific Languages written by Martin Fowler. As the name suggests it is about these little useful programming languages you can built very easily with Xtext. Unfortunately Martin decided to use more low level technology in order to explain how to implement such a language. So if you don't want to spend hours of hours dealing with complex low level technology just to end up with a parser but nothing else, here are the five steps needed to get a fully working implementation for the DSL he uses throughout the book (Mrs Grant's Controller). It doesn't get simpler.

Below you can find a detailed description. For those of you who prefer to just sit and watch, here are three screencasts I've recorded around this example.



Creating the project and defining the language



Defining the code generator and see its integration



Features, Features, Features



1) Download Eclipse and Install Xtext

For many people this is the hardest part. But believe me things have improved and it is now quite easy to get a working Eclipse distribution including the latest milestone of Xtext 2.0. :-)

Download an Eclipse Classic distribution for your platform from here :  
    http://www.eclipse.org/downloads/index-developer.php

Unzip and start Eclipse, choose Help -> Install New Software... and use the update site
    http://download.itemis.com/updates/milestones
in order to install the Xtext SDK feature

2) Use the Xtext project wizard

Select New -> Project..., and within the dialog choose Xtext Project. You can stick with the defaults on the next page, but make sure you choose the Experimental Features configuration in order to get the latest and greatest features, such as rename refactoring.

You should use the experimental features

3) Define the grammar

The Xtext grammar file is automatically opened. Just copy and paste the following grammar definition for Martin's statemachine DSL in.


grammar org.xtext.example.mydsl.MyDsl with org.eclipse.xtext.common.Terminals

generate statemachine "http://www.xtext.org/example/secretcompartments/Statemachine"

Statemachine :
     {Statemachine}
     ('events' 
          events+=Event+ 
     'end')?
     ('resetEvents' 
          resetEvents+=[Event]+ 
     'end')?
     ('commands' 
          commands+=Command+ 
     'end')?
     states+=State*
;

Event:
     name=ID code=ID
;

Command:
     name=ID code=ID
;

State:
     'state' name=ID
          ('actions' '{' actions+=[Command]+ '}')?
          transitions+=Transition*
     'end'
;

Transition:
     event=[Event] '=>' state=[State]
;


4) Run the generator

Now you need to run the generator which creates the parser and a couple of other parts of the language infrastructure. To do so right click into the editor and choose 
   Run As -> Generate Xtext Artifacts

5) Define the code generator

A DSL isn't worth much if you are not able to execute it somehow. Xtext has already provided you with a stub for a code generator written in our cool new template language Xtend (Xtend as well as the grammar language are also implemented with Xtext of course). 
To get a working implementation which produces a Java implementation of a defined statemachine, just copy the following into the file MyDslGenerator.xtend

package org.xtext.example.mydsl.generator

import org.eclipse.emf.ecore.resource.Resource
import org.eclipse.xtext.generator.IGenerator
import org.eclipse.xtext.generator.IFileSystemAccess
import org.xtext.example.mydsl.statemachine.Statemachine
import org.xtext.example.mydsl.statemachine.Event
import org.xtext.example.mydsl.statemachine.Command
import org.xtext.example.mydsl.statemachine.State

class MyDslGenerator implements IGenerator {
override void doGenerate(Resource resource, IFileSystemAccess fsa) {
fsa.generateFile(resource.className+".java", toJavaCode(resource.contents.head as Statemachine))
}
def className(Resource res) {
var name = res.URI.lastSegment
name.substring(0, name.indexOf('.'))
}
def toJavaCode(Statemachine sm) '''
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
public class «sm.eResource.className» {
public static void main(String[] args) {
new «sm.eResource.className»().run();
}
«FOR c : sm.commands»
«c.declareCommand»
«ENDFOR»
protected void run() {
boolean executeActions = true;
String currentState = "«sm.states.head.name»";
String lastEvent = null;
while (true) {
«FOR state : sm.states»
«state.generateCode»
«ENDFOR»
«FOR resetEvent : sm.resetEvents»
if ("«resetEvent.name»".equals(lastEvent)) {
System.out.println("Resetting state machine.");
currentState = "«sm.states.head.name»";
executeActions = true;
}
«ENDFOR»
}
}
private String receiveEvent() {
System.out.flush();
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
try {
return br.readLine();
} catch (IOException ioe) {
System.out.println("Problem reading input");
return "";
}
}
}
'''
def declareCommand(Command command) '''
protected void do«command.name.toFirstUpper»() {
System.out.println("Executing command «command.name» («command.code»)");
}
'''
def generateCode(State state) '''
if (currentState.equals("«state.name»")) {
if (executeActions) {
«FOR c : state.actions»
do«c.name.toFirstUpper»();
«ENDFOR»
executeActions = false;
}
System.out.println("Your are now in state '«state.name»'. Possible events are [«
state.transitions.map(t | t.event.name).join(', '].");
lastEvent = receiveEvent();
«FOR t : state.transitions»
if ("«t.event.name»".equals(lastEvent)) {
currentState = "«t.state.name»";
executeActions = true;
}
«ENDFOR»
}
'''
}

18 comments:

  1. It's really incredible the things Xtext 2 does!

    Many compliments! :)

    ReplyDelete
  2. Great job guys!
    This is so much better than the previous versions!
    I'd like to now where can I find some migration information so I can migrate the gramatics I have been developing previously?
    BTW, what about Xbase? I would like to know how can I allow the use of xbase sentences on my editor
    Thanks,

    ReplyDelete
  3. @Anonymous:
    There is a migration guide included in RC4 (see Eclipse Help). We will publish that to the website as well very soon.
    We haven't yet written a how-to for Xbase integration, but the domain model example has Xbase integrated and should help to understand what to do. Anyway, a how-to is needed and will be written within the next couple of weeks.

    ReplyDelete
  4. Hi Sven,

    I have massive java.lang.OutOfMemoryError: PermGen space Problems with the generated eclipse UI in the state machine example. Any tips?

    ReplyDelete
  5. This is pretty usual; make sure to use this vm arguments in the launch configuration:

    -XX:MaxPermSize=256m

    hope this helps :)

    ReplyDelete
  6. I've seen you can generate the Syntax diagrams of the gramatic using Xtext 2.0, so the user can see in a graphical way the options the language offers, how can I generate my own gramatic syntax diagram?

    ReplyDelete
  7. Thy syntax diaram is displayed in a view. So it is not generated and also not persisted somehow it is just a projection of the grammar. If you want your own you'll have to use Xtext's grammar language.

    ReplyDelete
  8. Very nice!

    Some tips for others:
    - eclipse.ini after -vmargs add: -XX:MaxPermSize=256M
    - In the eclipse you start from eclipse: create a new Java Project if you want to see automatic code generation work. With a normal project I did not get my statemachine code generated in /src-gen
    - The help option in eclipse contains an incorrect statemachine example. You must add 'end' before resetting doorOpened D1OP if you copied it from there.
    The README.TXT in the example project of the statemachine is correct.

    ReplyDelete
  9. Truly amazing work!

    Congratulations and thank you to Sven for tirelessly developing Xtend, Xpand, and Xbase! :)

    ReplyDelete
  10. Hi, Thank you so much for magnificently useful tool.

    I have just began Xtext, and had some problems;

    I have a List writerNames, which contains 10000 writer's names, one of them is "Stefan King" .
    I dont want to define all of them by one by, because i have ten more existing lists like this. Is there any way to get a list into my grammar?

    Because While I am coding with my dsl in my
    " something.mydsl " file. I want that content assist completes Stefan King", if i typed "Ste" or comes up with some names which start with "Ste"

    How can I solve this?
    Thanks a lot

    ReplyDelete
  11. @Caner01: Please use the newsgroup for more detailed questions. If it's not a croos link and you just want proposals, you should implement it in the ProposalProvider. If it's a cross reference it belongs into the GlobalScopeProvider.

    ReplyDelete
  12. Thanks for your time putting up the tutorials.
    Can you point me to a sample code which shows how to run the code generator from a Java class?

    Thanks

    ReplyDelete
  13. @Anonymous: in the generator package there should be a class called "Main" which contains what you are looking for.

    ReplyDelete
  14. Yeah, I tried out that but ended up with following stack trace.
    Exception in thread "main" java.lang.RuntimeException: Cannot create a resource for 'DomainModelGenerator.xtend'; a registered resource factory is needed
    at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResource(ResourceSetImpl.java:394)
    at org.eclipse.xtext.resource.SynchronizedXtextResourceSet.getResource(SynchronizedXtextResourceSet.java:23)
    at org.example.domainmodel.generator.Main.runGenerator(Main.java:50)
    at org.example.domainmodel.generator.Main.main(Main.java:32)


    It seems I need to register resource factory. Can you show me how to do this ?

    Thanks for your time.

    ReplyDelete
  15. The main class is not meant to get an Xtend file as a parameter but a path to a state machine model. Please use the forum at Eclipse for further technical questions :

    http://www.eclipse.org/forums/index.php?t=thread&frm_id=27&

    ReplyDelete
  16. Hi Sven,
    in Xtext 1 I installed my DSL as a plugin like described in http://wiki.eclipse.org/Xtext/GettingStarted#Deployment
    While developing the Xpand templates I could modify the templates, rerun the DslGenerator and then the changes were applied immediately in the generated Code.
    Is there a similar way in Xtext2, or do I have to spawn a new Eclipse / reinstall the plugins after every Template/Xtend change?
    Was this only possible, because Xpand was interpreted?

    Thank you

    ReplyDelete
  17. I'm trying to parse an ecore model from a script conform to a grammar using Xtext. the following link http://www.eclipse.org/Xtext/documentation.html#TutorialUnitTests present a code to adapt to your project. The drawback is where to put this code.

    ReplyDelete
  18. You need to call StandaloneSetup (generated for your grammar) once, which will register a ResourceFactory with EMF.

    After that you should be able to load it with plain EMF API, i.e. create a ResourceSetImpl and call 'getResource(myURI, true)'.
    Please ask in the Xtext forum if this doesn't help enough.

    ReplyDelete