Thursday, July 02, 2009

Xtext scopes and EMF index in action

This is a post about scoping and how to use the EMF index for that. It is in some sense a practical follow up on another blog post about the general idea behind indexing and scoping in Xtext. The topic is somewhat advanced and bleeding edge. This post describes the needed steps to get the current index based default scoping up and running. I've prepared a small screencast demonstrating the result in action. The example language can be downloaded from here.




Today, the common way to do cross resource references in Xtext is to do it via resource URIs. That is if you want to reference a model element (EObject) from another resource, you typically put the whole resource on the scope by adding a corresponding import. Example:
import "platform:/resource/my.project/src/othermodel.dsl"

//.. refer to elements from othermodel.dsl
The corresponding default scoping is very simplistic. Every object in the current resource and in the referenced resources can be referenced by its simple name (as long as it has a 'name').

Although this is very easy to understand, it has it's limitation when it comes to more sophisticated design. If you for instance want to hide some elements or have duplicate simple names in different packages (this can be the case if you use elements, which are developed by others).

In many programming languages we have the notion of namespaces, which are much more flexible and powerful. Java, for instance, is file system agnostic. Although it forces you to put the files into folders which correspond the packages, it ultimately is just based on namespaces (packages, types).
That said Java's namespace mechanism is also a bit limited. For instance I cannot have imports in nested namespaces but only per file. And I cannot nest packages but only classes and interfaces.

Scala and C# both allow to have multiple nested packages within one file and you can put imports per namespace, so that imported names are only visible within that namespace.

In order to demonstrate how to use the index together with Xtext, I've implemented a DefaultIndexBasedScopeProvider which implements a similar semantic. There's a small example I've prepared, where you can see how it can be used. It is mainly a matter of configuring the different implementations with Guice. Programming is not needed as long as you're happy with the defaults.

Here's how it works

The index registeres a builder, which is invoked on resource changes. In order to make your model elements visible, you'll have to contribute a so called Indexer using an extension point.
<extension point="org.eclipse.emf.index.indexer">
<indexer class="org.eclipse.xtext.example.DomainmodelExecutableExtensionFactory
:org.eclipse.xtext.index.DefaultDeclarativeResourceIndexer"
fileextensions="dmodel"/>
</extension>
Please ignore the ExecutableExtensionFactory, which is declared in order to make any executable extension Guice aware, that is you can use dependency injection. This is a different topic and might be covered by another blog post.

The actual class to be instantiated is the one after the colon (':'): The DefaultDeclarativeResourceIndexer, which delegates to an instance of IQualifiedNameProvider, which itself is injected. This means that its implementation can be arbitrarily changed.
The contract of a name provider is very simple: it computes a qualified name for an element, if it returns null, the element is not indexed and hence not referable.

By default we use a DefaultDeclarativeQualifiedNameProvider, which if not otherwise specified looks up a simple name (if there's an attribute 'name') and concatenates it to the qualified name of its parent. It's named 'declarative' because you're able to change the described default behavior per type by just adding a method like this:
String qualifiedName(MyType foo) {
// compute different qualified name for MyTypes
// ...
}
It will automatically dispatch to this method as soon as it has to compute a qualified name for an instance of MyType.

With this in place we'll have our elements automatically indexed as long as they are in a project, which have the index nature enabled. Being indexed means that they are globally visible by their qualified name, which is comparable to how public Java elements are globally visible as soon as they are on the classpath.

What's next?

In order to use the index and have it injected into your components (e.g. your scope provider) you'll have to configure the singleton instance from the index bundle into your Guice module. In the example the corresponding binding goes into the UI module and looks like this:
public IndexStore bindIndexStore() {
return EmfIndexUIPlugin.getDefault().getIndexStore();
}
With that in place you can inject the index store by just adding a dependency in your code:
@Inject
private IndexStore store;
Guice will automatically put the instance into such declared dependencies.
Now that we have a binding for IndexStore we can add the index based scoping to the runtime module:
public IndexStore bindIndexStore() {
return new PersistableIndexStore();
}

@Override
public Class bindIScopeProvider() {
return DefaultIndexBasedScopeProvider.class;
}
Note the additional IndexStore binding, which is overridden by the binding we previously added to the UI module, but is needed in order to use this stuff at runtime (i.e. without running within exquinox). So it gets active as soon as you run without UI.

How the DefaultIndexBasedScopeProvider works

The DefaultIndexBasedScopeProvider
- looks up EAttributes with name 'importNamespace'
- and translates the globally unique qualified name into shorter ones using those import statements.

By default qualified names with or without a wildcard at the end are supported. For an import of a qualified name the simple name is made available as we know from e.g. Java, where
import java.util.Set;
makes it possible to refer to 'java.util.Set' by its simple name 'Set'.
Contrary to Java the import is not active for the whole file but only for the namespace it is declared in and its child namespaces. That is why you can write the following in the example DSL:
package foo {
import bar.Foo
entity Bar extends Foo {
}
}

package bar {
entity Foo {}
}
Of course the declared elements within a package are as well referable by their simple name:
package bar {
entity Bar extends Foo {}
entity Foo {}
}
Of course the following would as well be ok:
package bar {
entity Bar extends bar.Foo {}
entity Foo {}
}
Disclaimer
All this is in a very early stage. The index is not finished and its architecture is not settled down yet. Also the scope provider implementation might be changed in future (I'm sure it will).
Additionally, there are other things around this which we have to work on before considering this mature.

But as I know that there are a lot of bleeding edge users out there, I wanted to share the current state, so you might find a starting point to play with it. The index is an enabler for more advanced functionality in Xtext and in EMF based development in general. So expect it to become an important part in Eclipse Modeling.

Feedback is highly appreciated and should either go to the newsgroup for Xtext (the scoping part) or to the EMFT newsgroup (the index part), because the index project is a component under EMFT.

10 comments:

  1. I tried to listen to you screencast, but couldn't hear a sound. Only picture. Is there no sound? Do I need a special codec? I used QuickTime on Windows XP.

    Thanks,
    Malte Finsterwalder

    ReplyDelete
  2. No sound. Sorry.
    I commented the screencast during the webinar. So you might want to watch that instead : http://live.eclipse.org/node/705

    ReplyDelete
  3. Hi.
    I downloaded and imported your project but references to external objects can't be found. ("Couldn't resolve reference to Type String"). References to datatypes inside the same document are resolved. I installed the newest version of xtext (0.7.1). Is there anything i forgot?

    Roland

    ReplyDelete
  4. Hi,

    I downloaded your example. The newest Xtext is installed but it doesn't reference the external datatypes (Couldn't resolve reference to datatype String). However references to variables inside the same document can be resolved. I tried to debug the code but emf makes this a pain. Do you have any hint for me?

    Thanks,
    Roland

    ReplyDelete
  5. Did you enable the EMF Index nature (find that action in the context menu of your project)?
    Integration of index will become a major focus in the upcoming development. Note that this is just a demo.

    ReplyDelete
  6. Thanks for your reply. That did the trick!

    Roland

    ReplyDelete
  7. Hi Sven

    This looks very interesting. However, what if I wanted to make elements in an existing EMF model instance (XMI file) available in the context assist, instead of Xtext elements in an Xtext grammar file? These EMF model instances would be instances of a meta model which contains all of the same elements as in the ecore file generated (thus the Xtext grammar only covers a subset of the other meta model's elements).

    Do I simply need to add something like the indexer declaration in your post to the plugin.xml, but change the fileextensions attribute to "xmi"? Is there a specialised indexer available for XMI files?

    Thanks
    Frank

    ReplyDelete
  8. @Frank,

    you could simply register a resource indexer for your meta model, or even a generic one for all kinds of EMF resources.

    ReplyDelete
  9. Hi Sven,

    could you help me to find the extension point org.eclipse.emf.index.indexer? What plugin do I need to import?

    Kind regards,

    Simon

    ReplyDelete
  10. @Simon, the post is outdated. The builder infrastructure in Xtext 1.0 is different. Please consult the documentaion or use the forum/newsgroup to ask any questions.

    ReplyDelete