Xtext scopes and EMF index in action

This is a post about scoping and how to use the EMF index for that. It is in some sense a practical follow up on another blog post about the general idea behind indexing and scoping in Xtext. The topic is somewhat advanced and bleeding edge. This post describes the needed steps to get the current index based default scoping up and running. I've prepared a small screencast demonstrating the result in action. The example language can be downloaded from here.




Today, the common way to do cross resource references in Xtext is to do it via resource URIs. That is if you want to reference a model element (EObject) from another resource, you typically put the whole resource on the scope by adding a corresponding import. Example:

import "platform:/resource/my.project/src/othermodel.dsl"

//.. refer to elements from othermodel.dsl

The corresponding default scoping is very simplistic. Every object in the current resource and in the referenced resources can be referenced by its simple name (as long as it has a 'name').

Although this is very easy to understand, it has it's limitation when it comes to more sophisticated design. If you for instance want to hide some elements or have duplicate simple names in different packages (this can be the case if you use elements, which are developed by others).

In many programming languages we have the notion of namespaces, which are much more flexible and powerful. Java, for instance, is file system agnostic. Although it forces you to put the files into folders which correspond the packages, it ultimately is just based on namespaces (packages, types).
That said Java's namespace mechanism is also a bit limited. For instance I cannot have imports in nested namespaces but only per file. And I cannot nest packages but only classes and interfaces.

Scala and C# both allow to have multiple nested packages within one file and you can put imports per namespace, so that imported names are only visible within that namespace.

In order to demonstrate how to use the index together with Xtext, I've implemented a DefaultIndexBasedScopeProvider which implements a similar semantic. There's a small example I've prepared, where you can see how it can be used. It is mainly a matter of configuring the different implementations with Guice. Programming is not needed as long as you're happy with the defaults.

Here's how it works

The index registeres a builder, which is invoked on resource changes. In order to make your model elements visible, you'll have to contribute a so called Indexer using an extension point.

<extension point="org.eclipse.emf.index.indexer">
<indexer class="org.eclipse.xtext.example.DomainmodelExecutableExtensionFactory
                  :org.eclipse.xtext.index.DefaultDeclarativeResourceIndexer"
            fileextensions="dmodel"/>
</extension>

Please ignore the ExecutableExtensionFactory, which is declared in order to make any executable extension Guice aware, that is you can use dependency injection. This is a different topic and might be covered by another blog post.

The actual class to be instantiated is the one after the colon (':'): The DefaultDeclarativeResourceIndexer, which delegates to an instance of IQualifiedNameProvider, which itself is injected. This means that its implementation can be arbitrarily changed.
The contract of a name provider is very simple: it computes a qualified name for an element, if it returns null, the element is not indexed and hence not referable.

By default we use a DefaultDeclarativeQualifiedNameProvider, which if not otherwise specified looks up a simple name (if there's an attribute 'name') and concatenates it to the qualified name of its parent. It's named 'declarative' because you're able to change the described default behavior per type by just adding a method like this:

String qualifiedName(MyType foo) {
  // compute different qualified name for MyTypes
  // ...
}

It will automatically dispatch to this method as soon as it has to compute a qualified name for an instance of MyType.

With this in place we'll have our elements automatically indexed as long as they are in a project, which have the index nature enabled. Being indexed means that they are globally visible by their qualified name, which is comparable to how public Java elements are globally visible as soon as they are on the classpath.

What's next?

In order to use the index and have it injected into your components (e.g. your scope provider) you'll have to configure the singleton instance from the index bundle into your Guice module. In the example the corresponding binding goes into the UI module and looks like this:

public IndexStore bindIndexStore() {
 return EmfIndexUIPlugin.getDefault().getIndexStore();
}

With that in place you can inject the index store by just adding a dependency in your code:

@Inject
private IndexStore store;

Guice will automatically put the instance into such declared dependencies.
Now that we have a binding for IndexStore we can add the index based scoping to the runtime module:

public IndexStore bindIndexStore() {
 return new PersistableIndexStore();
}
  
@Override
public Class bindIScopeProvider() {
 return DefaultIndexBasedScopeProvider.class;
}

Note the additional IndexStore binding, which is overridden by the binding we previously added to the UI module, but is needed in order to use this stuff at runtime (i.e. without running within exquinox). So it gets active as soon as you run without UI.

How the DefaultIndexBasedScopeProvider works

The DefaultIndexBasedScopeProvider
- looks up EAttributes with name 'importNamespace'
- and translates the globally unique qualified name into shorter ones using those import statements.

By default qualified names with or without a wildcard at the end are supported. For an import of a qualified name the simple name is made available as we know from e.g. Java, where

import java.util.Set;

makes it possible to refer to 'java.util.Set' by its simple name 'Set'.
Contrary to Java the import is not active for the whole file but only for the namespace it is declared in and its child namespaces. That is why you can write the following in the example DSL:

package foo {
  import bar.Foo
  entity Bar extends Foo {
  }
}

package bar {
  entity Foo {}
}

Of course the declared elements within a package are as well referable by their simple name:

package bar {
  entity Bar extends Foo {}
  entity Foo {}
}

Of course the following would as well be ok:

package bar {
  entity Bar extends bar.Foo {}
  entity Foo {}
}

Disclaimer
All this is in a very early stage. The index is not finished and its architecture is not settled down yet. Also the scope provider implementation might be changed in future (I'm sure it will).
Additionally, there are other things around this which we have to work on before considering this mature.

But as I know that there are a lot of bleeding edge users out there, I wanted to share the current state, so you might find a starting point to play with it. The index is an enabler for more advanced functionality in Xtext and in EMF based development in general. So expect it to become an important part in Eclipse Modeling.

Feedback is highly appreciated and should either go to the newsgroup for Xtext (the scoping part) or to the EMFT newsgroup (the index part), because the index project is a component under EMFT.