The MPS meet-up in Munich sprung some discussions about how feature branches and language migrations get along. It seems that there is quite some uncertainty how to approach this. In this blog post I shed some light on the topic and show what we found to work in practice. I assume that you already have some background in MPS. I won’t explain the details of how to write migrations or other aspects of MPS that play a role in the overall picture.
Let’s first explain what I mean by the two main topics I’m going to talk about here: Feature Branches and Language Migrations.
Feature Branch
Here I’m going to stick with Martin Fowlers definition:
The basic idea of a feature branch is that when you start work on a feature (or UserStory if you prefer that term) you take a branch of the repository to work on that feature. In a DVCS, you’ll do this in your personal repository, but the same kind of thing works in a centralised VCS too.
So in general it’s a branch that you create specifically to work on your feature without putting the changes directly into the main development branch. In effect, you isolate the development of your feature from everybody else’s work as long as you are not yet done.
Language Migration
In the context of MPS, language migration refers specifically to migration scripts that are written by the language developer in the migration aspect of a language.
There are also two other reasons why MPS would run migrations on the project. One is refactoring logs, which are automatically generated by MPS when, for instance, a concept is moved into a different language. While you can’t really influence the content of such auto-generated scripts, most of the suggestions of this post also apply to this kind of migrations.
The second type is called update module dependency versions. It is basically updating the transitive dependency versions in the language/solution file. This migration is always safe to execute; even if it causes a merge conflict, the MPS merge driver is pretty good at solving these automatically.
DOs
All of the following points mentioned here aren’t strict rules. They should be considered more like guidelines to keep in mind and applied when needed. Sometimes it makes sense to divert from them. But diverting should be an deliberate decision.
Update Dependencies Explicitly
One main reason why migrations are required is because some upstream dependency has changed and the updated version contains migrations. When this happens on two branches and the migrations are executed concurrently on them, merging them will cause merge conflicts. This typically happens when using a build system that also manages your dependencies like Gradle or Maven. A unintended update is avoidable by making the update of an upstream dependency an explicit step. If you are using Maven or Gradle and you specify your dependency by using a version matcher (for instance: 1.0+), this can cause your build to fetch a newer version of a dependency without noticing. If there are multiple feature branches active at the same time it can happen that these migrations are then executed on two branches. This is very likely because developing multiple features in parallel is one of the reasons why feature branches are used in the first place. When merging these changes back this will certainly cause merge conflicts because the migration will do the same changes on two branches.
These conflict are avoided by making the update an explicit step and not simply letting it happen during a random build on some developers machine. If you are using Gradle it is easy to handle. This plugin supports locking your dependencies from unintended updates during the build while still allowing you to use a version matchers. I won’t go into the details how it works here but it basically stores the exact version a version matcher has resolved to and uses this version until the user instructs it to update the version again.
Write Resilient Migrations
Writing resilient migrations applies more to language developers than to language users but many projects use their own languages to dog food their development.
What do I mean by resilient migration? Resilient means for me that the migration doesn’t entirely depend on the MPS infrastructure, most importantly the language version number, to determine which part of the model requires migration. It automatically results in migrations that are rerunnable. Of course this is not always possible because in some situations it’s impossible to detect if parts of a model need migration without relying on the language version that MPS stores alongside with the model. Especially in cases where the migration is required to correct/change semantics without a related structure change of the meta model this is impossible. But in many cases it is possible to detect this independent of MPS and it also results in a better experience for the end user.
Let me give an example:
When adding a new child to an existing concept, the child should have a default value. You might have a concept constructor or a node factory that correctly initialises this child for each new instance of the concept. Now you need a migration that sets the value to the default for all existing instances. This migration will get executed by MPS automatically for each Module (Solution or Language) that uses an older version of the language. In the implementation of the migration we could simply assume that all instances of our concept need a default value for the new child. Thinking like this is natural because we assume that the instances are in their old state and we now need to update all of them. This also results in the fact that we could never execute the migration again after is has run, it would simply override all instances where the user choose a difference value than the default. But lets face it, the assumption that users will execute your migration right away when they first see the migrations assistant popup isn’t always correct. Users will simply click cancel and delay migrations, might play around with the update and then later run the migrations. If, before the migrations are executed, the user has already created new instances of the concept and selected a different value than the default the migration will also overwrite them.
To prevent this problem, the migration can be written in a slightly different way which then makes it rerunnable and thus can also handle the situation where the user delay the execution. The migration could simply apply itself only to nodes where the newly created child is still empty. As there is no way for the user to construct a model where the child is not set this must be a node that still requires attention from the migration.
Don’t forget to mark the migration as rerunnable 😉.
Migrate Before Merge
Once we have resilient migrations we can make use of them. For instance to prevent outdated models from showing up on the main development branch even when MPS could not automatically determine that migrations were required. Before a feature branch is merged, simply check if anything requires migration by selecting the run rerunnable migrations entry form the migration menu. This will execute all migrations marked as rerunnable. Make sure you have committed all your changes before running this. Afterwards commit the migration result in a single commit.
I wouldn’t recommend adding this item to your normal feature branch merging habit but it can help in situations where migration where executed on incoming changes to ensure everything is up to date.
Commit Migrations Separately
When migrations are executed, put them into a separate commit that only contains the result of the migration. Worst case, this allows this commit to be reverted or dropped from the history, if necessary. It also allows easier review of the migration result.
Short Living and Isolated Branches
One key point aside from technical concern in MPS is that feature branches should be as short lived as reasonably possible. It allows to keep the branch isolated during the time it is active. Ideally you don’t need to merge your development branch into it while the feature branch is active. Of course this is not always possible but reducing the amount of merges also reduces the amount of potential migrations required.
Apply your Migrations before Merging
If migrations have been written on a feature branch that affect parts of the project, these migrations should get executed as part of the merge process of the feature branch. This way you ensure that in there are no migrations to be executed on the main development branch. Combined with the explicit update of dependencies this is quite effective to prevent accidental migrations.
DONTs
The points below are common pitfalls that we have observed when working with migrations in MPS in general and aren’t really specific to feature branches but still worth while considering.
Migrate when Solving Merge Conflicts
When solving merge conflicts MPS will also pop up the migration dialog if it detects migrations that are executable. This often happens when languages that had a conflict are built from the IDE to ensure the conflict resolution has worked out and the merged changes contained a new migration. It’s important not to apply this migration while merging. First solve the conflicts, commit the merge and then apply the migrations. This way it’s easy to commit the migration result as a dedicated commit.
If you agree that running migrations during a merge is generally a bad idea, you should upvote an MPS feature request to prevent this here.
Merge “master” and Migrate
Merging your main development branch and apply the incoming migration without thinking about it will sooner or later cause harm in your codebase which can be a pain to address. This might be one of the reasons which lead to the initial statement that feature branches are dangerous because of migrations. Whenever migrations are going to be executed, think about if it actually makes sense to execute them right away. MPS does not allow you to execute migrations selectively (yet). You might be in the middle of writing a migration and therefore running the migration from the incoming changes might also execute your (incomplete) migration. In these cases skip them and execute them when you think it’s a good point in time to do so.
Take Away
I hope I was able to show that language migrations in conjunction with feature branches aren’t something to be scared of. However, it is important to put deliberate thought into writing your migrations as it will benefit the developer of the language, and even more the end user.
Languages aren’t much different from other dependencies in software development. Updates to dependencies should be done deliberately to prevent problems with uncoordinated updates. In order to avoid large changes that might require integration against lots of migrations, make sure you work in small increments and merge feature branches early and often.
