There are a number of questions around testing in JetBrains Meta Programming System (MPS) that I get a lot. For example: How does MPS affect the way I do testing? How can I do TDD in MPS? Should I test this in MPS?
In this post I will try to answer these questions, but also enable you to answer these questions for yourself. To answer these questions we have to take a step back and reflect on why we do testing in the first place. This post is not a guide on how to implement tests in MPS, it focuses on the bigger picture: what you want to test, and why.
For this post I assume that you have some basic knowledge about JetBrains MPS and are familiar with its terminology. If you are unfamiliar with MPS I suggest taking a look at this ongoing video series I‘m doing on Twitch or to checkout the „Fast track to MPS“ tutorials by JetBrains.
Why Tests?
One questions we need answer before we can look at what we should test is why do we test at all. Obviously we don‘t do testing for the purpose of having tests. The reasons why we do testing mostly fits on one of these three categories:
Validation
Tests validate that the software is in accordance with a specification, norms or legal requirement. Examples here might be the law demanding that passwords are stored encrypted or that exported data from the software fits a schema.
Maintenance
This is probably the most obvious reason why we test. We want to ensure that changes to the software do not accidentally change the way the software behaves. For instance, a calculation is working correctly or an error message is presented to the user if an input was invalid. Obviously there is some overlap here with testing because of validation but the motivation is different. The tests are not performed because a change in the behavior of the software would violate a specification, but because the behavior change in general needs to get detected. In some cases that would mean the software still behaves according to the specification but the test fails.
Quality
It sounds pretty obvious that we want to improve the quality of our software through tests but what does that actually mean?
Do we get better quality because a tests fails? Mostly not. In fact when doing Test Driven Development (TDD) with its test first approach, even during development the test cases rarely fail. How do we get better quality then when not through tests that find bugs in our implementation during development?
The fact that you have to write the test first requires you to think about the actual problem and how to test it. It enforces thinking.
As Mike Gehard has put it: TDD stands for Thought Driven DevelopmentTDD = Thought Driven Development_Edit description_medium.com
The whole part on why testing is heavily inspired by this talk by Micheal Feathers. If you want to get a deeper understanding on why and maybe why not to test something I really recommend watching it.
Testing in MPS
JetBrains MPS ships with built-in support for testing various aspects of a language. For instance editors, type systems, constrains or scoping. The newest addition is support for generator tests, which at the time of writing is still very basic. These built-in testing facilities also integrate nicely with the well-known ways of writing unit tests in plain Java.
One thing that MPS requires is that you think about your problem and how to encode it. Its dedicated language aspects and their DSLs enforce a certain separation of concerns. To encode a simple language concept that also reports errors to the user, you need to divide it into at least it‘s structure, some constrains for its values and other means of checking supported byMPS. Doing this often results in much deeper understanding of the problem and its solution. For me, the idea that tests force me to think about a problem, more than I would otherwise do, is much less prominent in MPS than in for instance in Java.
In this aspect, MPS is similar to programming languages with powerful type-systems like Scala or Haskell. When you want to encode a problem in way that is native to these language you need to think about the problem a lot and get a better understanding of it this way. These languages also often make illegal state unrepresentable. With its individual DSLs for the different aspects of a language MPS does something similar but even goes a bit further: it also enforces a software architecture. You can‘t skip some layer easily in these DSLs. For instance you can‘t do editor-related operations in the constrains aspect of a concept. Of course this is not bullet-proof and with enough (misguided) energy you can still do these things, but doing so is usually much harder than doing the right way.
Of course there are exceptions. Not everything that is build with MPS is implemented with language aspects or DSLs that require you to get the abstractions right. There are also parts that are written in plain Java where a approach like TDDs test first is beneficial.
For the other reasons to write tests, validation and maintenence, MPS is not much different from other software development tools or approaches. If something is missing from the built-in test support or does not fit the needs it‘s often easy to test these things manually by interacting with the respective APIs. Hiding these APIs is often also possible via extensions to the testing languages.
Because it‘s easy to interact with programs written in MPS programmatically, adding new kinds of tests is often possible. Getting new metrics about the tests or their coverage is also easy. These metrics can be much more domain specific that for instance the typical line/branch coverage information you get for programming languages. Well chosen metrics can help a lot to gain valuable insights about the quality of the tests. In some domains e.g. medical, these metrics can also help to verify and qualify the tool/approach to certification authorities.
At the moment MPS does not support property-based testing for anything related to languages. Of corse it‘s possible to plugin a property-based testing framework for Java code written in MPS.
What to Test?
Now that we know why we test it is easy to decide what to test, right? Not quite, there are a lot of factors that you should take into account.
Project
The purpose and context of the project has significant influence on what needs testing. A project with the purpose to prove a concept or that is still exploring the domain, most probably does not need to test for changes in behavior as behavioral changes are intended at this point. Especially with MPS’ ability to quickly evolve languages testing editor behavior it pretty much not what you want to do that at this stage of a project. In contrast, a project that aims to build a DSL to program pacemakers, with a good understanding of the domain and a lot of users in the field has a stronger requirement on what to test. In the latter case you probably don‘t want to change the behavior of the software unintentionally.
Risk
Another factor to take into account is the actual damage if something goes wrong. What are the consequences if a certain aspect of a language does not work as expected? One good example for this are constraints on a concept that limits where it can be instantiated. If these constrains aren‘t working as intended what are the consequences? In most cases it will lead to an error in the downstream processing of the model, an interpreter will fail to produce a result or a generator will generate code that does not compile. In this case an error in the language produces a usability problem, the user can enter something that is unsupported, but will not produce a program/artifact that behaves in a erroneous way. The effort for testing these constrains is also not trivial and therefore most of the time it’s not done.
Maintenance
All code that is written needs to be maintained and this also applies to tests. Test must evolve along with the software they test. As an example, consider editor tests in MPS. They depend on the layout of the editor, when an editor definition is changed these tests tend to fail and require maintenance. The same applies for tests that check scopes of references, these tests need very well defined input model to work reliably.
Don‘t Test the Infrastructure
Awareness of what your are testing it important. In a Java project, would you test that method invocation on JVM works correctly? No. The same applies to MPS. As an example a editor definition that uses a grammar.wrap needs no editor test, the only thing the test would test is that the grammar cell wrap generator works correctly. A complex hand-written contribution to the transformation menu? You most probably want to test that.
Conclusion
If we take a step back from the concrete implementation of tests and look at the motivation behind testing we can see that the usage of MPS largely doesn’t affect testing. Though one aspect is different: the motivation „forced thinking“ from TDDs „test first“-approach is less prominent than in classical programming environments.
The decision of what you test is in the end much more influenced by the context than by the tool you use. Testing doesn’t come for free, the decision whether it is worth the cost depends on the project and risks involed. As with all things in software development there is no silver bullet . Think about the context and the risks involved and the take cautious decision what needs testing and what does not.
