My Opinion on XML Based Testing in Magento

Magento architect Anton Kril asked about opinions on XML based tests:

#magento twitter, what are your thoughts about writing tests in XML?

— Anton Kril (@AntonKril) 20. Juni 2017

My answer does not fit into a Tweet, so here’s a short blog post.

My first thought was “please no, there’s enough XML already”. But the explanation made me think.

Magento covers a scenario with test. Extension that modifies that scenario, also modifies its test with XML that is merged to original test

— Anton Kril (@AntonKril) 20. Juni 2017

Currently the core test suite does not have much value in custom projects, because the site specific settings, themes and modules will break the tests. But what if we can easily override parts of the functional or integration tests for any modifications we make? If that works it could be extremely useful.

But I’m still sceptical, if it can work. It’s definitely a huge effort to implement and on top of that I expect it to be hard to learn and difficult to debug. My experience with the current PHP based integration test framework is that as long as it works as it should, it’s nice, but as soon as it doesn’t, it’s frustrating and you can waste many hours on a single tests. An additional layer of abstraction possibly makes it even worse.

I was curious if something like this has been implemented successfully before and did some research for existing attempts of XML based testing frameworks. I found some papers from the late 90s, early 2000s, when XML was en vogue and envisioned as the one format to rule them all. What can I say, XML is still here but this vision has not come true.

The actual tools I found reflect this, most look merely like PoCs for academic purpose, or are not continued anymore:

• Anteater, last version 0.9.16 from 2003
• Apache Latka, last version 1.0, from 2010
• Jxunit, last version 3.1.2, from 2013
• Satix, last version 1.0.1, from 2014

Then there’s also XML2Selenium, under active development, current version 1.39.2, from 4/2017, started in 2013. But judged by Google search results it is not super popular. For example there are only three results on StackOverflow.

Another interesting case is BizUnit, a testing framework for Microsoft BizTalk Server (Disclaimer: I have not even heard about that until today). It looks like they had an XML format for tests until version 4, but now moved to XAML in version 5. But the purpose of this format is serialization. The files are not supposed to be written by hand, only written and read by the test framework.

Which reminds me of the original purpose of XML

The design goals for XML are:

1. XML shall be straightforwardly usable over the Internet.
2. XML shall support a wide variety of applications.
3. XML shall be compatible with SGML.
4. It shall be easy to write programs which process XML documents.
5. The number of optional features in XML is to be kept to the absolute minimum, ideally zero.
6. XML documents should be human-legible and reasonably clear.
7. The XML design should be prepared quickly.
8. The design of XML shall be formal and concise.
9. XML documents shall be easy to create.
10. Terseness in XML markup is of minimal importance.

Source: W3C Recommendation 10-February-1998 – (1.1 Origin and Goals)

The only reference to human users is “should be human-legible and reasonably clear”. The rest of the goals lean towards an exchange format between software across multiple platforms.

In that original sense, test cases in XML only make sense if they were generated by software, not hand-crafted. But I acknowledge that XML is established as configuration format. So the question is, can these XML test cases be viewed as configuration, just like layout, DI configuration etc. in Magento?

For that, they should be on a high abstraction level, accompanied by PHP code that maps directly to the XML elements. I don’t want to see code transformed 1:1 into XML like in this abomination from 1999:

<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE testsuite SYSTEM "test.dtd">
<testsuite name="Scanner Test">
  <preamble>
    <set id="scanner"><constructor class="Scanner"/></set>
  </preamble>
  <test name="Test 1">
    <preamble>
      <set id="vector"><constructor class="java.util.Vector"/></set>
    </preamble>
    <action>
      <set id="return">
        <method id="scanner" name="scan">
          <get id="vector"/>
          <primitive>
/**
 * The _Role_ interface.
 *
 * A role organizes a set of permits into a package.
 *
 * @author Todd Sundsted
 * @version 0.1
 *
 */
          </primitive>
          <primitive>0</primitive>
          <primitive>-1</primitive>
        </method>
      </set>
    </action>
    <result>
      <equal>
        <get id="return"/>
        <primitive>136</primitive>
      </equal>
      <equal>
        <method id="vector" name="size"/>
        <primitive>2</primitive>
      </equal>
      <equal>
        <method id="vector" name="elementAt"><primitive>0</primitive></method>
        <primitive>@author Todd Sundsted</primitive>
      </equal>
      <equal>
        <method id="vector" name="elementAt"><primitive>1</primitive></method>
        <primitive>@version 0.1</primitive>
      </equal>
    </result>
  </test>
</testsuite>

(Sorry Todd, I am sure you had the best intentions)

Conclusion

So what would I expect from an XML based testing framework?

• XML should be on high level, not going into details
• XML should only be used to compose tests and to define test data
• There should be actual PHP test code for each XML element. The mapping should be obvious.
• Ideally, the whole thing should be optional.

Consequential, XML based testing only makes sense for functional tests. Maybe to some degree for integration tests, but I don’t believe, that works.

And if I take the question literally:

writing tests in XML?

Writing tests, no! Just as much XML configuration to be able to extend / replace / remove tests and test data? If you get it right, it could be a good thing.