One of the most important aspects of using the Continuous Integration methodology is having a quick and reliable QA system that can keep up with the rapid-fire deployments that we are striving towards.  My objective was to achieve this very thing.  It was no small task, and the work is never finished. However at this stage of development, the main concepts have been designed and implemented to a point where I thought it would be worth sharing.

This post is a story of how we went from a traditional Jenkins QA setup that ran tests sequentially, slowly and unreliably to one that ran parallelized tests in a quick, scalable and more reliable way.  Our team maintains automated tests for API, Web, Mobile Web, and Native Apps, which makes our needs and problem-sets pretty wide-reaching.  Yet, with our strategy, we were able to positively impact every testing platform significantly.  By the end of the effort, we ended up with whole suites of tests that ran in under a minute or two (yes, this includes web tests).  This is down from test runs that could take 10 minutes to over an hour to complete.  Despite the ever-increasing coverage that we offer, our test runtimes are significantly shorter than a sequential setup.  With our current setup, we can run hundreds of individual test cases in parallel.

Even if your company is not adopting a CI strategy for their releases, you may find some pieces of this strategy beneficial to your testing solution.  There are a lot of moving pieces that ultimately led us to where we are today.  I have broken them up into milestones in an effort to give each one the attention it demands to properly communicate the level of effort involved.

To show the speed improvement between sequential and parallel runs, I built a simple web test that is repeated N times.  The web test simply opens up a browser.  This first image is of 50 such tests running sequentially.

With the same setup, here are the same 50 tests ran in parallel.  Notice the decrease in runtime.

As the number of tests increase, the runtime continues to increase for both sequential and parallel job runs, but at different rates.  Here, we have 100 tests running in sequence.

Finally, here are the same 100 tests running in parallel.  This example is very limited based on the memory of the slaves and the predictable nature of the tests, so results can vary in the wild.