As a software developer and testing expert with over 20 years in this industry, I’ve seen tools and solutions grow and disappear from the market. That’s OK, and that’s the life-cycle and “price” of innovation. Unfortunately, I have also seen some wrong practices when it comes to choosing such tools, especially around the testing domain.
In this post, i would like to make some order in what needs to be top of mind for managers and practitioners when trying to either change and existing solution or pick a new one.
Focus on the Process
I cannot stress enough the importance of the existing processes that are being followed within the organization that is evaluating a new tool. Process means, cadence of releases, how are testers, developers, and business work and collaborate?. Process also means, Agile vs. Waterfall vs. other methods like ATDD/BDD. Process also refers to the quality and the definition of done.
If the teams aim to release on a weekly basis, and their test automation suite is either unstable enough, the lab isn’t able to cope with the size of the tests and platforms, or the time it takes to analyze a regression test report is too long, this needs to impact the decision process as well as the evaluation criteria process.
In most organizations that I’ve met, the teams are distributed and share different skills in both development and testing. This is also a driver to the way test development and execution is processed.
Consider Overall Organizational Objectives
Teams need to ask themselves what good looks like? Do they aim towards a “right” test automation coverage of over 80%? What is the maximum time that they have to test within each iteration or sprint? Before asking the questions around what can the tool do for me, teams need to feel solid about their objectives and where they plan to grow their teams.
Just doing a checklist of tools functionalities isn’t relevant, and cannot resist over time since the domain is changing constantly, the tools sometime becomes obsolete and your own objectives might change as well.
Bottom line: “Don’t let the tool functionality drive your selection criteria, but rather let your own objectives and process fit do it“
In addition, teams ought to be focusing as part of the continuous improvement agenda on measurements, metrics and visibility into how they operate over time. For such, the organization needs to set objective metrics that matches where the teams are aiming to be in 6 months, 12 months, 2 years etc.
There are various market reports like the recent Google DORA DevOps report that was showing variance between Elite DevOps organizations and lower maturity organizations. These kind of market objective metrics can serve as baseline for setting the overall goals.
Get Down to Business – What can The Tool do For Me?
Only after making sure across the board the the above questions are checked, and you have a good understanding on where you wanna be, what does your team needs to be successful and meet their goals, than go and create the right list of selection criteria for your testing tools.
Such list from my experience ought to include the following 6 buckets and be able to cover the functionalities of test creation, test analysis, and test environment (lab) for proper coverage and execution.
Some tools come with AI and ML capabilities, some have record and playback, some require advance coding skills. You should consider all and in many cases, probably combine few tools to get to the 1 single goal you’ve set.
For each of the above categories whether you’re evaluating an open-source or a commercial too, you should focus on the test creation and analysis of each. Not all frameworks and tools were created equally.
If to only focus on web test automation with Java Script, you can observe the plethora of test frameworks as listed by Vitali Zaidman on Medium to see how may solutions are there per each category.
There are various Java Script Tools that…
- Provide a testing structure (BDD/ATDD) (Mocha, Jasmine, Jest, Cucumber)
- Provide assertion functions (Chai, Jasmine, Jest, Unexpected)
- Generate, display, and watch test results (Mocha, Jasmine, Jest, Karma)
- Generate code coverage reports (Istanbul, Jest, Blanket)
- Provide a browser or browser-like environment with control of scenario execution, UI testing, and more (Protractor, Nightwatch, Phantom, Casper, Selenium, WebDriver.IO, TestCafe)
- Provide mocks, spies, and stubs (Sinon, Jasmine, enzyme, Jest, testdouble)
BTW, the above constantly shift and changes as the market matures
Bottom Line
As highlighted in this post, it is about YOU and not the tool you’re evaluating. Try focusing on how your process needs to look like, see what objectives the business and management has for your projects, and than see how tools can fit into your processes. Don’t follow other mistakes that lists features and functionalities of their tools and make it your decision criteria, but rather come prepared with yours and see how tools help meet your goals. Don’t get me wrong here, features are critical for the success and enable your test automation coverage, however not every tool also fits your entire processes that starts from the creation of tests through analysis and maintenance.
Happy Transformation and Testing!
