In my last post, I talked about how to test the right things. This week, we continue our discussion of jumpstarting test automation with a breakdown where—and when—to run tests.
The answer to these questions will really depend on which stage of testing you’re in.
Waterfall is the legacy that we are hopefully moving away from. For example, if you’re in waterfall, you may be doing a lot of manual testing. In this scenario, everyone may be working together towards a major release, possibly doing joint bug-hunting sessions together to prepare. In this situation you’re likely doing local testing on desktops, and it’s common to have shared testing environments. Due to the nature of manual testing, you just won’t have time to test more often.
If you’re in fast waterfall, OK—that’s a good start when moving from manual testing to automation. It allows for a safe environment to learn how to automate and do proof of concepts for different approaches. Fast waterfall is where automation begins. Here, we start enabling testing to happen more often, perhaps one or more times per day. Tests are triggered manually and often combined with manual testing that takes place during the pre-release phase. We need to learn how our tests are evolving and working properly. In this scenario, we may still be testing locally or you may have built your own grid, and you may begin to dabble in cloud-based testing.
Once you move to Continuous Integration, automation dominates. With CI, all high-value tests are automated, everyone can execute them, they are part of the CI pipeline and give fast feedback. This is the true benefit of automation. Manual testing will still happen, but will be used more often for debugging or exploratory testing. A cloud infrastructure is vital for success at this stage, because maintaining your own test grid is difficult and expensive—and you need comprehensive browser and operating system coverage.
With CD, the development process is fully automated. All teams should be able to do testing and development. Manual is used for exploratory testing only. Tests are also used to validate production deployments. When are you running tests? All the time: pre-commit, commit, after merge/pull requests, after merging, release, etc.
The result is that automated tests are fully resilient and trustworthy, and can be used in all environments. Again, at this stage, you should be trusting a cloud-based testing platform to ensure you can be confident in the tests across multiple browsers, devices and operating systems.
Next time, I will talk about how to inspire people to join your mission to automate testing. If you just can’t wait, check out the webinar that I presented recently that covers the 5 steps to jump-starting your test automation. See you next time!
If you enjoyed this post, check out the previous installments of Diego’s “5 Steps to Jumpstart Test Automation” series: