We’ve been talking to mobile test automation for a while now on our blog, but how does test automation fit into the development process and how can it help with mobile app testing? Let’s take a quick look at why companies are adopting agile development, what challenges have to be faced when testing mobile apps and how automation can help to achieve a continuous delivery method.
We live in a pretty hectic world where everything is due yesterday and we can’t really disconnect from work. We expect things to be delivered faster. That is also true for IT processes: we’ve seen the whole development process shift from a waterfall model to continuous integration one and even further to a continuous delivery approach.
But why do companies use agile development?
Using agile development means to release product updates more frequently, adapting to the changing priorities on the go, while increasing productivity and the quality of the product. All of this will also impact directly how and when testers test the software. Here’s a graphic from “10th Annual State of Agile Report”.
What are the benefits for the company that implements agile?
There are three main benefits that have dominated for the previous 5 years in the same research:
It’s easy to manage changing priorities,
Increases team productivity
And project visibility
If you use a waterfall development model, you will have to wait for developers to finish writing the code in its entirety before the code can move on to the testers.
With more agile methods like continuous integration or continuous delivery, dev teams and QA teams will work at the same time. As soon as developers write a piece of code, fix a feature or add a feature, the testers can run the tests to check if everything is still working and that the new parts are working properly as well. At that point, the developers check the code again or change something and a new set of tests needs will be run.
Typical agile development cycle
Along the lifecycle of a software or app, testing is done often and after determined code writing. This way you can build a functioning app from the very beginning.
At different stages of the app development, you will have to check for different things, running a different kind of (automated) tests.
The first set of tests will be about unit tests and regression tests which are run on a few devices before the build.
The second set of test will be run after the build. These tests are typically integration tests, stress tests and performance tests and need to be executed on a larger number of devices because you’re about to release your app.
The last set of test is run after the release. These tests are a continuous set of tests and are meant for monitoring, back-end connections and if needed, you can also integrate a crash analytic tool like HockeyApp.
Testing isn’t just QA anymore, but it becomes a part of development and operations, also known as DevOps. We will have a DevOps-QA team that is responsible for the software, in every stage of its lifecycle.
Everything we have said until now of the development cycle is not really restricted to mobile devices. Desktop applications and software have the same development cycle. Mobile development has its own added obstacles that need to be overcome.
Challenges of mobile app testing
Compared to classic software testing, mobile application testing has some specific challenges, starting with manual testing. Manual testing is in fact no longer a sustainable option when it comes to mobile devices, especially with Android, since more than 24.000 different devices have been detected until August 2015.
It looks quite impossible to perform manual tests on all those Android devices. Your users would also face long waiting times between the releases – executing manual tests means to test one device at a time, performing the same actions over and over again. It’s not only time consuming, but wearisome as well, leading to mistakes.Apart from the device fragmentation you can find other types of fragmentation:
Many different OSs
Many OSs versions
Device models fragmentation
Wide range of screen-sizes
One other challenge is the hardware itself. Emulators and simulators cannot substitute physical devices. You won’t get real performance stats, because you don’t really see what’s happening to the device like: battery drainage, overheating and the app crashing due to previously mentioned reasons. You have a few options here: you can buy the devices yourself or you can subscribe to a cloud service where you can find hundreds of devices, providing both older and newer models.
App quality has a direct impact on the app’s success. And who decides how good your app really is? The users are the ones who will judge your app and they are unforgiving: they get annoyed by even the smallest bug and they will voice out their opinions, especially the negative ones. 58% of users will read the reviews of an app before installing it, if there are many negative reviews, the app just won’t be installed. This will result in a decreased revenue. So why give users a reason to write bad reviews?
To make sure to release a-nearly-flawless app, you’ll need to test more. Automated testing allows you to perform more tests while reducing your testing time.
How can you benefit from automation and what are the challenges?
Mobile test automation plays a key role in continuous integration and especially in continuous delivery. The more a development cycle becomes agile, the more automated tests have to be executed.
In this environment, testing should be one of the most important tasks in the development cycle and it needs to be done often, instead of just the last mile before the release, with no time for further iteration.
When we speak about automation at this point, we talk about functional UI test automation, so basically we check what the user sees and touches.
One of the main benefits of test automation is the reach a wider range of devices with less testing time, enabling shorter release cycles. One other benefit is that you can test as deep as the network connection and many edge cases.
The initial investment to implement mobile test automation in the development cycle and the acquisition of know-how is something that you’ll have to take into consideration, better sooner than later. Your effort will show you a high ROI soon enough.
With automation, you can also scale your testing which is something you cannot do with manual testing.
The main downside is that you can only use test automation for about 80% of your testing; there are certain things that you have to test manually. Integrated testing, which is a combination of automated and manual testing, will still save up to 70% of testing time compared to performing only manual testing.
How do I implement mobile test automation?
To start with test automation you’ll need test scripts that cover your test cases. The scripts can be written in different programming languages using different frameworks. To speed up your testing we suggest using Appium, which is the mobile specific framework derived from Selenium.
It actually is the most versatile mobile UI testing framework supporting many client libraries (Python, Java, JavaScript, Ruby on Rails, PHP, C#). The strong suit of Appium is that it is cross-platform and cross-technology: think about having to write just one test script and running it on iOS, Android and Firefox OS and it will also work with both native and hybrid apps. If you have to start somewhere, then start with Appium!
Learn more about testing with Appium with this webinar “Automated your tests like a pro”!
This post was originally a speech I held at the conference Softwareforen Leipzig on the 5th of April, 2016. The focuses of the conference were agile development and automated testing.
