Have you ever worked on a web-based test team and switched to a mobile team and wondered if your life is about to get easier or harder? There are significant differences between testing mobile vs. web, and yes, one is MUCH harder than the other. Want to guess which one? Read on and see if you guessed correctly.
Let’s Compare
The table below shows the different facets of testing and where its execution is most challenging.
Scenario | Web is Harder | Mobile is Harder | It's a Draw |
Feature Functionality Testing For the case where you have a web application with a supporting mobile app, it is likely the app will have a subset of features that the web side does. When first developing a feature, it is new to the web development team. They have to go through the process of designing and building out the new feature, and training their team on the concept. When the supporting app is created, it would most likely rely on the existing web feature set, so the design and concepts should be easier for the team to comprehend. | X | ||
Feature Parity Speaking of features, when you are building a web app you only have to account for the features in one user world, such as the browser. The feature only needs to be developed once. When you create features on a mobile app, you now have to take into account not only each device platform’s capabilities, but even the user communities’ expectations. Apple and Droid users have different expectations for how their apps will work. And if you want to have feature parity in your apps, than the development process management becomes all the more difficult. From a test perspective you now have double the work. | X (mobile is harder) | ||
Deploying the Application When using CI, your test environments can constantly be updated for your use with the latest builds. You can also do this on the mobile side if you are using simulators and emulators, or a device cloud such as the one provided by Sauce Labs. But what about if you want to test on live, local devices? Most likely your team is using them throughout the day and they are not connected to any CI environment. You have to manually install the apps when you want to test the latest, which can be time-consuming, especially when you have multiple devices and platforms. | X (mobile is harder) | ||
Integration Tests Testing needs to account for backend services. Both web and mobile development teams need to run integration tests against their respective APIs. The time and effort to accomplish this are similar. | X (it's a draw) | ||
Automating with Page Object Locators When writing automation on a web application, you need to find the page object locators. You only need to write code to support one set of locators. If you are developing automated tests for mobile apps, you now need to work with two different dev teams to determine the locators. (note, this is an ideal place to have cross-team standards to build consistency between the apps). And as stated previously, the features might not be in sync, causing testers to write multiple tests for similar features. | X (mobile is harder) | ||
Tool and Support Community Web automation is a very mature industry. Not only are the tools in place to support most automation interactions, but the user community is available to answer any question for any level of user. While the mobile world of automation has come a long way, it still doesn’t have the amount of support that web does. | X (mobile is harder) | ||
Platform Complexities When talking about the test infrastructure, mobile is much more fragile. The different deployment rules between Apple and Droid can make for some tricky problems to solve. Not only that, but the multiple-platform issue comes into play again because now you are supporting two sets of infrastructure that can push updates at any time, causing your tests to unexpectedly break. | X (mobile is harder) | ||
Test Strategy When you think of testing on the Web, your strategy usually takes into account the different supported browsers and maybe the underlying operating systems (OS). With mobile, you have to take into account the OS versions for each platform, as well as device types. While Apple is pretty stable and their user community is up-to-date on the OS, Droid can have a ton of different configurations that your user community supports. | X (mobile is harder) |
Did You Guess Right?
When I first posed the mobile vs. web question to an automation architect who has worked in both arenas, he immediately replied “mobile is 110% harder.” I have to agree.
But that is not bad news. First, mobile tooling is still evolving, and does not have the maturity that web tooling does. Secondly, there is already plenty available of resources that can help, such as real-device clouds and more and more robust testing frameworks like Appium and Robotium. While you can’t do much to change the nature of app stores, testing and CI tools for mobile are only getting better.
Joe Nolan (@JoeSolobx) is a Mobile QA Team Manager with over 12 years of experience leading multi-nationally located QA teams, and is the founder of the DC Software QA and Testing Meetup.
