Mobile Testing Tools - 11 Open Source Frameworks Compared
Calaba.sh, Robotium, Selendroid, Appium and more
A huge number of mobile testing tools have been developed in recent years to support mobile development. As more companies are developing mobile products and the marketplace is seeing more devices, platforms, and versions, testing your mobile apps is vital. When it comes choosing what mobile testing tool is right for you, there is a huge array of options, each with different strengths and weaknesses.
As a big supporter of open source technology, Sauce is committed to furthering the open source cause, and we believe there are a number of great open source mobile testing frameworks that should be considered when devising a mobile testing methodology. All have different ways of tackling mobile automation, and the right solution for your project will ultimately come down to which framework fits your needs best. We’ll discuss some aspects of some of the different open source mobile automation tools, as well as why we chose Appium as the main mobile automation framework supported on Sauce.
Why Should You Do Mobile Automation?
It’s clear that anyone making apps should test them. Failure to catch bugs or regressions can cost companies thousands of dollars a day, and releasing broken apps can frustrate and alienate your users. But why should you automate your mobile apps? The reasons for doing automated mobile testing are arguably the same as for traditional web development.
Speed of Testing
While manual testing can be useful, it is a slow and human resource-heavy process. By automating your testing, you can run a suite of tests that would take a manual tester hours to complete in minutes or seconds. Speeding up your testing can allow you to expand your test coverage so you can be more confident that you’re releasing bug-free code.
Not only is manual testing slow, but increasing your testing can be difficult. With automated mobile testing tools, increasing which platforms you are testing on and running significantly more tests is easy. The ability to reuse tests over and over also increases your test capabilities. And if you choose a cloud testing platform like Sauce Labs, you can scale instantly without needing to buy or install more servers or purchase more VM licenses.
Automated testing can save you time and money, since you can spend fewer resources on manual testing. Add to that the cost savings of using Sauce’s cloud platform instead of maintaining in-house infrastructure, and automated mobile testing can be a cost-effective solution for ensuring you’re releasing bug-free apps.
Some Options for Mobile Testing Tools
Mobile testing on Sauce is powered by Appium, an open source mobile automation framework for testing native and hybrid apps. However, there are many different options for mobile testing, all with different philosophies and approaches. We’ll discuss Appium’s philosophy and why we choose to support it, as well as explore some of the other tools available and the differences.
We believe that Appium’s approach to mobile automation makes it a strong mobile testing tool that will continue to improve. With the mobile marketplace fragmented between OS platforms and app types, there is need for a truly cross-platform mobile automation tool, and Appium aims to be that tool.
Appium’s philosophy follows four principles:
1) Test the same app you submit to the marketplace. You shouldn’t need to compile third party apps or frameworks into your app’s code that needs to be stripped from the app before submitting it to the app store. The reason for this is simple - if you test one version of your app and submit another version, there is a risk for bugs you didn’t foresee being released into the marketplace. Therefore, to minimize this risk, we believe it is prudent to test the same app you release.
2) Write your tests in any framework, using any language. People choose which language to write in for many different reasons, and many even write in multiple languages in a given day. This means that any tool that limits what languages and frameworks it supports will be limiting to those using it. Appium strives to support as many languages and frameworks as possible, to give developers the most flexibility.
3) Use a standard automation specification and API. Developing standards for specifications and API benefits everyone in the open source ecosystem. There are already existing specifications and standards for web testing tools, and Appium’s philosophy is that mobile automation tools should reuse and adapt these existing standards to determine what the standards for mobile testing tools should be.
4) Build a large and thriving open-source community effort. For both the health of the tool and the benefit of anyone wanting to test, it’s a good idea to encourage a robust community to support and improve the project. Appium strives to build a vibrant community on the belief that that community will further the technology for everyone.
It’s because of this approach that Sauce has made the effort to support Appium as a mobile automation framework. We believe that this philosophy translates to a well-thought out vision that drives the tool forward. We also believe that this philosophy sets Appium apart from other open source frameworks.
As you can see below, of the most popular mobile automation tools in the ecosystem today, they all specialize in iOS or Android. Appium, though, bridges the gap.
And, when comparing different mobile testing tools, none of the other popular frameworks follows Appium’s four principles.
Sauce Labs is committed to the health of the open source ecosystem, and we believe Appium is a great option for those looking for a mobile testing tool. Sign up for a free trial to start testing.
You can also read more about some of the other open source testing tools available, like Calabash, KIF, and Robotium, and dig into how they differ from Appium.
Sauce Labs - Scalable Mobile Testing