iOS App Testing by Sauce Labs
iOS app testing can be a process fraught with complexity, not least because there are different types of apps and different types of testing tools. Knowing how to line up the right tool for your app and team is extremely important. This page will put the general issues relating to the iOS testing scene in context and shed light on the specifics of Sauce Labs’ offering. (Note - we do not cover questions of beta testing and crowd testing of iOS apps; we address only the testing that you -- the app creator or QA professional -- perform yourself.)
It’s first necessary to talk about what we mean by “iOS apps.” The “iOS” part is obvious—iOS is Apple’s mobile operating system deployed on iPhones and iPads. iOS apps come in three kinds:
- Web apps: these are simply websites, mobile versions of web applications that might also be available to users running desktop browsers. Web apps are accessed via Safari, Apple’s mobile operating system. Web apps may or may not be “responsive,” meaning the design of the application may adjust to nicely accommodate the smaller screen size of a mobile device.
- Hybrid apps: these are a mix of the previous two kinds of app. There is a native UI component known as a “webview,” which can be embedded into native apps. A webview is like a borderless, chromeless version of Safari that gives a user a transparent window into web content from any URL, even local URLs. Hybrid apps are popular because they give developers the ability to develop some or all of their app logic using web technologies, but retain access to some of the native APIs (like camera or media library access, for example). In many cases, the native portion of a hybrid app is so “thin” that it’s invisible; in this case, the native portion is there simply to enable the application to live on the App Store, or for use of non-UI APIs like the ones mentioned previously.
Sauce Labs supports testing of all three of these iOS app types. We are able to do this because we rely on Appium, an open-source mobile test automation framework that comes with the ability to work with all of these app types. We also offer cloud-based services for running automated mobile tests at scale. Subscriptions start with small plans with unlimited manual testing and various packages of automated testing minutes that scale to large enterprise plans with unlimited testing time, team management and extensive reporting.
Oh wait – if you haven’t heard of Sauce Labs: our cloud testing platform runs more than a million tests a day for companies like Firefox, the BBC, salesforce.com, Yelp, and thousands of other organizations that are serious about testing, and that are helping us make our solution awesome. The same proven platform supports iOS (and Android) for both manual and automated testing on mobile simulators, emulators and real devices.
What Do You Mean When You Say “iOS App Testing”?
Now we have a very clear idea of what an iOS app is. What about “iOS app testing”? iOS testing is normally made up of some or all of the following components or practices. Which ones depend on your maturity level with automation and the organizational structure of your dev and/or QA team:
- Manual testing - this is testing by hand, the “old-fashioned” way. You test apps manually by firing them up on your device or on an iOS simulator, and navigating through different pre-determined test scenarios using your own finger, eyes, and brain. Apart from verifying working functionality according to pre-determined scripts, manual testing is also used in an exploratory fashion, or to help figure out what the test scenarios should be in the first place. There are service providers, including Sauce Labs, that can help you perform manual testing on multiple iOS configurations without having to set up a device lab yourself, which can be complex, expensive, and time-consuming.
- Automated testing - for any iOS app with some degree of complexity, you will inevitably get tired of manually testing their entire functionality every time a change is made in the app code. Even if you didn’t get tired of it, the time it would take to manually verify an increasingly large number of scenarios starts to slow down your development and release cycle. The solution is to write test scripts which walk your app through the test scenarios in an automated fashion, and make the appropriate verifications along the way. There are two main types of automated tests: end-to-end tests and unit tests.
- End-to-end tests - in this type of test, the entire application stack is exercised, from the UI to the app logic to any network requests or backend server functionality. In other words, a real user experience is simulated. This is therefore the “highest fidelity” kind of automated test you can achieve, since it most closely represents actual use conditions. Since the entire stack is exercised, this kind of test gives information about whether all components of the system work together. How are end-to-end tests written and run? Typically this test involves launching the app on a simulator or real device, then automating interaction with the user interface to perform certain user actions (such as touching, swiping), and then checking the test result. For example, you can run an end-to-end test that 1) logs into the app and 2) tries to upgrade from a free to paid subscription. This process is a highly important user flow, and one that must be tested as thoroughly as possible. Because system tests like these are “costly” in terms of computing resources and also more complex to run, you always need to prioritize and test only the most important workflows in your application. Another challenge is that system tests run differently on different iOS configurations (for example, on an iPad vs. on an iPhone). That means you have to run each test several times - once for each configuration you intend to support. Assuming you’ve chosen Appium (which is in fact based on XCUITest—Apple’s official UI testing library) as your UI automation engine, there are two general ways you can run end-to-end tests:
- Running UI automation yourself - you can use your own real devices or iOS simulators to run these scripted end-to-end tests. However, the number of tests you can possibly run, and the different iOS configurations you are able to test will be limited by the number of machines and mobile devices you actually own. In addition, you must install and manage the infrastructure on your own.
- Running UI automation in the cloud - Sauce Labs is one of several providers that allow you to run end-to-end tests on the provider’s servers in the cloud -- you need only have your app and a test script. Your test script doesn’t even get sent to Sauce Labs; the test commands merely execute sequentially on a Sauce Labs device, from your local machine. The number of servers at the cloud provider far exceeds the number any one tester can set up. Also, the cloud provider has already set up hundreds of iOS version-device combinations, so you cover more options and run more tests more quickly. But there's a price to all this versatility, of course: you’ll need to pay the cloud provider, usually based on the number of tests you perform and in accordance with test complexity, since complex tests may take longer to run and use more device time.
- Unit tests - a unit test involves coding to test a single component of an app, taken in isolation. The test typically passes to the component a series of variables, and checks that the component returns the correct values. Running unit tests is faster and far less “costly” than running full system tests, because you don't have to initiate a full instance of iOS for each test. Instead, you can run a large number of tests together on a local machine. Apple’s XCode development environment lets you create and run unit tests for iOS apps, utilizing their XCTest framework. It is often used together with Kiwi, an Objective-C unit testing framework, which aims to improve the readability of tests and make test running and error reporting more seamless.
- Test frameworks - whether it's unit tests or full-blown system tests, to make it all work together, you are going to need a testing framework that enables the reuse of testing code and handles some of the boilerplate of getting tests running. Your testing framework will rely on one (or potentially more) automation libraries. Examples of these automation libraries are: XCUITest, Earl Grey, and Appium, which Sauce Labs supports as an open source project. There are important differences between these libraries. See this page for a full comparison. Test frameworks are typically responsible for properly encapsulating UI information to make tests readable and maintainable. For example, check out the “Page Object Model” as an example of a framework-level strategy.
- Continuous integration - if you perform automated testing on a large scale, you probably use a CI system such as Jenkins, Microsoft VSTS or Travis. Those systems typically request and execute all these tests, record their results, and feed them into an agile development process. Of the iOS testing frameworks, some, including Appium, enable you to connect to a CI system and run automatic iOS tests in the context of your agile development process. Mobile CI is in its infancy relative to CI for web apps, and does not yet always work as smoothly as CI with traditional testing tools. The industry is moving forward, however, and with projects like Appium, CI is that much more achievable for mobile, since Appium provides exactly the same kind of output as Selenium, which is well-understood in terms of its CI integrations.
Testing your iOS ApplicationS
Create an account and start testing your iOS apps today! Get started with our 14 day free trial.