Top 5 iOS Testing Frameworks

Posted Sep 11th, 2020

This post was originally posted by Vince Power in Sept 2017 and revised by Matthew Heusser for Sauce Labs Sept 2020

As mobile development moves toward maturity, a multitude of test automation tools have appeared to help fill the testing gap. Today we'll go over some of the most popular testing  frameworks for iOS (in alphabetical order), their strengths, risks or weaknesses, features, test coverage options, and how they are licensed.

This article will focus on free and open-source tools that are designed for programmers.

1. Appium

Supported by the OpenJS Foundation, Appium describes itself as an “open source test automation framework for use with native, and mobile web apps. It drives iOS, Android, and Windows apps using the WebDriver protocol.”

Sauce Labs is very active in the development of Appium. Jonathan Lipps, formerly the Director of Open Source at Sauce, is the Appium project lead, and a number of Sauce Labs developers frequently contribute to the project. Your Appium tests can run across emulators, simulators and real devices on the Sauce Labs platform.

Strengths

  • You can choose your own underpinning framework, like XCTest, for unit testing.

  • Cross-platform support allows for reuse of test scenarios across mobile and web channels.

  • You can write tests with your favorite tools using any WebDriver-compatible language such as Java, Objective-C, and JavaScript.

  • The entire framework is open source

  • Appium is the de facto standard for WebDriver development on iOS

Weaknesses

  • Appium relies on a cascade of open source components; each must be installed in a version that supports the others. 

Sample Code for WebDriver

driver.findElement(By.id("com.example.app:id/radio0")).click();
driver.findElement(By.id("com.example.app:id/radio1")).click();
driver.findElement(By.id("com.example.app:id/radio2")).click();
driver.findElement(By.id("com.example.app:id/editText1")).click();
driver.findElement(By.id("com.example.app:id/editText1")).sendKeys("Simple Test");
driver.findElement(By.name("Answer")).click();

2. Apple Xcode Test

From the XCTest documentation:

“Use the XCTest framework to write unit tests for your Xcode projects that integrate seamlessly with Xcode's testing workflow. Tests assert that certain conditions are satisfied during code execution, and record test failures (with optional messages) if those conditions are not satisfied. Tests can also measure the performance of blocks of code to check for performance regressions, and can interact with an application's UI to validate user interaction flows.”

Strengths

  • Tooling is incredibly easy to embed in the application

  • Provides unit, UI, and performance testing

  • Uses Objective-C and Swift so developers can use their existing skillset

  • Can run tests automatically as a “bot” in the CI/CD process on Xcode Server

Weaknesses

  • XCode tests are "closer" to the code programming language, so testers will need to be more technical 

  • Programming languages limited to object-C and Swift

XCode Test may be better for production programmers to write tests against their own code

Sample Code

- (void) testAdditionPerformance {
[self measureBlock:^{
// set the initial state
[calcViewController press:[calcView viewWithTag: 6]]; // 6
// iterate for 100000 cycles of adding 2
for (int i=0; i<100000; i++) {
[calcViewController press:[calcView viewWithTag:13]]; // +
[calcViewController press:[calcView viewWithTag: 2]]; // 2
[calcViewController press:[calcView viewWithTag:12]]; // =
}
}];
}

3. Calabash

From calaba.sh:

“Calabash is an automated testing technology for Android and iOS native and hybrid applications.”

Originally developed by Xamarin, which is now a division of Microsoft, Calabash was released as open source in April 2017. The project is currently in search of a maintainer. 

Strengths

  • Uses Cucumber to describe the test scenarios using BDD. BDD Scenarios, with their "Given/when/then" format, can be a helpful collaboration tool, as they describe workflow and can be understood by programmers and business people alike.

  • Cross-platform support allows for reuse of test scenarios across iOS, Android, as well as traditional web applications.

  • A number of default behaviors on iOS, such as "swipe left" are predefined

Risks and Weaknesses

  • The support and future of Calabash is unclear

  • The example above simply defines the BDD scenarios. While the example above is a BDD feature, someone still needs to write the code to perform the action, which will look more like the Appium example above.

Sample Cucumber Feature

Feature: Refund item

Scenario: Customer returns a faulty microwave
Given Customer has bought a microwave for $100
And she has a receipt
When she returns the microwave
Then Customer should be refunded $100

4. Detox

From the Detox Documentation:

High velocity native mobile development requires us to adopt continuous integration workflows, which means our reliance on manual QA has to drop significantly. Detox tests your mobile app while it's running in a real device/simulator, interacting with it just like a real user.

Strengths

  • Runs on emulators and real devices

  • Test runner independent, Detox can run on anything that can run JavasScript

  • Debuggable: Includes async-await to run step throughs just like production code

  • Made for CI: Detox is designed to run inside a Continuous Integration loop

Weaknesses

  • Does not run on iOS physical devices (yet)

  • Detox is not nearly as popular or well-supported as the name-brand tool sets

Sample Code

describe('Login flow', () => {  
 it('should login successfully', async () => {
	await device.reloadReactNative();  
	await element(by.id('email')).typeText('john@example.com');
	await element(by.id('password')).typeText('123456');
	await element(by.text('Login')).tap(); 
	await expect(element(by.text('Welcome'))).toBeVisible();
	await expect(element(by.id('email'))).toNotExist();
  });
 });

5. EarlGrey

From EarlGrey's documentation:

EarlGrey is a native iOS UI automation test framework that enables you to write clear, concise tests.

With the EarlGrey framework, you have access to enhanced synchronization features. EarlGrey automatically synchronizes with the UI, network requests, and various queues; but still allows you to manually implement customized timings, if needed.

EarlGrey’s synchronization features help to ensure that the UI is in a steady state before actions are performed. This greatly increases test stability and makes tests highly repeatable.

EarlGrey works in conjunction with the XCTest framework and integrates with Xcode’s Test Navigator so you can run tests directly from Xcode or the command line (using xcodebuild).

Strengths

  • Easy to include in an iOS project, either directly or using CacaoPods

  • A flexible framework, with powerful synchronization features across internal components

  • Full framework is open source

  • Integrated with XCode

Weakness

  • Developed by Google, EarlGrey is designed to automatically integrate with Google Analytics. In trade for providing your test data to Google, the company provides you with a complimentary test result dashboard. The integration with Google Analytics is not optional.

Sample Code

// Objective-C
- (
void)testInvokeCustomSelectorOnElement {
[[EarlGrey selectElementWithMatcher:grey_accessibilityID(@"id_of_element")]
performAction:[GREYActionBlock actionWithName:@"Invoke clearStateForTest selector"
performBlock:^(id element, NSError *__strong *errorOrNil) {
[element doSomething];
return YES; // Return YES for success, NO for failure.
}
]];
}

6. OCMock

Homepage: ocmock.org

OCMock provides a framework to create stub objects in your iOS app. It comes in two options—a static library that is used for iOS development, and a framework used for OS X development.

Strengths

  • One of the easiest ways to add mock objects to existing unit tests

  • Uses Objective-C so developers can use their existing skillset

  • Full framework is open source

Weaknesses

  • OCMock is not a full test solution; it is a way of making XCode tests better.

Sample Code—Adding to an XCTest

- (void)testExample {
XCTAssertEqual(0, [self.calculator calculateTax]);
id mockSubscriptionManager = OCMClassMock([SubscriptionManager
class]);
}

Honorable Mention

Frank is a BDD framework for iOS and Macintosh that sits on top of Cucumber. If you have concerns about Calabash, frank might be another option to consider. KIF, short for Keep It Functional, is a test framework that leverages accessibility hooks, which are typically plain text, to drive the device. Kif could be an option to ensure accessibility, as if the accessibility hooks are not done, KIF certainly won't run.

Conclusion: Which iOS Testing Framework Should I Use?

Based on your needs and experience with other languages, the iOS testing framework may present itself. For example, production programmers may be led toward XCode tools that run in their preferred languages, while test automation programmers may be more interested in tools that feel like selenium and support Ruby or Python. Some tester creators may need to try all the frameworks listed to see which one works best for you.

Either way, at this point, the market is mature enough that there is a framework out there that fits the needs of almost anyone, and serves a variety of tasks—whether you simply need to automate testing service calls in iOS, or want to run UI tests in parallel across multiple mobile and web platforms.

Vince Power is a Solution Architect who has a focus on cloud adoption and technology implementations using open source-based technologies. He has extensive experience with core computing and networking (IaaS), identity and access management (IAM), application platforms (PaaS), and continuous delivery. His article was updated in 2020 by Matthew Heusser, the dean of software test writing.


Written by

Matthew Heusser


Topics

Frameworks