Top 5 iOS Testing Frameworks

Posted by Vince Power in Mobile Development & Testing

As the market matures in the mobile app space, iOS application developers have a multitude of iOS testing frameworks to leverage. Let’s go over some of the most popular frameworks (in alphabetical order), their strengths, features, test coverage options, and how they are licensed.

Appium

From appium.io:

“Appium is an open-source tool for automating native, mobile web, and hybrid applications on iOS and Android platforms. Native apps are those written using the iOS, Android, or Windows SDKs. Mobile web apps are web apps accessed using a mobile browser (Appium supports Safari on iOS and Chrome, or the built-in “Browser” app on Android). Hybrid apps have a wrapper around a “webview”—a native control that enables interaction with web content.”

Sauce Labs is very active in the development of Appium. Jonathan Lipps, Director of Open Source at Sauce, is the Appium project lead, and a number of Sauce Labs developers frequently contribute to the project. Your Appium tests can run across emulators, simulators and real devices on the Sauce Labs platform.

Strengths

  • You can choose your own underpinning framework, like XCTest, for unit testing.
  • Cross-platform support allows for reuse of test scenarios across mobile and web channels.
  • You can write tests with your favorite tools using any WebDriver-compatible language such as Java, Objective-C, and JavaScript.
  • Full framework is open source

Sample Code for WebDriver

driver.findElement(By.id("com.example.app:id/radio0")).click();
driver.findElement(By.id("com.example.app:id/radio1")).click();
driver.findElement(By.id("com.example.app:id/radio2")).click();
driver.findElement(By.id("com.example.app:id/editText1")).click();
driver.findElement(By.id("com.example.app:id/editText1")).sendKeys("Simple Test");
driver.findElement(By.name("Answer")).click();

Apple Xcode Test

From developer.apple.com:
“Use the XCTest framework to write unit tests for your Xcode projects that integrate seamlessly with Xcode's testing workflow. Tests assert that certain conditions are satisfied during code execution, and record test failures (with optional messages) if those conditions are not satisfied. Tests can also measure the performance of blocks of code to check for performance regressions, and can interact with an application's UI to validate user interaction flows.”

Strengths

  • Tooling is incredibly easy to embed in the application
  • Provides unit, UI, and performance testing
  • Uses Objective-C and Swift so developers can use their existing skillset
  • Can run tests automatically as a “bot” in the CI/CD process on Xcode Server

Sample Code

- (void) testAdditionPerformance {
[self measureBlock:^{
// set the initial state
[calcViewController press:[calcView viewWithTag: 6]]; // 6
// iterate for 100000 cycles of adding 2
for (int i=0; i<100000; i++) {
[calcViewController press:[calcView viewWithTag:13]]; // +
[calcViewController press:[calcView viewWithTag: 2]]; // 2
[calcViewController press:[calcView viewWithTag:12]]; // =
}
}];
}

Calabash

From calaba.sh:
“Calabash enables you to write and execute automated acceptance tests of mobile apps. It’s cross-platform, supporting Android and iOS-native apps. It’s also open-sourced and free, developed and maintained by Xamarin.”

(Editor's Note: Xamarin announced that they have discontinued active development of Calabash as of April, 2017).

Strengths

  • Uses Cucumber to describe the test scenarios using BDD. This is a great feature for QA teams because it makes information clear and easy to digest.
  • Cross-platform support allows for reuse of test scenarios across mobile and web channels.

Sample Cucumber Feature

Feature: Refund item

Scenario: Customer returns a faulty microwave
Given Customer has bought a microwave for $100
And she has a receipt
When she returns the microwave
Then Customer should be refunded $100

EarlGrey

From the Google Open Source Blog:
“Brewing for quite some time, we are excited to announce EarlGrey, a functional UI iOS testing framework. Several Google apps like YouTube, Google Calendar, Google Photos, Google Translate, Google Play Music and many more have successfully adopted the framework for their functional testing needs.”

Strengths

  • Easy to include in an iOS project, either directly or using CacaoPods
  • A flexible framework, with powerful synchronization features across internal components
  • Full framework is open source

Sample Code

// Objective-C
- (void)testInvokeCustomSelectorOnElement {
[[EarlGrey selectElementWithMatcher:grey_accessibilityID(@"id_of_element")]
performAction:[GREYActionBlock actionWithName:@"Invoke clearStateForTest selector"
performBlock:^(id element, NSError *__strong *errorOrNil) {
[element doSomething];
return YES; // Return YES for success, NO for failure.
}
]];
}

OCMock

Homepage: ocmock.org
OCMock provides a framework to create stub objects in your iOS app. It comes in two options—a static library that is used for iOS development, and a framework used for OS X development.

Strengths

  • One of the easiest ways to add mock objects to existing unit tests
  • Uses Objective-C so developers can use their existing skillset
  • Full framework is open source

Sample Code—Adding to an XCTest

- (void)testExample {
XCTAssertEqual(0, [self.calculator calculateTax]);
id mockSubscriptionManager = OCMClassMock([SubscriptionManager class]);
}


Conclusion: Which iOS Testing Framework Should I Use?

Based on your needs and experience with other languages, the iOS testing framework to select may be obvious. Or you may need to try all the frameworks listed to see which one works best for you.

Either way, at this point, the market is mature enough that there is a framework out there that fits the needs of almost any developer, and serves a variety of tasks—whether you simply need to automate testing service calls in iOS, or want to run UI tests in parallel across multiple mobile and web platforms.

Vince Power is a Solution Architect who has a focus on cloud adoption and technology implementations using open source-based technologies. He has extensive experience with core computing and networking (IaaS), identity and access management (IAM), application platforms (PaaS), and continuous delivery.

Discuss: Top 5 iOS Testing Frameworks
0 Comments

Free Trial

Get access to a free 14-day trial version, or contact Sales for more information.