Top 5 iOS App Testing Frameworks

Posted Oct 28th, 2022

Learn about the top test automation frameworks for testing iOS apps, including the benefits and advantages of each plus sample code.

A huge number of mobile testing tools have been developed in recent years to support mobile development in response to the explosive growth of the mobile app and device market in recent years. When it comes to choosing what mobile test automation framework is right for you, there is a huge array of options, each with different strengths and weaknesses.

This post covers some of the most popular free and open source frameworks for iOS app testing (in alphabetical order), including their benefits, disadvantages, features, test coverage options, and sample code.

1. Appium

Appium is one of the leading open source tools for controlling a native, hybrid, or mobile app on iOS mobile, Android mobile, and Windows desktop platforms. It allows testers and developers to create automated tests of mobile applications, helping them deliver quality software faster, with less risk.

Appium Benefits

  • Open source and free to use

  • Supports any WebDriver-compatible language including Java, Ruby, Python, and JavaScript

  • Built on the same JSON wire protocol as Selenium, making the transition easier for mobile developers and QA testers who are familiar with Selenium

  • Allows for testing of native, mobile web and hybrid apps, and can be run across both the iOS and Android operating systems

  • Backed by a large and active community that provides users with consistent support and troubleshooting

  • Choose your own underpinning framework, like XCTest or XCUITest, for unit testing

  • Cross-platform support allows for reuse of test scenarios across mobile and web channels

  • De facto standard for WebDriver development on iOS.

Appium Disadvantages

  • Requires users to learn the concepts of native app/ selectors and have a reasonable understanding of the Appium architecture, which adds to the learning curve

  • Relies on a cascade of open source components; each must be installed in a version that supports the others

Appium Sample Code for WebDriver

driver.findElement(By.id("com.example.app:id/radio0")).click();
driver.findElement(By.id("com.example.app:id/radio1")).click();
driver.findElement(By.id("com.example.app:id/radio2")).click();
driver.findElement(By.id("com.example.app:id/editText1")).click();
driver.findElement(By.id("com.example.app:id/editText1")).sendKeys("Simple Test");
driver.findElement(By.name("Answer")).click();

2. Detox

Detox is a JavaScript-based test framework for React Native Applications only which, like Appium, aims to be cross-platform. It is meant to be used as a Node.js library in your test code, so tests must be written in JavaScript. Detox is also fully open source.

Detox Benefits

  • Runs on emulators and cloud-based real devices

  • Supports cross-platform tests

  • Like Espresso and Earl Grey, Detox tries to ensure your app is in an idle state before allowing automation to continue

  • Underlying technology is precisely Espresso and Earl Grey, so it builds off of Google’s good work

  • Test runner independent, Detox can run on anything that can run JavasScript

  • Debuggable: Includes async-await to run step throughs just like production code

  • Made for CI: Detox is designed to run inside a Continuous Integration loop

Detox Disadvantages

  • Does not run on iOS physical devices (yet)

  • Partially supports running tests on physical Android devices

  • Detox is not nearly as popular or well-supported as the name-brand tool sets

Detox Sample Code

describe('Login flow', () => {  
 it('should login successfully', async () => {
	await device.reloadReactNative();  
	await element(by.id('email')).typeText('john@example.com');
	await element(by.id('password')).typeText('123456');
	await element(by.text('Login')).tap(); 
	await expect(element(by.text('Welcome'))).toBeVisible();
	await expect(element(by.id('email'))).toNotExist();
  });
 });

3. EarlGrey

Earl Grey is Google’s answer to XCUITest for testing iOS apps, and is an iOS UI automation framework which also happens to be open source! (However, using Earl Grey does report anonymous statistics back to Google). Earl Grey is for iOS only, and tests must be written in Objective-C or Swift. The main benefit of Earl Grey is that it brings Espresso’s synchronization features to iOS testing, to once again ensure that the automation is not trying to do something in the app while the app is busy.

EarlGrey Benefits

  • Easy to include in an iOS project, either directly or using CacaoPods

  • A flexible framework, with powerful synchronization features across internal components

  • Full framework is open source

  • Integrated with XCode

EarlGrey Disadvantages

  • Programming languages limited to object-C and Swift

  • EarlGrey integrates automatically with Google Analytics. In trade for providing your test data to Google, the company provides you with a complimentary test result dashboard. The integration with Google Analytics is not optional.

EarlGrey Sample Code

// Objective-C
- (
void)testInvokeCustomSelectorOnElement {
[[EarlGrey selectElementWithMatcher:grey_accessibilityID(@"id_of_element")]
performAction:[GREYActionBlock actionWithName:@"Invoke clearStateForTest selector"
performBlock:^(id element, NSError *__strong *errorOrNil) {
[element doSomething];
return YES; // Return YES for success, NO for failure.
}
]];
}

4. XCUITest

XCUITest is a test automation framework used for UI testing of mobile apps and web applications on iOS devices such as iPads and iPhones. It is part of Apple’s testing framework.

XCUITest also provides a framework that allows developers to programmatically identify and interact with UI elements from other testing tools. XCUITest replaced the older UIAutomator technology and is the only supported UI interaction library for iOS as of 2022.

XCUITest Benefits

  • Your application and test code can be written in the same language (Objective-C or Swift), edited entirely within XCode, and stored in the same repository.

  • Because XCUITest is so closely coupled with iOS, tests may execute faster compared with other frameworks.

  • XCode offers the ability to “Record” tests; generating test code by watching interactions a user has with a connected Simulator or Real Device.  This recorded test code can then be tweaked to give a reliable, repeatable test, saving time during test creation.

  • XCUITest allows testers to find elements by their title, label, value, or placeholder value.  XCUIElements can also have a unique "accessibility identifier" specified for use in testing only, which can make finding elements fast and easy.

XCUITest Disadvantages

  • XCode needs to be installed on every machine where your team develops XCUITest, including tester machines and CI/CD environments. 

  • XCUITest code can’t be run separately from the XCUITest framework; you must run the tests using the XCUITest runner.

  • Programming languages limited to object-C and Swift.

XCode Sample Code

- (void) testAdditionPerformance {
[self measureBlock:^{
// set the initial state
[calcViewController press:[calcView viewWithTag: 6]]; // 6
// iterate for 100000 cycles of adding 2
for (int i=0; i<100000; i++) {
[calcViewController press:[calcView viewWithTag:13]]; // +
[calcViewController press:[calcView viewWithTag: 2]]; // 2
[calcViewController press:[calcView viewWithTag:12]]; // =
}
}];
}

5. OCMock

OCMock provides a framework to create stub objects in your iOS app. It comes in two options: a static library that is used for iOS development, and a framework used for OS X development.

OCMock Benefits

  • One of the easiest ways to add mock objects to existing unit tests

  • Uses Objective-C so developers can use their existing skillset

  • Full framework is open source

OCMock Disadvantages

  • OCMock is not a full test solution; it is a way of making XCode tests better.

OCMock Sample Code: Adding to an XCTest

- (void)testExample {
XCTAssertEqual(0, [self.calculator calculateTax]);
id mockSubscriptionManager = OCMClassMock([SubscriptionManager
class]);
}

Which iOS Testing Framework Should I Use?

Based on your needs and experience with other languages, the iOS testing framework may present itself. For example, production programmers may be led toward XCode tools that run in their preferred languages, while test automation programmers may be more interested in tools that feel like selenium and support Ruby or Python. Some tester creators may need to try all the frameworks listed to see which one works best for you.

Either way, at this point, the market is mature enough that there is a framework out there that fits the needs of almost anyone, and serves a variety of tasks—whether you simply need to automate testing service calls in iOS, or want to run UI tests in parallel across multiple mobile and web platform.

Get Started with iOS App Testing

Get started running iOS tests quickly and easily with Sauce Labs, which provides access to a complete selection of simulated and physical iOS devices and configurations. Sign up for a Sauce Labs free trial to see how it works.


Written by

Sauce Labs


Topics

FrameworksReal Device TestingMobile TestingiOS TestingAppiumXCUITest