A huge number of mobile testing tools have been developed in recent years to support mobile development in response to the explosive growth of the mobile app and device market. When it comes to choosing the right mobile test automation framework for you, there are several options, each with different strengths and weaknesses. This post covers some of the most popular free and open source frameworks for iOS app testing (in alphabetical order), including their benefits, disadvantages, features, test coverage options, and sample code.
1. Appium
Appium is one of the leading open source tools for controlling a native, hybrid, or mobile app on iOS mobile, Android mobile, and Windows desktop platforms. It allows testers and developers to create automated tests of mobile applications, helping them deliver quality software faster, with less risk.
Appium Benefits
Open source and free to use
Supports any WebDriver-compatible language including Java, Objective-C, and JavaScript
Built on the same JSON wire protocol as Selenium, making the transition easier for mobile developer and QA testers who are familiar with Selenium
Allows for testing of native, mobile web and hybrid apps, and can be run across both the iOS and Android operating systems
Backed by a large and active community that provides users with consistent support and troubleshooting
Choose your own underpinning framework, like XCTest or XCUITest, for unit testing
Cross-platform support allows for reuse of test scenarios across mobile and web channels
De facto standard for WebDriver development on iOS.
Appium Disadvantages
Requires users to learn the concepts of native app/ selectors and have a reasonable understanding of the Appium architecture, which adds to the learning curve
Relies on a cascade of open source components; each must be installed in a version that supports the others
Appium Sample Code for WebDriver
driver.findElement(By.id("com.example.app:id/radio0")).click();
driver.findElement(By.id("com.example.app:id/radio1")).click();
driver.findElement(By.id("com.example.app:id/radio2")).click();
driver.findElement(By.id("com.example.app:id/editText1")).click();
driver.findElement(By.id("com.example.app:id/editText1")).sendKeys("Simple Test");
driver.findElement(By.name("Answer")).click();
2. Detox
Detox is a JavaScript-based test framework for React Native Applications only which, like Appium, aims to be cross-platform. It is meant to be used as a Node.js library in your test code, so tests must be written in JavaScript. Detox is also fully open source.
Detox Benefits
Runs on emulators and cloud-based real devices
Supports cross-platform tests
Like Espresso and Earl Grey, Detox tries to ensure your app is in an idle state before allowing automation to continue
Underlying technology is precisely Espresso and Earl Grey, so it builds off of Google’s good work
Test runner independent, Detox can run on anything that can run JavasScript
Debuggable: Includes async-await to run step throughs just like production code
Made for CI: Detox is designed to run inside a Continuous Integration loop
Detox Disadvantages
Does not run on iOS physical devices (yet)
Partially supports running tests on physical Android devices
Detox is not nearly as popular or well-supported as the name-brand tool sets
Detox Sample Code
describe('Login flow', () => {
it('should login successfully', async () => {
await device.reloadReactNative();
await element(by.id('email')).typeText('john@example.com');
await element(by.id('password')).typeText('123456');
await element(by.text('Login')).tap();
await expect(element(by.text('Welcome'))).toBeVisible();
await expect(element(by.id('email'))).toNotExist();
});
});
3. EarlGrey
Earl Grey is Google’s answer to XCUITest for testing iOS apps, and is an iOS UI automation framework which also happens to be open source! (However, using Earl Grey does report anonymous statistics back to Google). Earl Grey is for iOS only, and tests must be written in Objective-C or Swift. The main benefit of Earl Grey is that it brings Espresso’s synchronization features to iOS testing, to once again ensure that the automation is not trying to do something in the app while the app is busy.
EarlGrey Benefits
Easy to include in an iOS project, either directly or using CacaoPods
A flexible framework, with powerful synchronization features across internal components
Full framework is open source
Integrated with XCode
EarlGrey Disadvantages
Programming languages limited to object-C and Swift
EarlGrey integrates automatically with Google Analytics. In trade for providing your test data to Google, the company provides you with a complimentary test result dashboard. The integration with Google Analytics is not optional.
EarlGrey Sample Code
// Objective-C
- (void)testInvokeCustomSelectorOnElement {
[[EarlGrey selectElementWithMatcher:grey_accessibilityID(@"id_of_element")]
performAction:[GREYActionBlock actionWithName:@"Invoke clearStateForTest selector"
performBlock:^(id element, NSError *__strong *errorOrNil) {
[element doSomething];
return YES; // Return YES for success, NO for failure.
}
]];
}
4. XCUITest
XCUITest is a test automation framework used for UI testing of mobile apps and web applications on iOS devices such as iPads and iPhones. It is part of Apple’s testing framework.
XCUITest also provides a framework that allows developers to programmatically identify and interact with UI elements from other testing tools. XCUITest replaced the older UIAutomator technology and is the only supported UI interaction library for iOS as of 2022.
XCUITest Benefits
Your application and test code can be written in the same language (Objective-C or Swift), edited entirely within XCode, and stored in the same repository.
Because XCUITest is so closely coupled with iOS, tests may execute faster compared with other frameworks.
XCode offers the ability to “Record” tests; generating test code by watching interactions a user has with a connected Simulator or Real Device. This recorded test code can then be tweaked to give a reliable, repeatable test, saving time during test creation.
XCUITest allows testers to find elements by their title, label, value, or placeholder value. XCUIElements can also have a unique "accessibility identifier" specified for use in testing only, which can make finding elements fast and easy.
XCUITest Disadvantages
XCode needs to be installed on every machine where your team runs XCUITest, including tester machines and CI/CD environments.
XCUITest code can’t be run separately from the XCUITest framework; you must run the tests using the XCUITest runner.
Programming languages limited to object-C and Swift.
XCode Sample Code
- (void) testAdditionPerformance {
[self measureBlock:^{
// set the initial state
[calcViewController press:[calcView viewWithTag: 6]]; // 6
// iterate for 100000 cycles of adding 2
for (int i=0; i<100000; i++) {
[calcViewController press:[calcView viewWithTag:13]]; // +
[calcViewController press:[calcView viewWithTag: 2]]; // 2
[calcViewController press:[calcView viewWithTag:12]]; // =
}
}];
}
5. OCMock
OCMock provides a framework to create stub objects in your iOS app. It comes in two options: a static library that is used for iOS development, and a framework used for OS X development.
OCMock Benefits
One of the easiest ways to add mock objects to existing unit tests
Uses Objective-C so developers can use their existing skillset
Full framework is open source
OCMock Disadvantages
OCMock is not a full test solution; it is a way of making XCode tests better.
OCMock Sample Code: Adding to an XCTest
- (void)testExample {
XCTAssertEqual(0, [self.calculator calculateTax]);
id mockSubscriptionManager = OCMClassMock([SubscriptionManager class]);
}
Which iOS Testing Framework Should I Use?
Based on your needs and experience with other languages, the iOS testing framework may present itself. For example, production programmers may be led toward XCode tools that run in their preferred languages, while test automation programmers may be more interested in tools that feel like selenium and support Ruby or Python. Some tester creators may need to try all the frameworks listed to see which one works best for you.
Either way, at this point, the market is mature enough that there is a framework out there that fits the needs of almost anyone, and serves a variety of tasks—whether you simply need to automate testing service calls in iOS, or want to run UI tests in parallel across multiple mobile and web platform.
Get Started with iOS App Testing
Get started running iOS tests quickly and easily with Sauce Mobile, which provides access to a complete selection of simulated and real iOS devices and configurations.
