CI’s not a new thing. Wikipedia says the phrase was first used back in 1994, way before modern mobile apps. Today it’s commonplace in many dev shops for developers to expect that their code is automatically tested when they commit and even automatically deployed to a staging environment. For mobile developers though, it’s been a slow road to adopting many of these same practices. In large part, this is because mobile brings with it a whole set of unique challenges that make implementation tough. Nevertheless, tools have evolved a lot and mobile dev teams get a lot of value and goodness from having a solid CI system in place. Here are my top 3 reasons for using CI with the mobile products I work on:
No Touch Configuration: Human beings are terrible at creating releases. They inevitably misconfigure something. I want PROD releases to literally be a single button press.
Automated OTA Distribution: During development there’s always somebody that needs a new build of the app and having to manually create a build each time is super time consuming. With CI I can press a button and have it automatically build, code sign and push them an email w/ an install link.
Builds And Tests: If code is committed that doesn’t compile or breaks tests I want to know ASAP.
What Makes CI for Mobile Different?
Infrastructure:
As we all know, iOS is an huge part of the mobile ecosystem. If you’re at all familiar with iOS development then you’re almost certainly well acquainted with Xcode and I’ll bet you a warm cup of coffee you’re doing all your development on a Mac. Know how I know? Of course, you can only build (and compile) iOS applications on Mac hardware. What that means for setting up a CI system is that if you’re going to support iOS you’re going to have to get a dedicated physical Mac machine. No EC2 for you.
Compilation/Code Signing: Configuring an environment to compile a mobile app is quite different than setting up CI for most web environments. There are often a number of dependencies that need to be in place to make the compile work, like Xcode or Android SDK versions. Once a compilation has completed and you’re left with an APK (Android) or IPA (iOS) file there’s a code signing step that’s required before an app can be installed on a device. This is an important and non-trivial step that has no analog in a web environment. Any CI solution that isn’t specifically designed to handle this step will not be able to produce builds that run on devices. Furthermore, since code signing certificates/keystores are sensitive pieces of data the signing process needs to be designed to protected them.
Testing: Testing mobile applications bears some resemblance to web testing. There are both Functional and Unit testing frameworks for iOS and Android. However, running Functional tests on a mobile application requires the use of an Emulator/Simulator or a physical device. In either case, automatically starting/stopping emulators and simulators is difficult and error prone and physical devices become bottlenecks in a hosted environment. There just isn’t a corollary to this on other platforms.
Deployment:Deployment for mobile typically means either distribution to testers or publishing your app to an App Store. For web developers getting your app our there is as easy as committing your code, especially if you’re doing some form of Continuous Delivery. For mobile this is a multi step process that requires building, signing and uploading your app to a service like Hockey App.
Simulators, Emulators and Real DevicesWhen it comes to testing mobile applications there seems to be a lot of confusion around when/how often to test using real devices vs simulators/emulators. The actual decision regarding which to use is subject to a few variables that vary by product/team. First, however it’s often helpful to explain the benefits and differences of each of these options.
Simulators vs Emulators: It’s important to note that these are not the same thing and while the Android SDKs ship with Emulators, Xcode (iOS) uses Simulators. The biggest difference between the two is that Android Emulators actually use the same instruction set (usually ARM) that the target device uses. On the other hand iOS Simulators run using the instruction set on your dev machine (x86). What this means is that the same build that runs on an Android emulator will run on an actual device. Builds compiled to run in an iOS simulator however will not run on an iOS device. The downside to emulators is that they’re very slow while simulators are much faster.
Real Devices: There’s really no substitute for testing your application on the same physical hardware that your users are running. In practice Simulators and Emulators will catch many issues but, and this is especially true w/ Android, the variations and sometimes bugs found on the devices themselves can only be found by testing w/ the actual device. Furthermore, if your application has a reliance on specific physical hardware (i.e. Bluetooth Radio) it may be impossible to test without physical hardware.
So, when/how often should you test w/ each option? (Disclaimer, opinions to follow) In my experience, during the development process the variables to optimize for are speed and cost. In other words, when your developers are committing a lot of code you want your tests to be fast and cheap. On the other hand, when you’re getting ready to prep a release build it’s ok if your tests take longer to finish. What’s important is that you’re getting as much coverage and accuracy as possible. As such, my recommendation to most people is that they use simulators/emulators on their feature branches and physical devices on their master and release branches. Tools For Mobile CI There’s been a lot of progress over the last year or two on mobile CI tools. Testing frameworks have come a long way and now there are a handful of hosted CI platforms designed to make setup as easy as possible. Here are a few of my favorite tools:
Hosted CI Platforms:
Ship.io: The first CI solution to market focused specifically on mobile. Ship has an easy to use setup process, and supports both Simulators and Real Devices for testing.
Greenhouse CI: A newcomer, also focused specifically on CI for mobile applications. Greenhouse has a well designed UI that makes it easy to use for less technical team members.
Travis CI: Popular, robust CI platform with support for OSX based infrastructure. Configuration process is driven by a YAML file so it’s a bit less friendly than the UI offered by Ship and Greenhouse. That said, some folks prefer YAML.
Testing Frameworks:
XCTest: This unit testing framework first debuted in Xcode 5 and replaced the older OCUnit framework. Since it’s baked right into Xcode it’s become the “default” go-to framework for many people. XCTest has a solid feature set and is relatively straightforward to learn and start using.
JUnit: The corresponding “Out Of The Box” solution for Android developers. Similiarly it is tightly integrated with Eclipse and Android Studio but can also be run from the command line using Grandle or Ant.
Appium: Developed by Sauce Labs, Appium is a popular cross-platform framework that allows you to write tests that run on both iOS and Android. Another big plus is that it’s based on Selenium so folks who have worked w/ this in the past will find it a familiar transition.
KIF: “Keep If Functional” (KIF) is an iOS-only Functional Testing framework with a strong focus on usability. If you’re looking to write tests that reproduce user behavior at the UI-level (vs unit tests) I strongly recommend taking a look at this one.
In the end setting up CI for your mobile dev team is all about picking the right tool for the job. There is no one-size fits all solution and the depth and complexity of your system should match the needs and budget of your team. Large, well-funded organizations have very different engineering and business needs than small, early stage startups. The approach that I’ve seen be the most successful is to start with obvious pain points that everyone on the team knows about and automating them on a gradual basis rather than sinking lots of time into a large, potentially unnecessary process. -By Kevin Rohling Kevin Rohling is the Co-Founder of Emberlight and Ex-CEO of Ship.io (formerly CISimple), the first platform to offer Mobile CI as a Service.
