TINT uses Sauce to automatically test key functionality and secondary features, saving $20,000 monthly.
The Challenge
TINT (www.tintup.com) is a software company that provides a digital platform that enables companies to display any social media feeds anywhere. Customers use the TINT platform to collect user-generated content, brand it, and display it on anything from websites to massive screens at music festivals.
To make sure its platform works flawlessly for customers, TINT puts a premium on software testing. “Our software powers some of the largest screens in the world,” says Ryo Chiba, Co-founder and Co-CTO, TINT. “Obviously, we need to catch bugs before they show up on those screens. If we don’t, that doesn’t look good for our customers or us.” TINT, however, had significant challenges with its testing environment. The company originally relied on manual testing, which failed to catch all bugs and took too much developer time. More recently, TINT started using the Selenium automated testing framework, utilizing a Selenium Firefox plug-in to generate tests. “That was an improvement, but it still wasn’t good enough because we had very little control over the tests themselves,” Chiba says. “As a result, we still had to supervise test runs, which was time-consuming. And bugs were still getting through. We were getting a lot of cancellation requests from customers.”
For TINT, the solution to its challenges was clear. “We needed better automated testing and much more test coverage,” says Chiba.
The Solution
After it explored various automated testing solutions, TINT decided on Sauce Labs, a cloud-based solution that supports Selenium. “The automated testing space is in the early development stages, and right now Sauce Labs is the industry standard,” says Chiba. “We were very confident in choosing Sauce.”
Every day 11 TINT developers interact with Sauce Labs. “Sauce Labs is a key part of our build system. Whenever we write new features, we write Selenium tests for them and run those tests on Sauce Labs,” says Chiba. TINT runs close to 2,000 functional tests a day using 10 Sauce Labs virtual machines. “We run about 20 builds a day, and each build has 89 tests,” says Chiba. Those include tests for key features such as the company’s editor, which marketers use to build social hubs and change their look and feel. “Selenium goes through and connects data feeds and modifies the look and feel of a social media hub sample, ensuring it renders properly in different scenarios,” Chiba says. TINT also uses Selenium and Sauce Labs to test secondary features, such as analytics and customer payment flow scenarios.
Sauce Labs is integrated with TINT’s TeamCity continuous integration (CI) system, so builds kick off automatically. “We can deploy stable builds every day, and that wouldn’t be possible without Sauce Labs,” says Chiba.
The Results
By using Sauce Labs, TINT has increased its test coverage. “Prior to using Sauce Labs, we were running only six very basic smoke tests,” says Chiba. “With Sauce Labs, we have 89 tests per build, and those tests include coverage for our editor feature and many of our ancillary features.” The result? A better platform for customers. “Using Sauce with Selenium enables us to identify more regressions, and it makes all parts of our product more stable,” says Chiba.
TINT is also able to catch bugs earlier. “Every week, we catch several bugs in a build. Those are bugs that could immediately affect our bottom line,” Chiba says. “And we can now catch regressions as they’re introduced, as opposed to days later when we’ve lost context and it takes twice as much time to fix the bug.” Also, because testing no longer requires supervision, TINT developers can focus more of their time on new features.
In addition, TINT saves money by using Sauce Labs. “If we didn’t have automated testing, we’d be spending up to half a week of additional time trying to catch regressions,” says Chiba. “Between that and potential lost customers, we’re probably saving $20,000 a month.”
Overall, with Sauce Labs, TINT can deliver a higher quality platform to its customers. “We have definitely improved the quality of our product by using Sauce Labs,” says Chiba.
