Code.org Achieves 100% Test Automation With Sauce Labs, Delivering Flawless Apps for Millions

The scope of Code.org’s impact is never more evident than throughout its annual Hour of Code campaign, during which hundreds of thousands of teachers across the globe set aside an hour for their students to engage with more than 500 computer tutorials from dozens of activity providers, including 20 created by Code.org. Roughly 15 percent of all students in the world have participated in the Hour of Code since its inception -- a level of scale owed in no small part to a modern automated testing strategy anchored by the world’s most comprehensive continuous testing cloud.

But while Sauce Labs now serves as a crucial behind-the-scenes tool Code.org leverages to power initiatives like the Hour of Code campaign, that wasn’t always the case. In fact, there was a time before Sauce Labs when testing was a bottleneck for Code.org.

Automated testing across numerous browser/OS combinations has always been an absolute must for Code.org, especially considering the vast range of equipment - both old and new - found in classrooms across the world. But without the breadth of test coverage it truly needed, confidence was low and manual testing was common. When tests failed, as they often did, developers would have to burn precious cycles trying to pinpoint and remedy the root cause of those failures. It was a slow process that often left Code.org’s development team stretched thin and having to take extra precautions to ensure the quality of its programming.

“The experience of our students and teachers means everything to us,” says Anthony Suarez, CTO at Code.org. “Our programs simply have to work flawlessly, and the only way to ensure that they do is through testing. In years past, that meant locking down development up to a month before an event like Hour of Code to avoid the risks associated with introducing new code late in the process.”

Furthermore, as a nonprofit organization with limited resources, Code.org has no full-time testers or quality assurance team members supporting its 20-person development team. This means that any time allocated to testing is time not spent focused on the core mission of expanding access to computer science in schools and increasing participation by women and underrepresented minorities.

To better serve students and teachers worldwide and continue expanding to new school districts, Code.org needed a cloud-based testing solution that spanned the thousands of potential browser/OS combinations it encounters in classrooms and worked seamlessly with any continuous integration system, including Travis, CircleCI, and Drone.

“It’s essential for us to be able to ship changes every day, and often more frequently than that,” says Brad Buchanan, software engineering manager at Code.org. “We knew we needed to rethink the way we approached testing to make that happen.”

Code.org turned to Sauce Labs, provider of the world’s most comprehensive and trusted continuous testing cloud, to modernize its testing strategy and deliver the automation and scale it needed.

From the beginning, the fit was nearly perfect.

“The ability we now have to iterate quickly and do so knowing that when we push a new or updated program to production that it’s been comprehensively tested, that’s played a huge role in what we’ve been able to accomplish,” says Buchanan.

Sauce Labs enabled Code.org to scale its testing strategy in a manner that matches the rapid expansion of its computer science learning curriculum. Today, according to Buchanan, 100 percent of Code.org’s tests are automated, and the days of running sequential tests are a thing of the past, replaced by widespread use of parallel testing, with the team often executing up to 200 tests concurrently. Test suites that used to take up to 90 minutes to run are now executed in less than 20, and the team runs tens of thousands of test cases with Sauce Labs each day.

With the speed and scalability that Sauce Labs affords fueling its development efforts, Code.org recently surpassed 900 million hours of student learning since its founding, and it continues to innovate and add to its growing curriculum. For example, following extensive testing on the Sauce Labs platform, Code.org launched a new tutorial for Hour of Code 2019 called “AI for Oceans.” The tutorial introduces students to artificial intelligence by having them train a machine learning model that executes directly in their browser.

“We’re able to deploy new educational features to production each day, sometimes multiple times each day, and do so knowing that each release has passed an extremely high threshold for quality,” said Buchanan. “We’re incredibly confident in the quality of the software we’re releasing.”

So confident, in fact, that despite its continued growth and global presence, Code.org still only needs a single Support Program resource to support millions of students and teachers, and still confidently releases software without the benefit of any full-time testers or QA resources.

Moreover, in the past year, as the reach and depth of their programming have continued to expand, the Code.org team has begun leveraging the new analytics capabilities in the Sauce Labs Continuous Testing Cloud to drive an even more optimized automation experience. With visibility into how tests are performing through analytics, the team now averages a 98 percent pass rate on executed tests. This increased efficiency has enabled Code.org to encourage more open-source contributors to submit pull requests that can be tested quickly and with confidence.

“The end result of all of this is that we can be more responsive to teacher feedback and provide a much richer and more rewarding experience for our students,” said Suarez. “That’s ultimately why we’re here, and why Sauce Labs continues to support us with their platform. They believe in our mission and are committed to helping us achieve it.”