Code.org reduced testing time by 78% and now delivers high-quality code faster for millions of students
Because computer labs and equipment varies widely from school to school, cross-browser and cross-platform coverage is essential to the success of Code.org’s web-based application.
Code.org also needed to test more features without using manual processes, and turned to Sauce Labs to run hundreds of tests in parallel each day against multiple browsers.
The organization cut the time required to test each build by 78%, from nearly 90 minutes to 20 minutes, giving developers more time to work on new features.
By adding Sauce Labs, Code.org can encourage more open-source contributors to submit pull requests that can be quickly tested with confidence.
Although computers, tablets, and smartphones dominate today’s world, computer science is not a standard part of the curriculum in most U.S. schools.
Code.org wants to change that. The nonprofit organization trains teachers who teach 8 million students with full-length courses through its open source, online learning platform Code Studio and 100 million with the Hour of Code, a campaign to encourage students to try one hour of computer science.“We’re trying to get every school to teach computer science, so that every student in every neighborhood has the opportunity to learn about this 21st-century field that is changing our economy and our society,” says Hadi Partovi, CEO at Code.org. The organization has reached more than 8 million students and it has trained 20,000 teachers nationally in over 100 districts. Code.org also encourages diversity - 43 percent of its students are young women and 37 percent are black or Hispanic.
Code.org is dedicated to rigorous software testing. “We push code every single day and our curriculum is being used in schools on a daily basis, so testing is really important,” says Laurel Fan, Developer at Code.org. It is especially critical that the organization catch and fix bugs before the Hour of Code, an annual event during which millions of K-12 students and teachers worldwide are challenged to code for one hour. “We get a large amount of traffic then, from a wide variety of devices,” says Partovi. “We need to make sure everything works perfectly.”
However, ensuring quality was challenging with the organization’s previous testing environment. “We could only run 10 tests in parallel and things weren’t always reliable, so we were constantly having to manually rerun tests,” Fan says. As a result, the organization needed an automated testing solution that would provide more test coverage and better support for cross-browser tests. “We’re in classrooms with computer labs that were sometimes set up 10 or 15 years ago,” says Brian Jordan, Software Engineer at Code.org. “They end up having IE 9, IE 10, Chrome, or Firefox. We need to support every single browser.”
Being able to run our tests in parallel on Sauce Labs across all our browsers means that we can push our code out with way more confidence.
To meet its needs, Code.org started using Sauce Labs, a cloud-based software testing solution. Every day, 10 Code.org software engineers use Sauce Labs to run hundreds of tests in parallel, on multiple browser and platform configurations.
The organization’s engineers run end-to-end tests including functionality tests for 57 different features of the Code.org website. “We test everything — from the individual tutorial when you drag and drop blocks and run our code to all sorts of assertions that are testing things throughout our curriculum,” says Jordan.
Sauce Labs fits perfectly into Code.org’s approach of continuous integration. “The way we make sure our tutorials work in a cross-platform scenario is by using Sauce Labs, and it’s not just about testing our stuff, it’s about continuously integrating that testing into our development process,” says Partovi
Having all these tests running on Sauce Labs helps us move faster and get more done. Without this many tests, I don’t know how we could be collaborating and creating new features so quickly without taking huge detours to try to go figure out where something went wrong.
By having on-demand access to hundreds of Sauce Labs VMs, Code.org is able to test more features on more browsers than it could before. “Being able to parallelize with Sauce Labs means that we can cover more browsers and more features,” says Jordan. “When a new browser comes out, like Internet Explorer Edge, we can just add that on.” With Sauce Labs, Code.org tests also run much faster. “It used to take almost 90 minutes to run all our tests, and now we’re running all of them within 20 minutes,” says Fan.
Code.org also saves developer time through automated testing. “We don’t have an engineer sitting around every day trying to debug a day’s worth of commits,” says Jordan. And with more time, the organization’s developers can develop new features more quickly. “Having all these tests running on Sauce Labs helps us move faster and get more done,” says Jordan. “Without this many tests, I don’t know how we could be collaborating and creating new features so quickly without taking huge detours to try to go figure out where something went wrong.”
The organization can also provide better code. “We’re writing more tests, and that’s been great for code quality,” says Jordan. In addition, Code.org can better support contributors to the site. “All of our code is open source. Using Sauce Labs, now we can have more contributors on, without having to worry about whether something’s going to break,” Jordan says.
With higher-quality code, the Hour of Code is a better experience for students and teachers. “Being able to run our tests in parallel on Sauce Labs across all our browsers means that we can push our code out with way more confidence,” says Jordan.