Khan Academy
Industry:
Education
Company Size:
51 - 200

Khan Academy Ensures a World-Class Online Education

Khan Academy wanted to implement cross-browser, multi-device testing that helps fulfill their mission of providing a free, world-class online education for anyone, anywhere.

The company uses Sauce Labs to execute their entire end-to-end test suite up to 24 times per day.

Sauce Labs provides the reliable test results Khan Academy coders need to quickly write and deploy educational content that helps transform users’ lives.

Khan Academy
Industry:
Education
Company Size:
51 - 200

The Challenge

Khan Academy is a nonprofit dedicated to providing a free, world-class education for anyone, anywhere. What began in 2006 with founder Sal Khan recording educational videos for his cousin has grown into a web site and mobile app averaging 12 million monthly active learners.

Cross-browser functionality and mobile app support is very important to Khan Academy, since mobile devices frequently represent the only access users in developing nations have to educational content. “Mobile is important”, explains Software Engineer and infrastructure team member Dhruv Kapadia, “especially for us as an educational institution that wants to reach folks in the developing world.”

The team previously used a headless JavaScript framework to run end-to-end tests, but frequent test failures frustrated developers and destroyed the credibility of the testing process. “Once developers know these tests just fail a lot, they don’t really see a failure and think it’s a problem”, says Software Engineer and infrastructure team member John Sullivan. “They think ‘oh well, it’s failing again’.”


The Solution

Our team is small and we don’t want the overhead of managing infrastructure.

Dhruv KapadiaSoftware Engineer

After switching to Selenium to develop their tests, the team began looking for a testing infrastructure that developers would find easy to use and didn’t require self-hosted hardware. “Our team is small and we don’t want the overhead of managing infrastructure,” says Kapadia.

Sullivan says Sauce Labs stood out as the best option for Khan Academy. “I just searched online for good cloud-based Selenium solutions, and Sauce Labs came up pretty much immediately. What helped us make up our mind was the simplicity of Sauce’s interface and controls. And Sauce Labs seemed to be the clear answer if we didn’t want to host it ourselves. Nothing else came close,” Sullivan added.

Khan Academy makes at least two runs of their entire end-to-end test suite for every deploy they do. “We make about a dozen deploys every day, which totals about 24 test-suite runs,” says Sullivan. “We want to make sure every deploy is covered by our full test suite.”

The team has a 10-minute time limit for running tests, so speed and reliability are key. “It’s comforting to know that we can just throw more Sauce Labs VMs at the problem and if we need to,” says Sullivan. And Sauce Labs gives Khan Academy a big advantage in quickly resolving issues. “With Sauce Labs, it’s much easier and faster to investigate a failed test.”

Khan Academy has integrated Sauce Labs into its Jenkins CI Server, where developers get direct links to test results. Sullivan says Sauce Labs’ video playback feature is a favorite among developers. “Sauce gives us videos of all of the end-to-end tests we run. It’s really, really great being able to see what your tests have done. That’s one of the features that just gets all the developers here very excited.”


The Results

Sauce gives us videos of all of the end-to-end tests we run. It’s really, really great being able to see what your tests have done. That’s one of the features that just gets all the developers here very excited.

John SullivanSenior Software Engineer

Being able to test reliably across a wide variety of device and browser combinations without worrying about the underlying infrastructure has allowed Khan Academy to move much more quickly than in the past. Sullivan cites his team’s previous experience testing an SAT test-prep program on a wide variety of devices. “We would find old computers and try them out, and it was a very manual, slow process. We could have just toggled some flags on Sauce Labs and made this less time consuming for all of us.”


Ultimately, Sauce Labs lets Khan Academy develop more educational content without sacrificing quality. “The mission is what drives a lot of us here to provide great education for folks,” says Kapadia. “ Sauce really helps us move faster so that we can get new features out to our users.”


“I think that Sauce Labs helps Khan Academy’s future,” Sullivan says, with an enthusiastic smile. “And hopefully the future of the world, which is kind of what we’re going for.