As part of an initiative to improve customer e-commerce experiences and developer productivity, the Walmart team adopted continuous testing best practices using Selenium and a proprietary open source quality automation platform
Walmart now executes more than 50,000 automated tests per day and accelerated their deploy cycles from twice per month to multiple times daily
The Challenge
Walmart has long been a disruptor in the retail industry by revolutionizing the supply chain to offer lower prices. They also continue to make big investments in their digital strategy to help customers save time.
Through this transformation, Walmart Labs (Walmart’s technology arm) was finding it difficult to seamlessly transition away from legacy practices. There was a misalignment on goals across the organization and teams were using their own tools and technologies. They found themselves relying on time-consuming manual testing, which only allowed updates to deploy twice per month on a three-week development cycle, leaving them unable to keep up with the pace of app improvement they wanted. Furthermore, because their tests were completely manual, teams weren’t able to scale testing to ensure coverage across the multitude of browsers, operating systems, and mobile devices that their customers would be accessing their apps from.
They knew they needed to increase application quality and increase deployment velocity with zero friction for the dev teams, which led to the adoption of automated testing.
The Solution
In their move to automated testing, Walmart realized there were no test management tools that worked at the scale that they required. So in 2015, they started developing a proprietary open source quality automation platform developed to support native, web and backend testing services. This now is a central tool that supports functional testing, performance testing, mocking and data insights across all Walmart.com, Grocery and backed store systems. Walmart also understood that in order to be successful they needed to change the entire culture of how their teams developed and tested their apps. This led to the creation of their Center of Excellence - the DXT team. Their focus was to choose the best testing frameworks and processes to complement their automation platform. Once selected, they standardized and rolled these out to various teams by having everyone working similarly.
Walmart realized that open source software gave them the latest tools, flexibility and support they needed. So they chose the open source frameworks Selenium, Appium, Espresso, and XCUITest for automated testing. These allowed them to develop automated tests with widely adopted frameworks used by hundreds of thousands of organizations worldwide. This was important as they could build and scale an automated testing organization knowing that they could hire engineers that had experience in these technologies who could be productive immediately.
Walmart also leveled up their testing by implementing the Sauce Labs platform, the world’s largest cloud-based solution for web and mobile app testing. Sauce Labs gave them access to hundreds of browser and operating system combinations, increasing test coverage and saving time. Walmart's automation platform imports data from their own runners and Sauce Labs test details pages to display a quality dashboard so QA and engineers can quickly view the quality status of their builds.
As Walmart standardized their testing frameworks and began managing them with their own automation platform, Sauce Labs provided the tools that allowed them to scale their testing, enhance productivity, improve code quality and speed delivery. And with these systems in place, each team at Walmart knows that they are using the best tools, and they don’t have to constantly reinvent the wheel when they add a new app to their suite.
The Results
Today, Walmart has over 40 different projects around the world using their proprietary open source automation platform to manage all of their tests. Used in conjunction with the Sauce Labs platform these teams have run over 14 million tests in the past 7 years - about 50,000 tests per day currently. These tools have given teams the time to build out more sophisticated automation, with an increase from 200 test cases in 2016 to 500 test cases with every build to production in the case of one team.
Thanks to Sauce Labs, Walmart is also able to ensure that their apps work on over 700 browser/OS combinations, 172 device emulators and over 300 unique real devices, and all without having to manage any infrastructure internally.
However, this increase in quality and coverage has not come at the expense of speed. With continuous testing practices in place, teams at Walmart are deploying updates to their apps daily on every front end build. This velocity is only possible with the confidence knowing their code is well-tested and covered across every browser/OS/device combination.
This change in culture and process left not only the the technical teams happy and more productive, but it has also got the executive team excited. Within a year’s time, the efforts of the DXT team, along with the implementation of Sauce Labs, saved 750,000+ man hours that would have been spent on manual testing/updating support for new browsers and operating systems.
