Financial Services Organization Accelerates Release Cycles by 4x Without Sacrificing Quality

One of the world’s oldest and largest banks, with a history dating back over 200 years and 250,000+ employees, serves millions of consumers, small businesses and many of the most prominent corporate, institutional and government clients.

The digital team based in New York City is responsible for the organization’s flagship website, along with the mobile banking application. Their products serve over 77 million clients, and deliver one of their most interactive digital experiences. Financial services is a fiercely competitive market, and the team understood that it was critical to continually deliver flawless digital experiences at the speed users demand. To support this effort, the digital team delivers ongoing releases to address small bug fixes, which are complemented with quarterly major releases with new features and experiences. For both types of releases, speed is key. But as they modernized their delivery pipeline for CI/CD, testing bottlenecks were starting to emerge. In 2017, the team ran 2 million transactions through their regression cycle before a major release, which took 6 days to run due to mostly manual processes. In addition, their small subset of automated tests were reporting back a 50-60% failure rate.

The team knew that in order to succeed with their growing CI/CD practice, they needed a complete overhaul in the way they approached testing, and that access to the proper resources was critical to their success.

With a mandate from the organization’s Chief Development Officer, the digital team began a massive undertaking in automating their tests using Selenium and Appium. The main goal of this exercise was not only to reduce the amount of time it took to test, but also open up opportunities to address quality earlier in the pipeline by implementing pull request testing processes for development teams. This required tight collaboration between their Shared Services organization and individual development teams, as developers assumed responsibility for creating feature tests, while Shared Services looked after test architecture and ensured processes were flowing smoothly.

With the volume of automation growing at a rapid pace, leadership understood that testing resources were going to be critical to ensure the success of this initiative. They had experimented with building their own internal infrastructure, but were frustrated with its high flake rate, its time-intensive maintenance process, and the inability to scale on demand. At one point the team reported that to increase the size of their internal grid by 50% it would take 8 weeks of work for their team of 4. This simply wouldn’t work for the velocity they were hoping to achieve.

After doing extensive research, the digital team decided to partner with Sauce Labs to deliver a Continuous Testing Cloud solution to their teams. They knew that Sauce would be able to provide a comprehensive platform that provided every browser and operating system, along with mobile emulators and simulators, that could scale instantly to their needs. They also knew that Sauce Labs was the only vendor in the market that could pass their rigorous security process. And finally they understood that Sauce Labs provided not only a testing platform, but a team of industry-leading experts who provide best-in-class support and services that ensure their teams are enabled for success.

Today the digital team is running roughly 2.6 million tests per week on the Sauce Labs Continuous Testing Cloud. They are reporting accelerated release cycles, all while continuously increasing the number of tests they are executing throughout the pipeline. Their major quarterly release cycle which once ran 2 million transactions in 6 days now runs 8 million transactions in 36 hours - a 4x time savings. And this number will only continue to grow as they add more tests to their suite. With this incredible increase in testing volume, the digital team is releasing better experiences to users with increased confidence knowing that they are well-tested.

Sauce Labs has also helped break down barriers between once siloed teams. Developers are feeling more productive and continue to deliver new features at speed without any bottlenecks or slow testing cycles. Shared Services can continue to ensure that the pipeline is running smoothly without having to worry about maintaining any infrastructure, and instead can focus on establishing best practices. Overall, this has allowed the team to truly create a culture where everyone is committed to quality.

The digital team also values the services that Sauce Labs provides in addition to its platform. With access to teams that can advise on best practices, help set goals for testing success, and provide ongoing training and support, Sauce Labs has been a strong partner in helping the digital team realize true digital transformation through testing. By proving the value of automated testing at scale, this organization continues to lead the financial services market by continually delivering flawless experiences to their customers.