One of the world’s oldest and largest banks, with a history dating back over 200 years
and 250,000+ employees, serves millions of consumers, small businesses and many
of the most prominent corporate, institutional and government clients.
The digital team based in New York City is responsible for the organization’s flagship website, along with the mobile banking application. Their products serve over 77
million clients, and deliver one of their most interactive digital experiences. Financial
services is a fiercely competitive market, and the team understood that it was critical
to continually deliver flawless digital experiences at the speed users demand. To
support this effort, the digital team delivers ongoing releases to address small bug
fixes, which are complemented with quarterly major releases with new features and
experiences. For both types of releases, speed is key. But as they modernized their
delivery pipeline for CI/CD, testing bottlenecks were starting to emerge. In 2017, the
team ran 2 million transactions through their regression cycle before a major release,
which took 6 days to run due to mostly manual processes. In addition, their small
subset of automated tests were reporting back a 50-60% failure rate.
The team knew that in order to succeed with their growing CI/CD practice, they
needed a complete overhaul in the way they approached testing, and that access to
the proper resources was critical to their success.
With a mandate from the organization’s Chief Development Officer, the digital team
began a massive undertaking in automating their tests using Selenium and Appium. The
main goal of this exercise was not only to reduce the amount of time it took to test, but
also open up opportunities to address quality earlier in the pipeline by implementing
pull request testing processes for development teams. This required tight collaboration
between their Shared Services organization and individual development teams, as
developers assumed responsibility for creating feature tests, while Shared Services looked
after test architecture and ensured processes were flowing smoothly.
With the volume of automation growing at a rapid pace, leadership understood that
testing resources were going to be critical to ensure the success of this initiative. They
had experimented with building their own internal infrastructure, but were frustrated
with its high flake rate, its time-intensive maintenance process, and the inability to
scale on demand. At one point the team reported that to increase the size of their
internal grid by 50% it would take 8 weeks of work for their team of 4. This simply
wouldn’t work for the velocity they were hoping to achieve.
After doing extensive research, the digital team decided to partner with Sauce Labs
to deliver a Continuous Testing Cloud solution to their teams. They knew that Sauce
would be able to provide a comprehensive platform that provided every browser
and operating system, along with mobile emulators and simulators, that could scale
instantly to their needs. They also knew that Sauce Labs was the only vendor in the
market that could pass their rigorous security process. And finally they understood
that Sauce Labs provided not only a testing platform, but a team of industry-leading
experts who provide best-in-class support and services that ensure their teams are
enabled for success.
Today the digital team is running roughly 2.6 million tests per week on the Sauce Labs
Continuous Testing Cloud. They are reporting accelerated release cycles, all while
continuously increasing the number of tests they are executing throughout the pipeline.
Their major quarterly release cycle which once ran 2 million transactions in 6 days now
runs 8 million transactions in 36 hours - a 4x time savings. And this number will only
continue to grow as they add more tests to their suite. With this incredible increase in
testing volume, the digital team is releasing better experiences to users with increased
confidence knowing that they are well-tested.
Sauce Labs has also helped break down barriers between once siloed teams.
Developers are feeling more productive and continue to deliver new features at speed
without any bottlenecks or slow testing cycles. Shared Services can continue to ensure
that the pipeline is running smoothly without having to worry about maintaining any
infrastructure, and instead can focus on establishing best practices. Overall, this has
allowed the team to truly create a culture where everyone is committed to quality.
The digital team also values the services that Sauce Labs provides in addition to its
platform. With access to teams that can advise on best practices, help set goals for
testing success, and provide ongoing training and support, Sauce Labs has been a
strong partner in helping the digital team realize true digital transformation through
testing. By proving the value of automated testing at scale, this organization continues
to lead the financial services market by continually delivering flawless experiences to