Black Friday: Retail Holiday or Developers’ Nightmare?

In the United States, Thanksgiving is right around the corner. For many of us, this serves as a chance to see family, play some football, and stuff ourselves with an unholy amount of food. Thanksgiving also indicates an unofficial kickoff to the holiday season and by the time the last plate of leftovers is conquered we all have that one enthusiastic relative that is dusting off a Mariah Carey album. (Sorry about the trademark loss Mariah, you’ll always be the "Queen of Christmas” to us.)

For many brands and their systems engineers, Thanksgiving is the eve of their most stressful day of the year. While others are lining up outside a mall at midnight to “politely move” their neighbors aside for the shot at a heavily discounted TV, most back-end folks are hoping that their servers hold up during shopping’s Armageddon. Black Friday and Cyber Monday may be holidays to some, but to your friendly neighborhood developer or QA engineer, it’s a game of watching how much weight the ice can take before it finally cracks. 

User experience is paramount this time of year, and one of the measures we use to track success is uptime. But as we saw with multiple high-profile outages in the past week, it doesn’t matter if your platform has 99.99% uptime if that .01% downtime comes at the worst moment imaginable. A mishap when the entire world is watching can have cascading effects. We all know outages are bad for brand equity, but when they create a complete feeling of distrust, an outage could indirectly cause the demise of an entire company. This is clearly an area that we need to take seriously.

These late November shopping holidays have become sort of the norm. Much of the world has “Singles’ Day,” which runs on 11/11 and drives record revenue for platforms like Alibaba. North America claims the weekend after Thanksgiving. Some prudent brands have decided to offer their sales earlier in part to reduce the stress of driving millions of consumers to their platform in a single day. But what if they don’t? Are there ways that testing can save these platforms from an utterly disastrous outage?

Yes, but it goes beyond standard performance testing. Chaos testing or testing for failover and security testing is also of the utmost performance when dealing with a stressful event such as this. If one area falters, the whole house of cards is likely to come crashing down.

Adobe forecasts that $210 billion is at stake during this holiday season with the period of Thanksgiving and Cyber Monday accounting for $35 billion of that. Cyber Monday alone is predicted to bring in $11 billion with Black Friday coming in close behind at $2 billion. With data like that we could extrapolate further to see the potential fiscal impact of being down an hour. An ambitious data nut could probably break it down to the minute. Suffice it to say, on the two days of the year when $20 billion are at stake, you probably don’t want your servers to crash.

If we look at a Total Addressable Market over these two days as $20 billion, we can divide by 48 to see that this number shrinks to $417 million. For every hour that you are down, consumers are spending $417 million and you are receiving no slice of the pie. For every minute you are down, consumers are spending $7 million and once again you are seeing none. Mid-tier retailers only get a small percentage of that revenue anyway, but a giant like Amazon is looking at an actual revenue loss of millions of dollars for every minute they are down. 

Speaking more broadly, IDC estimated a few years ago that an outage among Fortune 1000 organizations is $100,000 per hour. Historically, the average cost of outages among these organizations falls between $1.25 billion and $2.5 billion annually, a number that has certainly increased with the rapid growth of eCommerce. Imagine being the Director of QA and telling your CEO that a 10-minute outage cost your company 7 figures–truly a black Friday indeed.

The worst part is these estimates are only tracking actual revenue (or lack thereof). The long-term damages can be far-reaching and even more devastating, a customer experience failure damages trust, brand equity, and even a sense of security. Our own research shows that many users will abandon a brand after a negative experience and then they’ll go straight to Reddit or Twitter to tell all their friends how much your platform sucks.

But enough of the fear-mongering! The holidays are supposed to be a time of joy and relaxation, right? (Well maybe not in this author’s family, but I digress.) Surely, I have not authored this article merely to say the end is nigh and there is nothing to be done about the impending digital apocalypse.

Let’s walk through three ways systems engineers can avoid a Turkey Day meltdown. 

With Akamai finding that over 83% of web traffic will involve API calls, developers and SREs are challenged to exponentially scale their functional and performance testing for unpredictable traffic spikes and patterns. At the same time, the tests must be reliable in all environments and easy to improve throughout all handoffs. By investing in improving performance testing and API load testing scalability and team collaboration, companies are more likely to provide mobile apps and APIs that are reliable throughout unpredictable traffic spikes and patterns.

In A False Sense of Security, a free eBook from Sauce Labs, we discuss why five well-known enterprises were failed by their performance and load tests, which were trusted to protect millions of dollars in revenues and reputation. You’ll learn how the companies fully embraced a cloud-driven approach to testing and monitoring to almost immediately increase coverage, accelerate debugging and enable smarter, faster development. 

Too often, developers and SREs rely on traditional performance tests and synthetic monitors that were never made for the mobile-first and API-first world. A False Sense of Security reveals the advantages of using a cloud platform to merge functional and load tests into holistic tests with negative/positive assertions plus coverage of API side-effects. This approach is a right step toward delivering mobile apps and APIs this holiday season that will go live without a “false sense of security.” 

One thing about hackers is, they love to kick you when you’re down. The moment your site goes “offline” an entire can of worms is opened on the security front. Bad actors can and will look for new ways into your system and exploit new vulnerabilities. That’s why security testing is intrinsically linked to performance testing when we speak to outages. 

Do you know the old chaos theory? When a butterfly flaps its wings halfway across the world all your servers crash and you miss your Q4 earnings? Something like that. Anyway, the same can apply to your platform: when certain components crash, it can have wide-ranging effects across your entire system. Designing your system to fail in predictable ways can minimize the damage of a potential outage so security and performance teams should be closely aligned to ensure that failover systems are resilient and operate as intended. 

Systems will crash all across the globe this weekend, and while not all will be preventable, better planning and synchronicity among various QA teams may have helped. Giving you these tips this late in the game almost feels cruel and harkens to the Alanis Morisette song “Ironic:” “It’s the good advice, that you just didn’t take.” Perhaps I’ll write next year’s Black Friday blog in March, so you have some time to execute these plans. As for this year? I’ll borrow a turn of phrase from the football games that you will be watching during your fifth plate of stuffing. If you didn’t plan this year, you can always hope for a Hail Mary.

Learn how Sauce Labs can help with API performance and load testing, or sign up for a free Sauce Labs trial to see for yourself.