Every engineer knows the process of a build going red and rerunning it before it comes back green. No code or configuration changed. The test just decided to pass this time.
A flaky test is an unreliable software test that produces both passing and failing results under the same conditions. Reducing flaky tests means finding their root causes rather than retrying past them.
This guide covers what causes flaky tests, how to detect them, how to fix them, and how to prevent them from recurring.
What are flaky tests?
A flaky test produces inconsistent results across identical test runs. Same code, same configuration but a different outcome. Run it 10 times and it passes eight, with the two failures pointing to nothing in the application. The test itself, or the environment it runs in, is the variable.
A concrete example: an async UI check waits for an element to render after an API call. Locally, the response arrives in 50ms, and the test passes. In CI, a slower runner adds 200ms of latency, so the element isn’t ready when the assertion fires. Consequently, the test fails. Nothing is wrong with the application. The test just assumed timing that does not hold in all cases.
Flaky tests are not the same as failed tests, though. A genuine failure indicates a bug in the code under test. Flakiness, on the other hand, suggests an unreliable test or an unreliable environment. The distinction matters because the fix is different — one is a code change, the other is a test engineering problem.
Not all flaky tests are bad tests. Some expose real timing bugs, race conditions, memory leaks, incorrect assumptions, or concurrency issues in the production code. Deleting a flaky test outright is risky if the flakiness is a symptom of an actual defect the test was built to catch.
Why do flaky tests hurt CI/CD pipelines?
The damage compounds slowly, then shows up everywhere at once.
Trust erodes first. Once reruns become routine, engineers stop believing red builds as failure notifications become background noise. When a real regression arrives, it sits in the same queue as the flaky tests, and no one investigates quickly. The result? Defects reaching production through a pipeline that technically ran every test.
Engineering time drains next. Triaging false alarms and rerunning pipelines eat up hours every week on their own, and babysitting the test suite on top of that adds more hours that show up nowhere on a sprint board. Google has reported that roughly 1 in 7 of its tests exhibit some form of flakiness. At that scale, even a few minutes of investigation per flaky failure adds up to substantial lost capacity.
Merges slow down. When the full suite must rerun to get a green, every merge takes longer. Developers queue behind each other, and release trains slip. In pipelines with long end-to-end suites, a single flaky test can add 20–30 minutes of rerun time per pull request. Multiply that across a team making 10 merges a day, and flaky tests silently consume hours of pipeline capacity.
Worst of all, retry logic masks real problems. A test that intermittently fails due to a genuine race condition in the application code will pass on retry, and the race condition ships to production. The pipeline reported green, so the defect was never investigated. Unfortunately, the most expensive cost of flaky tests is not the time wasted on false alarms but the real bugs that hide behind them.
What causes flaky tests?
Causes cluster into a handful of patterns. Most flaky tests trace back to one of these five categories, and knowing which category a flaky test belongs to can cut the debugging time in half.
Race conditions and timing problems
Asynchronous operations checked before they complete are the single most common source of flaky tests. A test asserts that an element is visible before the rendering finishes, or checks a value before the API call returns. Fixed sleeps (Thread.sleep(2000)) instead of explicit waits on a condition create a lottery: The test passes when the system is fast enough and fails when it is not.
Different threads or services completing in different orders across test runs produce the same effect. A test that depends on service A responding before service B works fine until the load on service A changes.
Test order dependency and shared state
Tests that rely on state left behind by previous tests pass or fail depending on execution order. Run the suite sequentially and test 47 passes because test 46 set up the data it needs. Randomize the order and test 47 fails because its precondition was never created.
Shared fixtures, databases, or caches without proper cleanup are the usual culprit. Teardown logic that doesn’t fully restore a clean slate leaves residual state for the next test to inherit.
External dependencies
Third-party API calls, network variability, and rate limits inject nondeterminism into any test that reaches outside its own process. A test that calls a live payment gateway will flake whenever that gateway is slow, throttled, or down for maintenance.
Anything the test does not control (DNS resolution, CDN latency, external test data services, etc.) becomes a hidden variable. The test is now measuring the network as much as the application.
Environment and infrastructure issues
Differences between local machines and CI runners are a factory for flaky tests. A developer's laptop has 16GB of RAM and an SSD. The CI runner has 4GB and a shared disk. Tests that pass comfortably on one might choke on the other.
Resource contention matters too. Parallel jobs sharing the same ports, file paths, or database instances step on each other. Memory leaks from earlier test runs slow subsequent tests past their timeout thresholds.
Unstable test data and randomness
Unseeded random data generators produce different inputs on every run, and some of those inputs trigger edge cases the test wasn’t designed for. Time-of-day assumptions and timezone dependencies cause the same problem, and so do hardcoded dates that expire ("valid until 2025-12-31") — all of it creates flaky tests that work fine … until they don’t.
How do you detect flaky tests?
Flaky tests reveal themselves in run history. Detection is a statistics problem, not a debugging problem.
Rerun failed tests automatically and record when outcomes flip on identical code. A test that fails on the first run and passes on the second, with no code change in between, is flaky. Treat a pass-on-retry as a flake signal, not a fix.
Track per-test history in CI to compute a flake rate, the percentage of runs where the test produced inconsistent results, per test, per suite, and per environment. A test with a 15% flake rate over the last 30 days is not intermittently broken. It’s reliably flaky, and it needs attention.
Dashboard the flake rate as an operational metric, the same way you track build time and deploy frequency. Alert on spikes after infrastructure changes, dependency upgrades, or a wave of new tests landing at once. A sudden increase in flaky tests after a CI runner change points straight at the environment.
Use randomized test ordering in scheduled runs to smoke out order-dependent flaky tests. If a test passes in the fixed order and fails in a randomized run, it has a hidden dependency on shared state.
Machine learning approaches can flag likely flaky tests from failure patterns before humans notice them. Sauce Labs test analytics, for example, automatically surface failure patterns and flake trends across runs, showing which tests are unreliable without requiring the team to build their own tracking infrastructure.
How do you fix flaky tests?
Fixing flaky tests involves conducting root-cause analysis and applying targeted code fixes. A rerun only collects evidence toward the diagnosis.
Reproduce the failure. Run the flaky test in a loop under CI-like conditions (same resources, same parallelism, same environment) until the flakiness shows up. If it only fails in CI and never locally, the environment difference is a clue.
Collect evidence. Logs, screenshots, video, and network traces from failing runs, compared side by side against passing runs, usually reveal the divergence point. Without artifacts, debugging a flaky test is guesswork.
Isolate. Mock external dependencies and pin test data to rule out environment noise. If the test stops flaking with mocked externals, the external dependency is the root cause.
Fix the actual cause. Replace fixed sleeps with condition-based waits. Add proper cleanup between tests, and seed random data generators. Pin timezone and locale settings in the test configuration. The fix should address why the test flakes, not suppress the symptom.
Validate. Run the fixed test dozens or hundreds of times before returning it to the gating suite. A test that passes once after a fix isn’t proven stable, but a test that passes 200 times in a row is.
Decide: Fix the test or fix the code. If the flakiness reflects a real concurrency bug or race condition in the application, the production code is what needs the fix. Stabilizing the test without addressing the underlying defect just hides it better.
Symptom | Likely root cause | Immediate action | Long-term fix |
Passes locally, fails in CI | Environment difference (resources, timing, network) | Compare CI and local environment specs | Match CI environment resources or add explicit waits |
Fails only in parallel runs | Shared state or port contention between tests | Run the test in isolation to confirm | Isolate test data, use unique ports per session |
Fails at specific times of day | Time zone, date, or time-of-day assumption | Check for hardcoded dates or TZ dependencies | Pin timezone in config, use relative dates |
Passes on retry | Timing race or transient external dependency | Log both failing and passing runs for comparison | Replace sleeps with condition waits, mock externals |
Fails only in full-suite runs | Order dependency or cumulative state leak | Run the test alone and in randomized order | Add proper setup/teardown, remove shared state |
How do you prevent flaky tests?
Preventing flaky tests is cheaper than fixing them. A few practices that keep flakiness out of the suite from the start.
Write deterministic tests. Use explicit waits tied to conditions rather than fixed sleeps or hardcoded timeouts. Seed random data generators. Mock clocks and time zones. Control test data at the test level, not at the suite level.
Avoid network assumptions. Never assume an action finishes in a set time (e.g., number of milliseconds). Wait for the actual element or API response instead.
Keep tests self-contained and idempotent. Design each test to create, use, and clean up its own data. Any test should be able to run on its own, in any order, without relying on state from a previous test.
Separate fast unit tests from slower integration tests and give integration tests dedicated, isolated environments. Unit tests that touch the network or the filesystem are integration tests in disguise, and they will flake. Keeping the layers clean reduces the surface area for flaky tests in the suite.
Containerize runs. Use tools like Docker to ensure local dev, staging, and CI pipelines run identical OSs and dependencies.
Quarantine flaky tests instead of deleting them. Move them out of the gating suite so they stop blocking merges, but set an SLA to fix or retire them within a sprint. Isolated and temporarily disabled chronically unstable tests should be visible on a dashboard so the team sees how many are waiting for fixes and how old the oldest ones are.
Assign ownership. "You wrote it, you fix it when it flakes" creates accountability. Review flake metrics alongside code coverage in pull request reviews. If a new test has a 10% flake rate in its first week, send it back to the author before it enters the gating suite.
Cap retry logic. One automatic rerun for data collection is reasonable. Retry-until-green is an antipattern that hides flaky tests and the defects they sometimes expose. If a test needs three retries to pass, call it what it is: failing, with extra steps.
How Sauce Labs’ AURA platform helps teams reduce flaky tests
Consistent infrastructure is the prerequisite for reliable tests. Half of the "flaky tests" teams debug are just inconsistent environments producing inconsistent results.
Sauce Labs provisions real devices and browsers in a clean, known state for every session. With no residual cache from a previous run, no shared disk, and no port contention from a parallel job on the same machine, Sauce Labs removes the environment variable from the equation. That way, when a test fails, the failure actually points to the test or the application.
Even better, reliability starts earlier than execution with AURA, Sauce Labs' AI-Unified Release Assurance platform. Sauce AI for Test Authoring — part of the “build” phase of the lifecycle — generates tests from plain-language intent using condition-based waits instead of the fixed sleeps and timing guesses that cause a large share of flaky tests in the first place. Fewer flaky tests get written into the suite to begin with.
Test analytics rank tests by failure rate and flag flaky patterns across suites and over time. A test that fails 12% of the time over 14 builds is visible on the dashboard without anyone maintaining a spreadsheet. During the “analyze” stage, Sauce AI for Insights takes this further, using failure patterns across execution history to surface which tests are unreliable and why.
Parallel execution runs on isolated sessions so that concurrency doesn’t introduce the shared-state contention that causes flaky tests in self-managed grids. Each session is independent, which is the same isolation your tests should have.
Debugging artifacts ship with every run: video, screenshots, command logs, and HAR files. When a flaky test fails, the artifacts from that failure are available for comparison with a passing run. Intermittent failures become reproducible evidence instead of anecdotes, and that evidence transforms flaky test triage from a guessing game into root-cause analysis.
The same discipline that keeps flaky results out of a gate carries into production. Sauce Error Reporting monitors crashes and errors that slip past a suite entirely, closing the loop between what a test suite verified and what's actually happening once a release is live.
Sauce Labs’ AURA provides one chain of evidence, running from business intent to production confidence.
Start building a flake-free test suite
Flaky tests are a solvable engineering problem. Measure the flake rate. Fix the root cause instead of rerunning, and run tests on infrastructure that behaves the same way every time. When the suite gets more trustworthy with each fix, trust in the pipeline makes continuous delivery truly work.
Try Sauce Labs free or book a demo to see test analytics on your own suite.
