Sauce Labs is taking the next step to give you visibility into your test results with the release of the Insights module of Test Analytics.
Our first release, Test Trends, gave developers visibility into test pass/fail rates over time. Insights extends that by diving deeper into the results of an individual test with details on pass/fail trends to determine flakiness.
One of the most frustrating QA issues is a test that sometimes passes and sometimes fails, or has widely different run times. If this happens repeatedly, when the test hasn’t changed, then the root cause is often networking / timing issues. Insights helps you visualize this by showing test run times vs. time - which makes it easy to see if your test is flaky. In the image below, you can see the run times of a single test that often passes (green dots) but sometimes fails (red dots) - the random pass / fail and variation in run times is a good indication of a flaky test.
Once you suspect network issues are the cause of flakiness, you can use our Extended Debugging command to rerun the test and expose the networking / timing issues with HAR files and JSON logs. Now you have the data to understand precisely why the test fails and and act quickly to fix the test.
Test Insights helps you quickly identify a test’s root-cause failures and reduce time spent in troubleshooting. Insights provides 30 days of performance data for a single test across key metrics such as:
- Error rate
- Failure rate
- Run time changes
The result is you you can identify what to fix first and prioritize your resources so you can ultimately release better software faster.
Test Insights is available to all enterprise customers of Sauce Labs today. Just look for a new tab on the left side of your Sauce Labs Dashboard.
Details are in our Docs wiki.