Shifting Accessibility Testing Left

Automated testing early in the pipeline for both web and mobile applications has become the standard for most development teams. Catching issues when they are faster, easier, and cheaper to fix provides tremendous value to a business both financially and through the improved quality of end-user experience. Unfortunately, accessibility has not been included in early pipeline testing due to the inability of traditional accessibility tools to both scale across an enterprise and meet the velocity requirements of modern release cycles.

In order for enterprises to shift accessibility testing left we need a solution that meets a number of key requirements. It needs to find critical accessibility issues that impact users, be quick and easy to implement, be easy to maintain, scalable, fast, cover all the most popular web and mobile platforms, and provide clear and actionable results. Fortunately, the combination of Evinced’s modern accessibility solutions and the power, speed, and scalability of the Sauce Labs continuous testing platform combine to create an enterprise ready solution for shifting accessibility testing left.

Due to the limited effectiveness of syntax/static only code analysis, existing legacy tools detect only a small fraction of web and mobile accessibility issues automatically. However, the complexity of modern apps and websites have moved far beyond simplistic semantic code design making it impossible to detect many critical accessibility issues that would block functionality for many users. Evinced has embraced this challenge and used modern technologies like computer vision and advanced data algorithms to provide a solution that automatically detects issues previously only able to be discovered through time consuming manual testing.

Before analyzing the code, Evinced renders the page or state and analyzes it as a sighted user would. It uses advanced rule-sets, computer vision, AI, and other algorithms to build a structural semantic model of a webpage. It identifies the intents and actions (input fields, drop-downs, etc.), and only then looks at the code to see if it has been implemented in an accessible manner. This approach allows Evinced engines to detect Keyboard Accessible, Interactable Role, Accessible Name, Zoom/Reflow, Focus Indication, and Text over Complex Image issues that no other technology can find automatically.

The Evinced automation SDKs for both web and mobile require only 3-5 lines of code in order to add accessibility scans to all of your existing functional tests. This uniquely positions Evinced SDKs for an enterprise implementation as there is no direct interaction with test code. This has distinct advantages. Because all of the needed setup is in auxiliary classes or methods, the tests stay exactly as the developer intended. This also means there is nothing needed to setup or maintain tests. This greatly reduces the friction created with the introduction of new tools and dramatically reduces the time it takes to start seeing a return on investment in accessibility testing.

1 @BeforeClass 2 public void setUp() { 3 driver = new EvincedWebDriver(new ChromeDriver()); 4 // Start the Evinced Engine 5 driver.evStart(); 6 }

1 @AfterClass 2 public void tearDown() { 3 // Stop the Evinced Engine 4 Report report = driver.evStop(); 5 // Output the Accessibility results in JSON or HTML 6 EvincedReporter.writeEvResultsToFile("accessibilityReport", report, EvincedReporter.FileFormat.JSON); 7 driver.quit(); 8 }

We invite you to read more about our suggested enterprise SDK implementation in our Plug and Play A11y Testing at Scale blog post as well as how to implement your existing Sauce Labs tests to add accessibility tests in our Evinced Selenium SDK + Sauce Labs post.

Evinced provides SDKs to test for accessibility on all of the major platforms but by adding the power of thousands of Sauce Labs browser, operating system and mobile device combinations we can test accessibility on any platform that an enterprise’s end user would use to access the application.

This coverage is especially important for mobile testing. On Android, breadth of coverage is extremely important due to the large segmentation in the Android market (Samsung, Google, LG, HTC, OnePlus, Huawei, etc). All these manufacturers take the default Android OS image from Google and make it their own. This creates the need for additional testing to make sure we give the best experience possible to all users. For iOS, once you upgrade your iOS device it is impossible to roll it back to a previous version. Proper testing requires a library of iOS devices on all the versions a potential customer might be using to ensure a great experience.

Fortunately, Sauce Labs, where Appium was born, has a comprehensive cloud based device pool where we can execute tests using the Evinced Engine to pinpoint accessibility issues on nearly any device and operating system combination. Read more about how to take your existing Sauce Labs mobile tests and add add accessibility scans in our Evinced Appium SDK + Sauce Labs blog post.

The Evinced automation SDKs have been designed to facilitate running a single test or all your tests by adding just 3 lines of code. With the instant scalability of Sauce Labs we can run all of our tests in parallel in the time it takes your longest test to finish removing testing, including accessibility, as a bottleneck in your pipeline. Along with running each of your tests in parallel, Sauce Labs also allows you to run all of your tests across multiple browsers, browser versions, and device types simultaneously. This level of speed and coverage is a powerful addition to any development process.

At the conclusion of the test a rich and comprehensive JSON or HTML report is generated to track issues in any reporting tool. Evinced provides a single, easy to understand, report that contains all of the information to identify, prioritize, reproduce, and remediate any issue. Reports include highlighted screenshots, WCAG success criteria, severity, URL, element identifier, html snippet, suggested fixes, and an identifier for cross scan issue tracking.

Read more about how the Evinced solve many of the core issues with legacy accessibility tools in our Evinced Selenium SDK vs Traditional Tools blog post. With the addition of videos, screenshots, logs, and performance information provided by Sauce Labs we have a comprehensive set of test reports that will help developers debug and identify issues extremely quickly and allow them to spend more doing what they do best, developing!

Evinced provides accessibility solutions that automatically detect up to 20x more critical accessibility issues than legacy solutions, can be implemented with as little as 3 lines of code, and provide a consolidated and actionable report. This in combination with the instant scalability and comprehensive browser/device and operating systems provided by Sauce Labs we have a solution that meets all the needs of a modern enterprise development team. Check out the Evinced and Sauce Labs pages for more information on getting started!