User Interface (UI) Testing. The idea is simple — automate some UI tests to ensure your application is still behaving as expected. Usually your first set of tests — running green, no doubt — make you all cheer and pat yourselves on the back. Then you open up the framework to more people. Despite the reviews (so many reviews), the failures start to come, and they don’t stop. Or they run green and then fail and then run green again. And then fail again. So why are they so unstable? Is it bad scripts? Environment issues? Sometimes you just don’t know, and you think you are going to lose your mind. Let’s take a look at some common and potential issues you may be facing.
Architecture, Environments and Settings
Is your infrastructure designed for stability? Are you using on-premise or cloud instances? What may have saved you a dollar upfront could cost you many more down the road - so your testing environment is important. Understand if your tests require particular system settings. Tests failing because of unwanted server variables is a waste of everyone's time. We found that out the hard way a long time ago. You may need to have some isolated tests that cannot be run on the same server so the majority of your other tests can pass. (Or maybe decide how important the test really is.) Or let’s say your testing frameworks are stable, but what about the tools or libraries you are importing? Are you pinning your stack to a version of these tools or libraries? A new version can completely break everything.
Many times, teams are pressed to have tests run as fast as possible. The way to accomplish this is parallelization (read more on that here). That said, if your automation architecture design was not planned out to support it, you may find difficulties.
Back-end Stability
How confident are you in your back-end acceptance tests? Without a stable base, it is hard to be confident in the UI built on top of it. When you write end-to-end (E2E) workflow tests, you are hitting your back-end. If they aren’t passing, you can’t be sure if your UI E2E tests are providing value, or if they are being impacted by deeper issues. Consider using mock API testing as part of your UI testing strategy. This reduces some of the impact of back-end issues, and gives more flexibility to the QA team.
Timing Issues
Consider how complex your system under test is. How long does it take to build? Does your back-end get out of sync with the UI and perhaps have changes that may not be in the UI yet (or vice versa)? In fast-paced modern development these types of issues can come up more and more. Also consider how you are setting up data. For example, let’s say you need to create a user, a student, an assignment, and a submission for that assignment. If your test is to grade that submission, but your setup hasn’t completed for some reason, your test will fail because it can’t find the submission. Think about how the team can help. Can your developers and automation champion design and deploy solutions for mock testing? Can DevOps help deploy those solutions with configuration management automation? Collaborating with the broader team for both timing and backend issues will help everyone.
Knowing What to Test: Just Because You Can, Doesn’t Mean You Should
Seriously think about what you are testing through the UI, and why. The more tests you have, the more of a pain your test suite will be to keep running green, and maintain. You should make higher-level decisions about the goals of your end-to-end tests. It is unlikely that the goal is to test every possible scenario. (Testing critical workflows is much more likely.) No matter what, there’s a constant struggle of balance that you need to consider. Try to focus your tests on UI components and business logic. For example, I just had a conversation with a developer — and, while I’m thrilled this person was thinking about testing — this person wanted to test how to cancel an item delete in two different ways. Rather than testing every combination, I recommended simplification, because i knew it would not have a critical workflow impact. In this case, I asked the developer, “Why not just focus on UI component/unit tests instead?” Automation provides a ton of benefits. But in order to make sure those benefits are realized over a long period of time, some planning is required. Consider your business goals and prepare for the things that can commonly go wrong.
Ashley Hunsberger is a Quality Architect at Blackboard, Inc. and co-founder of Quality Element. She’s passionate about making an impact in education and loves coaching team members in product and client-focused quality practices. Most recently, she has focused on test strategy implementation and training, development process efficiencies, and preaching Test Driven Development to anyone that will listen. In her downtime, she loves to travel, read, quilt, hike, and spend time with her family.
