A little over a year ago, my team was facing a problem. Without an automated regression and our team size reduced, how could we make sure that we had a good release? Although we certainly did testing within our teams, we just didn’t have the manpower to thoroughly check. We started using an untapped resource in our testing — one that I had not heard about before. The concept was not new to the industry, but it was new to us. And it was effective. So what was our new magical resource? A bug bash!
Exploratory Testing On Steroids
The bug bash brings not only your testers to the table, but also people that may not have any testing background - developers, project managers, product owners, support… the list could go on. Put aside your day-to-day, ho-hum tasks, and use your product! Depending on the project, we would open the build up to several groups — not just to our project teams, but also to people that work with clients day in and day out, who could give us a new perspective that perhaps we had not considered. If you work in a world where you have very little automation, a bug bash allows you to expand what can be tested by others, and it’s a quick way to add resources (where we are constantly being asked to do more with less). Conversely, if you have all automation, this allows you to get human eyes on the product. We found that our bug bashes gave us fresh eyes on the system. Whether people worked on one feature team and tested another area, or came from outside the department to help and saw the product for the first time, we found that tired eyes missed things, and fresh eyes found them. This lead to consistency across our product, and got us out of the mundane. The biggest benefit of all, in my opinion, was how much we found. Yes, it was a little scary, but we found bugs before clients did! Overall, we found a shade under 20% of the overall bugs for our release. I’d wager to say this was more than what our automated regression used to find heading into a release. (Did I mention it was effective?)
A few lessons learned...
Of course, as useful as it is, we’ve had a few hiccups along the way, and are adjusting where we can. I just wanted to share a few things, should you choose to go down this path:
Keep it Fresh! USE THIS TOOL WISELY. It’s a powerful weapon against bugs, but it can very easily be over-used, and then you lose the effectiveness. Initially we had a bug bash here and there. The less often we had it, the more participation we had. Then, we went weekly. (Don’t do this.) The idea was to test as we delivered work in the sprint. Well, there is such a thing as too much of a good thing. I LOVE testing, and I was burnt out from it. Avoid bug bash burnout.
Keep it fun! Gamify it! In the beginning, we had prizes! Best bug, most critical bug, most number of bugs found… We really had a competition on our hands. It worked. You don’t have to spend money to get people to compete, but try to find a way to make people want to report just one more bug, dive a little deeper, and stay involved. Some people work on pride alone — get a live dashboard up, show live results as bugs come in!
Keep it focused! We also had better participation (and results) when we would focus the bug bashes on particular areas. Places we knew were weak and needed deeper testing, or whatever was new for the sprint. The more broad the search, the more lost your testers are going to be. Have some charters, some general guidance, and let people loose. We also had sign-up pages so people could pick from a list of features we wanted tested, and they could see where others were testing.
Keep it Social! We had group chats available during each session, and ensured Subject Matter Experts were answering questions from people who were not familiar with the features under test. Honestly this probably made the biggest difference in preventing duplicates or no-fix issues because someone didn’t understand how things should work. It was also eye-opening to those of us that may have been staring at a feature for a few months and, while easy to us, it was maybe not so easy and intuitive to others. (And isn’t usability important?) Yes, duplicates have happened (a lot, even with the chat rooms), but if you have someone who is not in QA testing, to be honest, they probably aren’t going to follow all of your best practices to a T. Just try to mitigate it.
Bash Away
Try it! I always seek to improve, and even now, over a year later, we are still tweaking our process as we move to Continuous Delivery and work to find the right cadence, but we are getting there! Have you had successful bug bashes? I’d love to hear about them (and the horror stories too). We can all learn from each other!
