You Can’t Have AIOps Without Automated Testing

AIOps, an IT operations strategy that leverages machine learning and data analytics to automate decision-making, is a hot topic. It’s so hot, in fact, that the AIOps market is projected to grow at a rate of more than 25 percent per year for the next several years.

Like all trendy new technologies, however, AIOps requires a careful approach in order to be implemented successfully. Part of that approach entails setting up an effective automated testing strategy for AIOps applications. Without automated testing, you run the risk of not being able to ensure the quality of AIOps software at a pace that keeps up with your AIOps tools themselves.

To prove the point, I’d like to take a look in this post at an example of a common type of AIOps application — a chatbot — and consider how and why to automate testing for it. I’ll focus on Botium as an AIOps testing solution, but the lessons below apply broadly to any AIOps automated testing strategy or framework.

AIOps stands for Artificial Intelligence in IT Operations. It uses advanced algorithms and AI techniques to analyze Big Data from various IT and business operations tools, speeding up service delivery, increasing IT efficiency, and delivering a superior user experience. AIOps breaks away from siloed operations management.

AIOps is all about applying machine learning algorithms to the vast amounts of data available in order to provide insights and make a higher level of automation possible. This means that IT Ops no longer needs to extensively depend on human operators for modern software management. Solutions powered by AIOps get their data from a variety of resources and give analytics platforms access to this stored data.

A common example of a type of AIOps application in use in the real world today is a chatbot. Chatbots are apps that have conversations with humans, using machine learning to share relevant information and issue intelligent responses to questions from humans.

Because chatbots rely centrally on machine learning, ensuring adequate chatbot performance requires careful testing to verify that they respond as intended.Software testing for chatbots

Originally, chatbots were typically tested manually. A group of human operators would interact with them, and test analysts would study the results. Naturally, this was an inefficient process; to achieve sufficient test coverage, hundreds and even thousands of testers were needed.

Botium — the open source testing tool which bills itself as the “Selenium for Chatbots” — was created in part to solve this problem. Botium allows testers to automate the testing of chatbot behavior using APIs. Tests can be executed in seconds, and they are easily repeatable with every software update that flows down your chatbot delivery pipeline.

To make the lives of software testers even easier, Botium provides a complete management toolbox with predefined test environments and test sets with thousands of immediately runnable tests.

Because Botium is the Selenium for chatbots, regression testing is the first possible use that comes to mind. (Regression testing costs a lot of time and should be automated where possible.) Beyond regression testing, Botium is also suitable for security testing with the special test dataset for the OWASP Top 10.

For this article, we will use the Botium Box free trial.Botium Box has a simple Quickstart Wizard to build up a communication channel with your chatbot and configure testing:

Let’s walk through these steps.

Go to Quickstart in the left menu.

Now you can connect to a chatbot from a predefined set or via new connection settings.

For now, choose a chatbot from the predefined set. Choose Chatbot - 01Gea.

Click “Next.”

Here. you can select the test set for testing the chatbot 01Gea Enter de/Banking

Click “Next.”

No device set is required for testing the chatbot. (If you want to have a device set, this is possible for other datasets, and Sauce Labs is a device lab and cloud provider for Botium.)

Now, click “Save” and start testing.

Here, the test results are shown in a dashboard that lets you see the status of the test cases visually and textually.

For this test, 256 test cases were executed in less than seven minutes. A manual process would have taken much longer. The reliability of the artificial intelligence used in this chatbot is also verifiable in a faster and more structured way compared to a manual process.

Testing an AIOps tool like a chatbot manually requires enormous time and effort. And not testing it at all would leave end-users at risk of an AIOps experience that falls far short of delivering on its promised value.

Fortunately, new testing tools like Botium are helping to address this challenge by making it easy to execute automated tests for AIOps applications. As AIOps grows in importance, automated testing solutions will remain critical for ensuring that AIOps tools actually do what their designers intend.

Cordny Nederkoorn is a software testing and content marketing consultant with over 10 years of experience in finance, e-commerce and web development. He is also the founder of TestingSaaS, a testing and content marketing agency for companies related to Software as a Service (SaaS). With TestingSaaS, he has built a community to share thoughts and methods on making SaaS better and safer.