In a perfect world, all defects within an application would be rooted out and resolved throughout a meticulously defined testing process in the development cycle, resulting in no defects reaching production. Our world, however, is far from perfect, and it is likely that not all bugs will be caught prior to each production deployment. Furthermore, it is possible (and even likely, in many cases) that new features, even those that are bug-free, will not always be well-received.
These are just a few reasons why you should test in production. While you should also perform software testing at earlier stages of development, production testing is an important safeguard for early detection of issues that could cause significant problems for production code.
In this article, I explain what I mean and what I don’t mean when I refer to production testing. In addition, I discuss why production testing is an essential part of an effective application development strategy, as well as provide tips for implementing a testing process in a production environment.
What is Testing in Production?
The most practical way to indicate what we mean when we say “testing in production” is to explain what we don’t mean. Testing in production does not mean releasing code that has not been tested in an effort to get a feature out the door and hoping that it works when accessed by the customer. To effectively implement a strategy of production testing, it is still essential to leverage practices such as shift-left testing throughout the development lifecycle. But the idea is that after all of this, testing does not simply stop. Production testing refers to continuous testing and monitoring of the production environment after a production deployment. This provides several key advantages that will enhance the performance of an application in the long run:
Testing/monitoring in production allows the DevOps team to adequately prepare to recover from disastrous circumstances, rather than simply attempting to prevent them. The fact of the matter is that at some point, a defect will end up in production. If the team nurtures their ability to react to a bug, then the recovery time will be much quicker. Production testing will help to ensure that defects in production are detected and resolved more quickly through continuous monitoring.
Production testing can also allow for key user experience analysis. A/B testing, described more in-depth later in this article, can be utilized in a production environment to allow for analysis that indicates whether or not a recently adjusted feature is well-received.
While testing at all other points in the development lifecycle is both essential and effective, it is likely that the conditions of a production environment cannot quite be replicated. This means that it is likely not possible to prepare in full, and provide complete test coverage for an application prior to deploying to production. Edge cases that are not present due to data or traffic limitations in a test or staging environment may go untested. These edge cases are more likely to be put to the test when the code is deployed to production. Continuously monitoring the production environment can help to ensure that these edge cases do not cause any issues—and should they cause any issues, can help to ensure they are recognized and resolved in a timely manner.
A Few Tips for Production Testing
Implementing production testing can be done in a variety of ways. As mentioned above, A/B testing can provide valuable data related to analyzing user experience. When implementing an A/B test, the idea is to split a feature in your application into two separate versions, A and B.
For instance, imagine that changes are being made to a registration feature for your web application, and the goal is to figure out if the new registration version leads to a higher percentage of completed registrations. Version A would exist as your control (the original registration implementation), and version B would contain your changes to the registration feature. An A/B test would utilize a mechanism that recorded the number of registrations started for each version and the number of registrations completed. If 10% of users started the registration process with version A and completed their registration, yet 18% of users completed the registration utilizing version B, then it is likely that version B provides a better user experience. This type of testing cannot be done prior to production, but can enhance the viability of your application when done effectively in production.
Another type of production testing that equates more to production monitoring is Application Performance Monitoring. One major advantage for an organization that employs this type of monitoring is the increased ability to identify issues within the application that may not be easily identifiable in other environments. An example of this is the ability to identify slow pages within a web application. A web page may load without issue in an environment with a smaller data set (such as a test environment).
However, that same page, when deployed to a production environment, may load much more slowly. Monitoring the time it takes for a web server to process requests can help to identify pages where the application is particularly slow. And while this may not be a show-stopping bug that jumps off the screen at a user, it should still be fixed to provide a better user experience.
Conclusion
Application testing is a high priority for any software development organization. Most organizations prioritize testing as much as possible prior to production deployments (and rightfully so) to ensure the smoothest transition post-deployment. But production testing can provide some unique benefits for a DevOps organization that should not be overlooked. From preparing the team to respond to disastrous circumstances in production to providing a better user experience for the customer, production testing is becoming more and more of an essential part of application testing as time goes on.
Scott Fitzpatrick is a Fixate IO Contributor and has over 6 years of experience in software development. He has worked with many languages, including Java, ColdFusion, HTML/CSS, JavaScript and SQL.
