Knowing your subject

This article follows-up with the series of  articles about tackling test complexity, adding a view on the importance of product knowledge, when coping with the complexity of testing. Let’s assume that you understand already, who your customers are;)

I managed testing of a large integrated suite of software products on one of my past projects. Each product had its years of individual history already, when the decision was made to release them as a suite. And each of them was quite complex on its own, even without considering integration with other products. The number of installation issues reported by customers started increasing to an unacceptable level after a couple of years of the products suite’s existence. The whole engineering organization became concerned about the issue. First hypotheses about the cause assumed that the released installer was defective or that the product documentation was unclear. These were not proven however. So, we decided to conduct a profound root-cause analysis.

Unclarity of documentation or installer defects were indeed pointed out as the symptom of individual support cases, when we examined them. However, going a step deeper opened a completely new perspective on these issues. We learned that customers were deploying the product suite in unintended ways. Ways that the installer was not capable of handling and the documentation had not dealt with. The root-cause of the problem was that each individual product could have been configured in plenty of different ways, but only a portion of those configurations was practical for deployments within the integrated product suite. Our product documentation gave no guidance to customers whatsoever about recommended deployment configurations of the integrated suite, our developers and testers didn’t consider this aspect of the product suite, when developing and testing it. The problem became apparent, because the number of customer escalations kept increasing, while newer and newer versions of the product suite were released and more and more customers decided to adopt the product. It grew out of hand, because the project team missed considering an important angle of product knowledge or at least missed convincing upper management that that angle was important to consider and fund.

If you want to understand the potential scope of testing and work on addressing its complexity, you need to understand your product first and see it from different angles. Think about technologies that implement the product, but don’t ignore to look at uses cases that those technologies are used for! Think about it’s usage, but don’t neglect to understand, how it gets deployed! Think about the product’s details, but don’t forget to consider it as a whole! Sure, putting together the different views to a coherent plan is often a huge challenge, as the classic cartoon shows. Omitting to consider them however may bring you even more troubles.

(source: http://www.projectcartoon.com/cartoon/3)

Here is a couple of thoughts about ingredients for understanding your product:

• contribution from across the organization, not only testing (support, professional services, marketing, development, management) to get a wholistic view on the product
• experienced testers, who know the product and have a clue about angles from which product quality should be examined
• historical data, such as defects and customer escalations
• new features of the new release that you are planning to test
• if possible, do a bit of hands-on exploration of the product before you nail down your test plans

These are not going to guarantee that your test plan will be perfect, but they can significantly increase your chances to create a test plan that addresses your needs.