Free trial

Knowing your subject

This article follows-up with the series of  articles about tackling test complexity, adding a view on the importance of product knowledge, when coping with the complexity of testing. Let’s assume that you understand already, who your customers are;)

I managed testing of a large integrated suite of software products on one of my past projects. Each product had its years of individual history already, when the decision was made to release them as a suite. And each of them was quite complex on its own, even without considering integration with other products. The number of installation issues reported by customers started increasing to an unacceptable level after a couple of years of the products suite’s existence. The whole engineering organization became concerned about the issue. First hypotheses about the cause assumed that the released installer was defective or that the product documentation was unclear. These were not proven however. So, we decided to conduct a profound root-cause analysis.

Unclarity of documentation or installer defects were indeed pointed out as the symptom of individual support cases, when we examined them. However, going a step deeper opened a completely new perspective on these issues. We learned that customers were deploying the product suite in unintended ways. Ways that the installer was not capable of handling and the documentation had not dealt with. The root-cause of the problem was that each individual product could have been configured in plenty of different ways, but only a portion of those configurations was practical for deployments within the integrated product suite. Our product documentation gave no guidance to customers whatsoever about recommended deployment configurations of the integrated suite, our developers and testers didn’t consider this aspect of the product suite, when developing and testing it. The problem became apparent, because the number of customer escalations kept increasing, while newer and newer versions of the product suite were released and more and more customers decided to adopt the product. It grew out of hand, because the project team missed considering an important angle of product knowledge or at least missed convincing upper management that that angle was important to consider and fund.

If you want to understand the potential scope of testing and work on addressing its complexity, you need to understand your product first and see it from different angles. Think about technologies that implement the product, but don’t ignore to look at uses cases that those technologies are used for! Think about it’s usage, but don’t neglect to understand, how it gets deployed! Think about the product’s details, but don’t forget to consider it as a whole! Sure, putting together the different views to a coherent plan is often a huge challenge, as the classic cartoon shows. Omitting to consider them however may bring you even more troubles.


Here is a couple of thoughts about ingredients for understanding your product:

  • contribution from across the organization, not only testing (support, professional services, marketing, development, management) to get a wholistic view on the product
  • experienced testers, who know the product and have a clue about angles from which product quality should be examined
  • historical data, such as defects and customer escalations
  • new features of the new release that you are planning to test
  • if possible, do a bit of hands-on exploration of the product before you nail down your test plans

These are not going to guarantee that your test plan will be perfect, but they can significantly increase your chances to create a test plan that addresses your needs.

independent engineering qa system engineering system testing testing

Leave a Reply

Related articles


Let’s make LLMs generate JSON!

In this article, we are going to talk about three tools that can, at least in theory, force any local LLM to produce structured output: LM Format Enforcer, Outlines, and Guidance. After a short description of each tool, we will evaluate their performance on a few test cases ranging from book recommendations to extracting information from HTML. And the best for the end, we will show you how forcing LLMs to produce a structured output can be used to solve a very common problem in many businesses: extracting structured records from free-form text.

Notiondipity: What I learned about browser extension development

Me and many of my colleagues at profiq use Notion for note-taking and work organization. Our workspaces contain a lot of knowledge about our work, plans, or the articles or books we read. At some point, a thought came to my mind: couldn’t we use all this knowledge to come up with project ideas suited to our skills and interests?

From ChatGPT to Smart Agents: The Next Frontier in App Integration

It has been over a year since OpenAI introduced ChatGPT and brought the power of AI and large language models (LLMs) to the average consumer. But we could argue that introducing APIs for seamlessly integrating large language models into apps developed by companies and independent hackers all over the world can be the true game changer in the long term. Developers are having heated discussions about how we can utilize this technology to develop truly useful apps that provide real value instead of just copying what OpenAI does. We want to contribute to this discussion by showing you how we think about developing autonomous agents at profiq. But first a bit of background.