Free trial

Missing the forest for the trees

Understanding customer needs for being able to do good testing sounds like commonplace that doesn’t need a special discussion. And testing projects often omit to discuss them indeed. Testing teams often start their project involvement with reading functional (or non-functional) specifications, skipping the customer view. I’m still mentioning it on the first place in the introduction of approaches to tackle testing complexity.

Imagine that you need to test the database that we discussed in the same article. You got to make choices, which of the possible combinations you test and which of them not, given you cannot realistically cover all of them. Imagine that you know that 98% of your customers are small and mid enterprises with up to 1.000 employees. Naturally, you might conclude that the majority of your customers will prefer:

    1. small size deployments with 1-2 instances of the database (over large-scale, highly available and distributed database deployments) to save on hardware and software cost
    2. will prefer to install the software on their own (as opposed to inviting an external vendor to do the deployment) to save on service cost
    3. will prefer to operate the database deployment in production on their own (as opposed to engaging a support services vendor) to save on service cost

Given these requirements, you might conclude that your testing should focus on 1) quality of small scale deployments, 2) superb installation experience and on 3) ease of serviceability. This is of course an oversimplification of all the factors that one needs to consider in the real world, but I hope it is good enough to make the point.

At the same time, you might not know that the minority of your customers (2% represented by big enterprises) are generating 98% of your revenues. Such a ratio isn’t a rare case in the real world. Now what? This is completely changing the game. High-availability likely becomes a priority. Usability of installation is nice, but perhaps not that important as one would think, because those big customers prefer to employ external vendors, who are experts in doing deployments and who may prefer interfaces that enable automation of large scale deployment over an easy to use graphical, but manual interface. Those customers are also likely to engage an external service vendor (most probably the software vendor itself) to provide support services, so they don’t need to waste time on it. Consequently, testing will focus on completely different targets than you might have thought based on the initial information.

Understanding your customer can make a huge difference, how good or bad you are in testing. You may easily miss the forest for the trees, when jumping right away on analyzing functional specifications or, God forbid, on test automation without understanding your customer. The trick is that it is often difficult to find reliable and usable customer data. Anyway, try to do your best to find them. It’s worth the effort. Try to talk to marketing, support, professional services or any other applicable departments, if data are not readily available. They will not necessarily provide you with hard customer data, they may still be able to give you some soft metrics and a useful perspective on the way your customers use your product.

qa system engineering system testing

2 Responses to “Missing the forest for the trees”

  1. Karol says:

    Good point well presented.

Leave a Reply

Related articles


Let’s make LLMs generate JSON!

In this article, we are going to talk about three tools that can, at least in theory, force any local LLM to produce structured output: LM Format Enforcer, Outlines, and Guidance. After a short description of each tool, we will evaluate their performance on a few test cases ranging from book recommendations to extracting information from HTML. And the best for the end, we will show you how forcing LLMs to produce a structured output can be used to solve a very common problem in many businesses: extracting structured records from free-form text.

Notiondipity: What I learned about browser extension development

Me and many of my colleagues at profiq use Notion for note-taking and work organization. Our workspaces contain a lot of knowledge about our work, plans, or the articles or books we read. At some point, a thought came to my mind: couldn’t we use all this knowledge to come up with project ideas suited to our skills and interests?

From ChatGPT to Smart Agents: The Next Frontier in App Integration

It has been over a year since OpenAI introduced ChatGPT and brought the power of AI and large language models (LLMs) to the average consumer. But we could argue that introducing APIs for seamlessly integrating large language models into apps developed by companies and independent hackers all over the world can be the true game changer in the long term. Developers are having heated discussions about how we can utilize this technology to develop truly useful apps that provide real value instead of just copying what OpenAI does. We want to contribute to this discussion by showing you how we think about developing autonomous agents at profiq. But first a bit of background.