Free trial

Tackling complexity

Software complexity is one of the most significant challenges thatcomplexity and testing a software tester may face. Testing software that is complex usually requires a breadth of knowledge and experience. The goal that the tester needs to accomplish with regards to complexity is: to test the software in a reasonable time and at a reasonable cost.

As an example, think about ways to test upgrades of a database system with the following support matrix that gives altogether billions of combinations to test. Can this test scope be reasonably covered? Maybe. Let’s see several techniques that can help addressing complexity.

Operating Systems the database system should run on various versions of Windows, Solaris, RedHat Linux, Ubuntu Linux and Suse Linux
Deployment types the database system can be deployed as stand-alone, client-server or client server with multiple redundant instances of the server
Upgrade from 2 prior versions of the database system
Integrated products the database system should work with multiple versions of associated software products from different vendors (application server, sso, identity management, soa, portal, plug-in for integration with sso)
Interfaces thousands of GUI, CLI and API that exercise features of the database system

I don’t intend to suggest a single practice for addressing testing complexity here. The specific test approach always depends on the software in hand and other factors, such as available resources. One usually needs to employ a good mix of multiple test techniques to test both effectively and efficiently. Instead, let me discuss techniques and approaches that we’ve used in practice, hoping that you could leverage them for creating your own mix and extend them based on your own experience.

  1. Understand your customer’s needs – to understand your test objectives
  2. Understand your product – to understand your test scope
  3. Get involved in product development early – to reflect past testing experience with product complexity in subsequent product releases
  4. Identify testing priorities – to know, where to focus
  5. Rotation of test configurations – use the advantage of iterative processes to increase coverage
  6. Domain partitioning – sample your test scope to avoid executing tests that add low value
  7. Pair-wise testing – a brute force approach that dramatically decreases the number of test combinations to test
  8. Exploratory testing – to use the experience of your test experts to complement the pre-scripted way of testing
  9. Test automation – to speed up and lower the cost of execution

Each of them offers evident business benefits, if used appropriately. Each of them needs certain knowledge and involves some risk as well. I’m going to talk about these in the following series of articles.

What is software complexity?
This article discusses complexity from the point of view of a possibly huge number of test combinations. Not from the point of view of complexity of the code structure nor the business-processes that the software implements.

And, BTW, most of these approaches are described in books and taught on trainings about testing. I’m going to refer to those books and articles at least and reflect to their descriptions critically based on practical experience.

qa system engineering system testing testing

Leave a Reply

Related articles


Let’s make LLMs generate JSON!

In this article, we are going to talk about three tools that can, at least in theory, force any local LLM to produce structured output: LM Format Enforcer, Outlines, and Guidance. After a short description of each tool, we will evaluate their performance on a few test cases ranging from book recommendations to extracting information from HTML. And the best for the end, we will show you how forcing LLMs to produce a structured output can be used to solve a very common problem in many businesses: extracting structured records from free-form text.

Notiondipity: What I learned about browser extension development

Me and many of my colleagues at profiq use Notion for note-taking and work organization. Our workspaces contain a lot of knowledge about our work, plans, or the articles or books we read. At some point, a thought came to my mind: couldn’t we use all this knowledge to come up with project ideas suited to our skills and interests?

From ChatGPT to Smart Agents: The Next Frontier in App Integration

It has been over a year since OpenAI introduced ChatGPT and brought the power of AI and large language models (LLMs) to the average consumer. But we could argue that introducing APIs for seamlessly integrating large language models into apps developed by companies and independent hackers all over the world can be the true game changer in the long term. Developers are having heated discussions about how we can utilize this technology to develop truly useful apps that provide real value instead of just copying what OpenAI does. We want to contribute to this discussion by showing you how we think about developing autonomous agents at profiq. But first a bit of background.