Introduction: variability as a challenge in software testing
Software product lines (SPLs) allow companies to derive numerous variants of a software product from a common platform. This promises enormous benefits: tailored solutions for different customers, reuse of core components and shorter time to market. But where there is a lot of variability, particular challenges lie in wait – especially in testing. Conventional testing approaches that have proven themselves in individual projects quickly reach their limits in an SPL environment. The reason is the sheer number of combinations: every additional option or function can potentially interact with every other one. The number of possible product variants grows exponentially, which makes testing the entire variability practically impossible.
In a traditional software project without extreme variability, the test team concentrates on one defined system or a few execution variants. Test cases are derived from requirements, and the aim is to cover all important functional paths. With a software product line, however, there is no single system – but a whole spectrum of possible versions of the same system. A conventional approach would now require every variant to be tested in full. It is intuitively clear that this does not scale with dozens or even hundreds of features. Either gigantic test resources would have to be mobilised, or compromises in test coverage would have to be accepted. Neither is acceptable in practice, especially not in safety-critical areas. This is where a partner like bluesolve comes in, having recognised early on that new strategies and tools are needed to carry out SPL testing efficiently and in a targeted way.
Conventional testing vs. variant diversity
Why exactly does a conventional testing approach fail when variant diversity is high? One central problem is combinatorics. Suppose an SPL offers 20 optional features that can be combined freely – that alone theoretically results in over a million possible variants. In real product lines, the numbers are often much higher still. Conventional tests would try either to cover all combinations (impossible at this scale) or to restrict themselves to a few sample configurations. The latter carries enormous risks: untested feature combinations can show unexpected misbehaviour. The interaction between functions in particular is a frequent cause of bugs that remain hidden in individual component tests. So if not every relevant combination is checked, there is a danger that certain customer installations will have problems that nobody discovered beforehand.
A further obstacle of the classic approach is redundancy and inefficiency. Many variants share large parts of their functionality. A naive approach, however, would run tests for each variant separately, which means that shared functions are tested again and again. This wastes time and resources. At the same time, subtle differences between variants can be overlooked if one wrongly assumes that successfully testing one product variant automatically guarantees that another is free of defects. In practice, test managers have found that such a “copy-and-paste” testing approach becomes unmanageable for SPLs.
The issue becomes particularly critical in safety-critical environments. In domains such as medical technology, automotive or aviation, changes are subject to strict regulatory requirements. Every possible configuration must meet certain approval criteria. This means that no company can afford to leave any delivered variant untested. Here, conventional testing would lead to endless test cycles. The consequence: companies often proactively restrict the variability of their products in order to reduce the testing effort – which, however, runs counter to the actual benefit of SPLs.
Case study: a complex SPL from the field of medical technology
An impressive insight into this problem is provided by a case study from medical technology, in which an extensive software product line had grown over many years. The platform in question, used in imaging and diagnostics, comprised around 900 optional features and was operated in approximately 35,000 installations at end customers. It is easy to imagine what an enormous number of possible variants could theoretically be realised with it. A conventional testing approach inevitably reached its limits here: the volume of required test cases and test configurations was simply unmanageable.
Those responsible for the project faced a dilemma. On the one hand, the system was to remain flexible enough to cover a wide range of customer requirements; on the other hand, the quality of every single delivered configuration had to be reliably assured. To solve this challenge with traditional means, there was ultimately only one way out: drastically restricting variability. In practice, this meant that customers were offered only four preconfigured product variants – new features were delivered only in a quarterly release cycle.
In other words: out of hundreds of potential feature combinations, four permitted standard configurations were defined and fully tested. Customer installations had to adhere to these predefined variants in order to stay within the quality framework. However, this restriction also slowed down the innovation cycle considerably. More was simply not feasible with classic testing processes.
This example shows clearly how conventional testing ultimately limits innovative strength and flexibility. In theory, the SPL could have delivered significantly more variants – but for lack of testable coverage, a large part of the variability remained unused. For customers, this meant fewer choices; for the company, it meant leaving valuable market potential untapped and having to live with long release cycles.
It became clear: a new methodological approach was needed to restore the balance between variability and testing effort. This is exactly where modern strategies come in – often supported by specialised service providers such as bluesolve, who professionally support companies in managing such scenarios.
Feature modelling: mastering variants systematically
The first key to success was the use of feature modelling. This means explicitly modelling all the features of a product line and their dependencies on one another, usually in the form of a feature diagram. A feature model shows hierarchically which functions are optional or mandatory, which alternatives are mutually exclusive and which combinations are possible (or forbidden). This tool comes from the software product line engineering methodology and creates transparency about variability.
Why is this so important for testing? Well, without a formal feature model, the test team would have to work out for itself which combinations of functions are meaningful or valid. With dozens or hundreds of features, this is almost impossible. The feature model ensures that the rules of variability are known: for example, it prevents time being wasted on tests for constellations that must never occur anyway (for instance because, according to the model, feature A and feature B are mutually exclusive). At the same time, the model serves as a basis for generating test cases in a targeted way – more on this shortly in the section on combinatorial testing.
In the case study mentioned above, an extensive feature model was created that mapped all ~900 features and their relationships. This modelling alone was a major effort, but an indispensable one: it provides the map for navigating the variant landscape. With a feature model, it is possible, for example, to determine automatically how many theoretically valid product variants exist or which features most frequently occur together. Such information feeds into the test strategy. In its consulting projects, bluesolve always recommends first building a clean variant model. Experience there shows: only what has been modelled can be tested systematically. Through professional variant modelling – a core part of bluesolve’s services – companies lay the foundation on which all further optimisations build.
In addition, feature modelling promotes a shared understanding between development, testing and management. When everyone involved has a consistent picture of the product line, test objectives can be derived more precisely. For example, the model makes it possible to recognise which features are critical for one another and require particular attention. These semantic dependencies (see below) can be documented in the model. Modern tools, often supported by AI, even help to uncover anomalies or redundant dependencies in the feature model. Here, too, bluesolve is a competent partner: from selecting suitable tools to training employees in feature modelling.
Combinatorial testing: intelligent selection of test configurations
Perhaps the most powerful instrument for getting the flood of variants under control is combinatorial testing. Instead of bluntly trying to check every possible combination of features, mathematically sound strategies are used to achieve maximum coverage of feature interactions with a small number of targeted test cases. A well-known approach is so-called pairwise testing (or, more generally, t-wise testing). This ensures that every possible combination of t features occurs together in at least one test configuration. The most common choice is t=2 (pairwise), i.e. every pairing of features appears in at least one test case. The key point: many malfunctions in systems with many variants already arise from the interplay of two options. Higher-order interactions (of three or more features) are of course also possible, but empirical studies and practical reports show that pairwise testing can already find a large proportion of interaction faults. Increasing t to 3 (three-wise) raises coverage further, but also the number of tests required. Here, a pragmatic compromise has to be found.
In the context of the case study mentioned above, a combinatorial test strategy was applied to make the enormous number of features manageable. With the help of special algorithms (keyword: covering arrays), test configurations were generated that cover all critical feature combinations without having to test every possible variant individually. For example, a single test run might activate a dozen features at once and thus verify many pairwise combinations in one go. The entire test case catalogue was optimised so that no important two-way (or, where applicable, three-way) combination remained untested. As a result, the number of required test configurations could be reduced dramatically – reports often speak of savings by orders of magnitude, while coverage of important interactions is ensured. Millions of potential variants may thus become only a three-digit number of test cases, or even fewer, depending on the model’s restrictions. For the test team, this means a quantum leap in efficiency.
It is important to stress: combinatorial testing does not replace all tests, but complements them. Alongside these generated variants, basic functional tests, negative tests, performance tests etc. must still be carried out. But as far as variant coverage is concerned, the combinatorial method creates a systematic assurance that could hardly be achieved manually. A positive side effect is objectivity: instead of deciding on gut feeling which combinations to test, a reproducible, traceable selection process is used. Especially towards auditors or approval bodies (in the case of medical technology, for example, TÜV or the FDA), this makes it easier to justify why sufficient test coverage can be claimed even with 900 features.
Such methods sound complex, but fortunately tools and expert knowledge for them are now available. bluesolve, for example, has experience in introducing pairwise testing tools in large organisational structures. They help to generate test configurations automatically from the feature model that offer optimal combination coverage. bluesolve also helps to set the right parameters for t-wise testing (i.e. whether pairwise is sufficient or whether three-wise makes sense in certain places) and to integrate the generated tests into the existing test process. The result: companies can exploit the variability of their SPL much more broadly without having to invest proportionally more testing effort.
Deployment-based testing: focusing on real customer configurations
Alongside the statistical-combinatorial optimisation, the experience report revealed a further success factor: deployment-based testing. This approach directs the testing focus to configurations that have actually been delivered or are realistic for customers. The idea behind it: not every theoretically possible feature combination is relevant in the field. In practice, certain typical configuration patterns often emerge (e.g. standard packages, frequently chosen options, regional variants etc.). In deployment-based testing, such real use cases are tested as a priority.
In the case of the 35,000 installations in our case study above, the team was able to analyse which variant combinations actually occur at customers. It presumably turned out that customers do not activate anywhere near all 900 features in a random mix, but that each installation follows a particular profile. This insight can be used to refine the test strategy: first, all variants that exist in reality are tested in full (because this is where an error is most immediately business-critical). Beyond that, tests can be defined for conceivable new combinations, but specifically where there is a need.
For this purpose, the concept of “Customer-Required Subgraphs” was introduced – in German roughly “kundenbezogene Teilgraphen” of the feature model. The thinking behind it is to divide the large feature model into smaller, manageable areas, each of which is relevant to particular customers or deployment scenarios. Every concrete customer installation can be represented as a subgraph of the overall feature model that contains exactly the features the customer needs. By looking at these subsets, the variations within its relevant scope can be tested specifically for each customer (type). If, for example, a certain group of customers uses only 50 of the 900 features (in various combinations), the focus is on fully validating these 50 in interaction – without being distracted by the rest of the model. Should new customer requirements arise that call for new features or unusual combinations, a new “Customer-Required Subgraph” is defined for them and it, too, is subjected to the necessary tests in a targeted way.
This approach has two major advantages. First, it ensures that every configuration that reaches the customer has already been covered by the test department – an invaluable benefit in safety-critical environments. Second, over-testing is avoided: resources do not flow into exotic variants that no customer will ever see, but into realistic scenarios with immediate added value. Deployment-based testing thus closes the gap between the abstract variant space and the concrete situation in the field. It is, so to speak, risk-based testing for SPLs: testing is preferentially done where the risk (in the form of frequent use or particular criticality) is highest.
bluesolve can support companies with this pragmatic approach, too. By evaluating customer data, usage statistics or sales feedback, it is possible to find out which feature combinations are actually in demand. bluesolve also brings experience in structuring feature models and variant management so that new customer requests can be quickly turned into test cases. This keeps companies flexible: the product line can grow and serve new markets without the test team losing track. On the contrary, testing is always deepened exactly where new configurations come into play.
Semantic dependencies: taking domain interactions into account
Another important lesson from the experience report was that it is necessary to look beyond the purely formal dependencies in the feature model. Semantic dependencies are relationships or interactions between features that arise from their meaning, even if they are not explicitly stated as a rule in the model. Put simply: not all critical interactions are already covered by “requires” or “excludes” in the feature model. Some only become apparent when the specialist domain is considered.
For example, two optional functions A and B could be technically independent (i.e. combinable according to the model), but lead to dangerous behaviour in practical use if they are activated together. Perhaps both demand so much computing power that their combination overloads the system – a classic semantic dependency. Or feature C is only useful if feature D is also activated; otherwise the customer would receive a nonsensical configuration (even if the model does not forbid it). Such subtleties need to be uncovered and taken into account in testing.
In the project mentioned above, domain experts were therefore brought in to identify implicit constraints. These were incorporated into test planning as additional rules or notes. Some semantic dependencies presumably led to certain feature combinations being classified as particularly critical and given higher priority in the tests. Other dependencies may have resulted in the decision not to permit certain combinations at all (because they had no meaningful use case or carried risks). Either way: explicitly capturing these domain interactions was crucial to leaving no gaps in test coverage.
For the test strategy, this meant in concrete terms: where it was known that only the interplay of three or four particular features represents a potential problem point, a test case was specifically planned that checks exactly this constellation – even if a pure pairwise scheme might not have covered such a four-way combination. Semantic dependencies thus act like an additional layer on top of the normal variability model: they direct the tester’s attention to the scenarios that really matter.
bluesolve knows from experience that, especially in complex domains (such as medical technology), the specialist knowledge of developers and users is a goldmine for test optimisation. As part of its consulting services, bluesolve helps to capture this knowledge in a structured way and make it usable. Through workshops with stakeholders, for example, relevant dependencies are collected and integrated into the variant model or the test design guidelines. This produces targeted test scenarios that cover real risks – something that a purely automatic approach might overlook.
“Customer-Required Subgraphs”: testing subsets in a targeted way
The concept of Customer-Required Subgraphs, i.e. customer-specific subgraphs of the feature model, has already been touched on. This concept deserves special mention once more, as it is highlighted as an innovation in the Siemens Healthineers study. What exactly does it mean? Essentially, it is a clever division of the big picture. Instead of having to consider the complete feature tree with 900 nodes in every test run, it is divided into meaningful segments. Each segment (subgraph) represents the features that are relevant to a particular customer group or product configuration.
One can picture it like this: the entire product line may cover several product variants or modules that not all customers use at the same time. For example, laboratory customers of the software use only the features needed for laboratory diagnostics, while radiology customers use other feature sets. For each of these different usage contexts, a subgraph is defined in the feature model. Within this subgraph, the test strategies described above (feature modelling, combinatorial testing, etc.) are then applied again, but restricted to this context. Testing effort and test cases are thus managed per segment, which is more manageable than a monolithic approach.
The trick is that these subgraphs can overlap or build together on the same core. Nevertheless, the division allows a modular testing approach. When a new customer with a new use case is won, perhaps requiring a previously unused feature, the corresponding subgraph can be tested precisely, without having to touch the entire test library of the other areas. This increases the reusability of test results: what has already been checked in subgraph A does not need to be verified again for subgraph B, as long as B does not affect that part.
The “Customer-Required Subgraphs” thus contribute significantly to keeping the testing effort per newly supported variant low. In the experience of the case study above, this made it possible to overcome the rigidity of the originally only four permitted configurations. Looking ahead, it is conceivable that this methodology can safely support significantly more than four variants without descending into chaos – because only the differentiating features are additionally tested.
For a company, this means more flexibility and faster response to customer requests. Where previously every new variation entailed a huge tail of tests, new combinations can now be released with limited additional effort. And that is exactly what makes a successful software product line – being variable and yet manageable.
Through its work on such concepts (some of bluesolve’s developers were themselves involved in the case study), bluesolve has gained valuable know-how in how to define and use “Customer-Required Subgraphs” in practice. bluesolve passes this knowledge on to its customers to make their SPL test management more agile and scalable as well.
Why the conventional approach fails – and what the new one delivers
Let us summarise the findings: conventional testing fails with SPLs mainly because of a lack of scalability. The old methods are not designed to deal with hundreds of variables. They lack the systematic approach needed to make a targeted selection from the infinite set of combinations. The consequence was either exorbitant testing effort or risky gaps in quality assurance. In the worst case, as initially seen at Siemens Healthineers, diversity has to be artificially cut back – a step that leaves the full potential of a product line lying idle.
The new methodological approach, by contrast, combines several intelligent strategies into a holistic solution:
- Feature modelling provides the formal foundation and prevents unnecessary and invalid test combinations.
- Combinatorial test generation covers the essential interactions with significantly fewer test cases.
- A deployment focus ensures that no real use case slips through the net and prioritises the important variants.
- Semantic analysis brings in expert knowledge to also safeguard hidden dependencies.
- Segmentation into subgraphs (Customer-Required Subgraphs) structures the variant space into manageable parts and enables iterative extension.
This combination has proven extremely effective in practice. As a result, significantly more variants can be supported without the testing effort rising exponentially. Quality remains high because every relevant constellation is covered in a well-thought-out way – but the cost and time for testing fall drastically compared with the naive approach. Especially at a time when software product lines are becoming ever more extensive and dynamic, this new approach is an enabler: companies can offer their customers a greater choice and more frequent updates without having to shoulder an unbearable testing burden.
It is also worth addressing the cultural change: through this approach, test teams develop further – away from simply working through fixed, predefined tests towards becoming shapers of quality assurance who work closely with product line architects and variant managers. Tools take over the generation of many tests, while people plan the strategy, define meaningful combination rules and evaluate results. This change is made easier by external partners such as bluesolve, who provide assistance with their experience and their tools.
Mastering SPL testing challenges with bluesolve
Software product lines undeniably bring great benefits – but only if their variability can also be used effectively. Conventional testing quickly falls behind here, as the example of Siemens Healthineers impressively shows. The good news is: with the right methodological approaches and tools, variability can be tamed without loss of quality or exploding costs. Feature modelling, combinatorial testing, a deployment-based approach and taking semantic dependencies into account are the central building blocks here. Industry experience shows that such a structured approach is far superior to the classic one.
However, the path there is not a foregone conclusion. It takes expertise to set up a complex feature model, to find the right parameters for combinatorial tests and to restructure the test processes accordingly. This is exactly where bluesolve supports companies as a competent partner. bluesolve has already supported numerous projects in variant management and helped to take SPL testing to a new level. From the initial analysis of the product line, through the introduction of suitable test tools, to training the team in new approaches – bluesolve provides both advisory and operational support.
The result is more efficient testing, more satisfied development teams and, ultimately, more robust products for customers. Thanks to this support, companies can make full use of the advantages of software product lines and concentrate on their capacity for innovation, while quality assurance is set up to scale.
If your company, too, faces the challenge of taming test complexity in a product world with many variants, it is worth taking a look at bluesolve’s solutions. With the right partner at your side, you can overcome conventional limits in SPL testing and turn them into real competitive advantages.
Further reading: NIST – interactions involved in software failures.