Software testing is considered to be an integral part of the development process. Recently, different types of tests have emerged, and their complexity is increasing as well. What are modern estimation techniques in software testing and how are they used? Read this article to learn.
What Is Software Testing Estimation
Prior to beginning the testing, two issues that are significant for every client should be discussed: how long the process will take, and how much it will cost. To answer these questions, it’s necessary to estimate parameters such as “time,” “resources,” “cost ” and “human skills.”
Software testing estimation is a management activity to calculate and approximate time, resources and expenses needed to complete test execution in a specified environment.
It is a forecast that helps avoid exceeding time limits and overshooting budgets. Estimation is important to planning the project, allocating resources and optimizing activities, which results in high-grade software quality.
Let’s define why each parameter is essential for the team.
The success of the testing team is determined by its ability to meet the deadline. We should estimate testing efforts, in man-days or man-hours, to find out how long it will take. Managers calculate the optimal duration for each part of the project and set up a clear schedule adhering to which helps execute tasks in a timely manner. This is why time estimation is one of the key factors that helps build a good reputation and create an audience of loyal customers.
To perform a test, the team needs a definite set of resources (infrastructure, technical capabilities, specialists, time and money). If there are not enough resources, a task can’t be completed in time. It’s necessary to estimate the available assets and those that have to be added.
Testing, like any other work, requires expenses (including expenses towards hardware, software and human efforts). The overall budget should be evaluated and agreed upon beforehand, in order to assure the customer that the process will be kept within the budget, and no additional payments will be needed.
This evaluation considers the professional skills and experience of team members. In case they lack some knowledge, the process will slow down and the costs may increase.
As we see, all the aspects are intertwined and may influence each other: the time schedule depends on the resources, human skills and the budget; the budget, in turn, is affected by human skills, resources, and time; resources include time and professional knowledge as well.
Traditional Software Testing Estimation Techniques
Now let’s figure it out how to do software testing estimation. In recent years, different techniques, tools and methods have been developed for this very purpose. Their implementation and usage greatly simplify the estimation process.
Here we are going to discuss the most popular and effective estimation techniques in software testing.
Work Breakdown Structure (WBS)
The essence of this technique is to divide a complex test project into small components to obtain the following hierarchy: the project is broken down into sub-modules; each sub-module, in turn, is divided into functionalities, which are split into sub-functionalities. As a result, we get a very detailed and transparent structure (the sample is given below).
Since small tasks are much easier to estimate, they are allocated among the team members. Every person assumes responsibility for a given task and provides the result in a certain period of time. The project can, therefore, be completely checked step by step, without missing any detail. This approach eliminates situations in which some testers may perform the same task simultaneously, while other tasks may remain unaccomplished.
The WBS is the most powerful and effective method among QA estimation techniques. It enables a project manager to create a clear schedule and to plan a reliable budget. It also cultivates team cooperation and personal commitment.
This is a statistical method, but it also breaks down the task into subtasks (in this it is similar to WBS). Then, three possible scenarios should be estimated for each sub-task.
- The best case: assuming that you have a talented team and all of the necessary resources, and assuming that no problem occurs and everything goes right, you can complete the task, for example, in 100 man-hours (B). This is an optimistic scenario.
- The most likely case: assuming that you have a good team, enough resources, and almost everything goes right, although some problems may occur, you can complete the task in 150 man-hours (M). This is a normal scenario.
- The worst case: assuming that your team is not experienced, everything goes wrong and you have to solve numerous problems, you can complete the task in 200 man-hours (W). This is a pessimistic scenario.
Thus, you have three values:
B = 100
M = 150
W = 200
Now, you can calculate the average value for the test estimation (E) using the following formula:
E = (B + 4M + W) / 6
E = (100 + 4 x 150 + 200) / 6 = 150 man-hours
As the average value may fluctuate a little bit, to be more accurate, you need to calculate standard deviation (SD) — the limits within which E may change. The formula is as follows:
SD = (W – B) / 6
SD = (200 – 100) / 6 = 16.7 man-hours
You can present the final estimate as this: the team needs 150 +/- 16.7 person-hours to accomplish the sub-task.
Three-Point Estimation is one of the most effective methods for software testing when you have practice and data from the previous projects and an ability to apply them. The essence of this method is to find out the best and the worst working conditions for your team.
Function Point Analysis (FPA)
This technique indicates software functionality from the user’s perspective. It estimates the value of the total effort that can be considered as time, cost or size of the task. FPA is based on specification documents, such as SRS document or Design. Again, as with WBS, the project is split into modules. Each module — depending on its complexity — is assigned a functional point (FP). Simple tasks get lower points, difficult tasks get higher points. Total effort is calculated by the formula:
Total Effort = Total FP x Estimate Defined per FP
Estimate defined per FP is a given value that a test manager defines on the basis of the team experience. It is calculated with respect to time, cost or size (e.g. 10 hours/points, $100/points).
Let’s consider the total effort with respect to cost and take the estimate defined per FP as equal to $100/points.
The whole project is divided into three groups of modules:
- Complex modules (FP is 5) — 2 pieces
- Medium modules (FP is 3) — 10 pieces
- Simple modules (FP is 1) — 5 pieces
Total effort = 45 x 100 = $4,500
This means that to complete the project, you need $4,500.
FPA is not a really accurate method, as it involves risks and may give false results. Nevertheless, it is one of the more popular estimation techniques in software development and can be used in combination with other tools. Try to use this simple estimation template for your tasks, and you’ll see how it works.
This is one of the most widely used testing estimation techniques based on surveys of the experts involved in the testing process. The essence is that a panel of experts discuss the given task under the guidance of a manager and make anonymous personal forecasts (how many man-hours this task will take), providing the reasons for their opinions.
As a rule, after the first round, the range of answers is quite wide. Then, the experts are encouraged to revise their answers taking into account other members’ judgments. Several rounds may take place until the range of answers decreases and the average value can be calculated. The process is finished after a predefined criterion (i.e. after a limited number of rounds, or if the consensus is achieved, or when the results are stable).
Delphi technique is very simple and quite reliable due to the participation of experienced people and maintained anonymity. It gives qualitative and quantitative results and can be combined with other methods.
The above-described techniques are more suitable for Waterfall development and V-model development. They use a “bottom-up” approach when all the details and requirements should be defined and all the tasks should be estimated separately before you can plan the project schedule and budget. These development models, as well as the estimation approaches, have significant drawbacks due to the great speed of changes in the software industry. Thus, the number of tasks using these methodologies is currently decreasing.
More and more customers and providers prefer using agile development methodology, as it emphasizes continuous delivery, makes the process user-focused and gives a competitive edge. Agile estimation fundamentally differs from traditional techniques and uses a “top-down” approach.
It means that, initially, you process currently available data and provide a higher level estimation that is enough to start the project. Later on, as the project progresses, new information is revealed and more details are uncovered at lower levels. New knowledge is incorporated into the estimation, the values are refined and improved without stopping the development process. This approach makes the team capable of estimating new features effectively.
Principles of Agile Estimation:
- Collaboration — all team members are included in the process; no one person can be blamed for an incorrect estimate
- Quickness — it is even more important than accuracy
- Relative comparison — e.g. points are used instead of days or dollars and compared to each other
Kanban and Scrum are two popular methodologies for agile development.
Kanban is a method of visualization. The whole workflow is organized on a Kanban board, enabling team members to track progress and see new features or difficulties as soon as they occur. Thus, the team can decide on the appropriate workload.
In Kanban, the estimation of the item duration is optional. After completing one item, team members just proceed with another one from the backlog. Some teams still prefer carrying out the estimation using a simple method: they ensure that all items are of equal size, therefore they can be completed in an equal period of time.
Scrum is a framework for structuring the process. Using Scrum, developers can break their work into sprints. At the beginning of each sprint, they make plans, estimate the size of stories, manage priorities and forecast releases. At the end of each sprint, working software is delivered.
Here is a list of some popular agile estimation techniques:
- Planning Poker
- Affinity Mapping
- Bucket System
- T-Shirt Sizes
- Dot Voting
- Ordering Protocol
There are also methods for estimating the size and effort of test automation, for instance, using Katalon Studio.
For the estimation of any testing project, we highly recommend following these rules:
- Base the estimation of the software requirements
- Make use of the data, experience and knowledge from previous projects (it helps get results cheaper and faster, up to 25%)
- Add reasonable buffer time
- Consider possible internal and external risks (downtime, skill improvements, lack of resources, etc.) and the complexity of the application
- Consider resources availability and team knowledge and skills
- Update the estimation if there are changes in the project
- Verify the estimation and ask an outside expert to review it
Ensure the Quality of Your Project
Estimation is an essential part of the test management that lays the foundation for the testing schedule and the budget. Different estimation techniques in software engineering are used through the whole software testing lifecycle and help efficiently run the project and accomplish it in due time. Certainly, there can’t be one right and appropriate technique for estimation, as every task has its own requirements and conditions.
The available methods are not fully accurate and can provide only approximate values. But even the most experienced professional isn’t able to give a completely spot-on forecast, as there are always unpredictable difficulties once the project is underway. Therefore, test case estimation techniques are extremely helpful in obtaining good results and reducing the time and financial expenses.
SaM Solutions is a company with vast experience in software development and testing processes. Our skilled team is able to find an agile approach to every customer. We help our clients to reach the highest standards of quality involving QA specialists in all stages of a project. We provide end-to-end services for setting up testing processes that include planning, estimation, the creation of test scenarios for various environments, execution of manual and automated tests, and providing detailed reports and statistical data. Contact us for more information.
Thanks a lot for the article. Very informative.
Another reason why estimation is very important, is because it allows you to “fail early” – meaning you can identify weak points or potential failures in design before going live. This obviously makes it easier to create a better product prior to launching.
Getting into the agile development/framework part of the article and it would have been very useful if there was a more in-depth comparison of Kanban vs Scrum. Would like to see another blog post, specifically on that topic.
Trying to learn more about FPA and the section in this article was a good primer on the subject. But when it mentioned that “it’s not really accurate,” I was wondering if there’s an optimal alternative or not…
When it comes to calculating the ideal estimate value for a development project (or any project, for that matter), using a three point estimating methodology is a really great way to approach it. Going into the difference between triangular and beta distribution would’ve been nice, though.
Making effective, and efficient, work breakdown structures is something that should be one of the top priorities during the testing phase. Thanks for the blog on this, covered the basics and everything else I was looking for.
My old company was mainly using waterfall development, but now the place I’m at is focused more on an agile process. So the part about delphi vs. agile was interesting, especially when it described the specific agile estimation techniques.
The part about Function Point Analysis (FPA) was insightful. Trying to learn how to approximate the cost of development (early in the development cycle) is challenging but this blog definitely made it easier to understand.
Testing is a vital part of the entire development lifecycle. Good to see this article go in-depth on a lot of areas that other blogs just skip right over (or only briefly mention). The section about three-point estimation was really helpful.
There are four major areas to keep in mind when estimating (at least on our team). Resources, times, skill, and costs are the four major areas we focus on. We also break it down further, but that’s the general way we approach it.
Easily one of the most important parts of test management. In my experience, it comes down to two main questions that will be asked. The first being how long the testing will take, and the second being how much it will cost.
I truly enjoy looking through on this web site , it holds superb content .
thank you for the article. it is very informative.
whoah this blog is excellent i really like reading your articles. Stay up the good work! You already know, many persons are looking around for this info, you could help them greatly.
I really like your writing style, excellent info, thank you for putting up
Thanks for sharing nice information about software testing.