Despite the massive investment of software testing, the different types of testing are relatively ineffective. In other particular, testing typically only identifies from one-fourth to one-half of defects; such as inspections, generally are more effective. Doing a test is one of the main reasons why we usually deliver software with approximately 2 to 7 defects per thousand lines of code. While this may look like an insignificant amount, we provide and place the result as significant software-reliant systems to operation with hundreds or thousands of residual defects. If we count the software as security defects, the rates are even more troubling.
There are significant problems with the efficiency and effectiveness of testing as it is currently performed in practice. In the course of three decades in developing systems and software; it involves the numerous independent of technical assessments in development projects. It is identifying and analyzing the testing related problems that engineers and managers have observed to occur during testing commonly.
We organize these problems into two groups:
- General testing problems that are not a specific type of testing, but apply to all different kinds of test.
- Test type-specific problems that are certain to a single type of testing processes like unit testing, integration testing, and system testing.
8 Categories of General Testing Problems:
- Test planning and scheduling problems often happen when no separate test plan, but rather highly incomplete and superficial summaries. We postpone significant testing until late in the development process, especially on projects using traditional sequential development cycles.
- Stakeholder involvement and commitment problems having the wrong testing mindset is having unrealistic testing expectations and having stakeholders who are committed and supportive of the testing effort.
- Management-related testing problems involve the impact of management. Management can fail to supply test resources or place inappropriate external pressures in one testing. Testing lessons learned are too often ignored, the same with the problems are done project-after-project.
- Test organizational and professionalism problems include an insufficiency of being independent, unclear testing responsibilities, and test proficiency.
- Test process problems often occur when doing tests and engineering processed the poorly integrated. Organizations sometimes approach taken to trial, regardless of the specific needs of the project. Testing may not be prioritized, so the functional testing, black-box system testing, or white-box unit and integration testing may be overemphasized. Testing of components, subsystems, and the system may begin before they are sufficiently mature for software testing.
- Test tools and environments problems include manual testing and software testing tools. There is an insufficient number of test environments. Some of the test environments also have poor quality or inadequate to the actual system that undergo testing. The method and software under test may behave differently during testing than during the operation. Some common problems are the tests were not delivered, or the test software, test data, and test environments were not in under sufficient configuration control.
- Test communication problems primarily involve test documentations. These types of problems often happen when we do not maintain test documents, or communication is concerning about taking place.
- Requirements-related testing problems related to the requirements that should be driving testing. The conditions are ambiguous, missing, incomplete, incorrect, or unstable. Lower-level requirements may be improperly derived from the higher-level sources. Verification methods may be unspecific, and also, the tracing between requirements and tests may be lacking.
Addressing General Software Testing Problems
There are obviously many problems with the way software, and software-reliant systems undergo testing. While the few projects are experiencing all of these problems, many projects exhibit several of them. While these test problems do not guarantee the software an excessive number of residual defects, these problems definitely pose serious risks that must be managed.
As an example of the results, the software testing problem “Wrong Testing Mindset” has been documented with the following information:
Some of the testers and stakeholders have the wrong testing mindset.
Some testers and stakeholders assume that the system or software works. We do testing to demonstrate that the system or software works properly than to determine where and how it fails. There is no testing of input data. Test input includes only values instead of boundary values and some corner case.
A big probability exists that the delivered system or software will contain significant residual defects related to abnormal behavior, and these defects will unnecessarily reduce its reliability and robustness. Customer representatives, managers, and developers will sense the fails of security that the system functions properly.
We teach testers that their job is to verify or prove the system or software how it works. Developers typically conduct their own unit-level testing. Management gave the testers the powerful impression that they don’t want to hear any bad news.
State in the project test plan that the primary goal of testing is finding the defects by causing the system to fail rather than to demonstrate that there are no defects. Provide test training that emphasizes the proper testing mindset. In inclusion to test cases that verify all nominal behavior, emphasize looking for errors where they are most likely to hide.
Inappropriate external pressures and communication concerning testing verbal and written communication occur between testers and stakeholders, including managers, requirements engineers, developers, and acquirers.
Benefits of Using the Catalog of Common Software Testing Problems
The analysis of common testing problems and solutions we can use as training materials is to learn better how to avoid, identify, and understand testing problems and mitigate them. This list can be used to categorize the problem types for metrics collection. They can be a checklist when:
- Producing the test plans and related documentation
- Evaluating contractor proposals
- To evaluate test plans and relevant documentation
- Evaluating the performed test process
- Identifying the test associated risks and their mitigation approaches