ST-Lecture02[5057313].pdf

Testing in the Software Life Cycle By: Dr. Keyvan Mohebbi  The General V-Model 2  Each of the software life cycle models implies certain views on software testing  From the viewpoint of testing, the general V-model [Boehm 79] plays an especially importan...

Testing in the Software Life Cycle By: Dr. Keyvan Mohebbi  The General V-Model 2  Each of the software life cycle models implies certain views on software testing  From the viewpoint of testing, the general V-model [Boehm 79] plays an especially important role  The main idea behind the general V-model is that development and testing tasks are corresponding activities of equal importance  The General V-Model… 3  The General V-Model… 4  The left branch represents the development process, including:  Requirements definition: The needs and requirements of the customer are gathered, specified, and approved. Thus, the purpose of the system and the desired characteristics are defined  Functional system design: maps requirements onto functions of the new system  Technical system design: designs the system architecture  Component specification: defines each subsystem, including its task, behavior, inner structure, and interfaces to other subsystems  Programming: Each specified component is coded in a programming language  The General V-Model… 5  For each specification and construction level, the right branch of the V-model defines a corresponding test level. It includes:  Component test: verifies whether each software component correctly fulfills its specification  Integration test: checks if groups of components interact in the way that is specified by the technical system design  System test: verifies whether the system as a whole meets the specified requirements  Acceptance test: checks if the system meets the customer requirements, as specified in the contract and/or if the system meets user needs and expectations  Testing Aspects 6  Every test contains both of the following aspects:  Verification:  Does a product fulfill its specification?  Is the system correctly built?  Validation:  Does a product solve the intended task?  Is it the right system?  Component Test 7  Test basis include:  Detailed design  Code  Data model  Component specifications  Typical test objects:  Components, units or modules  Code and data structures  Classes  Database scripts  The software components are tested individually and isolated from all other software components (to prevent external influences on components)  In order to perform the component test, the tester may use a test driver (i.e., a program that calls the component under test and then receives the test object’s reaction)  Component Test-Test objectives 8  Testing the functionality  The input/output behavior of the test object  Testing robustness  Focuses on items either not allowed or forgotten in the specification (negative tests)  The component’s reaction should be an appropriate exception handling  Component Test-Test objectives… 9  Efficiency test  how efficiently the component uses computer resources (such as use of memory, computing time, disk or network access time, and the time required to execute the component’s functions and algorithms)  In contrast to most other nonfunctional tests, a test object’s efficiency can be measured (e.g., memory usage in kilobytes, response times in milliseconds,…)  Efficiency tests are seldom performed for all the components of a system. Efficiency is usually only verified in efficiency-critical parts of the system or if efficiency requirements are explicitly stated by specifications (e.g., in testing embedded or real-time systems)  Maintainability test  Includes all the characteristics of a program that have an influence on how easy or how difficult it is to change the program or to continue developing it  The following aspects are most important for testing maintainability: code structure, modularity, quality of the comments in the code, adherence to standards, understandability, and currency of the documentation  Component Test-Test Strategy 10  The tester usually has access to the source code, which makes component testing the domain of white box testing  The tester can design test cases using her knowledge about the component’s program structures, functions, and variables  In reality, component testing is often done as a black box testing (e.g., if software system consists of countless elementary components)  Test first or Test-driven development (TDD) is a modern approach in component testing  The idea is to design and automate the tests first and program the desired component afterwards  This approach is iterative. The program code is tested with the available test cases. The code is improved until it passes the tests  Integration Test 11  Test basis include:  Software and system design (Architecture)  Sequence diagrams  Interface and communication protocol specifications  Use cases  Workflows  External interface definitions  Typical test objects:  Subsystems  Databases  Infrastructure  Interfaces  APIs  Microservices  Integration Test… 12  Test objects for the integration test are divided into: 1. Assembled components The most important test objects of integration testing are internal interfaces between components 2. External systems (e.g., database systems, networks, hardware) 3. Acquired components  During integration testing, additional tools, called monitors, are required  Monitors are programs that read and log data traffic between components  Monitors for standard protocols (e.g., network protocols) are commercially available  Special monitors must be developed for the observation of project- specific component interfaces  Integration Test-Different Levels 13 1. Component integration tests or integration test in the small will test the interfaces between internal components or between internal subsystems 2. System integration tests, higher-level integration testing, or integration testing in the large focus on testing interfaces between different systems and between hardware and software  Integration Test… 14  Can the component test be omitted?  Because there is no suitable access to the individual component, some failures cannot be provoked and many faults, therefore, cannot be found  If a failure occurs in the test, it can be difficult or impossible to locate its origin and to isolate its cause  Integration Test-Test objectives 15  Wrong interface formats  Problems can arise when attempting to integrate two single components with incompatible interface formats  Typical faults in data exchange  A component transmits syntactically incorrect or no data. The receiving component cannot operate or crashes  The communication works but the involved components interpret the received data differently  Data is transmitted correctly but at the wrong time, or it is late (timing problem), or the intervals between the transmissions are too short (throughput, load, or capacity problem)  Nonfunctional tests may also be executed during integration testing, if attributes such as reliability, performance, and capacity are important  Integration Strategies 16  In which order should the components be integrated in order to execute the necessary test work as efficiently (i.e., as quickly and easily) as possible?  Different software components are completed at different times, weeks or even months apart  No project manager or test manager can allow testers to sit around and do nothing while waiting until all the components are developed and they are ready to be integrated  Integration Strategies… 17 General integration strategies  Top-down integration: The test starts with the top level component of the system that calls other components but is not called itself  Stubs replace all subordinate components  Successively, integration proceeds with lower level components  The higher level that has already been tested serves as test driver  Advantage: Test drivers are not needed or only simple ones are required, because the higher–level components that have already been tested serve as main part of the test environment  Disadvantage: Lower level components not yet integrated must be replaced by stubs. This can be very costly  Integration Strategies… 18  Integration Strategies… 19  Integration Strategies… 20 General integration strategies  Bottom-up integration: The test starts with the elementary system components that do not call further components, except for functions of the operating system  Larger subsystems are assembled from the tested components and then these integrated parts are tested  Advantage: No stubs are needed  Disadvantage: Higher-level components must be simulated by test drivers  Integration Strategies… 21  Integration Strategies… 22 General integration strategies  Ad hoc integration: The components are being integrated in the (casual) order in which they are finished  This means, as soon as a component has passed the component test, it is checked whether it fits with another already tested component, or if it fits into a partially integrated subsystem  If so, both parts are integrated and the integration test between both of them is executed  Advantage: This saves time, because every component is integrated as early as possible into its environment  Disadvantage: Stubs as well as test drivers are required  Integration Strategies… 23 General integration strategies  Non incremental integration – big-bang-integration: The integration waits until all software components have been developed. Then, everything is thrown together at once  Disadvantages: The waiting time for the big-bang to finish is a lost in test execution time It will be very difficult and time-consuming to localize and correct defects  Integration Strategies… 24  Integration Strategies… 25  Top-down or Bottom-up integration in their pure form can only be applied to program systems that are structured in a strictly hierarchical way (In reality, this rarely occurs)  So, a more or less individualized mix of the above mentioned integration strategies may be chosen  System Test 26  System testing checks if the integrated product meets the specified requirements  In the lower test levels, the testing was done against technical specifications  The system test, looks at the system from the perspective of the customer and the future user  The test basis includes all documents such as system requirements, specifications, risk analyses, …  System Test… 27  The system test tests the system as a whole in an environment as similar as possible to the intended production environment  Instead of test drivers and stubs, the hardware and software products that will be used later should be installed on the test platform (hardware, system software, device driver software, networks, external systems, etc.)  The system test not only tests the system itself, it also checks system and user documentation, like system manuals, user manuals, training material, and so on  Testing configuration settings as well as optimizing the system configuration during load and performance testing must often be covered  System Test-Test objectives 28  System test validates whether the complete system meets the specified functional and nonfunctional requirements  Problems in System Test Practice 29 1. If there are no requirements, then all behaviors of a system would be valid and assessment would be impossible 2. While the testers identify the original requirements, they will discover that different people may have completely different views and ideas on the same subject  Acceptance Test 30  The focus is on the client’s perspective  The acceptance test may be the only test that the clients are actually involved in or that they can understand  They may even be responsible for this test  Acceptance tests may also be executed as a part of lower test levels or be distributed over several test levels  Acceptance Test… 31  The test basis for acceptance testing can be any document describing the system from the user or customer viewpoint, such as:  User or system requirements  Use cases  Business processes  Risk analyses  Forms  Reports  Laws and regulations  Descriptions of maintenance  Acceptance Test… 32  There are four typical forms of acceptance testing: 1. Contract acceptance testing 2. User acceptance testing 3. Operational acceptance testing 4. Field testing (alpha and beta testing)  Contract Acceptance Testing 33  If customer-specific software was developed, the customer will perform contract acceptance testing (in cooperation with the vendor)  Based on the results, the customer considers whether the software system is free of deficiencies and whether the service defined by the development contract has been accomplished and is acceptable  The test criteria are the acceptance criteria determined in the development contract  User Acceptance Testing 34  Such a test is especially recommended if the customer and the user are different  It is necessary to organize a user acceptance test for each user group  Because the acceptance test is too late, to prevent any disaster, it is advisable to let a number of representatives from the user groups examine prototypes of the system early  Operational Acceptance Testing 35  It assures the acceptance of the system by the system administrators  It may include testing of:  Backup and restore  Installing, uninstalling and upgrading  Recovery  User management  Data load and migration tasks  Checks for security vulnerabilities  Performance testing  Field Testing 36  If the software is supposed to run in many different operational environments, it is very expensive or even impossible for the software producer to create a test environment for each of them. In such cases, the software producer may choose to execute a field test  The objective of the field test is to identify influences from users’ environments that are not entirely known or specified and to eliminate them if necessary  Field Testing… 37  This test is especially recommended for the COTS systems  The producer delivers the software to preselected customers  These customers then either run test scenarios prescribed by the producer or run the product on a trial basis under realistic conditions and give feedback to the producer  Such testing of preliminary versions by representative customers is also called alpha testing or beta testing  Alpha tests are carried out at the producer’s location  Beta tests are carried out at the customer’s site  Field Testing… 38 Dogfood Test  A new term (by Microsoft and Google) that refers to a kind of internal field testing where the product is distributed to and used by internal users in the company that developed the software  “if you make dogfood, try it yourself first”  Maintenance Testing 39  The first deployment marks only the beginning of the software life cycle  Once it is installed, the software will often be used for years and is changed, updated, and extended many times  The test strategy for testing a changed system is the same as for testing every new product version:  Every new or changed part of the code must be tested  Additionally, in order to avoid side effects, the remainder of the system should be regression tested  Maintenance testing can be difficult because the system’s specifications are often out of date or missing, especially in the case of legacy systems  Testing in Iterative and Incremental Development 40  After each release, the project starts over, running through all the phases  Do we have to completely rerun all the test levels for every release of the product?  Yes, if possible!  As with maintenance testing, anything new or changed should be tested, and the remainder of the system should be regression tested  Types of Testing 41  Different types of testing can be distinguished on each test level: 1. Functional testing 2. Nonfunctional testing 3. Testing of software structure 4. Testing related to changes  Functional Testing 42  Functional requirements describe “what” the system must be able to do  Functional testing includes all kind of tests that verify a system’s input/output behavior (mainly used in system testing and other higher levels of testing)  To design functional test cases, the black box testing methods are used  Functional Testing Methods 43  Requirements-based testing uses the final requirements as the basis for testing  For each requirement, at least one test case is designed and documented in the test specification  Business-process-based testing or use-case-based testing are other functional testing methods  Requirements-based testing focuses on single system functions. Business-process-based testing, however, focuses on the whole process consisting of many steps  Nonfunctional Testing 44  Nonfunctional requirements describe “how well” or with what quality a function or the system should work  Some of the nonfunctional tests (usually in system testing) include:  Load test: Measuring of the system behavior for increasing system loads (e.g., the number of users that work simultaneously, number of transactions)  Performance test: Measuring the processing speed and response time for particular use cases, usually dependent on increasing load  Volume test: Observation of the system behavior dependent on the amount of data (e.g., processing of very large files)  Stress test: Observation of the system behavior when the system is overloaded  Checking of the documentation: For compliance with system behavior (e.g., user manual and GUI)  Testing of Software Structure 45  Structural techniques (structure-based testing, white box testing) use information about the test object’s internal code structure or architecture  Typically, the followings are analyzed:  control flow in a component  the call hierarchy of procedures  the menu structure  the structure of abstract models of the software (finite state machine)  Structural techniques are most used in component and integration testing  Testing Related to Changes 46  When changes are implemented, parts of the existing software are changed or new modules are added  Tests must show that earlier faults are really repaired  Additionally, there is the risk of unwanted side effects  Repeating other tests in order to find them (regression testing)  Regression testing may be performed at all test levels and applies to functional, nonfunctional, and structural test  How much retest and regression test 47  The question is how extensive a regression test has to be. There are the following possibilities: 1. Rerunning of all the tests that have detected failures whose reasons (the defects) have been fixed in the new software release (defect retest, confirmation testing) 2. Testing of all program parts that were changed or corrected (testing of altered functionality) 3. Testing of all program parts or elements that were newly integrated (testing of new functionality) 4. Testing of the whole system (complete regression test)