Software Testing PDF

Introduction to software testing Software testing overview Introduction and welcome Welcome to Software Testing! I'm Matt Rudge, and along with Anna Greaves, we’ll guide you through your first steps into software testing. In this unit, you'll learn how to create and implement a test strategy to ensure software works as required, and how to produce test reports to address any issues. The unit has two sections: 1. Theory: Anna will cover what testing is, why it's needed, common software issues, core testing principles, different testing types, and how to validate user input in forms. 2. Practical: I’ll reintroduce Python and demonstrate white box and black box testing. You'll write tests, solve coding challenges, and build code. By the end, you’ll be challenged to write your own tests. Ready? Click "Next" to begin! _________________________________________________________________________ Why do we need testing? What is it? An introduction to testing and its importance. What does it do? Shows what can go wrong when something isn’t tested properly. Why do you need it? To grasp the importance of testing. Why do we need testing? Testing means checking how well something works. Examples: Car safety tests (e.g. MOT) check brakes and lights. Online forms to ensure data is processed correctly. Quizzes to test software development knowledge. Before exploring software testing, consider a real-world example where testing was inadequate. Challenge Approximate time: 10 minutes Watch a video about the Tacoma Bridge collapse. What caused it? How could it have been prevented? If identified early, could the issue have been fixed? Explanation Approximate time: 5 minutes The Tacoma Narrows Bridge collapsed due to wind twisting the structure. If wind tests had been done on a model, the issue could have been spotted and corrected, saving millions. This case highlights why thorough testing is crucial to avoid unforeseen problems. _________________________________________________________________________ Types of software defects What is it? An introduction to software errors. What does it do? Errors and bugs affect the usability of software-based products and services. Why do you need it? To consider potential issues when creating or using software. What can go wrong? (Types of software defects) Software consists of programs and operating information used by a computer. As computers only follow programmed instructions, they can encounter bugs and errors. Software testing identifies potential problems so they can be fixed. Challenge Approximate time: 20 minutes Think about five instances where software you use daily (e.g. phone apps, TV, laptops, banking systems) didn’t work properly. Write down what happened each time. Explanation Approximate time: 20 minutes Here are five main types of software defects: 1. Functional error Occurs when software doesn’t perform its intended function. ○ Example: A cancel button not working on a booking form. 2. Crash Software shuts down unexpectedly. ○ Example: A navigation app closing in the middle of a drive. 3. Acknowledgement message error You receive incorrect or no feedback from the system. ○ Example: No confirmation email after signing up for a service. 4. Calculation error Incorrect results from software calculations. ○ Example: A total cost not adding up correctly on a travel website. 5. Control flow error The software skips or repeats steps incorrectly. ○ Example: A game jumping from level one to level four without following the sequence. These are common errors, but there are many other types that will be covered later in the module. _________________________________________________________________________ Verification and validation What are they? The two main activities of software testing: Verification and Validation. What do they do? Verification ensures the software is built correctly (does what it’s supposed to do). Validation ensures the right software is built (meets user needs). Why do you need it? To ensure software is usable, functions as intended, and satisfies both technical and user requirements. Verification and validation Software testing checks that software: Works efficiently and effectively. Matches expected technical and user requirements. Benefits: Saves time and money by identifying and fixing problems early. Ensures software is secure so users trust their data is safe. Improves user experience by ensuring the software works properly. Verification Asks: "Are we building the software correctly?" Ensures software meets technical specifications. Validation Asks: "Are we building the correct software?" Ensures the software meets user expectations and is fit for purpose. Both methods ensure product quality across industries. For instance, the Titanic's rivets were made of cheap metal, contributing to its sinking. Proper verification could have prevented this issue. Challenge Approximate time: 15 minutes Test a supermarket checkout machine: Test 1: Scan a product to check if the software registers it. Test 2: Ask staff if the checkout machine is easy to use. Which test is verification? Which is validation? Explanation The main goal of software testing is to uncover errors or any missing defined requirements in software. But software testing is more than finding defects. It also aims to improve the software in terms of usability, efficiency and accuracy. Verification checks if software meets technical requirements.Verification testing can be done throughout the process of building software. ○ In Test 1, scanning a product verifies if the system registers it correctly. Validation refers to the set of tasks and activities that ensure the software meets the functional or user requirements.Validation testing is often done after the software has been completed, so that it is fully available for a user to test. ○ In Test 2, testing usability by staff validates if the system is easy to use. Test your knowledge Approximate time: 15 minutes For an e-book reading device, determine if the following are verification or validation: 1. Forward/backward buttons flip pages correctly. 2. Backlight is gentle on users' eyes. 3. Device switches between books in under 0.1 seconds. 4. Device is comfortable to hold. Solution 1. Buttons flipping pages correctly: Verification (meets technical requirement). 2. Backlight being gentle: Validation (user experience). 3. Switching books quickly: Verification (meets performance requirement). 4. Comfort of the device: Validation (user feedback needed). _________________________________________________________________________ Non-technical skills that software testers gain What is it? Non-technical skills that software testers gain. What does it do? Provides advice on the skills you will learn and use as a software tester. Why do you need it? Knowing the non-technical skills you will learn is useful. Role of a software tester: To ensure that a software product works as expected for users. They work alongside software developers, who design and code the software and fix errors identified by testers. Team Size: Varies by company size; in small companies, there may be one developer and one tester, while in large companies, separate teams must coordinate. Technical Skills Required: Software testers need technical skills such as software development skills and programming implementation knowledge. Key Non-technical Skills for Software Testers: 1. Analytical, reasoning, and logic skills: Software testers have good reasoning, analysing, and logic skills. When a tester uses these skills during testing, they can look for errors, understand complex systems, assess, and test any unusual application behaviour. 2. Communication skills: Software testers have good verbal and written communication skills. Test reports created by the software tester should be straightforward and easy to understand, enabling the software development team to fix errors. 3. Time management and organisational skills: Necessary for working within tight deadlines and effectively managing workloads. 4. A positive attitude: Vital for maintaining morale, as testers find flaws in colleagues' work while fostering a supportive team environment. 5. Creative thinking: Enables testers to think of multiple ways a product could fail, enhancing product robustness. Less Useful Skills: Teaching, project management, and graphic design skills are generally not as relevant in a software tester's role. Test Your Knowledge: Identify useful non-technical skills to avoid conflict when a testing team harshly presents defects to developers, damaging communication. Solution: Better communication skills and a positive attitude would have prevented this situation. _________________________________________________________________________ Seven principles of software testing What are they? The seven principles of software testing. What do they do? They provide advice on how to approach testing. Why do you need them? To refer to when building a software solution. Thorough software testing can be difficult and time-consuming. Software development teams aim to maximise their testing budget efficiency. The seven principles of software testing offer general guidelines to help software developers recognize the limits and potential flaws in testing processes. Explanation Approximate time: 15 minutes If you selected all the statements, well done! Each relates to one of the seven principles of software testing defined by the International Software Testing Qualifications Board (ISTQB) to guide testers in best practices. 1. Testing shows the presence of defects, not their absence: Detecting defects does not mean all have been found; users may encounter defects after the software is deployed. 2. Exhaustive testing is impossible: For every defect, there can be an almost infinite number of conditions that might cause it. For example, a password input field that throws an error can have millions of character combinations. Thus, it is impossible to run tests for every potential condition due to the cost and time involved. The cost and time required to conduct such exhaustive testing would become impractical for realistically creating any software. 3. Early testing saves time and money: Early testing in the software development life cycle is both time and cost-effective. Identifying and fixing issues during the coding process, or even before coding starts, is far quicker and less expensive than addressing them at the end of the life cycle. 4. Defects cluster together: Defect clustering in software testing means that a small part of the software contains a lot of problems.Typically, 80% of problems come from 20% of the software. So software testers spend a lot of time testing the 20% of the software that has the most problems. 5. Beware of the Pesticide Paradox: Similar to pesticides losing effectiveness over time, repeating the same tests during development may stop uncovering new defects. To detect new issues, existing tests and test data must be updated, and new tests may need to be created. 6. Testing is Context-Dependent: Different software types require unique testing approaches. For example, testing a banking application differs from testing an e-commerce or advertising application. Each carries specific risks, so a one-size-fits-all approach to testing is inadequate. 7. Absence of Errors Fallacy: It is a fallacy to believe that finding and fixing numerous defects guarantees software success. The software might work perfectly but, if it is difficult for the user to operate, does not fulfil their needs and expectations, or it is inferior compared to other software, then it will still go unused. Test Your Knowledge A team of software testers is working on an online shop for selling t-shirts. Below are case study examples. For each, identify which of the seven principles of software testing the testers should have remembered: 1. Case Study 1: The testing team created a detailed plan and identified hundreds of defects, all fixed by developers. Upon launch, users struggled with navigation and opted for a competitor. 2. Case Study 2: A software tester tests the password field for a login form by attempting to list every potential password. After a month, he still has trillions of combinations left to consider. 3. Case Study 3: The software testing department creates a set of detailed tests to test for all of the software’s requirements and functionality. These tests are re-run each time developers make fixes, but they are not updated to check for new issues. Consequently, the tests find no further problems, yet issues remain when the software is launched. 4. Case Study 4: The tester waits until development is complete before beginning testing. This leads to the discovery that users cannot create login accounts, requiring a rewrite of the login functionality and causing significant project delays. 5. Case study 5: A software tester has created and run a test plan, she discovered 10 defects. The software tester states that her tests prove that she has found all of the defects in the software, but this is not the case. 6. Case study 6: A software tester spends an equal amount of time testing each feature of the website. Even though the payment functionality of the software has been very problematic for the software developers to create, because it accepts many different types of payment. 7. Case study 7:A software tester tries to save time by reusing the test plan he made for a different piece of software he tested a year ago. The previous software was for an online banking application. As a result, the testing plan does not check many of the features of the t-shirt shop software were working, and many defects are missed. Solution 1. Case Study 1: Absence of errors fallacy 2. Case Study 2: Exhaustive testing is impossible 3. Case Study 3: Beware of the pesticide paradox 4. Case Study 4: Early testing saves time and money 5. Case Study 5: Testing shows the presence of defects, not their absence 6. Case Study 6: Defects cluster together 7. Case Study 7: Testing is context-dependent _________________________________________________________________________ Software testing life cycle The software development life cycle What is it? The stages that the creation and maintenance of software goes through. What does it do? Provides a guide to the phases of successful software development. Why do you need it? Use it as a guide to determining the activities and phases employed when creating software. Software Development Life Cycle (SDLC) This lesson covers two life cycles: Software development life cycle (SDLC). Software testing life cycle. These are interlinked as testers and developers collaborate closely. The SDLC is a structured method that ensures software is of high quality and meets client requirements. It provides a step-by-step guide for designing, developing, and maintaining software, with each phase having specific processes and outcomes. Challenge Approximate time: 15 minutes Take a look at the image that lists the phases of the software development life cycle: ○ What are the six phases called? On a new project, which phase would begin the software development process? What do you think each phase might involve? ○ Why do you think the process is cyclical (i.e. goes round in circles)? ○ Why do you think they are in that particular order? Explanation Approximate time: 15 minutes While there are variations, the most common approach to the software development life cycle includes six phases: 1. Requirement gathering and analysis Before building software, developers need to have a clear picture of what it is they intend to build and how it will work. During this initial phase a list of requirements is collected to define things about the software such as: ○ What it is for. ○ How it should behave. ○ What type of users it is for. Once the requirements are gathered and put into a requirements document, they are analysed to see if the software can be developed. The analysis tries to determine the following: ○ The technical feasibility to build the software. ○ If the project is cost-effective. ○ If there is a team available to create the software. ○ If the project can be completed on time. 2. Design During the design phase, high level decisions are made about how the software will work. This will include defining: ○ Technical specifications such as what programming languages will be used. ○ How much the software will cost to make. ○ What kind of resources will be needed. ○ Which features should be worked on first, and which can be left for a later release. 3. Implementation Once the design phase is complete, the implementation phase begins. During implementation, software developers begin writing the code to build the software. The work is divided into small sections called features, and these are shared among the software developers. This is the longest phase in the software development life cycle. 4. Testing In this phase, testing is performed on the software until it meets the client’s requirements. In order to ensure that the software meets expectations, testers consult the documents created at the requirement gathering and analysis phase. While testing commonly follows implementation we will see later in the unit that testing can also begin during the implementation phase. This relates to the early testing saves time and money principle of software testing. 5. Deployment Once the software is tested to the point where the requirements are met, it is released for actual use out in the world as a fully functioning product. Releasing software is often referred to as deployment. 6. Maintenance After software has been released the developers take care of any issues that are identified after deployment, as well as any new features that may be needed. Cyclical Process The need for new features discovered during the maintenance phase often triggers the beginning of a fresh round of the software development life cycle. So the circle of phases may begin again with a new requirement gathering and analysis phase. The order of the cycle The phases of the software development life cycle are in this specific order because each phase leads into and informs the next phase. Following this process ensures that the goals of the project are clear and measurable, and that these goals are carefully tested and checked as the software is built. Test your knowledge Approximate time: 15 minutes Read through the examples below, which phase of the software development life cycle does each one belong to? ○ The team decides which features should be worked on first, and which can be left for a later release. ○ The team collects information about who and what the software is for. ○ After the software is released, an issue is identified by a user and this is assigned to a developer to fix. ○ The software developers build the code to create the software. ○ The software is released to the world as a fully functioning product. ○ The software testers check that the software meets the defined requirements. Solution 1. Design. 2. Requirement gathering and analysis. 3. Maintenance. 4. Implementation. 5. Deployment. 6. Testing. ______________________________________________________________________ The software testing life cycle The software testing life cycle is a set of actions and tasks done during the testing phase of the software development life cycle. Both verification and validation are part of the software testing life cycle.The software testing life cycle is made up of six phases. Phases of the Software Testing Life Cycle (STLC) 1. Requirements Analysis ○ Testers refer to the requirements defined in the SDLC to determine necessary tests. ○ Involves identifying and documenting: The goals of the testing process. Which tests are needed for each software requirement. The types of tests required and how they will be executed. 2. Test Planning ○ During this phase, software testers use the requirements analysis document to create a test plan. ○ The test plan sets out: The testing budget. The timeframe or project deadline. Which resources are available for the testing process. A testing schedule. Any testing limitations. 3. Test Case Development ○ Once the test plan has been created, software testers begin developing the tests. They also create the data they will use to run their tests. ○ Once the tests and data are ready, the environment setup phase can begin. 4. Test Environment Setup ○ A testing environment is where the software can be tested in a way that will not permanently break anything for the user. ○ You can think of a testing environment like a laboratory, separated away from "real life" conditions, where the tests can be carefully controlled. ○ A software testing environment separates the software code away from the user and any data they might interact with, so that it can be carefully tested in a controlled setting. ○ During the environment setup phase, the software testers: Prepare the test environment with the required hardware and software. Ensure that the environment works as expected. Set up the tests and test data in the test environment. 5. Test Execution ○ During this phase, the tests are run on the software within the testing environment. ○ The process of running the tests includes: Running the test code. Creating a test report that outlines any bugs and issues that have been found. Sending the test report to the software developers so that the identified issues can be fixed. Retesting any features that have been fixed by the development team. 6. Test Cycle Closure ○ This final phase involves: Reviewing findings from the testing process. Analysing test results to identify severe issues. Evaluating the success of the testing cycle based on goals defined in the Requirements Analysis. Reporting results to the client. Order of the Cycle The phases are structured in a specific order; each phase leads into the next, ensuring that goals and requirements are clear and measurable throughout the STLC. Test Your Knowledge Read through the examples below. Which phase of the software testing life cycle does each one belong to? Write the code for the tests and design the test data. Review the information gathered from the testing process. Identify which tests are needed for each of the software requirements. Create a test report for the software developers. Set up the tests and test data in a separate location where the testing conditions can be controlled. Create a testing schedule. Solution Test case development. Test cycle closure. Requirements analysis. Test execution. Environment setup. Test planning. ______________________________________________________________________ Test strategies Testing visibility The two approaches to test visibility What is it? An introduction to black box and white box testing. What does it do? Black box testing only considers the external behaviour of software. White Box testing considers the internal functioning of software. Why do you need it? To help make decisions about how to test software. When testing a software system, it is sometimes referred to as a "box". Outside the box: This is everything that the user interacts with (e.g., buttons, navigation, input fields). For instance, when using a website, the navigation buttons, dropdowns, and forms you use are outside the box. Inside the box: Refers to the internal functioning of the software, including the code and logic that create the features the user interacts with. Software tests can have one of two approaches to visibility, depending on if the test will focus on what is inside or outside of the box: The black box approach: The box is treated as solid, and what is inside it cannot be seen. The test does not peek inside the box and is only concerned with the external interface that the user interacts with. The white box approach: The box is treated as see-through or open where everything inside it can be seen. The test inspects the software code inside the box in detail, and the test is created around how the code was built. Challenge: Imagine you have a calculator that you want to test. Consider how many possible ways you can calculate the result of 4 from the inputs of 2 and 2 (e.g., 2 + 2 = 4). If we test a calculator using black box testing, all three outputs would pass the test, as the result (4) is the same for each. Black box testing only checks the final result. However, with white box testing, passing the test requires not only the correct result but also that the correct method was used to calculate it. For example, although 2+2, 2 x 2, and 22 all give the answer of 4, when we use the + symbol, it would be expected that an add() function was used. When the x symbol is used, it would be expected that the multiply() function was used, and so on.The key distinction is that white box testing focuses on how the results are produced, while black box testing is happy as long as the correct result of 4 is obtained.In the coming topics, we will discuss the processes and advantages and disadvantages of both black and white box testing.In practice, it is best to use a combination of both approaches to ensure software is as defect-free as possible. _________________________________________________________________________ Black box / Functional testing What is it?A type of test that only considers the external functionality of software.What does it do?Tests the results from the software for the user, without considering the code and logic within the software.Why do you need it?So that you can verify that the software solution meets the user requirements. Black box testing is sometimes referred to as functional testing, because this testing approach is concerned with how the software functions from the user's perspective. Challenge Approximate time: 20 minutes An alarm clock contains several components such as the clock display, alarm and a snooze button, as well as the ability to change times, and dates etc. Imagine you had to identify any defects in this alarm clock without using tools. 1. How would you get information about how the alarm clock should work? For example, if you bought this alarm clock new from a shop, how would you find out how to operate it? 2. How would you test the ways that the clock should work? 3. If the result of a test produces a different result than was expected, does this mean you have found a defect? Explanation Approximate time: 15 minutes To test an alarm without tools you would: 1. Refer to the alarm clock manual, because it would list all the features of the clock and how they are supposed to work. 2. By trying to replicate the functionality of the clock by pressing buttons and doing other tasks. 3. When the result of these tasks appear different than expected, then you know the clock is not working as it should. Black box testing Similar to the alarm clock example, before performing black box, software testers must first understand how the software is meant to help its users with their needs. So, the black box process begins by first referring to the requirements document, which lists the functionality that the software is required to have. For the alarm clock example above, the requirements document was the clock manual. The tester would then test the functionality of the software based on each requirement. Advantages of black box/functional testing 1. Black box testing is fast because it only focuses on interaction with the user. 2. A black box software tester does not require a lot of technical knowledge, because the tests are carried out from the perspective of the user. 3. The same tests can be used even if the software code that runs it needs to be significantly changed. 4. It is effective in finding the defects that matter to users. Disadvantages of black box/functional testing 1. Black box testing has difficulty identifying rare situations that might cause severe defects. 2. Without precise functional specifications, it can be difficult to design good tests. 3. There is a high probability of repeating tests already performed by the programmer. 4. It can be hard to find the cause of a defect identified with black box testing, as it only identifies an issue, not where it lies within the software code. Manual vs Automatic black box testing It should be noted that all manual tests are black box tests. However, similar to using a robot to test the alarm clock example above, it is also possible to simulate inputs from users by using automated tests. These types of tests would require more technical skills to implement. _________________________________________________________________________ White box / Structural testing Unlike black box testing, white box testing is concerned with how the code within the software works. It is often referred to as structural testing. Structural testing tests the logic and structure of software code itself. Imagine again that you had to identify defects in an alarm clock, but this time you have tools to take it apart. 1. How would you get information about how the individual alarm clock components should work? 2. How would you test the ways that the clock components should work? 3. If the result of a test produces a different result than was expected, does this mean you have found a defect? Explanation (Approximate time: 15 minutes) To test an alarm with tools, you would: Refer to the manual to understand how the clock components should function. Use mechanical skills to remove individual components from the clock. Apply technical skills to test key components (e.g., clock display and alarm) and the interaction between them. White Box/Structural Testing Unlike black box testing, white box/structural testing requires technical knowledge of how the software code works. White box software tests are programs, i.e. code, written specifically to test software. Therefore, white box testing must be done by testers with software development experience. Similar to black box testing, the white box testing process also begins by referring to the requirements document, where the software tester can understand the overall design and purpose of the software. The tester identifies the testable sections of the software (referred to as components), for example, specific functions that can be tested. For each component, the software tester builds a series of tests and test data using code. These will test for each of the paths the code could follow. When the tests and test data have been written, the software tests are run to check for defects. Any tests that fail will identify defects in the specific components. Advantages of white box/structural testing 1. White box testing helps identify rare situations that might cause severe defects, i.e. edge cases. 2. Because tests focus on one component at a time, it is easier to find the cause of the defect in the software code. 3. Because they are always written with code and in a way that tests individual components, the tests can be re-run quickly and easily any time changes are made to the software. 4. Tests can be run earlier, as the software is being developed, because they can be run on individual components as they are developed with no need for a user interface. Disadvantages of white box/structural testing 1. Creating the tests for white box testing is slow, because they need to be written carefully to check for every path that the software may take. 2. White box testing might not find unimplemented or missing features. 3. White box testing requires technical skills and understanding of how software code works. 4. White box testing requires access to the code being tested. 5. White box tests need to be carefully maintained, they need to be updated any time changes are made to the components they test. Combining the best of both As you have seen, both black and white box testing approaches have advantages and disadvantages. Therefore, it makes sense to combine the use of both types to test software to ensure that the most defects are identified and fixed. _________________________________________________________________________ Test strategies Testing approach Manual testing What is it? Testing software manually by interacting with it as a user would. What does it do? A tester runs black box tests by clicking buttons, adding inputs, and recording the software's responses. Why do you need it? To identify defects that users might encounter. Manual testing is a black box testing method because it involves interacting with the software as the user would. Challenge: You will manually test the software in the provided runnable example. The console serves as the user interface; do not alter the code. 1. Click the Run Code button. 2. Enter the number 2 in the console. What result do you get? Is it correct? 3. Run the code again. 4. Enter the word cat in the console. What result do you get? Is it correct? 5. Test the software with your own inputs. Can you find any additional defects? def getSquare(num): num = int(num) return num * num num = input("Enter a number: ") square = getSquare(num) print(f"The square of {num} is {square}!") Advantages of Manual Testing Manual Testing Does Not Require a Lot of Preparation: Manual testing can be performed simply by using the software in a controlled manner, without the need for a formal test plan. This allows testing to occur whenever necessary. Developers often conduct manual tests from the user’s perspective during the development process, provided there is an interactive interface. Manual Testing Allows Software Testers to Be Creative: Manual testers can be creative and flexible in trying out things that are not in the original list of planned tests. For example, as a software tester runs one test scenario, they might think of a very interesting related scenario and try that too. This approach is called exploratory testing, as it allows software testers to explore all aspects of the software at the same time as they are testing it, and in this way come up with new and unexpected tests. Disadvantages of Manual Testing Manual Testing Can Be Very Time Consuming: Because manual testing involves a human working with the software from the perspective of the user, it is a time-consuming process. Each test case takes time for the tester to push buttons, input values, and record the results. Manual Tests Can Be Difficult to Accurately Repeat: The same test performed at two different times may not be run in exactly the same way. This can cause certain defects to be missed or reported as fixed when they were not. Test Your Knowledge Approx. Time: 20 minutes Click the runnable 2 button under the first runnable example to manually test this code. The software should: 1. Ask the user for a number. 2. Respond with the number the user entered and a statement about whether it is zero, positive, or negative. 3. Not throw any errors for unexpected inputs. Can you identify the four defects in this software? Click the Run Code button to execute the code. Remember to rerun the code each time you want to test a different input. You only need to identify the defects; you do not need to change the code or fix the errors. num = int(input("Enter a number: ")) if num > 0: print(f"{num} is a positive number") else: print(f"{num} is a negative number") if num == 0: print(f"{num} is zero") Solution Approx. Time: 5 minutes The defects identified in the tests are: Entering 0 results in two responses, one incorrect. It should only respond with "0 is zero". Entering a non-numeric value causes the software to throw an error. Entering a decimal number results in an error. Not entering any value also causes an error. _________________________________________________________________________ Automated testing What is it? A method of testing software using tests written with code. What does it do? A tester writes tests with code that run on individual components of the software. These tests can be run automatically and repeatedly.Why do you need it? To carry out white box testing, and create tests that are faster and more precise than manual testing. Automated testing is primarily a white box testing method as it involves direct interaction with the software code at the component level. Testers review the code and write small programs to test individual components. These small test programs can be run all together, at any time. Each time the test programs are run, the tester can see if any of them fail. More tests can be added to the test programs at any time. Challenge Approximate time: 15 minutes The code in the runnable example contains an automated test. 1. Click the Run Code button; what result do you see in the console? 2. Can you find the test in the code? What is the name of the function it is testing? 3. What do you think the assert keyword is for? def getSquare(num): num = int(num) return num * 2 def test(): result = getSquare(4) assert result == 16, f"Expected the result of getSquare(4) to be 16, got: {result}" test() Explanation Approximate time: 15 minutes When we run the code, we can see the following error: "AssertionError: Expected the result of getSquare(4) to be 16, got: 8". The code in the runnable example contains an automated test that begins on line 5. This test calls the getSquare function with the number 4 and checks that it returns the square of 4, which is 16. If the value returned from the getSquare function is not 16, the assert keyword raises an AssertionError, and the information about the test is provided in the error message. Since the getSquare function does not return the intended result, this constitutes a defect, which the automated test identifies. Advantages of Automated Testing It is fast to run and re-run automated tests Once software testers have written test programs, these can be executed at any time. Hundreds of tests for complex software can be run simultaneously, making it quicker and more efficient to run and re-run tests for the entire software compared to manual testing. Automated tests are consistent and precise Because automated tests are computer programs, they are always run in exactly the same way once created. The tests are precise and exact in what they are looking for, eliminating the potential for human error. A Disadvantage of Automated Testing Automated testing requires preparation Before automated tests can be run, they need to be written, which can be time-consuming to set up. However, once a test is written, it can be executed and re-run as many times as needed. Automated tests also require a comprehensive knowledge of software development practices and the programming language, which adds to the overall budget of a project. Automated Black Box Testing As you learned in the black box testing unit, it is possible to simulate user inputs using automated tests. Automated testing is considered black box testing when the tests: Check functionality for the user. And the tester cannot see the code. Test Your Knowledge Approximate time: 15 minutes How would you fix the code in the getSquare function so that the test would pass? Note: Once the test passes, the console will be blank when you run the code (i.e., you will not see the AssertionError message). Solution Approximate time: 5 minutes There are several ways to fix the code; one example is shown below. Note that the getSquare function now returns the correct value, and we have not needed to make any changes to the test code. def getSquare(num): num = int(num) return num * num # FIX: Changed from `return num * 2` to `return num * num` def test(): result = getSquare(4) assert result == 16, f"Expected the result of getSquare(4) to be 16, got: {result}" test() Note: The two jobs of detecting defects and fixing them are usually performed by separate teams. The software testing team prepares a report listing all the defects found, and then the software development team uses this list to improve their code. ______________________________________________________________________ The two approaches to measuring coverage What Are They? Test coverage and code coverage are the two ways we can measure how much of the software has been tested. What Do They Do? They indicate how much of the software requirements and software code have been covered by tests. Why Do You Need It? It is important to know where there are gaps in your testing coverage. When testing software, it is essential to understand: How much of it has already been tested. How much still requires testing. This is done by measuring coverage. There are two ways that coverage of the software can be measured, depending on whether you are measuring black box or white box testing. Challenge Approximate Time: 15 minutes You have been given social media software to test: The software has 50 requirements. For example, a user can create a new account, or a user can delete a post. The software has 10,000 lines of code. Which would you use to measure black box test coverage? The number of requirements tested. The number of lines of code tested. What about white box testing? Do you think it is more important that: The software has met all of its requirements, i.e., the software can do everything it is supposed to do. Every line of code is tested. Explanation Approximate Time: 15 minutes There are two ways to measure the coverage of the software, depending on whether you are measuring black box or white box testing. Test coverage measures black box tests, and code coverage measures white box testing. Test Coverage Test coverage measures the number of requirements that have been tested. If black box software testers have tested 48 out of 50 requirements, then they have 96% coverage with their tests. Missing two of the software requirements with black box testing would be a significant issue in determining if the software works as expected. However, this can be easily rectified by creating more tests for the missing requirements. Black box testing should aim to test 100% of the requirements. Code Coverage Code coverage measures the total number of lines of code that have been successfully checked by software tests. Therefore, if white box software testers have written tests that check 9,500 out of 10,000 lines of code, then they have 95% coverage with their tests. Missing 5% of the code when running white box tests is unlikely to pose a significant problem in determining if the software works as expected. While white box testing should aim for a high percentage of code coverage, it rarely achieves 100% coverage. It is more important to focus on testing the key parts of the software than to try to reach every line of code with software tests. ______________________________________________________________________ The testing process Project introduction Re-introducing Python What is it? The Python programming language. What does it do? It lets us write code and tests. Why do I need it? It shows how to use software to test software. Now that we’ve covered the theory of software testing, we’ll start writing code. Challenge Approximate time: 15 minutes We’ll build tests using Python’s assert keyword, as shown in Runnable 1: 1. What happens on line 1? 2. What does the assert check on line 3? 3. What will happen when you run it? 4. How would you fix the code so no error shows? 5. What happens after fixing and re-running? my_number = 42 assert my_number == 22, "Number is not 22" print("All tests passed!") Explanation Approximate time: 10 minutes Line 1 assigns 42 to my_number. assert checks if the condition my_number == 22 is True. If false, it raises an AssertionError with the message: “Number is not 22”. To fix: Change my_number to 22, or Change the assert condition to check for 42. After fixing, the code runs without errors, and "All tests passed!" prints to the console. Test Your Knowledge Approximate time: 15 minutes Runnable 2 adds another assert: 1. What’s the second assert checking? 2. What happens when you run it? my_number = 42 assert my_number == 22, "Number is not 22" assert isinstance(my_number, str), "Variable is not a string" print("All tests passed!") Solution Approximate time: 10 minutes The second assert checks if my_number is a string. Since my_number is an integer, the test fails. Python stops at the first failed assert, so the second isn’t run. This is an issue as we want all tests to run. We’ll resolve this later. Review Approximate time: 1 minute assert checks if a condition is true. Python stops after the first failure, which we’ll fix in the next lesson. ______________________________________________________________________ I will try...except What is it? The Python try/except block. What does it do? It "catches" errors without stopping the code. Why do I need it? It will allow us to build a full test suite. The last topic ended on a cliffhanger! How do we get Python to run all of our assert statements? Granted, it is not the most gripping of cliffhangers, but it is important in the context of our tests. Challenge Approximate time: 15 minutes Take a look at the code in runnable example 1: 1. Run the code, what happens? 2. Delete the assert on line 2, run the code again, what happens? 3. What do you think try is doing? 4. What do you think except is doing? 5. If the assertion has failed, why do you think the final print statement has still run? 6. Try fixing the error again; what output is printed to the console? Why is the output different this time? python Copy code my_number = 42 assert my_number == 22, "Number is not 22" try: assert my_number == 22, "Number is not 22" except AssertionError as error: print(f"Fail: {error}") print("All tests have been run.") Explanation Approximate time: 15 minutes As with the previous topic, the assert line has failed, an error has been logged to the console, and the code has stopped. By removing the assert on line 2, we can see that the code has finished running, and two print statements have shown. One shows a string value containing the error text, and the second shows the final print statement. In English if we "try" something, it means that we know we may fail, but we will attempt it anyway. It is the same in Python. The try keyword means: "Attempt to run this bit of code. It may result in an error, but try it anyway." Another name for an error in code is an exception. The except block is executed when an exception occurs in the try block. In our case, we ask Python to: 1. First, try running the assertion. 2. The code will run except if an AssertionError occurs, which is what happens when the assert fails. 3. If an AssertionError occurs, we store the error in a variable called error and then print it to the screen. Given what we now know about try and except, we would expect that the assertion will fail, so the error will be printed, and the code would stop. However, because we used a try/except block, the final print statement will still be executed. As the assertion did not fail, only the final print statement will be executed. Test your knowledge Approximate time: 15 minutes In the previous topic, we had two assert statements. The second one, you will remember was: assert isinstance(my_number, str), "Variable is not a string" Your challenge for this topic is to add the second assert using our new try/except method. Open runnable 2 and add the second try/except block beneath the first one, above the final print statement. my_number = 42 try: assert my_number == 22, "Number is not 22" except AssertionError as error: print(f"Fail: {error}") # Write your try/except block here print("All tests have been run.") Solution Approximate time: 5 minutes The solution is to add the following highlighted lines to the code: my_number = 42 try: assert my_number == 22, "Number is not 22" except AssertionError as error: print(f"Fail: {error}") try: assert isinstance(my_number, str), "Variable is not a string" except AssertionError as error: print(f"Fail: {error}") print("All tests have been run.") Review Approximate time: 1 minute We now have the basis for our test suite. We will build tests using the Python assert keyword inside try/except blocks. In the next topic, we will get a first look at the code we will be testing for the rest of the lesson. _________________________________________________________________________ Introduction to the calculator Introduction to the Calculator What is it? The calculator code. What does it do? It is a basic, but broken, calculator in code. Why do I need it? We will build tests to find out what is wrong with the code. For the rest of this lesson, we will focus on building tests for some broken calculator code. The developers were clearly in a hurry and left some bugs in the code. Rather than show you the code, we will use the principles of user testing and software testing to pinpoint the errors. You will not be expected to fix the code. Challenge Approximate time: 20 minutes The four calculator functions, add, sub, mul, and div, are imported from a hidden file, calc.py. The calculator prints the result to the console. 1. Look at the code in runnable 1. What do you expect the result to be? 2. Run the code. Did the result meet your expectations? 3. Change the two numbers in parentheses and run the code again. Do the results match your expectations? 4. Try calling the add function multiple times with different numbers. What happens? from calc import add, sub, mul, div # all functions take two arguments add(22, 20) Explanation Approximate time: 5 minutes The two numbers in parentheses are the numbers to be added together, in this case 20 and 22. Therefore, we would expect the result to be 42. The addition function appears to be working correctly because the result is 42. Depending on what you passed into the add function, you may have been able to force an error, for example by adding 2 strings instead of numbers, but largely the add function performs according to expectations. Calling the add function multiple times with different numbers prints out the result on separate lines. There is no limit to how often you can call the add function. Test Your Knowledge Approximate time: 15 minutes 1. Reset the code and then add the sub function. Try sub(4, 2). What do you expect the result to be? Does it perform according to your expectations? 2. Add the mul and div functions too. Pass in two numbers each time. Do these perform according to your expectations? 3. Change the numbers for each of the functions. Try some floating-point numbers as well as integers. Do they perform correctly? 4. How would you define this type of testing? Is it black box or white box testing? Solution Approximate time: 5 minutes We would expect the sub function to subtract 2 from 4, which would give us the result of 2. However, it returns 2.0. Could it be a bug, or is the calculator doing something different? The mul function seems to work correctly, but the div function does not work with whole numbers! Now we are seeing some odd behaviour with sub, and you may not have been able to get div working. It works if we pass in a floating-point number as the first number, e.g., 10.1, but that is clearly incorrect behaviour! This is black box testing because we do not know what is actually happening in the code. All we can do is provide inputs and see if what comes out is what we expected. Review Approximate time: 1 minute We have already identified that there are some issues with the calculator, but we are a long way from formalising what these are. As we go through this lesson, we will perform user tests and build automated, black-box tests to pinpoint the errors. We could say that the calculator does not perform according to the expectations, but it is much better if these expectations are laid out in a formal requirements document. We will create such a document in the next topic. _________________________________________________________________________ Building the requirements document What is it? A Software Requirements Specification (SRS) document. What does it do? It outlines the requirements that software must meet. Why do I need it? It serves as our source of truth for building tests. You probably already have an idea about how a calculator should behave, but what if another person has a different idea? How do we know whose idea is correct? For this, we need a Software Requirements Specification document or SRS. The SRS outlines exactly what the Functional and Non-Functional requirements of a piece of software are. Using this detailed specification, we can test the software to ensure it meets the requirements. Requirements analysis is the first step of the software testing life cycle. SRS documents can be very long and involved. We have produced a simplified one for this course to demonstrate the main features of an SRS document. Challenge Approximate time: 20 minutes A simple SRS document consists of five main sections: 1. Product description: In this section, we describe the intended purpose of the software system, and we list the most important capabilities of the system. 2. User characteristics: Here, we define who the intended end users are and what pre-existing knowledge they should already have. 3. Operational scenarios: This section details how the system is intended to be used by its end users to support them in their activities. This usually documents one or more specific uses of the software. 4. Functional requirements: This is where we describe what the software must do, the main functions it must perform. 5. Non-functional requirements: In this final section, we discuss other requirements, which may be regulatory or involving security or performance. With the above definitions in mind, download the SRS document or use the one provided by your teacher. The content in the document provided is out of order, and the headings are missing. Using the headings above to assign a correct heading to each of the paragraphs in the document. Then put the document in the correct order. Explanation The correct ordering of the sections is: 3, 1, 5, 2, 4. In the challenge document: 1. 3: Product description. 2. 1: User characteristics. 3. 5: Operational scenarios. 4. 2: Functional requirements. 5. 4: Non-functional requirements. Some of these sections may seem similar to each other, so it is important you understand the distinctions. Test your knowledge Since some of the sections may seem similar, let us look at them in more detail. 1. Which heading would the following statement come under? "The user is expected to be proficient at using the Windows operating system, and be familiar with copying and moving files." 2. Which heading would the following statement come under? "An Editor will use the system to adjust the content of the file and copy it to the backup folder." 3. Is the following statement an example of a functional or non-functional requirement? "The system is expected to run for 99.999% of the time over a one month period." 4. Is the following statement an example of a functional or non-functional requirement? "The software should allow a user to sign-up or log-in using their Google account." Solution 1. This would come under the heading of User Characteristics because it is outlining the skills we expect a user to have. 2. This would come under the heading of Operational Scenarios. This may seem similar to User Characteristics, but it is different because we are describing a proposed specific use of the system. User Characteristics defines who we expect to use the system. Operational Scenarios describes how they could use it. 3. This is a non-functional requirement, it is concerned with a requirement that does not affect the intended functionality of the software. 4. This is a functional requirement, the action taken from this statement will directly affect the functionality of the software. Review The SRS is an important foundation for building software. You now know the basics of a simple SRS document. In the next topic, we will build one for the calculator. _________________________________________________________________________ Challenge: Build your own requirements document What is it? Software Requirements Document (SRD) What does it do? Outlines essential software requirements. Why do I need it? Forms the basis for future testing. In the last topic, we covered Software Requirements Specification (SRS) documents and created a simple one for a web-based maths game. Challenge Approximate time: 60 minutes Using the provided template, create an SRS document for the calculator. Sections are outlined; complete the details for: Product description User characteristics Operational scenarios Functional requirements Non-functional requirements Don't stress about inventing a scenario; focus on the functional requirements. Solution Here’s my SRS for the calculator. Don’t worry if yours differs; prioritize the functional requirements for this lesson. _________________________________________________________________________ Experiment with the calculator What is it? User testing. What does it do? Tests the calculator software from an end-user perspective. Why do I need it? It's part of User Acceptance Testing (UAT), crucial for ensuring software meets requirements. Challenge Approximate time: 20 minutes With the SRS document, perform structured user testing. Earlier, testing was informal—now, use the SRS to verify where the calculator fails. User testing is, as it sounds, the stage where the user or client tests the software to ensure it meets the requirements. It is also known as "End-User" or "User Acceptance" testing (UAT). This testing can be performed by individual users or groups at different stages of development. It helps catch bugs and find unexpected behaviours, ensuring the software works as intended. Use the Functional Requirements from the SRS document to test each function as a user. Record where the calculator fails to meet the SRS specifications. Use black box tests: try unexpected inputs (e.g. strings or floating points). Functional requirements: 1. Add function: Adds two whole numbers. 2. Sub function: Subtracts one whole number from another. 3. Mul function: Multiplies two whole numbers. 4. Div function: Divides one whole number by another. 5. Error checking: Only whole numbers are valid, and ensure the calculator includes error checking to restrict inputs to whole numbers only. Explanation Approximate time: 10 minutes In testing each function using the requirements from the SRS document, you will have found a number of errors. In fact, none of the functions entirely meets the specifications: The add function has no error checking; it can add strings together. The sub function just is not subtracting! The mul function is only checking for one input as a whole number; it allows multiplying strings by numbers. The div function seems broken; it works incorrectly when a floating point number is passed as the first number. If you did not catch all of these errors, do not worry. Go back over the code and try it for yourself. Did you find more errors this time than when you tested the calculator without the SRS document? Test your knowledge Approximate time: 10 minutes Generally, this type of testing is done with a group of simulated or real users. Alpha testing: Early testing by an internal group of simulated users, a long time before the software is ready for release. It helps identify major issues early on. Beta testing: Testing with a group of actual users, not long before the software’s release. It helps to gather feedback on the product and identify any remaining bugs. Why is it good to have early, alpha testing? Alpha testing provides early detection of bugs and other issues. Professional testers can often identify problems that developers may have overlooked. Why is it a good idea to beta test software before release? Beta testing helps to polish up the final bugs before release. It improves the overall product and its features. Real-world users often use the software in different ways compared to developers, making their feedback vital. Solution Approximate time: 5 minutes Alpha testing: Provides early detection of bugs and other issues. Professional testers can often identify issues that the developers may have overlooked. Beta testing: Helps to polish up the final bugs before release. It helps to improve the overall product and its features. Real-world users often use the software in a different way to developers, so their feedback is vital. Review Approximate time: 2 minutes When we said "play with the calculator", we really meant "structured play". This is what user testing is! It involves experimenting with the product and testing things that the developers may not have thought about. Now that we have tested the calculator as a user, we will see how the asserts we learned about earlier can do a lot of this for us. _________________________________________________________________________ The testing process The testing approach A brief look at white box testing What is it? Black box and white box testing. What does it do? Provides different, but complementary ways of testing software. Why do I need it? Black box and white box techniques are an important part of the software tester’s arsenal. In an earlier topic, we said that we were going to focus on black-box testing for this lesson. However, before we do this, let us take a look at one white box testing technique, so that you are familiar with the difference between black box (functional) testing and white box (structural) testing. Challenge Approximate Time: 15 minutes Take a look at the code in runnable 1. Do not run the code yet, but just read through to see what it does. With the input variable set to 5, what do you expect the code to print out? With the input variable set to 1, what do you expect the code to print out? Run the code and experiment with changing the input variable to different numbers. Can you reach the final else clause? Experiment with changing the input variable to a string. Can we reach the final else clause then? Explanation Approximate Time: 5 minutes The code prints "The if condition was met" as expected. The first if clause has been executed. The code prints "The elif condition was met", again as expected. The elif clause has been executed. No, changing the input variable to different numbers never causes the final else clause to be executed. No, this just results in error messages. Therefore, we can see that the final else clause is never executed. White box (structural) testing In a sense, you have just performed a white box testing technique by analysing the code and determining which statements will be executed, and also which code will not be used. Note that the above exercise was merely to see what branches are accessible - we are not concerned with the output from the program. In a similar way, many white box testing techniques may not even run the code! They just perform analysis on it. There is a more formal way of doing this, though, which we will cover below. Test Your Knowledge Approximate Time: 20 minutes A white box testing technique, which is designed to eliminate redundant, or unnecessary code, is called statement coverage. This attempts to determine whether every line of code can or will be executed at least once. To calculate statement coverage, we use the following equation: Read through the code in runnable 1. 1. Not counting blank lines, how many lines of code will be executed if input is 1? 2. Not counting blank lines, how many lines of code will be executed if input is 10? 3. Are there any lines of code that will never get executed? 4. There are 7 lines of code in our example, not counting blank lines. Using your answers from questions 1 and 2, calculate the statement coverage for each case. Solution Approximate Time: 5 minutes If input is 1, four lines of code are executed: The original input declaration. The first if statement. The elif statement. The print statement on line 6. If input is 10, three lines of code are executed: The original input declaration. The first if statement. The print statement on line 4. There is no way for lines 7 and 8 to ever be executed. In the first case, we have statement coverage of 4 ÷ 7 * 100 = 57.1%. The second case has coverage of 3 ÷ 7 * 100 = 42.8%. Code that is never executed is called redundant or dead code and should be removed. Doing this will make our program run more efficiently and faster. Our example was quite simple. You could tell by looking at the code that some of it would never be executed, but this can be much more difficult in larger projects. You will be happy to know that there are software tools that can perform this analysis for us, so we would not need to read thousands of lines of code, looking for ones that would never be executed. Review Approximate Time: 1 minute We have taken a brief look at one white box testing technique. White box and black box testing are complementary techniques. Software testers use both white box (structural) tests and black box (functional) tests in their work. For the remainder of this lesson, we will build our tests using black box testing. This will simulate a real-world approach where we are interested in testing the functionality, rather than the structure, of our code. _________________________________________________________________________ Planning and documenting the tests What is it?The test plan document.What does it do?Outlines the overall plan of our tests. Why do I need it?It is a master document that allows us to set the scope and deliverables of our tests. We already have our Software Requirements Specification document, so we know what our software should do. But, how do we know what we will test? The answer lies in the test plan document. You are probably coming to realise that software testing involves a lot of documentation. There are three main sections of a test plan document: 1. Objectives - what are we hoping to achieve by testing the software? 2. Deliverables - what are we expected to deliver back to the development team or other stakeholders? 3. Testing strategy - what tasks we will perform and the type of testing we will do? For larger or more complicated pieces of software, a test plan can run to hundreds of pages. We will keep it simple for these lessons. Challenge Approximate time: 15 minutes Based on the above definitions, which heading would the following sentences come under? 1. "We will perform black box testing on each of the main functions of the program" 2. "Confirm that a descriptive error is returned if the input is not as expected" 3. "We will supply a test report in the form of a PDF file" Explanation Approximate time: 5 minutes 1. This would come under the Test Strategy heading. 2. This would come under the Objectives heading. 3. This would come under the Deliverables heading. Objectives First, we need to think about our objectives. Our objectives are what we hope to achieve by testing the software. The objectives can be very simple, such as "Given a range of inputs, verify that all of the functional requirements from the SRS document have been implemented". Deliverables The next section we need in our test plan document outlines the deliverables. The deliverables are what we are expected to deliver back to the development team or other stakeholders. Usually, this would be in the form of a bug report or a test report. You may also hear these referred to as artefacts. We will look at building a test report a bit later. Test strategy We also need to plan our test strategy. This is our overall approach to the testing. We need to think about the tasks we will perform and the type of testing we will do. Test your knowledge Approximate time: 15 minutes Let us think about our calculator software again. As always happens in software development, the deadline for release is fast approaching. We want to ensure that the calculator functions work properly because it would be embarrassing if it could not perform basic arithmetic! We have supplied a test plan template for you in this Google doc. Please log into your Google account and make a copy in "File" so you can use it. The steps you will need are: 1. Refer back to your SRS document for the calculator, and complete the Objectives section. This does not need to be too complicated, two sentences will suffice. Remember, this section outlines what we want our tests to achieve. Try to elaborate on the example objectives we mentioned above. 2. What deliverables will be provided to the Development team and stakeholders? (Note: We have already completed this section on the document provided). 3. What kind of testing approach would we use to achieve the objectives from the challenge? Would this be black box or white box? Should we perform tests using software, or is user testing sufficient? Use what you have learned to complete the Test Strategy section. Solution Approximate time: 5 minutes 1. The Objectives section could contain something like: "Given a range of inputs, verify that each of the calculator functions accepts whole numbers and returns the correct result. Confirm that a descriptive error is returned if whole numbers are not provided." You may have come up with something different to this, and that is fine. The important thing is that you are thinking about the purpose of our tests. 2. For example, "Upon completion of the tests, a Final Test Report document will be provided to the development team and other relevant stakeholders. The report will outline the tests carried out, the status of each test and comments that may help with fixing bugs." 3. The Test Strategy section could contain something like: "We will use black box, or functional, testing for each function. We will use an automated, software testing approach with a variety of inputs and record a pass/fail for each test." Again, you may have come up with something different. Do not worry if you have. The reason for using black box testing is that, based on the SRS document and the objectives, we are concerned with checking that the calculator functions as expected. Automated software testing allows us to perform a greater number of tests in a shorter time. Review Approximate time: 1 minute These three sections: objectives, deliverables and testing strategy are the main components of a test plan document. In the next topic, we will learn about test cases. _________________________________________________________________________ Build your own test plan What is it?Completing the test plan document.What does it do?Provides a detailed document to guide test creation.Why do I need it?It serves as the "source of truth" for all tests, helping testers, developers, and stakeholders. In the previous section, we covered the core components of the test plan. Now, we’ll finish building the document by adding test cases, which are individual tests to be performed on the software. Challenge (10 mins) Match the Test Case Titles with the correct Descriptions: Explanation (5 mins) The format of a test case is as follows: Test case ID: A unique number or value to identify the test. Module Name: The name of the module or function to be tested. Test Description: What we are testing. Prerequisites: Any special setup that is needed before performing the test case. Test Data: Data needed to run the test. Test Steps: The steps needed to perform the tests. Expected Results: What we expect as an outcome of the test. Comments: Extra information regarding the tests. Test Your Knowledge (20 mins) Task: Use the template document to create the following extra test cases: Create three more tests for the add function. For example, you can create tests for entering different data types (e.g., floats, strings, integers), or different values. You should have four test cases for the add function when you’ve completed this task. Important: When thinking about the test cases, don’t only create tests that you know will pass. Also, include tests that are likely to fail or return errors. Solution (5 mins) Check the completed test plan document to see the examples we came up with. Your test cases might differ, which is fine. The key is to think about a variety of scenarios. Review (1 min) In this topic, we completed our test plan document, which will guide us as we build our tests in Python. _________________________________________________________________________ Challenge: complete the test plan What is it? A challenge to test your knowledge and skills. Purpose: Assesses your understanding of test plans. Why it's needed: Develops Software Testing skills. Challenge In the previous topic, you created four test cases for the add function in the calculator. Now, create test cases for the remaining functions: sub, mul, and div. Challenge Steps 1. Approximate Time: 45 minutes. 2. Instructions: Make a copy of the provided test case plan document as a starting point. 3. Create Test Cases: ○ Add Function: 4 test cases (already created). ○ Sub Function: 4 test cases. ○ Mul Function: 4 test cases. ○ Div Function: 4 test cases. ○ Total: 16 test cases by the end. Steps for Each Function Sub Function: ○ Test that: Two whole numbers return the correct result. Two different whole numbers return the correct result. An integer and a string throw an error. Two floats return an error. Mul and Div Functions: ○ Repeat similar test cases as for the sub function. ○ Note: The error message in each case will be the same: "Error: The calculator can only accept whole numbers." Hints Sub Function Checks: ○ Verify tests for two whole numbers, distinct whole numbers, integer and string (error), and float inputs (error). Mul and Div Functions: Follow similar steps with varied inputs. Solution Refer to the example Test Plan document for guidance. Your answers may differ slightly. _________________________________________________________________________ The testing process Building a simple test suite in Python Converting the test plan to code Lesson aim Learn how to create a basic testing framework in Python. Lesson objectives To do this you will: Convert a test case to code. Create new test cases. Document the results of the tests. What is it? Writing our test cases in code. What does it do? Creates Python tests. Why do I need it? These are the tests we will use with our calculator application. Previously, we created our test plan document. Now, we can start converting the test cases to Python code. As demonstrated at the beginning of the previous section, we will use the assert keyword to do this. Challenge Approximate time: 15 minutes To begin, let us take a look at the CompletedTestPlan.pdf. This is probably a little different from the one you created, but we will use it for the remainder of the lesson. We are specifically concerned with the Test Cases section. 1. Compare the code in runnable example 1 with the first test case: ○ Which part of the code corresponds to the Prerequisites in the Test Plan? ○ Which part corresponds to the Test Data? ○ Where is the Expected Result? ○ Have the Test Steps been followed? 2. Run the code; did it throw an error? Note: If you are not sure about the code, you can refer back to the "I will try...except" and the "Introduction to the calculator" lessons. Explanation Approximate time: 5 minutes The prerequisite is that the add function is imported from calc.py, so line 1 corresponds to the Prerequisites. The two arguments passed to add on line 3, namely 2 and 2 correspond to Test Data. The Expected Result is 4; this is also on line 3 when we assert that add(2,2) should return 4. Yes, all of the Test Steps have been followed. No errors are thrown when you run the code; this means the test has passed! Test your knowledge Approximate time: 20 minutes Using what we have already learned, look at the second test case. What are the Prerequisites? Are they already met? What is the Test Data this time? What is the Expected Result? On line 6, add the try/except block for the second test case. Break the test by changing the expected result and run the code again. What happens now? Solution Approximate time: 5 minutes The Prerequisites are to import the add function from calc.py, this is already met. This means you do not have to re-import the add function again. This time the Test Data is that input 1 should be 20, input 2 should be 22. The Expected Result is 42. Your code should now look like the code displayed below: Breaking the code by changing the expected result in the first test to 5, for example, should print out the message: Fail: Did not get 42 for add(20,22) Remember to change the text of the message if you adjust the arguments that you pass in to the add function. Review Approximate time: 1 minute We have seen how to convert a single test case to code. In the following topics, we will write, perform and document all of the tests. ____________________________________________________________________ Build extra assertions What is it? Adding more assertions. What does it do? Write assertions with fail messages that state both what the expected result should be, and what the actual result was. Why do I need it? Allows you to drill down into more detail about why a test failed. Build extra assertions In the previous topic, we added code for the first two test cases in our completed test plan document. If you have got this far in the course, then you probably have a very good idea about what is coming next. Challenge Approximate time: 15 minutes Remember, in the requirements of our calculator app, we stated that we could only use whole numbers. This means that strings and floating point numbers (numbers with a decimal point like 3.14159) are not allowed. With this in mind, take a look at the following questions: Compare the Comments of the 3rd test in the test plan to the third test in the runnable example. What are we checking for this time? Should this test pass or fail, do you think? Run the code. Were you correct? Why do you think the test gave this result? Explanation Approximate time: 5 minutes This time we are checking to see if "Error: The calculator can only accept whole numbers" is returned. The test failed. This result can seem a little weird! Our test failed, but it failed by not giving an error message? Remember what we are testing for. This time, we sent in bad data (float values) expecting to get an error message back. So, because we did not get the error message, then our test failed. It might seem contradictory, but an error would need to be returned for this test to pass. Why do we need to return an error? Remember, sometimes we want our program to return an error if a user enters incorrectly formatted data the app will stop working and let the user know that something is wrong. Without this feedback, the user might continue on not knowing that they have made an error. In the above example, we are testing to see that the error was returned. If an error is returned, the test will pass; but if the error is not returned, the test fails. Note: The error that is returned is one written by the programmer, not a built-in Python error. The code in the calculator functions looks like this: Here, the code checks to see if the num variable is an int - an integer or whole number. If it is not, then it returns our custom error. It would be a good idea to make a note of this. Firstly, it will help you understand how the calculator functions should work. Secondly, it will help you with challenges later on. Test your knowledge Approximate time: 15 minutes Now, add the code for the fourth test case for the add function: Add a new try/except block at line 12. Add the assert and the correct comparison. Add the failure message. Run the tests to see if the new one passes or fails. What have we learned about the add function from these four tests? Solution Approximate time: 10 minutes Your code should now look like the second runnable example. From the results of our tests, we can infer that our add function is not working properly. To infer something means to come to a decision based on evidence or reasoning. Since the final two tests do not pass, this indicates that something is wrong with the function. In both cases, we were expecting errors to show but did not receive them, so the tests failed. As software testers, we do not need to fix the code. We simply make a report back to the software developers so that they can fix the bug. Review Approximate time: 1 minute We have now built all of the tests for the add function. But, you may have a nagging feeling that you want to know why some of the add function tests are failing. If so, congratulations! You are thinking like a software tester. In the next topic, we will learn how to use tests to drill down a bit deeper and see if we can learn why these errors occur. _____________________________________________________________________________________ Digging deeper What is it? Writing additional tests to diagnose faults in our code. What does it do? It provides us with extra tools to delve into software defects. Why do I need it? Allows us to dig deeper into code problems. We now have a failing test case. We expected to get an error message, but we got something else instead! Challenge Approximate time: 20 minutes To give a more detailed report on what is wrong with our code, we often want to investigate further into exactly what is being returned when we run the tests. A professional test suite would give you very descriptive failure messages, such as "Fail: Expected X, but got Y". Let us look how we could dig a little deeper into the test from the previous topic. Take a look at the code in the runnable example. 1. What is being stored in the expected_result variable? 2. What is being stored in the actual_result variable? 3. Can you see how the assert is different this time? How has the comparison changed? How is the error message different? 4. Run the code, what happens? Explanation Approximate time: 5 minutes 1. The expected_result variable contains what we expected to be returned from the add function, which in this case is the error message. 2. The actual_result variable contains what was actually returned from the add function. 3. Now, we are asserting that actual_result and expected_result should be equal. If they are not, then we have a different error message. This is more like the professional test suite we spoke about earlier - it tells us what it expected, but what actually happened. 4. You can see that our failure message now says: Fail: Expected: Error: The calculator can only accept whole numbers, but got: 20cat 5. What this means… As you can see, by using this second type of test which returns a more detailed output, we can delve deeper into our code, so that we can find out exactly what is wrong with it. We can then pass on this information to the Software Development team to fix. In this case, we can see that the result is concatenating, or combining, the number 20 and the word cat into 20cat. This is clearly not the result we would expect and demonstrates a more specific problem within our add function. You would not choose to write all of your tests using this method, but it demonstrates a useful technique for drilling down if you want to get more detailed information. 6. Test your knowledge 7. Approximate time: 15 minutes With the above additional testing strategy in mind: Adjust the actual_result variable to see why the third test fails. How does the actual_result differ from the expected_result this time? Vary the actual_result variable to try passing in two strings, or the first argument as a string and the second as a number. What is wrong with the add function? 8. Solution 9. Approximate time: 5 minutes To do this, we need to change actual_result to add(20.0,2.0). The third test fails because the actual_result is 20.02.0. The expected_result was the error message. If we do not call add with whole numbers, then it returns a concatenated string each time. This is not the expected behaviour, and it should have resulted in an error. 10. Review 11. Approximate time: 1 minute We have now built all of the tests for the add function. We have also seen how to build additional tests to try and find out what is causing the issue. Before you move on, make sure you are familiar with this code because we will use the same technique for the remaining three calculator functions. As yet, we have not documented whether these tests have passed or failed. We will do that in the next topic. _________________________________________________________________________ Making a test report What is it? The test report document. What does it do? It provides the results of our tests. Why do I need it? It is the final document required for the testing process. Making a Test Report We have learned how to write tests, but we have not yet recorded the results! As you are learning, a software tester needs to manage a significant amount of documentation. This final document, the test report, is a formal record of the test results. As its name suggests, it completes the set of three essential documents required in software testing. Challenge Approximate time: 15 minutes We have provided a sample test report document. It is divided into three sections: 1. Introduction 2. Overview of Test Results 3. Detailed Test Results 1.1 Overview - This section includes a brief introduction to the document and states its intended audience. 1.2 References - Links back to the SRS and Test Plan documents, as this document ties them all together. 2.1 Test Log - An overview of when the tests were conducted and by whom. 2.2 Test Assessment - A written assessment of what was tested and the conclusions inferred from the tests. It also includes a quantitative assessment, listing the number of tests run, passed, and failed. 3.1 Results Log - A detailed log of each individual test from the test plan document. With the above structure in mind, consider where the following paragraphs would fit in the document: 1. “Four tests were carried out on the add function. Two passed and two failed. The failures indicate insufficient error-checking in the add module, as an error was never returned.” 2. “The add function from the calculator module was tested on 30 September 2022. Testers were Johana Lopez and Marc Smith.” 3. “This document is the software test report for the calculator functions software development project. The intended audience is the development team and technical stakeholders.” 4. “Tests run: 4, Tests passed: 2, Tests failed: 2, Tests not run: 0, Pass percentage: 50%.” Explanation This statement would likely fit in section 2.2 Test Assessment. This sentence would belong in the 2.1 Test Log section. This paragraph is a good fit for 1.1 Overview. This data would also go into section 2.2 Test Assessment as a quantitative assessment. Test Your Knowledge Approximate time: 20 minutes 1. Make a copy of the sample test report in your own Google Drive. 2. Add the correct paragraphs from above to each section. 3. Add your own name as the tester in the Test Log section. 4. Using the test plan document and the results of our tests, complete section 3.1 Results Log. The first entry is completed for you. The code is available in the runnable example for you to re-run the tests. Solution Here is an example of how your completed document could appear. Keep your own version available, as we will need it to record the results of future tests. Review Approximate time: 1 minute In the past few lessons, we have seen how to create: Software Requirements Specifications (SRS) Document: Outlines what our app is supposed to do. Test Plan Document: Details how we will verify that the app features work as specified in the SRS Document. Test Report Document: Summarises the outcomes of the tests. We have begun work on the final document needed. Next, we will focus on completing the tests for the calculator. _________________________________________________________________________ Testing the subtract function What is it? Writing tests for the subtract function. What does it do? It tests the functionality of the function using software. Why do I need it? To practise writing Python assertions. It’s only fair to mention that the programmers of our calculator functions have been somewhat careless. None of the four functions work entirely as intended. In this topic, we will use our test plan document and Python assertions to test the sub, or subtract, function. Challenge Approximate time: 20 minutes Refer back to the test plan PDF we used previously. Write the code for the first sub test, which has the Test ID of 5 in the document. Here are some tips: Remember to check the prerequisites. Ensure the module name and test data are correct. Pay attention to the syntax of the try/except block. The test should pass when you run it. Place a print statement at the end that says, "All tests have been run." Refer back to the tests you wrote in previous topics if unsure. Explanation Approximate time: 5 minutes Your code should look like this: If your code worked, well done! If not, don’t worry; it takes a bit of practice to get the syntax right. Read over the solution and see if you can spot any mistakes. In programming, it’s all about the little details! Test Your Knowledge Approximate time: 20 minutes Let’s maintain the momentum and add the second test, with the ID of 6, from the test plan document. This test is slightly different from the first because this time, we are checking for an error to be returned. Consider the following questions: What is our test data this time? What format should our assertion statement take? If you feel confident, add the code below the first test and before the final print statement. Does this test pass or fail? Solution 1. Our Test data this time are the number 2 and the word "cat". 2. Our assert statement, without the try: part, would look like this: our code should now look like this: We are receiving an error when we pass in a string value, so this test also passes. Our subtract function is looking good! Review Approximate time: 1 minute In this topic, we have added the first two tests for the sub (subtract) function. Everything is looking good; we have a 100% pass rate so far, and the error message is functioning as expected. In the next topic, we’ll see why it is essential to perform multiple tests on a function. _________________________________________________________________________ Finding errors in the code What is it? Using tests to detect errors in the code. What does it do? It demonstrates the value of comprehensive testing. Why do I need it? To understand why multiple test cases are necessary, not just one or two. So far, we have performed two tests on our subtract function, both of which pass. We might think we have written enough tests; after all, the subtract function seems to work, and it returns an error if the input is incorrect. However, recall the pesticide paradox in our seven principles of software testing. To detect new defects, existing tests and test data may need changing, and additional tests may be required. Challenge Approximate time: 15 minutes Refer to the test case on row 7 in the test plan document. What is our test data this time? What is the expected result? Using what you have learned, write the code for this test case. Run the tests. Do they all pass? If they do not pass and your test is written correctly, what does this imply? Explanation Approximate time: 5 minutes For this test, the data values are 62 and 20. We expect the result to be: 62−20=4262 - 20 = 4262−20=42 Your code should look like this: No, not all tests pass. Our new test fails. The test is written correctly, and the test data and expected result are valid, indicating that there is an error somewhere in the subtract function code. Test Your Knowledge Approximate time: 20 minutes As we did with the add function, let’s investigate further to identify the issue. Often, we need to write extra tests to diagnose an error. Open the second runnable example tab and complete the code: 1. Complete the import statement on line 1. 2. Determine the expected result on line 3. 3. Retrieve the actual result on line 4. Once you have completed the code, run the test. What happens? Try running the test multiple times with different expected_result and actual_result values (e.g., set expected_result to 10 for sub(12, 2)). Can you deduce what is wrong with the subtract function? Solution 1. The import should be: from calc import sub

Software Testing PDF

Document Details

Tags

Related

Summary

Full Transcript

Upgrade to continue