lec1.pdf

Software Testing Prof. Meenakshi D’Souza Department of Computer Science and Engineering International Institute of Information Technology, Bangalore Lecture – 01...

Software Testing Prof. Meenakshi D’Souza Department of Computer Science and Engineering International Institute of Information Technology, Bangalore Lecture – 01 Software Testing: Motivation Hello everyone my name is Meenakshi. And this is a first lecture that I will be doing as a part of the course on software testing that I am currently offering for NPTEL. So, what are we going to do today? Basically I will talk to you about the kinds of software that we encounter in the present world, and what do expensive errors look like, and how to avoid these expensive errors by using testing. So, it is basically motivation module where we deal with how important testing is. (Refer Slide Time: 00:45) So, what a software in the current century look like? If you look all around you, it is not too difficult to understand that there is software. So, we use Paytm, we use banks online, we go to an ATM machine there is software, I make a call to my friend or my parents using my mobile phones, there is software. Software that controls the heating, ventilation then air conditioning in our homes and offices, there is software that tells you how electricity flows through a grids, how water flows through a network. Software that run in manufacturing industries, the software that help us to drive a car, autopilot and other kinds of software that help you to fly our planes, when the software that controls real network, the software even in toys that children use right. So, what is our expectation about such software? We all expect the software to be error free, not only that we want the software to respond fast right. The second I insert my card into the ATM machine and what the welcome screen the next second before I had link my id, I cannot be it long enough. And the other important thing to notice that if you consider a software like the autopilot of a plane, let us say there is an error in the software you cannot say that I will shut it down mid flight, I will rectify the bug and I will restart to be able to run right. It is no longer feasible to be able to shut down system because there is an error in the software. We want the system to be able to continue to done in the autopilot case it needs to be able to continue to fly the aircraft safely and help us to land safely. (Refer Slide Time: 02:22) So, I would like to talk to you about some popular errors that have occurred in the past. This was the error is the error is the error in European space agency rocket called Ariane 5, it occurred exactly 21 years ago and the important thing to note that this error is because of a software bug. So, there was this rocket called Ariane 5 which was being controlled by software that is launched by European space agency. So, this software had the following error. So, it was trying to squeeze in data corresponding to a 64-bit floating point number into a memory space that is allotted for a 16-bit integer. So obviously it is not going to be able to succeed in doing that. And because such rockets have safety critical systems they always have backup software, but in this case the problem was the backup software also had exactly the same error. So, this resulted in transmission of incorrect altitude data to the aircraft and this rocket Ariane 5, went and plunged into the Atlantic Ocean within 36 seconds after it was launched. So, that is about 15 years of total effort and you can imagine that millions of Euros that would have been lost. (Refer Slide Time: 03:33) So, the next example that I would like to discuss about is another unfortunate example that happened, again because of a software error. So, in this case 6 patients lost their life due to buggy software in a machine that gives radiation therapy to cancer patients. So, there was a race condition error in this software. So, this software ended up calculating more dosage of radiation then what was needed and unfortunately these patients lost their life because of an overdose of radiation. (Refer Slide Time: 04:05) So, the next kind of example that I would like to discuss with you it is software that is expensive that costssomebody a lot. You all would have heard of this Intel Pentium processors right. So, there was this particular P4 Pentium processor, the mathematician in the US, when he was trying to do research towards its prime numbers, he found that this Pentium 4 processor was doing floating point division wrongly. So, when it was announced an Intel investigator did realize that all the P4 processors that it had released to the market had the same error. So, the only solution left was to be able to recall all these processors and that cost Intel a lot of money. So, this does not bring us to the end of expensive errors. So, if you Google you know you can talk, you can find Toyota breaks, crashes all kinds of crashes happened. (Refer Slide Time: 04:56) So, here are some few reports that talk about how expensive errors can be and what is the cost of inadequate testing. National institute standards and technology in the year 2002 released a report which basically discussed the impact of software testing in US. So, it says that inadequate software testing causes the US economy anywhere between 22 and 59 billion every year. Not only that the report goes ahead and says that if there are better approaches that can be found to do software testing, then you could bring down the amount of these losses to almost half of what it is. The other popular categories of losses are due to web applications which we use all the time. So, one particular thing that I would like to discuss with you now happened almost 10 years ago. Amazon had this online discount sale, and because of a software error what happened was that it ended up giving double the amount of discount and by the time Amazon realized it, it is too late that are already lost lot of money. The next fact that I would like to drought your attention is a report that Symantec, the security organization published in 2007. So, it says that security related vulnerabilities that occur in a financial transactions, in our online wallets and so on mainly occur no because of cryptographic errors, they occur because of software errors. (Refer Slide Time: 06:21) So, we know expensive errors and software can be. So, let us try to look at the various kinds of software and typically how they are developed. You might have heard this buzzword called agile methodologies right. So, agile methodologies basically insist that people who develop the code, that is the developers, also have to unit test a code. Typically, developers do not have any great knowledge on testing, but agile methodology believes that the developer himself or herself the best person to identify the error in the code. So, that puts a lot of pressure on developers to be able to know testing well. Now, let us look at the various kinds of software that we will deal with. If you consider a software that typically runs in your cars that helps you to do breaking automatically, or does a cruise control or a software that runs in your aeroplanes, such software is what is called embedded control software. So obviously, such software runs in what are called safety critical systems where a failure can be catastrophic. So, safety critical software have to meet regulatory standards where, by which regulatory authorities decide if a software has been tested enough and it is fit for release. So, this just means that almost double the effort goes into testing the software. Other kind of software is what is called enterprise software. So, what would be a typical example of an enterprise software let us see a software that run city bank right or a software that managers are Indian railways ticketing system right. What do such software have to deal with, the basically the first complex thing is that they deal with fairly huge data basis that have a lot of data and the software is critically dependent on the server and backup server running all the time right. The third popular category of software is what is called web applications. Things like social networking sites, amazon online and so on and so forth. We all know how important it is for these software to be correct. Finally, I would like to end this slide with the point what free software suppose you pick up a piece of software from the internet for free. Just because it is available for free you are not willing to accept the fact this software could have errors. So, paradoxically we expect free software also to be error free. (Refer Slide Time: 08:41) This is a very important slide. So, this is a part of study that was done at Carnegie Mellon university a few years ago. So, this slide discusses about how expensive testing can get as we go down software development. So, if you see a typical software development initially you write requirements and then you do design right. And then your unit test your software and then put together the modules, that you have tested and do what is called integration testing then you put the software with the system or hardware that it is supposed to run on and in your system testing and finally, release the software. So, suppose there was an error in the requirements, and it was found that and there then what the slide tells you is that gives you the cost - the cost is fairly low right, but suppose there was an error that was found in requirements or design, and it went all the way through testing, integration testing, system testing, did not get detected at all. The software called released and the error was found there. Then what this slide tells you is that the cost fixing that can be very high, it is not only the cost fixing the error the consequences of that error as we saw through the examples can also be really bad right. (Refer Slide Time: 09:49) So, what are the facts and myths about testing? There is a popular saying by Edgar Dijkstra who said that never think that you can use testing and say that I have proved my software to be correct. It is wrong to say the testing prove software to be correct. The main goal of testing is to be able to find errors in the software right. Using testing I can never say that I have tested my software and it is fully correct, it is a wrong statement to make. The other wrong statement to make is that a developer typically thinks that you know “ have written the code, I have debugged it well and my quality auditor my SQA in my company has inspected the code. So, there is no need to test it”. That typically will not work; it still needs to be tested. Another popular thought that people have especially these people who sell software testing tools is to be able to say that you just download and install my tool it can do magic; it will do every kind of testing you can think of. That is not true. The main goal of testing tools is to be able to help you execute the test cases and record the result. Now to be able to design test cases, you will have to be able to design cases to find errors effectively. Typical Pareto principle applies here 80 percent of the errors come in 20 percent of the code or design. So, test case design which is what the course is about needs a human to be able to do it right. (Refer Slide Time: 11:18) So, I would like to end this module by discussing about certain process maturity levels in testing. Why is it important to look at process maturity levels? It is important because it tells you what role testing plays in the software development that you do. Broadly there are 5 levels beginning from 0 and going all the way till 4. (Refer Slide Time: 11:42) So, what does level 0 tell you? Level 0 tells you that there is basically nothing called testing. Developers write their code, they debug their code they ensure that it is correct and that is it, they go ahead and release it right. So, it is clear that this does not really help to develop software that is considered to be fully safe and reliable. (Refer Slide Time: 12:01) The next comes level one thinking. Level one thinking a developer thinks that he or she has written a piece of code and their goal to test is to be able to show that the code is correct. As we saw little while ago I clearly cannot use testing to show that a piece of software it is correct right. So, let us see a particular piece of code has an integer variable. To be able to exhaustively tested on the 32 bit processor I need to be able to give every value which is 2 power minus 32 to 2 power 32 plus or minus 1, and even when an extra fast PC this is going to be able to take several years to do. So, unless I test it for every value I cannot say that the testing is correct. So, here there is nothing like test engineers and they even if there are they do not have goals and they just show that the software is not failed, but the underlined listen it is not failed because it is correct or it is not failed because you have designed the test cases wrongly, that is never clear. (Refer Slide Time: 13:00) The next level is level 2 thinking when you begin to believe that the goal of testing is to be able to identify failures or errors in the software. This is the beginning of positively using testing, but it organizations that are typically at this level there is lot of tiff between developers and testers, they belong to different teams and then one does not want to help the other, and there is confusion even though the goal of testing is fully realized. We move on to level 3 where they not only realize that the goal of testing is to find errors, but they also work together and say that we will not only find errors, we will make sure that we reduce errors to the extent possible in the software. (Refer Slide Time: 13:28) And that is also conscious understanding that when I release the software there is a bit of risk involved. The risk could be high or the risk could be low and my goal is to make the risk as low as possible. (Refer Slide Time: 13:55) Level 4 thinking is where large organizations strive to be in, here testing becomes a mental discipline. So, there is positive thought and efforts in the organization level to make sure that testing teams' efforts are taken into consideration to develop software that is as error free as possible. An organization like Microsoft which try to achieve excellence in level 4 thinking and this is what is the desired level that we would like to achieve in testing. (Refer Slide Time: 14:28) Now, what is this course got to do with all the levels and terminologies that we saw till now? So, this course will help you to think that I want to be able to do testing at levels 2 3 or 4, which means what that the goal of my testing is to be able to find errors. So, to find errors, I have to first define what are my testing objectives in technical terms, and I have to be able to figure out how to effectively design test cases to be able meet or cover these test objectives. So, in the course we will look at algorithms and techniques that will help you to formulate test objectives and design test cases that will help you to meet these objectives. So, the next module that we will be seeing, we will introduce the various terminologies that exist in testing and also clarify about what we would use is a part of this course. Thank you.