Flash and JavaScript are required for this feature.
Download the video from iTunes U or the Internet Archive.
Topics covered: Testing and debugging
Instructor: Prof. Eric Grimson, Prof. John Guttag
11: Testing and Debugging
The following content is provided under a creative commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.
PROFESSOR: One of the things that you should probably have noticed is as we're moving in the terms, the problem sets are getting less well defined. line And I've seen a lot of email traffic of the nature of what should we do with this, what should we do with that? For example, suppose the computer player runs out of time. Or the person runs out of time playing the game. Should it stop right away? Should it just give them zero score? That's left out of the problem set. In part because one of the things were trying to accomplish is to have you folks start noticing ambiguities in problem statements. Because that's life in computing. And so this is not like a math problem set or a physics problem set. Or, like a high school physics lab, where we all know what the answer should be, and you could fake your lab results anyway. These are things where you're going to have to kind of figure it out. And for most of these things, all we ask is that you do something reasonable, and that you describe what it is you're doing. So I don't much care, for example, whether you give the human players zero points for playing after the time runs out. Or you say you're done when the time runs out. Any of that -- thank you, Sheila -- is ok with me. Whatever. What I don't want your program to do is crash when that happens. Or run forever. Figure out something reasonable and do it. And again, we'll see this as an increasing trend as we work our way through the term.
The exception will be the next problem set, which will come out Friday. Because that's not a programming problem. It's a problem, as you'll see, designed to give you some practice at dealing with some of the, dare I say, more theoretical concepts we've covered in class. Like algorithmic complexity. That are not readily dealt with in a prograing problem. It also deals with issues of some of the subtleties of things like aliasing. So there's no programming. And in fact, we're not even going to ask you to hand it in. It's a problem set where we've worked pretty hard to write some problems that we think will provide you with a good learning experience. And you should just do it to learn the material. We'll help you, if you need -- to see the TAs because you can't do them, by all means make sure you get some help. So I'm not suggesting that it's an optional problem set that you shouldn't do. Because you will come to regret it if you don't do it. But we're not going to grade it. And since we're not going to grade it, it seems kind of unfair to ask you to hand it in. So it's a short problem set, but make sure you know how to do those problems.
OK. Today, for the rest of the lecture, we're going to take a break from the topic of algorithms, and computation, and things of the sort. And do something pretty pragmatic. And we're going to talk briefly about testing. And at considerable length about debugging. I have tried to give this lecture at the beginning of the term, at the end of the term. Now I'm trying it kind of a third of the way, or middle of the term. I never know the right time to give it. These are sort of pragmatic hints that are useful. I suspect all of you have found that debugging can be a frustrating activity. My hope is that at this point, you've experienced enough frustration that the kind of pragmatic hints I'm going to talk about will not be, "yeah sure, of course." But they'll actually make sense to you. We'll see.
OK. In a perfect world, the weather would always be like it's been this week. The M in MIT would stand for Maui, instead of Massachusetts. Quantum physics would be easier to understand. All the supreme court justices would share our social values. And most importantly, our programs would work the first time we typed them. By now you may have noticed that we do not live in an ideal world. At least one of those things I mentioned is not true. I'm only going to address the last one. Why our programs don't work. And I will leave the supreme court up to the rest of you. There is an election coming up.
Alright, First a few definitions. Things I want to make sure we all understand what they mean. Validation is a process. And I want to emphasize the word process. Designed to uncover problems and increase confidence that our program does what we think it's intended to do. I want to emphasize that it will increase our confidence, but we can never really be sure we've got it nailed. And so it's a process that goes on and on. And I also want to emphasize that a big piece of it is to uncover problems. So we need to have a method not designed to give us unwarranted confidence. But in fact warranted confidence in our programs. It's typically a combination of two things. Testing and reasoning. Testing, we run our program on some set of inputs. And check the answers, and say yeah, that's what we expected. But it also involves reasoning. About why that's an appropriate set of inputs to test it on it. Have we tested it on enough inputs? Maybe just reading the code and studying it and convincing ourselves that works. So we do both of those as part of the validation process. And we'll talk about all of this as we go along.
Debugging is a different process. And that's basically the process of ascertaining why the program is not working. Why it's failing to work properly. So validation says whoops, it's not working. And now we try and figure out why not. And then of course, once we figure out why not, we try and fix it. but today I'm going to emphasize not how do you fix it, but how do you find out what's wrong. Usually when you know why it's not working, it's obvious what you have to do to make it work.
There are two aspects of it. Thus far, the problem sets have mostly focused on function. Does it exhibit the functional behavior? Does it give you the answer that you expected it to give? Often, in practical problems, you'll spend just as much time doing performance debugging. Why is it slow? Why is it not getting the answer as fast as I want it to? And in fact, in a lot of industry -- for example, if you're working on building a computer game, you'll discover that in fact the people working the game will spend more time on performance debugging than on getting it to do the right thing. Trying to make it do it fast enough. Or get to run on the right processor.
Some other terms we've talked about is defensive programming. And we've been weaving that pretty consistently throughout the term. And that's basically writing your programs in such a way that it will facilitate both validation and debugging. And we've talked about a lot of ways we do that. One of the most important things we do is we use assert statements so that we catch problems early. We write specifications of our functions. We modularize things. And we'll come back to this. As every time we introduce a new programming concept, we'll relate it back, as we have been doing consistently, to defensive programming.
So one of the things I want you to notice here is that testing and debugging are not the same thing. When we test, we compare an input output pair to a specification. When we debug, we study the events that led to an error. I'll return to testing later in the term. But I do want to make a couple of quick remarks with very broad strokes.
There are basically two classes of testing. There's unit testing, where we validate each piece of the program independently. Thus far, for us it's been testing individual functions. Later in the term, we'll talk about unit testing of classes. The other kind of testing is integration testing. Where we put our whole program together, and we say does the whole thing work? People tend to want to rush in and do this right away. That's usually a big mistake. Because usually it doesn't work. And so one of the things that I think is very important is to always begin by testing each unit. So before I try and run my program, I test each part of it independently. And that's because it's easier to test small things than big things. And it's easier to debug small things than big things. Eventually, it's a big program, I run it. It never works the first time if it's a big program. And I end up going back and doing unit testing anyway, to try and figure out why it doesn't work. So over the years, I've just convinced myself I might as well start where I'm going to end up.
What's so hard about testing? Why is testing always a challenge? Well, you could just try it and see if it works, right? That's what testing is all about. So we could look at something small. Just write a program to find the max of x and y. Where x and y are floats. However many quotes I need. Well, just see if it works. Let's test it in all possible combinations of x and y and see if we get the right answer. Well, as Carl Sagan would have said, there are billions and billions of tests we would have to do. Or maybe it's billions and billions and billions. Pretty impractical. And it's hard to imagine a simpler program than this. So we very quickly realize that exhaustive testing is just never feasible for an interesting program.
So as we look at testing, what we have to find is what's called a test suite. A test suite is small enough so that we can test it in a reasonable amount of time. But also large enough to give us some confidence. Later in the term, we'll spend part of a lecture talking about, how do we find such a test suite? A test suite that will make us feel good about things. For now, I just want you to be aware that you're always doing this balancing act.
So let's assume we've run our test suite. And, sad to say, at least one of our tests produced an output that we were unhappy with. It took it too long to generate the output. Or more likely, it was just the wrong output. That gets us to debugging.
So a word about debugging. Where did the name come from? Well here's a fun story, at least. This was one of the very first recorded bugs in the history of computation. Recorded September 9th, 1947, in case you're interested. This was the lab book of Grace Murray Hopper. Later Admiral Grace Murray Hopper. The first female admiral in the U.S. navy. Who was also one of the word's first programmers. So she was trying to write this program, and it didn't work. It was a complicated program. It was computing the arctan. So you can imagine, right? You had a whole team of people trying to figure out how to do arctans. Times were different in those days. And they tried to run it, and it ran a long time. Then it basically stopped. Then they started the cosine tape. That didn't work. Well they couldn't figure out what was wrong. And they spent a long time trying to debug the program. They didn't apparently call it debugging. And then they found the problem. In relay number 70, a moth had been trapped. And the relay had closed on the poor creature, crushing it to death. The defense department didn't care about the loss of a moth. But they did care about the fact that the relay was now stuck. It didn't work. They removed the moth, and the program worked. And you'll see at the bottom, it says the first actual case of a bug being found. And they were very proud of themselves. Now it's a wonderful story, and it is true. After all, Grace wouldn't have lied. But it's not the first use of the term "bug." And as you'll see by your handout, I've attempted tend to trace it. And the first one I could find was in 1896. In a handbook on electricity.
Alright. Now debugging is a learned skill. Nobody does it well instinctively. And a large part of being a good programmer, or learning to be a good programmer, is learning how to debug. And it's one of these things where it's harder. It's slow, slow, and you suddenly have an epiphany. And you now get the hang of it. And I'm hoping that today's lecture will help you learn faster. The nice thing, is once you learn to debug programs, you will discover it's a transferable skill. And you can use it to debug other complex systems. So for example, a laboratory experience. Why isn't this experiment working? There's a lecture I've given several times at hospitals, to doctors, on doing diagnosis of complex multi illnesses. And I go through it, and almost the same kind of stuff I'm going to talk to you about, about debugging. Explaining that it's really a process of engineering.
So I want to start by disabusing you of a couple of myths about bugs. So myth one is that bugs crawl into programs. Well it may have been true in the old days, when bugs flew or crawled into relays. It's not true now. If there is a bug in the program, it's there for only one reason. You put it there. i.e. you made a mistake. So we like to call them bugs, because it doesn't make us feel stupid. But in fact, a better word would be mistake.
Another myth is that the bugs breed. They do not. If there are multiple bugs in the program, it's because you made multiple mistakes. Not because you made one or two and they mated and produced many more bugs. It doesn't work that way. That's a good thing. Typically, even though they don't breed, there are many bugs. And keep in mind that the goal of debugging is not to eliminate one bug. The goal is to move towards a bug free program. I emphasize this because it often leads to a different debugging strategy. People can get hung up on sort of hunting these things down, and stamping them out, one at a time. And it's a little bit like playing Whack-a-Mole. Right? They keep jumping up at you. So the goal is to figure out a way to stamp them all out.
Now, should you be proud when you find a bug? I've had graduate students come to me and say I found a bug in my program. And they're really proud of themselves. And depending on the mood I'm in, I either congratulate them, or I say ah, you screwed up, huh? Then you had to fix it. If you find a bug, it probably means there are more of them. So you ought to be a little bit careful. The story I've heard told is you're at somebody's house for dinner, and you're sitting at the dining room table, then you hear a [BANG]. And then your hostess walks in with the turkey in a tray, and says, "I killed the last cockroach." Well it wouldn't increase my appetite, at least. So be worried about it.
For at least four decades, people have been building tools called debuggers. Things to help you find bugs. And there are some built into Idol. My personal view is most of them are not worth the trouble. The two best debugging tools are the same now that they have almost always been. And they are the print statement, and reading. There is no substitute for reading your code. Getting good at this is probably the single most important skill for debugging. And people are often resistant to that. They'd rather single step it through using Idol or something, than just read it and try and figure things out. The most important thing to remember when you're doing all of this is to be systematic. That's what distinguishes good debuggers from bad debuggers. Good debuggers have evolved a way of systematically hunting for the bugs. And what they're doing as they hunt, is they're reducing the search space. And they do that to localize the source of the problem. We've already spent a fair amount of time this semester talking about searches. Algorithms for searching. Debugging is simply a search process. When you are searching a list to see whether it has an element, you don't randomly probe the list, hoping to find whether or not it's there. You find some way of systematically going through the list. Yet, I often see people, when they're debugging, proceeding at what, to me, looks almost like a random fashion of looking for the bug. That is a problem that may not terminate. So you need to be careful.
So let's talk about how we go about being systematic, as we do this. So debugging starts when we find out that there exists a problem. So the first thing to do is to study the program text, and ask how could it have produced this result? So there's something subtle about the way I've worded this. I didn't ask, why didn't it produce the result I wanted it to produce? Which is sort of the question we'd immediately like to ask. Instead, I asked why did it produce the result it did. So I'm not asking myself what's wrong? Or how could I make it right? I'm asking how could have done this? I didn't expect it to do this. If you understand why it did what it did, you're half way there.
The next big question you ask, is it part of a family? This gets back to the question of trying to get the program to be bug free. So for example, oh, it did this because it was aliasing, where I hadn't expected it. Or some side effect of some mutation with lists. And then I say, oh you know I've used lists all over this program. I'll bet this isn't the only place where I've made this mistake. So you say well, rather than rushing off and fixing this one bug, let me pull back and ask, is this a systematic mistake that I've made throughout the program? And if so, let's fix them all at once, rather than one at a time.
And that gets me to the final question. How to fix it. When I think about debugging, I think about it in terms of what you learned in high school as the scientific method. Actually, I should ask the question. Maybe I'm dating myself. Do they still teach the scientific method in high school? Yes, alright good. All is not lost with the American educational system. So what does the scientific method tell us to do? Well it says you first start by studying the available data. In this case, the available data are the test results. And by the way, I mean all the test results. Not just the one where it didn't work, but also the ones where it did. Because maybe the program worked on some inputs and not on others. And maybe by understanding why it worked on a and not on b, you'll get a lot of insight that you won't if you just focus on the bug. You'll also feel a little bit better knowing your program works on at least something. The other big piece of available data we have is, of course, the program text. As you the study the program text, keep in mind that you don't understand it. Because if you really did, you wouldn't have the bug. So read it with sort of a skeptical eye. You then form a hypothesis consistent with all the data. Not just some of the data, but all of the data. And then you design and run a repeatable experiment.
Now what is the thing we learned in high school about how to design these experiments? What must this experiment have the potential to do, to be a valid scientific experiment? Somebody? What's the key thing? It must have the potential to refute the hypothesis. It's not a valid experiment if it has no chance of showing that my hypothesis is flawed. Otherwise why bother running it? So it has to have that. Typically it's nice if it can have useful intermediate results. Not just one answer at the end. So we can sort of check the progress of the code. And we must know what the result is supposed to be. Typically when you run an experiment, you say, and I think the answer will be x. If it's not x, you've refuted the hypothesis. This is the place where people typically slip up in debugging. They don't think in advance what they expect the result to be. And therefore, they are not systematic about interpreting the results. So when someone comes to me, and they're about to do a test, I ask them, what do you expect your program to do? And if they can't answer that question, I say well, before you even run it, have an answer to that.
Why might repeatability be an issue? Well as we'll see later in the term, we're going to use a lot of randomness in a lot of our programs. Where we essentially do the equivalent of flipping coins or rolling dice. And so the program may do different things on different runs. We'll see a lot of that, because it's used a lot in modern computing. And so you have to figure out how to take that randomness out of the experiment. And yet get a valid test. Sometimes it can be timing. If you're running multiple processes. That's why your operating systems and your personal computers often crash for no apparent reason. Just because two things happen to, once in a while, occur at the same time. And often there's human input. And people have to type things out of it. So you want to get rid of that. And we'll talk more about this later. Particularly when we get to using randomness. About how to debug programs where random choices are being made.
Now let's think about designing the experiment itself. The goal here, there are two goals. Or more than two. One is to find the simplest input that will provoke the bug. So it's often the case that a program will run a long time, and then suddenly a bug will show up. But you don't want to have to run it a long time, every time you have a hypothesis. So you try and find a smaller input that will produce the problem. So if your word game doesn't work when the words are 12 letters long, instead of continuing to debug 12 letter hands, see if you can make it fail on a three letter hand. If you can figure out why fails on three letters instead of 12, you'll be more than half way to solving the problem. What I typically do is I start with the input that provoked the problem, and I keep making it smaller and smaller. And see if I can't get it to show up.
The other thing you want to do is find the part of the program that is most likely at fault. In both of these cases, I strongly recommend binary search. We've talked about this binary search a lot already. Again, the trick is, if you can get rid of half of the data at each shot, or half of the code at each shot., you'll quickly converge on where the problem is.
So I now want to work through an example where we can see this happening. So this is the example on the handout. I've got a little program called Silly. And it's called Silly because it's really a rather ugly program. It's certainly not the right way to write a program to do this. But it will let us illustrate a few points. So the trick, what we're going to go through here, is this whole scientific process. And see what's going on. So let's try running Silly. So this is to test whether a list is a palindrome. So we'll put one as the first element, maybe a is the second element. And one is the third element. And just return, it's done. It is a palindrome. That make sense. The list one a one reads the same from the front or from the back. So that's good. Making some progress. Let's try it again. And now let's do one, a, two. Whoops. It tells me it is a palindrome. Well, it isn't really. I have a bug. Alright. Now what do I do? Well I'm going to use binary search to see if I can't find this bug. As I go through, I'm going to try and eliminate half of the code at each step. And the way I'm going to do that is by printing intermediate values, as I go part way through the code. I'm going to try and predict what the value is going to be. And then see if, indeed, I get what I predicted. Now, as I do this, I'm going to use binary search. I'm going to start somewhere near the middle of the code. Again, a lot of times, people don't do that. And they'll test an intermediate value near the end or near the beginning. Kind of in the hope of getting there in one shot. And that's like kind of hoping that the element you're searching for is the first in the list and the last in the list. Maybe. But part of the process of being systematic is not assuming that I'm going to get a lucky guess. But not even thinking really hard at this point. But just pruning the search space. Getting rid of half at each step.
Alright. So let's start with the bisection. So we're going to choose a point about in the middle of my program. That's close to the middle. It might even be the middle. And we're going to see, well all right. The only thing I've done in this part of the program, now I'm going to go and read the code, is I've gotten the user to input a bunch of data. And built up the list corresponding to the three items that the user entered. So the only intermediate value I have here is really res. So I'm going to, just so when I'm finished I know what it is that I think I've printed. But in fact maybe I'll do even more than that here. Let me say what I think it should be. And then we'll see if it is. So I think I put in one a two, right? Or one a two? So it should be something like one, a, two. So I predicted what answer I'm expecting to get. And I've put it in my debugging code. And now I'll run it and see what we get. We'll save it. Well all right, a syntax error. This happens. And there's a syntax error. I see. Because I've got a quote in a quote. Alright I'm just going to do that.
What I expected. So what have I learned? I've learned that with high probability, the error is not in the first part of the program. So I can now ignore that. So now I have these six lines. So we'll try and go in the middle of that. See if we can find it here. And notice, by the way, that I commented out the previous debugging line, rather than got rid of it. Since I'm not sure I won't need to go back to it. So what should I look at here? Well there are a couple of interesting intermediate values here, right? There's tmp. And there's res. Never type kneeling. Right? I find something to tmp. And I need to make sure maybe I haven't messed up res. Now it would be easy to assume, don't bother looking at [UNINTELLIGIBLE]. Because the code doesn't change res. Well remember, that I started this with a bug. That means it was something I didn't understand. So I'm going to be cautious and systematic. And say let's just print them both. And see whether they're okay.
Now, let's do this. So it says tmp is two a one, and res is two a one. Well let's think it. Is this what we wanted, here? What's the basic idea behind this program? How is it attempting to work? Well what it's attempting to do, and now is when I have to stand back and form a hypothesis and think about what's going on, is it gets in the list, it reverses the list, and then sees whether the list and the reverse were identical. If so it was a palindrome, otherwise it wasn't. So I've now done this, and what do you think? Is this good or bad? Is this what I should be getting? No. What's wrong? Somebody? yeah.
STUDENT: [UNINTELLIGIBLE]
PROFESSOR: Yeah. Somehow I wanted to -- Got to work on those hands. I didn't want to change res. So, I now know that the bug has got to be between these two print statements. I'm narrowing it down. It's getting a little silly, but you know I'm going to really be persistent and just follow the rules here of binary search, rather than jumping to conclusions. Well clearly what I probably want to do here is what? Print these same two things. See what I get. Whoops. I have to, of course, do that. Otherwise it just tells me that Silly happens to be a function. Alright. How do I feel about this result? I feel pretty good here. Right? The idea was to make a copy of res and temp. And sure enough, they're both the same. What I expected them to be. So I know the bug is not above. Now I'm really honing in. I now know it's got to be between these two statements. So let's put it there. Aha. It's gone wrong. So now I've narrowed the bug down to one place. I know exactly which statement it's in. So something has happened there that wasn't what I expected. Who wants to tell me what that bug is? Yeah?
STUDENT: [UNINTELLIGIBLE].
PROFESSOR: Right. Bad throw, good catch. So this is a classic error. I've not made a copy of the list. I've got an alias of the list. This was the thing that tripped up many of you on the quiz. And really what I should have done is this. Now we'll try it. Ha. It's not a palindrome. So small silly little exercise, but I'm hoping that you've sort of seen how by being patient. Patience is an important part of the debugging process. I have not rushed. I've calmly and slowly narrowed the search. Found where the statement is, and then fixed it. And now I'm going to go hunt through the rest of my code to look for places where I used assignment, when I should have use cloning as part of the assignment. The bug, the family here, is failure to clone when I should have cloned. Thursday we'll talk a little bit more about what to do once we've found the bug, and then back to algorithms.
Free Downloads
Video
- iTunes U (MP4 - 108MB)
- Internet Archive (MP4 - 108MB)
Free Streaming
Subtitle
- English - US (SRT)