Lecture 2: Asymptotic Notation; Recurrences; Substitution, Master Method | Video Lectures | Introduction to Algorithms (SMA 5503) | Electrical Engineering and Computer Science

Topics covered: Asymptotic Notation - Recurrences - Substitution, Master Method

Instructors: Prof. Erik Demaine, Prof. Charles Leiserson

Related Resources

Download this transcript - PDF (English - US)

And I don't think it matters And 11111 forever is the same My name is Erik Demaine. You should call me Erik. Welcome back to 6.046. This is Lecture 2. And today we are going to essentially fill in some of the more mathematical underpinnings of Lecture 1. So, Lecture 1, we just sort of barely got our feet wet with some analysis of algorithms, insertion sort and mergesort. And we needed a couple of tools. We had this big idea of asymptotics and forgetting about constants, just looking at the lead term. And so, today, we're going to develop asymptotic notation so that we know that mathematically. And we also ended up with a recurrence with mergesort, the running time of mergesort, so we need to see how to solve recurrences. And we will do those two things today. Question? Yes, I will speak louder. Thanks. Good. Even though I have a microphone, I am not amplified. OK, so let's start with asymptotic notation. We have seen some basic asymptotic notation. I am sure you have seen it in other classes before, things like big O-notation. And today we are going to really define this rigorously so we know what is true and what is not, what is valid and what is not. We are going to define, and unfortunately today is This will be true if c_1 is going to be really mathematical and really no algorithms today, which is sort of an anticlimax. But next lecture we will talk about real algorithms and will apply all the things we learned today to real algorithms. This is big O-notation, capital O-notation. We have f(n)=O[g(n)]. This means that there are some suitable constants, c and n_o, such that f is bounded by cg(n) for all sufficiently large n. So, this is pretty intuitive notion. We have seen it before. We are going to assume that f(n) is non-negative here. And I just want f(n) to be bounded above by g(n). We have seen a bunch of examples, but something like 2n^2=O(n^3) defined. And roughly this means if you drop leading constants and low order terms then this is less than or equal to that. So, big O corresponds roughly to less than or equal to. But this is the formalization. Another way to think of it formally, a funny thing about this notation is it is asymmetric. Normally, you think of equality being symmetric. If A=B then B=A. But it's not true here. We do not have n^3 being big O of n^2. We don't even have big O of n^3 equaling n^2. So, we will see exactly what that means in a second. But before we get there, this is a bit bizarre notation and you should always think about what it really means. Another way to think about what it really means is that f(n) is in some set of functions that are like g. You could define big O[g(n)] to be a set of functions, let's call it f(n), such that there exist constants. They are the same definition, I think, fancy here, c and n_o, such that we have the bound f(n) is between zero and cg(n). It is a bit of a long definition, and that is why we use the notation, to avoid having to write this over and over. You can think of instead of n^2 being equal to big O of n^3, what we really mean is that 2n^2 is in the set big O(n^3). When we write equal sign, we in some sense mean this in the set, but we are going to use equal sign. You could write this. And occasionally you see papers that write this, but this is the notation that we are going to use. That has the consequence the equal sign is asymmetric, just like this operator. We have some nifty ways that we actually use big O-notation. And it is using it as a macro. By the way, we have a lot to cover today, so I am going to go relatively fast. If anything is unclear, just stop, ask questions, then I will slow down. Otherwise, I will take this as all completely obvious and I can keep going at full speed. The convention, this is intuitive, I guess, if you do some macro programming or something, but it's a bit more mathematical. We have defined big O-notation and it equals big O of something. And so we have only defined big O when on the equal sign we have big O of some function. But it is useful to have some general expression on the right-hand side that involves big O. For example, let's say we have f(n) = n^3 + O(n^2). This is attempting to get an error bound. This is saying f(n) is basically n^3 but there are these lower order terms that are O(n^2). And so this means that there is a function, shorthand for a function, h(n) which is in O(n^2) or equals O(n^2) such that f(n) = n^3 + h(n). It is saying that there are some lower order terms that are bounded above by some constant times n^2 for sufficiently large n, and that is what is here. And then f(n) equals, now this is a true equality, n^3 plus that error term. This is very useful here. Essentially, I am expressing what the lead constant is and then saying well, there is other stuff and it's all at most n^2. Saying that f(n) therefore is also order n^3, but that is a bit weaker of a statement. This is a bit more refined. We won't need to use this too often, but it is useful. Sometimes we will see, like in last class we even had a big O inside a summation. So, you can use them all over the place. The point is they represent some function in that set. A bit less intuitive, and this is more subtle, is what it means to have big O on the left-hand side. It means the same thing, but there is some convention what equality means. And this is why equal sign is asymmetric. You should read equals like "is". Is means that everything over here is something over here. So, there is an implicit for all on the left-hand side and there exists on the right-hand side. This is a true statement. Anything that is n^2 + O(n) is also O(n^2), but not the other way around. So, this is a bit asymmetric. If you think about it, this is pretty intuitive but it is subtle so you should be careful. This says for any expansion of the macro on the left-hand side, which should be f(n), there is an expansion of the macro on the right-hand side such that we get equality. And what this allows you to do is if you have a chain of equal signs relations, a chain of "is"s, then the very first one is equal to or bounded by the very last one. So, you can chain equal signs the way you normally would. You just cannot flip them around. Good. So, that's big O-notation. Any questions about that? So, big O is great for expressing upper bounds. But we also want to talk about lower bounds. For algorithms, we usually care about upper bounds on their running time. Running times at most n^2 is at most n log n up to big O, but sometimes we need to express functions that are at least some quantity. For example, we will show that sorting requires at least n log n time in some model. So, we need some other notation for that. And the notation is big Omega-notation. And it is pretty symmetric. I will just write out the set definition here. And we are going to write f(n)= big Omega[g(n)] to mean f(n) is at least some constant times g(n) -- -- for sufficiently large n. So, I am basically just reversing the inequality relation between f and g, nothing surprising, just to have it there. A random example, and now we will get a little bit more sophisticated, root n= big Omega(lg n). And you should read this that up to constant factors root n is at least log n for sufficiently large n. So, omega sort of corresponds to greater than or equal to. Let me give you some analogies. We have big O, we have big omega, this is less than or equal to, this is greater than or equal to. And I am going to fill in some more here in a moment. It's nice to have all the usual operators we have. Normally we have strict less than, strict greater than and equal sign. And we want those sort of analogs in the asymptotic world where we ignore constant factors and ignore lower order terms. We have, for example, big Theta[g(n)]. This is a capital theta which means you write the horizontal bar in the middle as opposed to all the way through. I didn't invent Greek, so that is the way it is. Theta means that you are less than or equal to and you are greater than or equal to up to constant factors, so it is the inner section of these two sets, big O and big Omega. That is sort of like equal sign but, of course, this is very different. You have things like n^2 is big Theta of 2(n^2) because you ignore constant factors, but all of these other relations, OK, n^2 + O(n) = Theta(n^2), but this does not hold with theta because square root of n is really asymptotically bigger than log n. And some of the other examples we saw like n^2 versus n^3, those don't hold with T. And we have some strict notation which are the little o-notation and little omega-notation. There is no little theta because there is not notion of strict equality versus unstrict equality. Little o is going to correspond roughly to less than and little omega is going to correspond to greater than. This is a notation you will just have to get used to. And I am not going to define it precisely here because it is almost exactly the same. The difference is that instead of saying there exists constant c and n_o, you have to say for every constant c there exists a constant n_o. The relationship between f and g, this inequality must hold for all c instead of just for 1. And so n_o can now depend on c. You can assume that really n is sufficiently large, but this gives you a strict inequality. No matter what constant you put here, in front of g, let's say we are doing little o, f will be still less than c times g for sufficiently large n. We have some random examples. We are again ignoring constants. n^2 is always less than n^3 for sufficiently large n. And it is a bit subtle here. I mean in order to prove something like this, it will become intuitive after you manipulate it a little bit. You have to figure out what n_o is in terms of c. I think it something like 2/c. If we have less than or equal to, that should be right. As long n is at least this big, no matter how small of a c, you should think of c here as being epsilon now, in the usual epsilon and deltas. No matter how small c gets, still I can bound n^2 in terms of n^3, upper bound, but whenever you have theta you do not have either of these relations. For example, Ωn^2 = Theta(n^2) and it is not little o(n^2) and it not little omega(n^2) because it is exactly n^2. You will get some sense in order relation out of this, although there are some messy behaviors as you will see in your problem set. Any questions about asymptotic notation? That is the quick rundown. Now we are going to use it to solve some recurrences. Although we won't use it that much today, we will use it a lot more on Wednesday. OK. We will move onto the second topic of today, which is solving recurrences. You have probably solved some recurrences before in 6.042 or whatever discrete math class you have taken. We are going to do more and have some techniques here that are particularly useful for analyzing recursive algorithms, and we will see that mostly on Wednesday. There are three main methods that we are going to use here for solving recurrences. The first one is the substitution method. There is no general procedure for solving a recurrence. There is no good algorithm for solving recurrences, unfortunately. We just have a bunch of techniques. Some of them work some of the time, and if you are lucky yours will work for your recurrence, but it is sort of like solving an integral. You have to just know some of them, you have to know various methods for solving them. It is usually easy to check if you have the right answer. Just like with integrals, you just differentiate and say oh, I got the right answer. And that is essentially the idea of substitution method. Substitution method will always work, but unfortunately Step 1 is guess the answer. And you have to guess it correctly. That makes it a big difficult. You don't have to guess it completely. You can usually get away with not knowing the constant factors, which is a good thing because we don't really care about the constant factors. You guess the form. You say oh, it is going to be roughly n^2, and so it's some constant times n^2 presumably. So, you guess that. We are going to figure out the constants. You try to verify whether the recurrence satisfies this bound by induction, and that is the key. Substitution uses induction. And from that you usually get the constants for free. You figure out what the constants have to be in order to make this work. So, that is the general idea. You will see a few examples of this. Actually, the same example several times. Unfortunately, this is what you might call, I don't know. This is an algorithm, but it uses an oracle which is knowing the right answer. But sometimes it is not too hard to guess the answer. It depends. If you look at this recurrence, T(n) = 4T(n/2) + n, we should implicitly always have some base case of T of some constant, usually 1 is a constant, so we don't really care about the base case. For algorithms that is always the case. And we want to solve this thing. Does anyone have a guess to what the solution is? Ideally someone who doesn't already know how to solve this recurrence. OK. How many people know how to solve this recurrence? A few, OK. And, of the rest, any guesses? If you look at what is going on here, here you have T(n/2) and let's ignore this term more or less. We have n/2 here. If we double n and get T(n) then we multiply the value by 4. And then there is this additive end, but that doesn't matter so much. What function do you know that when you double the argument the output goes up by a factor of 4? Sorry? n^2,yeah. You should think n^2 and you would be right. But we won't prove n^2 yet. Let's prove something simpler, because it turns out proving that it is at most n^2 is a bit of a pain. We will see that in just a few minutes. But let's guess that T(n) = O(n^3) first because that will be easier to prove by induction. You sort of see how it is done in the easy case, and then we will actually get the right answer, n^2, later. I need to prove. What I am going to do is guess that T(n) is some constant times n^3 at most, so I will be a little more precise. I cannot use the big O-notation in the substitution method so I have to expand it out to use constants. I will show you why in a little bit, but let me just tell you at a high level what is important in not using big O-notation. Big O-notation is great if you have a finite chain of big O relations, you know, n^2 is big O(n^3) is big O(n^4) is big O(n^4) is big O(n^4). That is all true. And so you get that n^2 is big O(n^4). But if you have an infinite chain of those relations then the first thing is not big O of the last thing. You have to be very careful. For example, this is a total aside on the lecture notes. Suppose you want to prove that n = O(1). This is a great relation. If it were true, every algorithm would have constant running time. This is not true. Not in Wayne's World notation. You could "prove this by induction" by saying well, base case is 1 = O(1). OK, that is true. And then the induction step as well, if I know that n-1, so let's suppose that n-1 = O(1), well, that implies that n, which is (n-1) +1, if this is O(1) and 1 = O(1), the whole thing is O(1). And that is true. If you knew that (n-1) = O(1) and 1 = O(1) then their sum is also O(1), but this is a false proof. You cannot induct over big Os. What is going on here is that the constants that are implicit in here are changing. Here you have some big O of 1, here you have some big O of 1. You are probably doubling the constant in there every time you do this relation. If you have a finite number of doubling of constants, no big deal, it is just a constant, two the power number of doublings. But here you are doing n doublings and that is no good. The constant is now depending on n. So, we are avoiding this kind of problem by writing out the constant. We have to make sure that constant doesn't change. Good. Now I have written out the constant. I should be safe. I am assuming it for all k less than n, now I have to prove it for k equal to n. I am going to take T(n) and just expand it. I am going to do the obvious thing. I have this recurrence how to expand T(n). Then it involves T(n/2). And I know some fact about T(n/2) because n/2 is less than n. So, let's expand. T(n) = 4T(n/2) + n. And now I have an upper bound on this thing from the induction hypothesis. This is at most 4 times c times the argument cubed plus n. Continuing on here. Let's expand this a little bit. We have n cubed over 2 cubed. Two cubed is 8, so 4 over 8 is a half. So, we have Ωcn^3 + n. And what I would like this to be is, so at the bottom where I would like to go is that this is at most cn3. That is what I would like to prove to reestablish the induction hypothesis for n. What I will do, in order to see when that is case, is just write this as what I want, so this is sort of the desired value, cn3, minus whatever I don't want. This is called the residual. Now I have to actually figure this out. Let's see. We have cn^3, but only Ωcn^3 here, so I need to subtract off Ωcn^3 to get that lead term correct. And then I have plus n and there is a minus here, so it is minus n. And that is the residual. In order for this to be at most this, I need that the residual is non-negative. This is if the residual part is greater than or equal to zero, which is pretty easy to do because here I have control over c. I get to pick c to be whatever I want. And, as long as c is at least, oh, I don't know, 2, then this is a 1 at least. Then I have n^3 should be greater than or equal to n. And that is always the case. For example, this is true if c is at least what n is, but let's say n is at least 1 just for kicks. So, what we have done is proved that T(n) is at most some constant times n^3. And the constant is like 1. So, that is an upper bound. It is not a tight upper bound. We actually believed that it is n^2, and it is, but you have to be a little careful. This does not mean that the answer is n^3. It just means that at most n^3 is big O(n^3). And this is a proof by induction. Now, technically I should have put a base case in this induction, so there is a little bit missing. The base case is pretty easy because T(1) is some constant, but it will sort of influence things. If the base case T(1) is some constant. And what we need is that it is at most c times one cubed, which is c. And that will be true as long as you choose c to be sufficiently large. So, this is true if c is chosen sufficiently large. Now, we don't care about constants, but the point is just to be a little bit careful. It is not true that T(n) is at most 1 times n^2, even though here all we need is that c is at least 1. For the base case to work, c actually might have to be a hundred or whatever T(1) is. So, be a little bit careful there. It doesn't really affect the answer, usually it won't because we have very simple base cases here. OK, so let's try to prove the tight bound of O(n^2). I am not going to prove an omega bound, but you can prove an omega n squared bound as well using substitution method. I will just be satisfied for now proving an upper bound of n squared. Let's try to prove that T(n), this is the same recurrence, I want to prove that it is O(n^2). I am going to do the same thing. And I will write a bit faster because this is basically copying. Except now, instead of three, I have two. Then I have T(n) = 4T(n/2) + n. I expand this T(n/2). This is at most 4c(n/2)^2 + n. And now, instead of have 2 cubed, I have 2 squared, which is only 4. The fours cancel. I get cn^2 + n. And if you prefer to write it as desired minus residual, then I have cn^2 - (-n). And I want this to be non-negative. And it is damn hard for minus n to be non-negative. If n is zero we are happy, but unfortunately this is an induction on n. It's got to hold for all n greater than or equal to 1. This is not less than or equal to cn^2. Notice the temptation is to write that this equals O(n^2), which is true for this one step. cn^2 - (-n), well, these are both order n, or this is order n, this is order n squared. Certainly this thing is O(n^2), that is true, but it is not completing the induction. To complete the induction, you have to prove the induction hypothesis for n with this constant c. Here you are getting a constant c of like c + 1, which is not good. This is true but useless. It does not finish the induction, so you can sort of ignore that. This proof doesn't work, which is kind of annoying because we feel, in our heart of hearts, that T(n) = n^2. It turns out to fix this you need to express T(n) in a slightly different form. This is, again, divine inspiration. And, if you have a good connection to some divinity, you are all set. [LAUGHTER] But it is a little bit harder for the rest of us mere mortals. It turns out, and maybe you could guess this, that the idea is we want to strengthen the induction hypothesis. We assumed this relatively weak thing, T(k) is less than or equal to some constant times k^2. We didn't know what the constant was, that is fine, but we assumed that there were no lower order terms. I want to look at lower order terms. Maybe they play a role. And if you look at this progression you say, oh, well, I am getting something like n^2 and the constants are pretty damn tight. I mean the fours are canceling and the c just is preserved. How am I going to get rid of this lower order term plus n? Well, maybe I could subtract off a linear term in here and, if I am lucky, it will cancel with this one. That is all the intuition we have at this point. It turns out it works. We look at T(n) and this is 4T(n/2) + n as usual. Now we expand a slightly messier form. We have 4[c_1*(n/2)^2 - c_2*(n/2)] + n. This part is the same because the fours cancel again. So, we get c_1*n^2, which is good. I mean that is sort of the form we want. Then we have something times n, so let's figure it out. We have a plus 1 times n, so let's write it 1 minus c_2 over 2 times n. Oops, got that wrong. There is four times a two so, in fact, the two is upstairs. Let me double check. Right. OK. Now we can write this as desired minus residual. And we have to be a little careful here because now we have a stronger induction hypothesis to prove. We don't just need it is at most c_1*n^2, which would be fine here because we could choose c_2 to be large, but what we really need is c_1*n^2 - c_2*n, and then minus some other stuff. This is, again, desired minus residual. And minus residual, let's see, we have a minus 1 and we have a minus c_2. That doesn't look so happy. Plus c_2, thank you, because that again looked awfully negative. It is plus c_2. I am getting my signs, there is a minus here and there is one minus here, so there we go. Again, I want my residual to be greater than or equal to zero. And if I have that I will be all set in making this inductive argument. Office hours start this week, in case you are eager to go. They are all held in some room in Building 24, which is roughly the midpoint between here and Stata, I think, for no particular reason. And you can look at the Web page for details on the office hours. Continuing along, when is c_2 - 1 going to be greater than or equal to zero? Well, that is true if c_2 is at least 1, which is no big deal. Again, we get to choose the constants however we want. It only has to hold for some choice of constants. So, we can set c_2 greater than or equal to 1. And then we are happy. That means this whole thing is less than or equal to c_1*n^2 - c_2*n if c_2 is greater than or equal to 1. It is kind of funny here. This finishes the induction, at least the induction step. We proved now that for any value of c_1, and provided c_2 is at least one. We have to be a little more careful that c_1 does actually have to be sufficiently large. Any particular reason why? c_1 better not be negative, indeed. c_1 has to be positive for this to work, but it even has to be larger than positive depending. Sorry. I have been going so fast, I haven't asked you questions. Now you are caught off guard. Yeah? Because of the base case, exactly. So, the base case will have T(1) is c_1 time 1 squared minus c_2, we want to prove that it is at most this, and T(1) is some constant we have assumed. We need to choose c_1 to be sufficiently larger than c_2, in fact, so c_2 has to be at least 1. c_1 may have to be at least a hundred more than one if this is sufficiently large. And sufficiently large now means with respect to c_2. You have to be a little bit careful, but in this case it doesn't matter. Any questions about the substitution method? That was the same example three times. In the end, it turned out we got the right answer. But we sort of had to know the answer in order to find it, which is a bit of a pain. It would certainly be nicer to just figure out the answer by some procedure, and that will be the next two techniques we talk about. Sorry? How would you prove a lower bound? I haven't tried it for this recurrence, but you should be able to do exactly the same form. Argue that T(n) is greater than or equal to c_1*n^2 - c_2*n. I didn't check whether that particular form will work, but I think it does. Try it. These other methods will give you, in some sense, upper and lower bounds if you are a little bit careful. But, to really check things, you pretty much have to do the substitution method. And you will get some practice with that. Usually we only care about upper bounds. Proving upper bounds like this is what we will focus on, but occasionally we need lower bounds. It is always nice to know that you have the right answer by proving a matching lower bound. The next method we will talk about is the recursion-tree method. And it is a particular way of adding up a recurrence, and it is my favorite way. It usually just works. That's the great thing about it. It provides you intuition for free. It tells you what the answer is pretty much. It is slightly nonrigorous, this is a bit of a pain, so you have to be really careful when you apply it. Otherwise, you might get the wrong answer. Because it involves dot, dot, dots, our favorite three characters, but dot, dot, dots are always a little bit nonrigorous so be careful. Technically, what you should do is find out what the answer is with recursion-tree method. Then prove that it is actually right with the substitution method. Usually that is not necessary, but you should at least have in your mind that that is required rigorously. And probably the first few recurrences you solve, you should do it that way. When you really understand the recursion-tree method, you can be a little bit more sloppy if you are really sure you have the right answer. Let's do an example. We saw recursion trees very briefly last time with mergesort as the intuition why it was n log n. And, if you took an example like the one we just did with the recursion-tree method, it is dead simple. Just to make our life harder, let's do a more complicated recursion. Here we imagine we have some algorithm. It starts with a problem size n, it recursively solves a problem of size n/4, it then recursively solves a problem of size n/2, and it does n^2 work on the side without nonrecursive work. What is that? I mean that is a bit less obvious, I would say. What we are going to do is draw a picture, and we are just going to expand out that recursion in tree form -- -- and then just add everything up. We want the general picture, and the general principle in the recursion-tree method is we just draw this as a picture. We say well, T(n) equals the sum of n^2, T(n/4) and T(n/2). This is a weird way of writing a sum but why not write it that way. This is going to be a tree. And it is going to be a tree by recursively expanding each of these two leaves. I start by expanding T(n) to this, then I keep expanding, expanding, expanding everything. Let's go one more step. We have this n^2, T(n/4), T(n/2). If we expand one more time, this is going to be n^2 plus two things. The first thing is going to be (n/4)^2, the second thing is going to be (n/2)^2. Plus their recursive branches. We have T(n/16) and T(n/8). Here my arithmetic shows thin. This better be the same, T(n/8), and this should be T(n/4), I believe. You just keep going forever, I mean, until you get down to the base case where T is a constant. So, I am now going to skip some steps and say dot, dot, dot. This is where you have to be careful. We have n^2, (n/4)^2, (n/2)^2. Now this is easy because I have already done them all. (n/16)^2, (n/8)^2, (n/8)^2 again, (n/4)^2 and et cetera, dot, dot, dot, of various levels of recursion here. At the bottom, we are going to get a bunch of constants. These are the leaves. I would like to know how many leaves there are. One challenge is how many leaves in this tree could there be? This is a bit subtle, unlike mergesort or unlike the previous recurrence we solved, the number of leaves here is a bit funny because we are recursing at different speeds. This tree is going to be much smaller than this tree. It is going to have smaller depth because it has already done down to (n/16). Here it has only gone down to (n/4). But how many leaves are there in this recursion tree? All I need is an upper bound, some reasonable upper bound. I can tell you it is at most T(n^10), but that is a bit unreasonable. It should be less than n, good. Why is it less than n? Exactly. I start with a problem of size n. And I recurse into a problem that n/4 and a problem that says n/2. When I get down to one I stop. So, n/4 + n/2 = æn, which is strictly less than n. So, definitely the total number of leaves has to be at most n. If I start out with n sort of stuff and get rid of a quarter of it and then recurse, it is definitely going to be less than n stuff at the bottom. So, strictly less than n leaves. At this point, I have done nothing interesting. And then the second cool idea in recursion trees is you don't just expand this tree and see what it looks like and then say, well, God, how the hell am I going to sum that? You sum it level by level. That is the only other idea. It usually works really, really well. Here it is a bit complicated and I have to think a bit to figure out n^2 is n^2. That is the first level. Easy. The second level, I have to think a lot harder. There are three kinds of mathematicians, those who can add and those who cannot, and I am the latter kind so I need your help. Can you add these things together? It's n^2 over something. Please? (5/16)n^2. Now I really need your help. I think that one I could have done, but this one is a little bit harder. I will go look at my notes while you compute that. Any answers? 73/256. Anyone else confirm that? It seems a bit high to me. 73 does not sound right to me. 64? Closer. It is actually important that we get this right. The 256 is correct. I can tell. Everyone should know that 16^2 = 256. We are computer scientists. 25, good. We have two people saying 25, therefore it is correct by democracy. [LAUGHTER] 25 is also what my notes say, and I computed it at home. (25/256)n^2 is the right answer. Now, did anyone notice something magical about this progression? It squares each time, good. And, if we were going to add these up, you might call it? A geometric series, very good. So, it turns out this is geometric. And we know how to sum geometric series, at least you should. We started n^2. We know that at the bottom, well, this is not quite a level, we get something like n, but we are decreasing geometrically. So, the total, I mean the solution to the recurrence is the sum of all the numbers in this tree. If we added it up level by level and then add up all the levels that is going to give us the answer. This is the total computed level by level. It is just a cute way to compute it. It usually gives you nice answers like geometric answers. We have n^2(1 + 5/16 + 25/256 + ...). And, if we believe in fate and we see this three number recurrence, we know that we have the right answer. In general, it is going to be (5/16)k, at least we hope, and so on. And it keeps going. It doesn't go on infinitely, but let's just assume it goes on infinitely. That will be an upper bound that goes on forever. This is all times n^2. Now, if you are going to know one thing about geometric series, you should know that 1 + Ω + º, if you sum all the powers of 2 you get 2. We are computer scientists. We have got to know at least the binary case. This is like writing 0.1111111 in binary, actually, as 1, so this is 2. This is even smaller. We have 5/16, that is less than a half and then we are squaring each time, so this is even less than 2. If you want, there is a nifty formula for solving the general geometric series, but all we need is that it is a constant. This is O(n^2). It is also O(n^2). It is pretty obvious that it is O(n^2) because the top thing is n^2. So, there is our lower bound of n^2. And we have it within a factor of 2, which is pretty good. You actually get a better factor here. So, that is recursion-tree method. It is a little shaky here because we have these dot, dot, dots, and we just believe that it is geometric. It turns out most of the time it is geometric. No problem here. I would definitely check it with the substitution method because this is not obvious to me that it is going to be geometric. In the cases we will look at in a moment, it will be much clearer, so clear that we can state a theorem that everything is working fine. And still time, good. So, that was recursion-trees. There is one more method we are going to talk about, and you could essentially think of it as an application of the recursion-tree method but it is made more precise. And it is an actual theorem, whereas recursion trees, if the dot, dot, dots aren't obvious, you better check them. The sad part about the master method is it is pretty restrictive. It only applies to a particular family of recurrences. It should be T(n) = aT(n/b) + f(n). Am I going to call it f? Yes, I will call it f. In particular, it will not cover the recurrence I just solved because I was recursing on two different problems of different sizes. Here, every problem you recurse on should be of the same size. There are a subproblems. A way to think of this is a recursive algorithm. You have a subproblems. Each of them is of size n/b, so the total costs will be this. Then you are doing f(n) nonrecursive work. A few constraints. a should be at least 1, should have at least 1 recursion. b should be strictly greater than 1. You better make the problem smaller or else it is going to be infinity. And f should have some nice property. f(n) should be asymptotically positive. How many people know what asymptotically positive means? No one. OK, you haven't read the textbook. That's OK. I haven't read it either, although don't tell Charles. And he'd notice. And what might you think asymptotically positive means? That we can do a little bit better. Sorry? Yes, it means for large enough n, f(n) is positive. This means f(n) is greater than zero for n, at least some n_o, so for some constant n_o. Eventually it should be positive. I mean, we don't care about whether it's negative 1 for n=1, not a big deal. It won't affect the answer because we only care about the asympotics within. The master method, you gave it a recurrence of this form, it tells you the answer. That is the great thing about the master method. The annoying thing about the master method is that it has three cases. It is a big long. It takes a little bit longer to memorize than all the others because the others are just ideas. Here we need to actually remember a few things. Let me state the theorem. Well, not quite yet. There is one very simple idea, which is we are going to compare this nonrecursive work f(n) with a very particular function n^(log_b(a)). Why n^(log_b(a))? You will see later. It turns out it is the number of leaves in the recursion tree, but that is foreshadowing. So, it is either less, equal or bigger. And here we care about asymptotics. And we have to be a little bit more precious about less, equal or bigger. You might think well, it means little o, big Theta, or little omega. It would be nice if the theorem held for all of those cases, but it leaves some gaps. Let's start with Case 1. Case 1 is when f is smaller. And not just that it is little o, but it is actually quite a bit smaller. It has got to be polynomially smaller than n^(log_b(a)). For some positive epsilon, the running time should be this n to this constant log base b of a minus that epsilon, so it is really polynomially smaller than n^(log_b(a)). We cannot handle the little o case, that's a little bit too strong. This is saying it is really quite a bit smaller. But the answer then is really simple, T(n) = Theta(n^(log_b(a))). Great. That is Case 1. Case 2 is when f(n) is pretty much equal to n^(log_b(a)). And by pretty much equal I mean up to poly log factors. This is log base 2 of n to the power k. You should know this notation. For example, k could be zero. And then they are equal up to constant factors, for some k greater than or equal to zero. Less than will not work, so it is really important that k is non-negative. It should probably be an integer. It doesn't actually matter whether there is an integer, but there it is. It could n^(log_b(a)) times log n or just times nothing, whatever. Again, the solution is easy here, T(n) = Theta(n^(log_b(a))* lg^(k+1)(n)). Presumably it has to be at least times log k. It turns out it is log to the k plus 1 of n. That is Case 2. We have one more case which is slightly more complicated. We need to assume slightly more for Case 3. But Case 3 is roughly when f(n) grows bigger than n^(log_b(a)). So, it should be capital Omega, here is one place where we get to use omega, (n^(log_b(a)) + epsilon) for some positive epsilon. It should grow not just bigger but polynomially bigger. Here it was growing just a log factor bigger, poly log, and here it is a polynomial factor. In this case, we need another assumption about f because we worry a little bit about how quickly f grows. We want to make sure that as you go down the recursion f gets smaller. It would be kind of nice if f gets smaller as you go down, otherwise you are, again, trying to sum to infinity or whatever. I see why this is for some epsilon prime greater than zero. What I would like is that if I just sort of take the recurrence, this T(n) and just throw in fs instead, f(n) should be somehow related to af(n/b). What I would like is that f(n), which is at the top of the recursion tree, should be bigger than the thing at the next level down. The sum of all the values at the next level down should be bigger by some constant factor. Here I have the next level down is at most some 1 - e, something strictly less than 1, some constant strictly less than 1 times the thing at the top level. I need that to make sure things are getting smaller as I go down. Then T(n) = Theta[f(n)]. And that is the theorem. This is the master theorem or whatever you want to call it. It is not named after some guy name Master. It is just the master of all methods because it is very easy to apply. Let's apply it a few times. It is a bit much to take in all at once. And then I will give you a sketch of the proof to see that it is really not that surprising this is true if you look at the recursion-tree. But first let's just try using it. For example, we could take T(n) = 4T(n/2) + n. This is a, this is b, this is f(n). The first thing we should compute is n^(log_b(a)). This I think even I can do. Log base 2 of 4. Yeah, log base 2 I can do. This is n^2. OK, so is f(n) smaller or bigger than n^2? Well, f(n) = n. n^2 is clearly bigger by a polynomial factor. So, we are in Case 1. What is the answer? n^2, yeah. It is T(n^(log_b(a))), which here it is just n^2. Let's do some slight variation. I am going to keep a and b the same and just change f. Let's say T(n) = 4T(n/2) + n^2. This is like drill spelling. n^2 is asymptotically the same as n^2 even up to constants. What is the answer? This is Case 2. It is slightly harder. What is k in this example? Zero. The answer is? Survey says? n^2 log n. Good. And a couple more. T(n) = 4T(n/2) + n^3. What is the answer? n^3. This is Case 3. I know this is pretty boring. At this point we are just applying this stupid theorem. How about n^2/lg n? What is the answer? Good. In this case no one should answer. It is a big tricky. I forget exactly the answer. I think it is like n^2 log log n over log n, no? Oh, no. n^2 log log n, that's right. Yeah. But you shouldn't know that, and this doesn't follow from the master method. This is something you would have to solve, probably with the recursion-tree would be a good way to do this one, and you need to know some properties of logs to know how that goes. But here the master method does not apply. And so you have to use a different method. OK. The last thing I want to do is tell you why the master method is true, and that makes it much more intuitive, especially using recursion-trees, why everything works. This is a sketch of a proof, not the full thing. You should read the proof in the textbook. It is not that much harder than what I will show, but it is good for you to know the formal details. I don't have time here to do all of the details. I will just tell you the salient parts. This is the proof sketch or the intuition behind the master method. What we are going to do is just take the recursion-tree for this recurrence and add up each level and then add up all the levels and see what we get. We start with f(n) at the top after we have expanded one level. Then we get a different problems, each of n/b. And after we expand them it will f(n/b) for each one. They are all the same size. Then we expand all of those and so on, and we get another a subproblems from there. We are going to get like f((n/b)^2). That is sort of decreasing geometrically the size, and so on and so on and so on, until at the bottom we get constant size problems. This is a bit special because this is the base case, but we have some other constant at the bottom. We would like to know how many leaves there are, but that is a little bit tricky at the moment. Let's first compute the height of this tree. Let me draw it over here. What is the height of this tree? I start with a problem of size n. I want to get down to a problem of size 1. How long does that take? How many levels? This is probably too easy for some and not at your fingertips for others. Log base b of n, good. The height of this tree is n^(log_b(a)), because it is just how many times I divide by b until I get down to 1. That is great. Now I should be able to compute the number of leaves because I have branching factor a, I have height h. The number of leaves is a^h, a^log_b(n). Let me expand that a little bit. a^log_b(n), properties of logs, we can take the n downstairs and put the a upstairs, and we get n^(log_b(a)). Our good friend n^(log_b(a)). So, that is why Our good friend n^(log_b(a)) is so important in the master method. What we are doing is comparing f, which is the top level, to n^(log_b(a)), which up to theta is the bottom level. Now the leaves are all at the same level because we are decreasing at the same rate in every branch. If I add up the cost at the bottom level, it is Theta(n^(log_b(a))). I add up the things at the top level it is f(n), not terribly exciting. But the next level, this is a little bit more interesting, is af(n/b), which should look familiar if you had the master method already memorized, it is that. So, we know that af(n/b) has decreased by some constant factor, 1-epsilon prime. We have gone down. This is a constant factor smaller than this. And then you sum up the next level. It is going to be like a^2f(n/b^2). I see that I actually wrote this wrong, the parentheses. Sorry about that. It is not (n/b)^2. It is (n/b^2). So, this sequence, in Case 3 at least, is decreasing geometrically. If it is decreasing geometrically up to constant factors, it is dominated by the biggest term, which is f(n). Therefore, in Case 3, we get Theta[f(n)]. Let's look at the other cases, and let me adapt those cases to how much time we have left. Wow, lot's of time. Five minutes. Tons of time. What to do? Let me write that down. Case 3, the costs decrease. Now, this is a place I would argue where the dot, dot, dot is pretty obvious. Here, this is damn simple, it is a^kf(n/b^k). And, in Case 3, we assume that the costs decrease geometrically as we go down the tree. That was sort of backwards to start with Case 3. Let's do Case 1, which is sort of the other intuitively easy case. In Case 1, we know that f(n) is polynomially smaller than this thing. And we are sort of changing by this very simple procedure in the middle. I am going to wave my hands if this is where you need a more formal argument. I claim that this will increase geometrically. It has to increase geometrically because this f(n) is polynomially smaller than this one, you are going to get various polynomials in the middle which interpret geometrically from the small one to the big one. Therefore, the big one dominates because it is, again, geometric series. As I said, this is intuition, not a formal argument. This one was pretty formal because we assumed it, but here you need a bit more argument. They may not increase geometrically but they could increase faster, and that is also fine. So, in Case 3, you are dominated, I mean you are always dominated by the biggest term in a geometric series. Here it happens to be f(n) and here you are dominated by n^(log_b(a)) with a bottom term, oh, Theta. Case 2, here it is pretty easy but you need to know some properties of logs. In Case 2, we assume that all of these are basically the same. I mean, we assume that the top is equal to the bottom. And this is changing in this very procedural way. Therefore, all of the ones in the middle have to be pretty much the same. Not quite because here we don't have the log factor. Here we have a log to the k. We have n^(log_b(a)) times log to the kn. Here we don't have the log to the k. So, the logs do disappear here. It turns out the way they disappear is pretty slowly. If you look at the top half of these terms, they will all have log to the k. The bottom half they will start to disappear. I am giving you some oracle information. If you take logs and you don't change the argument by too much, the logs remain. Maybe halfway is too far. The claim is that each level is roughly the same, especially the upper most levels are all asymptotically equal. Roughly the same. And, therefore, the cost is one level, here like f(n) times the number of levels, h. And h is log base b of n. B is a constant so we don't care. This is Theta(lg n). And, therefore, we get T(n) = (n^(log_b(a)) lg^(k+1)(n)) times another log n. So, we get [f(n)lg n]. That is the very quick sketch. Sorry, I am being pretty fuzzy on Cases 1 and 2. Read the proof because you will have to, at some point, manipulate logs in that way. And that is all. Any questions? Or, you are all eager to go. OK. Thanks. See you Wednesday.