Lecture 36: Sampling Theorem

Flash and JavaScript are required for this feature.

Download the video from iTunes U or the Internet Archive.

Instructor: Prof. Gilbert Strang

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high-quality educational resources for free. To make a donation, or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR STRANG: OK, hi. So I've got homework nine for you. Ready to return at the end. Also, the department asked me to do evaluations, but that's the end of the lecture. Then, so everybody knows there's a quiz tomorrow night. And shall I just remember the four questions on the quiz? I mean, not the details but the general idea of the questions. Details OK too. Yes. Yeah, so there'll be one question on a Fourier series. And you should know the energy equality for all of these possibilities, connecting the function squared with the coefficient squared. A second one on the discrete Fourier transform. Cyclic stuff. The third one on the Fourier integral. And have a look at the applications to solving an ODE. I did one in class. The one in class was the one in the book -u''+a^2*u=f(x), so this will be. So have a look at that application. This is, of course, on minus infinity to infinity. and then a fourth question on convolution. OK. And this afternoon, of course, I'll be here to answer any questions from the homework, from any source, for these topics. Are there any questions just now, though? I'm OK to take questions. I thought I'd discussed today a topic that involves both Fourier series and Fourier integrals. It's a kind of cool connection and it's linked to the name of Claude Shannon who created information theory, who was a Bell Labs guy and then an MIT professor. So I should put his name in. Shannon.

OK, so this is, yeah. You'll see. So it's not on the quiz but it gives me a chance to say something important, and at the same time review Fourier series and Fourier integrals. So let me start with the problem. The problem comes for an A to D converter. So what does that mean? That means, this A is analog. That means we have a function, A for analog. And D for digital. So we have a function, like-- So f(x), say, all the way minus infinity to infinity, so we'll be doing, that's where the Fourier integral's going to come up. So that's analog. All x, it's some curve. And people build, and you can buy, and they're sold in large quantities, something that just samples that function. Say, at the integers. So now I'll sample that function and let me take the period of the sample to be one, so that I'm going to take the values f(n). So now I've got something digital that I can work with, that I can compute with. And, so the sampling theorem-- Well, I mean, the question is-- Yeah, the sampling theorem is about this question, and it seems a crazy question, when do these numbers -- That's just a sequence of numbers. This x was all the way from minus infinity to infinity. And similarly, n is numbers all the way from minus infinity to infinity, they're just samples. When does that tell me the function? When can I learn from those samples, when do I have total information about the function? Now, you'll say impossible. Right? So suppose I draw a function f(x), OK? And I'm going to sample it at these points. All the way, so these are the numbers, these are my f(n). Sampling at equal intervals, because if we want to use Fourier ideas, equal spacing is the right thing to have. So when could I recover the function in between? Well, you'd say, never. Because how do I know what that function could be doing in between.

So let me take the case when all the samples are zero. And let's think about that case. Suppose, what could the function be if all the samples, if these are all zeroes, forever. OK, well there's one leading candidate for the function, the zero function. Now, you'll see the whole point of the sampling theorem if you think about other, what other functions? Familiar functions, yeah. I mean, we could, obviously, any, all sorts of things. But since we're doing Fourier, we like to pick on the sines, cosines, the special functions, and think about those in particular. So, somebody said sines. Now, what function, so a sine function certainly, the sine function hits zero infinitely often. What frequency, so sine of what would give me the same answer? The same samples. If I put this sine function that you're going to tell me, so you're going to tell me it's sine of something will have these zero values at all the integers, at zero, one, two, minus one, minus two, and so on. So what would do the job? Sine of? Of what will hit zero. So I'm looking for a sine function, I guess I'm looking first for the function that just does that. And what is it? sin(pi*x). And now tell me some more. Tell me another function. Which will also, it won't be that graph. sin(2pi*x). And all the rest, OK? Let me just use a word that's kind of a handy word. Of course, let's put zero on the list here. OK. So this is where k, the frequency, usually appears. This is where k--

The word I want to introduce is alias. This frequency, pi, is an alias for this at frequency zero. Here's the, it's a different function but yet the samples are the same. So if you're only looking at the samples you're getting the same answer but somehow the function has a different name. So that frequency and this frequency, and all those others would be alias. Can I just write that word down, because you see it often. Alias. That means two frequencies, like pi and 2pi, and zero or whatever, that give you the same samples. OK, so now comes Shannon's question. So we have to make some assumption on the function. To knock out those possibilities. We want to know a limited class of functions. Which don't include these guys. So that within this limited class of functions, this is the only candidate and we have this possibility of doing the impossible. Of determining that if I know zeroes here, the function has to be zero everywhere. OK, now the question is what class of functions? We want to eliminate these guys, and sort of, your instinct is, you want to eliminate functions that, you know, if it's not zero then it's got to get up and back down in every thing. It could do different things in different intervals. But somehow it's got to have some of these frequencies. Pi or higher. Would have to be in there. So this is the instinct. That if I limit the frequency band, so I'm going to say f(x) is band-limited, can I introduce that word? I'll maybe take a moment just ask you if you've seen that word before. How many have seen this word, band-limited? Quite a few but not half. OK.

Band-limited means the band is a band of frequencies. So the function's band-limited when its transform, this tells me how much of each frequency there is. If this is zero, in some band. In some band, let's say, all frequencies below something. And let's not even put equal in there. OK. But that's not critical. Band-limited, I have to tell you the size of the band. And the size of the band, the limiting frequency is this famous Nyquist frequency, so Nyquist is a guy's name. And the Nyquist frequency in our problem here is pi. This is the Nyquist frequency. If we let that frequency, that's the borderline frequency. And there would be a similar Nyquist sampling rate. So Nyquist is the guy who studied the sort of borderline case. So the point is that if our, say, band-limited by pi, I have to tell you, so band-limited means there's some limit on the band. And our interest is when that limit is the Nyquist frequency. The one we don't want to allow, so we-- This is the point. So this will be the idea. That if we take this class of function, that band-limited-- Those are called band-limited functions, and they're band-limited specifically by the Nyquist limit. If we take those, then the idea is that then we can reconstruct from the samples. Because the only function that has zero samples in that class is the zero function. You see that class has knocked out, is not allowing these guys. Of course, haven't proved anything yet. And I haven't shown how to reconstruct. Well, of course, we quickly reconstructed the zero function out of those zeroes, but now let me take another obviously important possible sample.

Suppose I get zero samples except at that point, where it's one. OK, now the question is what function, f(x)-- Can I fill in, in between zero, zero, zero, zero, one, zero, zero, zero, can I fill in exactly one function that comes from this class? So that I have now the answer for this highly important sample? The sample that's all zeroes except for the delta sample, you could say. OK, so I'm looking for the function now which is one at that point. And zero at the others. So here's a key function. And I'll show you what it is. So the function, this function that I'm going to mention, will get down here, it'll oscillate, it'll go forever. It's not like a spline. Splines made it to zero and stayed there, right? The cubic spline, for example. OK, but I guess, yeah, that somehow that function, we're not in that league. We're in this band-limited league. In a way you could say that, I mean, what's the key connection between dropoff of the Fourier transform? So if the Fourier transform drops off fast, what does that tell me about the function? It's smooth, thanks. That's exactly the right word. If the Fourier transform drops off fast, the function is smooth. OK, this is really an extreme case. That it's dropped off totally. You know, it's not just decay rate, it's just zonk, out. Beyond this band of frequencies. And so that gives, you could say, sort of a hyper-smooth function. I mean, so smooth that you know everything by knowing the sample.

OK, now I'm ready to write down the key function, a famous function that has those samples. And that function is sin(pi*x)/(pi*x). I don't know if you ever thought about this function, and it has a name. Do you know its name? Sinc. It's the sinc function. Which is a little-- you know, the name's a little unfortunate. Mainly because you know, you're using those same letters S I N but it's a c that turns it, that gives it-- So this is called the sinc function. sinc(x). But the main thing is its formula. OK, well everybody sees that at x=1, the sin(pi) is zero, at x=2, the sin(pi) is zero, all these ones we've seen already. And now what happens at equals zero? Do you recognize that this function, as x goes to zero, is a perfectly good function? I mean, it becomes 0/0 at x=0. But the limit, we know to be one. Right? This sin(theta)/theta is one as theta approaches zero, right? So that does have that correct sample. And now what's the, I claim that that is a band-limited function. And you'll see that that function pushes the limit. It's right-- Nyquist barely lets it in. Now here's a calculation. So this is our practice. What is f hat of k for that function? Now, let me think how to do this one. So just to understand this better, I want to see that that is a band-limited function and what is its Fourier transform. OK, now once again here we have a function where if I want to do-- How best to do this? You could say well, just do it. As I would say on the quiz, just go ahead and do it. But you'll see I'm going to have a problem, I think. But this gives us a chance to remember the formula. So what's the formula for the Fourier integral transform of this particular? So my function is the sinc function, sin(pi*x) over sin(pi*x).

So how do I get its, I do a what here? e^(-ikx), and am I doing dx or dk? dx, right? And I'm going from minus infinity to infinity. And am I, do I have a 2pi? Yes or no? Who knows, anyway? Right. In the book I didn't? OK. Now, well, I don't know the answer. But so let's, it's much better to start with the answer, right, and check that-- So let me say what I think the answer is. I think the answer is, it's a function that's exactly as I say, it pushes the limit from, this is k. It's the square wave, it's zero, the height is one. It's the function that's zero all the way here, all the way there. I think that that's the Fourier transform of that function. And just before we check it, see how is Nyquist got really pushed up to the wall, right? Because the frequency is non-zero right, all the way through pi. But pi is just one point there. And anyway, I think we won't get into philosophical discussion about whether, you know, is that limited to pi? I don't know whether to put, you saw me chicken out here. I didn't know whether to put less or equal or not, and I still don't. But this is making the point that that's the key frequency. So this particular function, what I'm saying is this particular function has all the frequencies in equal amounts over a band, and nothing outside that band. And that's the Nyquist band.

OK, now why is this the correct answer here? I guess the smart way would be, this is a good function, easy function. So let's take the transform in the other direction. Start from here and get to here. Right? That would be convincing. Because we do know that that pair of formulas for f connected to f hat connected to f, they go together. So if I can show that I go from here, that that Fourier integral takes me from here to there, then this guy will take me back. So let me just do that. Because that's a very very important one that you should be prepared for. Right. So now what do I want to do? Here's my function of k, and I'm hoping that I recall-- Now, what do I do when I want to do the transform in the opposite direction? It'll be an e^(+ikx), right? d what? dk, now. From k equal minus infinity to to infinity. And now I think I do put in the 2pi, is that right? And the question is, does that bring back the sinc function? If it does, then this was OK in the other direction. If the transform's correct in one direction then the inverse transform will be correct. So I just plan to do this integral. f hat of k, of course, is an easy integral now. f hat of k is one over, between minus pi and pi, so I only have to do over that range where f hat of k is just a one. And now that's an integral we can certainly do. So I have 1/(2pi), integrating e^(ikx) will give me e^(ikx) over ix. Now, remember I'm integrating dk. Oh, look. See, we're showing this x now in the denominator. That we're hoping for. And now I have to do that between k is minus pi and pi. So this is like, so I get 1/(2pi), e^(i*pi*k) minus e^(-i*pi*x), right? Over the ix. OK so far?

I was doing a k integral and I get an x answer. And I want to be sure that this x answer is the x answer I want, it's the sinc function. OK, it is. Right? I recognize the sine, e^(i*theta)-e^(-i*theta), divided by two, I guess. Is the sine, right? So I have 1/(2pi). And here is ix-- Well, no, the i is part of that sine. So I'm just using the fact that e^(i*theta)-e^(-i*theta) is, this is cosine plus i sine, subtract cosine. But subtract minus i sine, so that will be 2i*sin(theta), right? We all know, and now theta is pi*x here. So I have two-- let me keep the i there, and 2i*sin(pi*x). You see it works. Just using this, replacing this by the sine, the two cancels the two. The i cancels the i, and I have sin(pi*x) over pi*x, that's the sinc function. OK, so that's the function we've now checked. We've checked two things about this function. It has the right samples, zero, zero, zero, zero, one at x=0 and then back to zero. It is band-limited, so it's the guy, if this was my sample, if this was my f(n), zero, zero, zero, one, zero, zero, zero, then I've got it. It's the right function. OK, we can create Shannon's sampling formula. Shannon's sampling formula gives me the f(x) for any f(n). Maybe you can spot that, now. So this is going to use the shift invariance. Oh yeah, let's-- Tell me what the, let's take one step here. Suppose my f(n)'s were zero, zero, zero, zero, and a one there. So suppose-- This is, for the exam too, this idea of shifting is simple and basic. And it's a great thing to be able to do.

So this wouldn't be the right answer. That function produced the one at zero. Tell me what function, copying this idea, will produce all zeroes except for a one at this point. So I'll put this question over here. Suppose I'm all zeroes at that point but at this point I'm up and then I go back to zeroes. What function is giving me that? You see it does decay because of the x in the denominator, it kind of goes to zero but not very fast. OK, what's that function? I replace x by x-1, right. So the function here is sin(pi(x-1)), divided by pi(x-1). I just change the x to x-1. Now again, at x=0, this is sin(pi), sin(-pi), it's safely zero, but at x=1 that's now the point where I'm getting 0/0. And the numbers are right to give me the exact answer, one. OK, so now we see what to do if the sampling turned out to give us this answer. And now can you tell me the whole formula? Can you tell me the whole formula, so now I'm ready for Shannon's sampling theorem. Is that f(x), if f(x) is band-limited, then I can tell you what it is at all x. So I'm going back to the beginning of this lecture. It's a miracle that this is possible. That we can write down a formula for f(x) at all x, only using f(n)'s. OK, now it's going to be a sum. From n equal minus infinity to infinity, because I'm going to use all the f(n)'s to produce the f(x). And now what do I put in there? So I want my formula to be correct. I want my formula to be correct in this case, so that if all the f's were zero except for the middle one, then I want to put sin(pi*x)/(pi*x) in there. The sinc function. And I also want to get this one right. If all the f's are zero, so there'll only be one term. If this is the term, I want that to show up. OK, what do I-- Yeah, you can tell me. Suppose this is hitting at n.

We'll just fix this and then you'll see it. Suppose that all the others, n, n-1, n+1, all those, it's zero, but at x=n, it's one. Now what should I have chosen? What's the correct-- I'm just going to make it easy for all of us to, yes. What's the good sinc function which peaks at a point n? Again, I'm just shifting it over. So what do I do? Put in, what do I write here? n. I shift the whole thing by n. So that's the right answer when this hits at n. So now maybe you see that this is going to be the right answer for all of them. You see that, we're using linearity and shift invariance. The shift invariance is telling us this answer for wherever the one hits. That's what we need. And then by linearity, I put together whatever the f is at that point, that would just amplify the sinc. And then I have to put them in for all the other values. That's the Shannon formula. That's the Shannon formula, and this function is band-limited, let's see. What's the-- Oh, yeah. What's the, do you see that this one, that this guy is band-limited? We checked, right? We checked that this one, sin(pi*x)/(pi*x), that was band-limited because we actually found the band. Now, that just gives us another chance to think. Allowed. What's the Fourier transform of this guy? My claim is that it's also in this band. Non-zero only in the band, and zero outside the Nyquist band. What is the transform of that? What happens if you shift a function, what happens to its transform? So that's one of the key rules that makes Fourier so special. If I took this sine, let me write this guy again. This was the un-shifted one. That connected to the, what am I going to call that, the box function. The box function, the square wave. Well, box is good. Now, what if I shift the function? If I shift a function, what happens to its Fourier transform? Anybody remember? You multiply it by, so if I shift the function, I just multiply this box function, this is a box function in the k, times something, e to the i shift, and the shift was one, right? Is it just e to the ik, d being the shift distance. Oh, the shift distance was n. Right. And possibly minus, who knows.

OK, but what's the point here? The point is that it's still zero outside the box. Inside the box, instead of being one, it's this complex guy. But no change. It's still zero outside the box. It's still band-limited. So this is the transform of this guy. And then the transform of this combination would be still in the box, multiplied by some messy expression. So what was I doing there? I was just checking that, sure enough, this guy is band-limited. And it's band-limited, it gives us the right f(n)'s, of course. Everybody sees that at x=n, let's just have a look now. We've got this great formula. Plug in x=n. What happens when you plug in at one of the samples, you look to see what this A to D converter produced at time n, and let's just see. So at x=n, the left side is f(n). Why is the right side f of that n? That particular n? Maybe I should give a specific letter to that n. So at that particular sample, this left side is f(N), and I hope that the right side gives me f at that capital N. That particular one. Why does it? You're all seeing that. At x equal capital N, these guys are all zero, except for one of them. Except for the one when little n and capital N are the same. Then that becomes the one. And I'm getting f at capital N. So it will give me that, for the n=1, the one term, yeah. I don't know if it was necessary to say that. You've got the idea of the sampling formula.

I could say more about the sampling, just to realize that the technology, communication theory is always trying to, like, to have a greater bandwidth. You always want a greater bandwidth. But if the bandwidth, which is this, is increased, well by the way, what does happen? Suppose it's band-limited by pi, oh, by pi/T. Let's just, I normalize things to choose samples every integer. Zero, one, two, three. And that turned out that the Nyquist frequency was pi. Now, what sampling rate would correspond to this band, which could be-- Well, let me just say what it is. That would be the Nyquist frequency for sampling every T. Instead of a sampling interval of one, if I sample every T, 2T, 3T, -T, my sampling rate is T, so if T is small, I'm sampling much more. Suppose T is 1/4. If T is 1/4, then I'm doing four samplings. I'm taking four samples, I'm paying more for this A to D converter, because it's taking four samples where previously it took one. How do I get paid back? What's the reward? The reward is if T is 1/4, then the Nyquist limit is 4pi. I can get a broader band of signals by sampling them more often. Let me just say that again, because that's the fundamental idea behind it. If I sample more often, say, so fast sampling would be small-- Fast samples would be small t and then a higher Nyquist. A higher band limit. More functions allowed. If I sample more often I'm able to catch on to more functions. If I sample-- And that's what, I mean, now, communications want wide bands. And this is where they get limited. I mean, this is, you could say, the fundamental, I don't know whether to say physical limit, sort of maybe Fourier limit on sampling theory. Is exactly this Nyquist frequency.

OK, questions or discussion about that. OK. So that's an example that allowed us to do a lot of things. I did want to ask for your help doing these evaluations. Let me say what I'm going to do this afternoon. I'm going to answer all the questions I can, and I planned, when there is a pause, and nobody else asks, I plan to compute the Fourier integral and Fourier series, say, Fourier series, for a function that has, it's going to be like the one today except this is going to have a height of 1/h, and a width of h. So that's, in case you're not able to be here this afternoon, I thought I'd just say in advance what calculations I thought I would do. So there's a particular function f(x), it happens to be an even function. We'll compute its Fourier coefficients, in and we'll let h go to zero. To see what happens. It's just a good example that you may have seen on older exams. OK, well can I just say a personal word before I pass out-- So evaluations, if you're willing to help, and just leave them on the table, would be much appreciated. I'll stretch out the homeworks. I just want to say I've enjoyed teaching you guys. Very much. Thank you all, and-- Thanks.