Lecture 25: Linear Gaussian Channels

• Download this transcript - PDF (English - US)

PROFESSOR: OK, well I'm going to start now. As I say, I know that the wireless course has its final at 11:00, so we might not see the people who are taking it. Is anyone here in the wireless course? OK. Q.E.D. Ipso facto ergo sum. OK, just to remind you, I think we have Problem Set 10 solutions today. In any case, they'll be available on the web. The final is next Tuesday. Do bring calculators if you have them. They won't be absolutely essential, but they might be useful. You may bring five sheets of notes. It's a good exercise to make these up. And otherwise, it's closed-book. Ashish will hold a review session tomorrow, 4:00 to 6:00, 26-328, which I'm sure will be excellent as usual.

All right, just to remind you, we're in Chapter 14. Most of the course we talked about coding for power-limited channels, where basically you could get away with binary codes with very little loss of optimality. For bandwidth-limited, additive white Gaussian noise channels, we've got to consider non-binary, multi-level codes, we've really got to go out into Euclidian space and use Euclidian distance as our metric rather than Hamming distance.

So we started out with a simple idea, build a lattice constellation. A lattice is an infinite set of points in end space that has a group property. That's it, a discrete set of points. We take a finite set of them by chopping out a region R, a compact region, which chops out a finite number of points in this lattice or in a translate of the lattice. And we went through this exercise of computing the union-bound estimate, which I remind you is a reasonable estimate for codes that are not too complex. We're not going to use this for capacity-approaching codes. But for codes, where..., basically dominated by minimum distance properties, how many nearest neighbors are there, the likeliest error events are all to the nearest neighbors, we can use the union-bound estimate, which basically just tallies the probability of error to the nearest neighbor events by the union-bound, which itself is inaccurate if you get too many near neighbor events.

And we've got this nice approximate expression that the probability of error per n dimensions, if we're signaling in n dimensions, is the number of nearest neighbors in the lattice times the Q to the square root of a function times an expression, which involves a coding gain of the lattice, and separably a shaping gain of the region times 3 times SNR norm. So this embodies all kinds of simplifications. It separates the effects of the lattice and the region. Basically, we can optimize each of these separately to optimize this expression.

And also, by the way, by talking about SNR norm, which is SNR normalized by 2 to the spectral efficiency, it makes this whole expression independent of rate or equivalently independent of spectral efficiency. We saw this back when we talked about m by m QAM, we got a universal expression. The probability of error was simply this expression with a 3SNR norm in here - regardless of what m was (regardless of how big the QAM constellation was), we got the same approximate expression. So this is also independent of the size of the constellations in this large constellation, high SNR approximation. Once you get to enough points, everything sort of washes out. So it's very useful.

And I remind you that the coding gain basically depends on the minimum square distance normalized by the density of the lattice, if you like, normalized to two dimensions. The shaping gain is basically the shaping gain of an n-cube divided by-- this is the normalized second moment of an n-cube, this is the normalized second moment of your region, which better be better than that of an n-cube, meaning smaller. And writing it out, it's equal to this, where this is the second moment of your region. And this is the volume, again, normalized to two dimensions.

So the baseline constellations, we showed these were invariant to Cartesian products. In any number of dimensions, n, we can take the integer lattice or any translate of the integer lattice and we can take a region, which is the n-cube -- which will chop out basically a square or cubic constellation, a regular constellation in n dimensions-- and we'll get that the coding gain is the shaping gain is 1. So we'll get just our baseline expression for any integer lattice shaped by an n-cube centered on the origin.

And this is what it looks like. This is back from Chapter Four or whatever. We get this universal curve, the baseline curve, which holds for m-PAM, m by m QAM, or you can Cartesian product these up into as high a dimension as you like. The constellation we're talking about is really m-PAM Cartesian product to the nth. For any of those constellations, if we normalize the probability of error to two dimensions-- which is what I'm recommending to you, normalize everything per two dimensions in the high SNR regime-- then we basically get the number of near neighbors in Z-squared, the QAM lattice, which is 4 times Q to the square root of 3SNR norm.

And it looks like this. We plot probability of error per two dimensions over SNR norm. I think at 10 to the minus 5, it's 8.4 dB. Does anybody remember? But it's somewhere around 9 dB. And the Shannon limit, as we recall, is for SNR norm, it's at 0 dB. So this is where we can play. Any code will fall somewhere between these two curves, and the name of the game is to get a code the gets as close to the Shannon limit as possible. And at least in this moderate complexity regime, the way we play the game is we try to optimize the shaping region, we try to optimize the lattice, we're going to get a certain gain from shaping, and a certain gain from coding, de-rated by the number of nearest neighbors, which is going to turn out to be an important effect for some well-known lattices because they have huge numbers of near neighbors. And that will give us some curve somewhere in here, and we plot that and we're done. We've written a paper for that particular lattice constellation, for its performance.

Today I'm going to talk quickly about shaping, because that's a very easy story, and obvious, and it won't take long. Then I'm going to talk about lattices, which is really a subject in mathematics, what's known about lattices. I guess the decoding part of that story is not mathematics. That's engineering. And then I'm going to talk about trellis codes, which are the analog of convolutional codes, and which, similarly to the way convolutional codes beat block codes for the bandwidth for the power-limited regime, trellis codes tend to beat lattice codes in terms of performance versus complexity for the bandwidth-limited regime.

So, shaping. Shaping is really an almost trivial story. This really wasn't recognized as a separable issue until perhaps sometime in the '80s, when trellis-coded modulation was invented. In '82 was Ungerboeck's paper. That was really the breakthrough in bandwidth-limited coding, that trellis codes were the first bandwidth-limited codes that were widely implemented. And after people analyzed them for a while, they realized that you could separate out the shaping effect from the coding effect, which is what I've shown by this expression.

So anyone care to guess what the optimum region is in n dimensions? I guess I've already mentioned it last time. The optimum region, what do we want to do? We want to minimize the normalized second moment-- well, a second moment with appropriate normalization, what would you do? Well, the optimum region for just about anything is an n-sphere, and it certainly is here. So we take Rn equals an n-sphere. If we have a two-dimensional sphere, what are its parameters? We certainly know that the volume of this region, if this is R, it's pi times R squared. And the second moment of this region, how do we do that? We integrate ... go to polar coordinates, 0 to 2pi, d pi, 0 to R, rdr. And then what are we measuring? The second moment is r squared, but we normalize it per dimension, which is 2.

Oh, and we've got to take the uniform distribution over v of R, and that's one over pi R squared. So have I got it now? This is the second moment of a uniform distribution, a probability distribution over the 2-sphere. And so this will be 1 over 2pi R squared times R 4th over 4, right? Is that all correct? And so p of R over v of R is already not normalized to two dimensions, so we don't have to re-normalize it. I didn't get the 2pi in here. I need some help, class. When it comes down to integration, I'm not necessarily the world's best.

So R squared over 4... R squared over 4pi R squared, so we get 1 over 4pi. Is that right? That looks about right because it's fairly close to 1 over 12. So what is the shaping gain of a 2-sphere? And I can just say 2-sphere, because I know it's invariant to scaling. Rotation isn't going to do much to it. It's 1/12 over this, the normalized second moment, 1 over 4pi, which is pi over 3, which is about 2/10 of a dB. OK, so I've improved the second moment by 2/10 of a dB, reduced it to 2/10 of a dB.

This is not very impressive. So of course, a lot of the early work was on shaping two-dimensional constellations. And this basically says if you get a large constellation, you don't even see this effect to, say, 64-QAM. If you have a 64-QAM constellation, 8 by 8-- Somebody at Paradyne noticed that you could improve it slightly by taking the four corner points out and putting them up here, without reducing the minimum distance. OK, so there is an improved, more spherical 64-point constellation. It's improved by about 0.1 dB, as I remember. And what this calculation says is if you go up to 256, 1024, and you simply take all the points in the QAM grid that are surrounded by a sphere of appropriate size to pick out 2 to the n points, for high, large constellations you get a gain of about 0.2 dB. So this is nothing to write home about.

What happens as n goes to infinity? We look up in a math book what the expressions are for the normalized second moment-- actually, you can do it just by this kind of calculation. It's easy enough. And just by scaling, you'll find that the shaping gain of the n-sphere goes to pi times e over 6, which is 1.53 dB. A famous number not only now in shaping, but also in quantization, where you're also interested in minimizing the second moment of a quantization cell. Or it's something like 0.229 bits per two dimensions, is another way of expressing it.

All right, so what is this? This is the maximum shaping gain you could get by the optimum region in the optimum number of dimensions, which is as large as you can go. So this says this is the ultimate shaping gain. You can never get a shaping gain of more than 1 and 1/2 dB. So it says when we play this game, we've got about 9 dB down here that we want to make up at 10 to the minus 5, 10 to the minus 6, we can get only 1 and a 1/2 dB of it by doing good shaping in large dimensions. That's the best we can ever do.

So that's a little bit discouraging. It says shaping is never going to give you that much. On the other hand, you can't get this 1 and a 1/2 dB in any other way. If you have n-cubed constellations, you will always be limited to be 1 and a 1/2 dB away from the Shannon limit. So if you want to approach the Shannon limit, you do have to address the shaping problem. You can't ignore it. 1 and a 1/2 dB is enough to be noticed in most applications, so you should do some moderate shaping.

Now, of course we can plot the curve. It's done in the notes for any n whatsoever, and you'll see that it goes up fairly steeply at first. It gets to about 1 dB at 16 dimensions, so if you shape in a 16 dimensional sphere, you can get 1 dB out of this maximum of 1 and a 1/2 dBs you can ever get. If you go to 32, it's 1.2 dB or something like that, it begins to slope out. So that gives you a good idea of what you can actually attain.

In the V.34 modem, for instance, a rather simple shaping scheme is used that gets up to about 0.8 dB of shaping gain. It's called shaping on regions. I just mentioned some techniques for shaping. As with a lot of things, once shaping was realized to be a separate subject from coding, there was a flurry of activity on it. People basically proposed a number of different schemes, particularly in the context of the V.34 modem standards development. Of which trellis shaping, I would say, is the most elegant. This is kind of a dual to trellis-coded modulation where we use trellis codes for shaping rather than coding. So that's elegant.

But a cruder and quite effective, just fine technique, shell mapping, was used in the V.34 modem. The basic idea here is that you take a two-dimensional constellation, you just divide it into shells like this. It turns out in two dimensions, if you take a constant delta here, that the average energy goes up, I think, linearly with delta or some way nicely with delta. You take a constellation bigger than you really need, so you have the possibility of some redundancy. So you identify each point in the constellation by which region it lands in. You then go up to 16 dimensions or something-- I think it is 16 dimensions in the V.34 modem-- and you assign each series of points a score depending on which shells its points land in. And you basically have a big table that chooses the lowest energy combinations from all the possible series of eight shells, which can be done fairly elegantly using generating function techniques.

I realize this is just hand-waving and you can't get it, but the effect is that you use this shell with very high probability. You sort of get a Gaussian-like probability on the shells. Let me just draw that like this. You don't use the points out in this outermost shell very much. You use this a lot, and they sort of have an overall Gaussian distribution.

And I should mention that this is an equivalent way of getting shaping gain. If we're shaping uniformly over an n-sphere in a high number n of dimensions and we basically have constellations which consist of two-dimensional QAM points, what is the projection of a uniform distribution over an n-sphere down onto two dimensions? If we look at each two-dimensional component of these n-dimensional vectors, what distribution will it have if we impose a uniform spherical distribution on the high dimensional constellation? It will turn out that it becomes Gaussian in two dimensions. Think of the projection of the 2-sphere down on one dimension. The projection of a 2-sphere on one dimension is something like this. It's not Gaussian, but there's more probability that one of the coordinates will be near 0 than it will be out at one of these maxima here. So this is just the same thing raised up to n dimensions.

So an equivalent way of looking at this problem is we want somehow to get a Gaussian-like distribution on a one- or two-dimensional constellation, and this is one way of doing it. And you can get the same pi times e over 6 factor by just looking at it from an entropy point of view, the entropy of a two-dimensional Gaussian distribution relative to the entropy of a uniform distribution over a square. That will give you the same factor if you do everything right.

OK so basically, shaping is easy. These techniques are not difficult to implement. You can get one dB of shaping gain quite easily through a variety of techniques. It's worth getting that one dB, you're not going to get it by any other means. But having done that, that's all you're ever going to get, and so end of story on shaping. So as I said, it was a flurry of activity in the '90s, and no one has looked at shaping since then.

So that's part of the story. Let's continue now with lattices, an old mathematical subject. So what we're interested in is, what's the densest lattice packing in n dimensions? This appeared in mathematics, long before it appeared in engineering, as Hermite's parameter. And Hermite said this was the right way to measure the density of a lattice, normalized in all the appropriate ways. So any version of a lattice is equivalent. A version is something that you get by scaling or rotating or even by Cartesian products, which I'm not sure how widespread that idea is.

So what's the densest lattice in two dimensions again? We want to find the densest lattice, in terms of coding gain or Hermite's parameter in n dimensions. Anyone care to guess what the densest lattice in two dimensions is? It's the hexagonal lattice. So in two dimensions, hexagonal, which reasons of mathematical history is denoted A_2. And its coding gain, I believe we computed it last time, and it was 0.6 dB.

So what's the most we could ever do by fooling around with two-dimensional constellations? Suppose I want the best possible 256 point QAM constellation? Well, according to these large constellation approximations, the best I could do would be to take the 256 lowest energy points in some translate of the hexagonal lattice. So I might fool around with the translation vector a little bit to try to find the absolute lowest energy point, but it really wouldn't matter very much regardless of what I did. I could expect 0.6 dB of coding gain and 0.2 dB of shaping gain, or 0.8 dB better than the 8 by 8 - sorry the 16 by 16 QAM constellation with the same number of points and the same minimum distance. This basically tells you what your options are. 2D, there's not much to do.

You remember way back in Chapter One, I gave you this problem about the 16 point, 2D constellation. That is precisely 16 points from a certain translate of the hexagonal lattice that fall within a circle that's just big enough to enclose 16 points. And it has, I forget, something like a 0.7 dB gain over the 4 by 4 square constellation. We computed that in the very first homework. So again, staying to two dimensions, things aren't very exciting. Turns out, it's most interesting to go in powers of two. In four dimensions, the densest lattice is something called D4, Schlafly's lattice, 1850-something. And its coding gain is precisely the square root of 2, which is 1.5 dB.

8 dimensions, it's E8, Gosset's lattice, about 1900. The D4 and E8 come, again, from the mathematical literature and have something to do with their relationship to certain groups. Lattices and finite groups are very closely connected. And this has a coding gain of 2 or 3 dB. 16 dimensions, it's something just called lambda 16. It's a Barnes-Wall lattice, as all of these are, actually. And it's 2 to the 3/2, or 4.5 dB. You see a certain pattern here?

In 32 dimensions, we have two lattices that I'll mention. There's the 32-dimensional Barnes-Wall lattice, which has a coding gain of 4.6 dB. Or there's actually Quebbeman in the '60s, I think, came up with a denser lattice. So this is still a subject of active investigation, particularly in dimensions like 19, where it's not so easy to find the densest lattice. And this, I forget, is maybe 6.2 dB. It's definitely denser. But it's something like that.

This basically goes to infinity. You can get denser and denser lattices just by following this chain here which is a chain of Barnes-Wall lattices. The nominal coding gain can be taken to infinity, just as we saw with kd over n. That goes to infinity for rate 1/2 Reed-Muller codes. But of course, Shannon says we can't ever get more than 9 dB of effective coding gain here. Or actually, we can't get more than 7 and a 1/2 dB if we take into account that 1 and a 1/2 is going to be given by shaping gain.

So what must be happening here? We must be having a huge number of near neighbors. The number of nearest neighbors in the hexagonal lattice is six. In D4 it's 24, in E8 it's 240, in lambda 16 I think it's 4320-- the correct numbers are in the notes. By the time we get up here, we get something like a 146,680 or something like that. So these lattices are designed to have a very high degree of symmetry. That means if you stand on any point, you look around you, they've pushed the neighboring points as far away from you as they can. But now there are a great many of them, so you get a uniform sky just filled with points around you, a number of dimensions.

So that's what the problem is and that means that the effective coding gain-- I again do the calculation. I know here we lose at least two dB. This is something like 3.8 dB. OK, so the effective coding gain actually sags and begins to top out as you get out here. Maybe nowadays you could think of implementing a larger lattice, but I think this is the largest anyone has ever thought about implementing. And what do we get? A coding gain of less than 4 dB -- an effective coding gain. So if you use that, you'd be, say, somewhere around 5 db here. We go over 6 dB, but then we go up by this normalized per two dimensions, so we divide by 16. That's still 10,000, which is still 13 factors of 2, which is 2.6 dB. You can see where the problem is.

So anyway, we get something that's really this moved way up a lot more, and as a result we get a much steeper curve. This method doesn't work up here, but this might be lambda 16 or lambda 32. And then we can get another 1 and a 1/2 dB of shaping gain. So maybe 4 dB, optimistically. OK, not really. So that's a little disappointing. Just by looking up in the math textbooks, we don't do very striking things.

I mention in particular the Barnes-Wall lattices, and if I had more time I would develop them because they're very much analogous to the Reed-Muller codes. They are constructed by a u, u plus v construction in the same way as Reed-Muller codes. They are length-doubling constructions that also double the distance, just as the Reed-Muller codes do now Euclidian distance.

But let me just give you the first couple of sentences in a discussion of the Barnes-Wall lattices, which were only discovered in '59, about the same time as the Reed-Muller codes. Suppose we take the QAM grid like this. An important observation is that if we take every other point in sort-of checkerboard fashion, what do we get? Imagine it's going to infinity in both directions. So this is a subset of half the points, basically. Half the density, twice the volume of points that, itself is a rotated and scaled version of the integer lattice. Take the centers on a checkerboard and they form a 45 degree rotated and square root of 2 scaled version of the integer lattice. If this had minimum squared distance 1 between points, then this has minimum squared distance 2 between points. So we could label the points red, black, red, black, red, black. If we did that, the red points would be a co-set of this sub-lattice and the black points would be a co-set-- meaning a translate-- of this sub-lattice.

So let me write that as follows. If we take Z-squared, it has a sub-lattice which we call RZ squared, r for rotation. Rotation by the Hadamard matrix, 1, 1, 1 minus 1. If I write d min squared, this has d min squared at 1, this has twice the d min squared. So I partition my lattice, or any translate of this lattice, into two co-sets of the sub-lattice with twice the minimum squared distance. This is the first steps in what Ungerboeck called set partitioning. So it's an obvious way to partition any QAM-type signal set, red and black points.

Having done that, I can do it again. Alright, do it again, what do I get? So now I'm going to take every other one of these points, which will leave me, say, these 4 points. Let me call these A, A, A, A. These points might be B, B, B, B, the other ones that I cut out. I could call these C, C, C, C and D, D, D, D. Now I've partitioned each of these subsets into two subsets. I've partitioned one into A, B and the other into C, D, such that - what does A look like? A is another translate of a lattice. This happens to be Z-squared scaled by 2. It's not rotated, it's just Z-squared scaled by 2. So we get down to 2 times Z-squared. And what's its minimum squared distance?

AUDIENCE: 4.

PROFESSOR: 4. So I've doubled the distance again. This is an elementary but important observation. I can continually go through these, partitioning into two subsets. Every time I do, I double the distance. I get another version of the integer lattice at a higher scale and, perhaps, rotated. So if I did it again, I would get eight subsets with minimum squared distance 8 between points in each subset. So it's just a repeated divide-by-2 scheme.

All right, now to get the Barnes-Wall lattices, you remember how we got the Reed-Muller codes. We took, say, two codes of length 2 and we put them together with the u, u plus v construction and we got codes of length 4 that had the minimum square distance of the higher distance code, if we chose them properly. That's the same thing we do here. If we put together these two lattices, if we choose them-- well, just use the u, u plus v construction where u is from Z-squared and v is from RZ-squared. So in words, that's what you do. If you're interested in looking it up, look it up in the notes.

What you get is a lattice, D4, that has normalized density halfway between that of Z-squared and RZ-squared, it turns out-- sort of analogous to the code-- and its minimum square distance is 2. Or if you put these two together, you get something called RD4, whose minimum square distance is 4. So you get something whose normalized volume is halfway between that of these two normalized volumes, but it has the same minimum distance as this guy, and that's where you get the coding gain from. This is denser but still has the same minimum distance.

If we do this again, go out to eight dimensions, we get, it turns out, E8. And this has minimum square distance 4. But it has the same normalized volume normalized back to two dimensions as this guy, which only had minimum squared distance 2. So that's precisely why this has a nominal coding gain of 2. We've doubled the minimum square distance without changing its density, normalized back to two dimensions.

All right, so you get all the Barnes-Wall lattices just by doing this. This turns out to be a sub-lattice of this guy. Not surprisingly, just as these lattices are nested, just as the Reed-Muller codes are nested, so you get a similar tableau. You get a similar construction that you can take out to 16 dimensions, 32 dimensions, however many dimensions you want. And every time you do, you increase the coding gain by a factor of square root of 2, so the coding gain eventually gets driven to infinity, and that's the story on Barnes-Wall lattices. So similarly to Reed-Muller codes, these are lattices that are the densest known in certain small dimensions like 4, 8, and 16. We don't know any denser lattices.

You start to get up to 32, 64 dimensions, then we do know of denser lattices. But they are not nearly as nicely structured, and maybe these are still the best from a complexity point of view. You can play similar games on building trellis representations of these lattices, as we did for Reed-Muller codes. They have similar kinds of trellises because they're based on the same construction.

So that's the story of Barnes-Wall lattices. The most famous lattice is the Leech lattice, which is a super-dense lattice in 24 dimensions. It has a coding gain of 6 dB. And unfortunately, it has 195,560 near neighbors or something like that. So you can see, it turns out it's a very close cousin of this 32-dimensional Barnes-Wall lattice. It's related to the 24-12-8 Golay code in the same way as the Barnes-Wall lattices are to the Reed-Muller codes. So the fact that the Golay code is so exceptional can be seen just as a projection of the fact that the Leech lattice is so exceptional. This somehow has the Golay code buried in it, if you can find it in here.

And the Leech lattice is particularly notable not just because it has remarkably high coding gain for its dimension-- it's remarkably dense for its dimension, exceptional-- but also because it has remarkable symmetries, so it's connected to some of the largest exceptional groups, the monster exceptional groups. And it was a key step in the classification of all finite groups to find the symmetry group of this lattice. Just a little liberal arts for you. So that's what we can get with lattices. By going out to the Leech lattice or the Barnes-Wall lattice-- this could equally well be the Leech lattice-- again, effective coding gain is only about 4 dB or maybe even less because of the exceptional number of near neighbors here. To one digit of significance, that's approximately its coding gain.

So how can we go beyond this? As I said already, the next great breakthrough was trellis codes for which Gottfried Ungerboeck became famous. Trellis codes, as I say in the notes, are basically-- first of all, Ungerboeck thought about the problem from a Euclidian distance point of view. So what you had to do to break out of Hamming distance thinking. He had the idea that in the bandwidth-limited regime, what you want to do is expand the constellation rather than expand the bandwidth. So to send four bits per second per Hertz. You could do that using uncoded 16-QAM constellation. Or Ungerboeck saw, well, let's make it into a 32-point constellation. Now we have some redundancy, now we can do some coding on this constellation.

He used the set partitioning idea, which I've just explained to you. Initially, he just looked at a QAM constellation and he said, let's divide it up into A, B, A, B, C, D, C, D, A, B, A, B, C, D, C, D. So let's partition it from A, B, C, D into A and D as I've done it here, and B and C are these two subsets. And we'll break this up into A and D and B and C. And then some theoretical people came along and said, well, this is just Z-squared, this is RZ-squared, and this is 2Z-squared.

Let's, first of all, partition our enlarged constellation. Now here's the setup we're going to use for coding. We are going to bring in our ...If we're trying to send-- I forget what notation I used in the notes-- b bits per two dimensions. Let's take one bit per two dimensions and put it into a rate 1/2 convolutional code. So we're going to get out two bits. Everything here is going to be per two dimensions because we're going to use a two-dimensional signal set. And we'll bring in our other b minus 1 bits down here. These are called uncoded bits.

For these two bits, we're going to go into a subset selector. Two bits are going to select among-- this going to be something from A, B, C, or D. It's going to tell us which subset to use. All right, then we'll use that in combination with these b minus 1 bits to select a point from a 2 to the b plus 1 bit QAM constellation. Two dimensions. So this is actually our signal point y, our two-dimensional signal point.

Do you see how the numbers work out? We want to send b bits per QAM point, per two dimensions. We're going to use a 2 to the b plus 1 point constellation, so 32 instead of 16, say. Take one bit in here, generate two streams, one redundant. This will tell us A, B, C, D, so that's two of the bits you need to select one of these points. Say, in a 16-point constellation, we'll provide a bit label that looks like this. For A, it will be 0, 0. b is 0, 1, c is 1, 0, D is 1, 1. That's always the low-order bits in any selection for A, B, C, D. And then we'll take two high-order bits, which then tell us which of these four groups it's in. So we'll have another 0, 0, 0, 1, 1, 0, 1, 1.

Now I'm talking about b equals 3. We have two bits that come in here and select one of these blocks, and then two bits that come out of here and select A, B, C, D within the block. There's two parts to the label. One of them determines A, B, C, D, the other one just determines some larger group.

These bits are called uncoded because there's no protection. If you sent point A over here and received point A over here, that would look like a perfectly legitimate code word. If we make an error of that far, which is d min squared equals 4, say. We make an error of size d min equal 2 all the way over here, then there's no further protection from this code because this code only determines the sequence of As, Bs, Cs, and Ds.

This is called a within-subset error. There's no protection on within subset errors. What the code does is it protects against smaller errors that are between subset errors like A to B, which has squared distance of 1 or A to D has a squared distance of two, and so forth. Are you with me? Nobody's asking questions. Maybe I haven't permitted it. Do you follow what the overall setup is?

I've done everything now, except choose an exact constellation. For the exact constellation, I'll just choose one of the best QAM signal sets. If it's large enough, I'll shape it like a circle. Or I might even try to put Gaussian shaping on it by some completely extraneous thing that works only on these clumps. So shaping takes place on the higher order bits. You can see that, given a large enough constellation just by choosing these clumps, I can make the constellation sort of have the shape of a circle. The subsets of A, B, C, and D points have to all be the same size in order for this scheme to work. But I can do that within a sort of circular constellation if the constellation is large enough. Or I can put some Gaussian distribution on these clumps.

So I do shaping down here. Initially, of course, Ungerboeck didn't worry about shaping. But if I try to do shaping, it's down on this index, which is considered to be the higher order bits that determine the growth structure of the constellation. These are the lower order bits, these two bits, and they determine the fine structure which is determined by this code. So the last thing I have to select is the code, and it turns out that the same code we used in the example is a good code to use here.

Suppose we use the four-state rate 1/2 example code that I've used before that is everybody's favorite example because it has such excellent distance properties. So it looks like this. So that's my rate 1/2 encoder. You remember as a binary code, it had minimum Hamming distance 5. And the only minimum distance code word was the impulse response. Let me write the impulse response as this way: 11, 01, 11. This is what we get out at time 0, time 1, and time 2.

Let me assign the following labels to these subsets. We'll make that 00, this one 11, this one 01, and that one 10. In other words, this is the complement of that. This is the complement of that. What is the nearest neighbor point in Euclidian space for this sequence? If I complement the first two bits coming out, then I'm going to go from one of these subsets to another, but within one of these RZ-squared subsets, which has minimum squared distance 2. So let's remind ourselves that the minimum squared distance within A, B, C, D is 1, within A, D or B, C is 2, within 2Z-squared is 4.

So I'm claiming-- there's actually a problem on this on Problem Set 10 if you tried to do it-- that any error of this type, 11, is going to cause me a Euclidian error of minimum squared distance 2, at least. It can only cause me to go from A to D, which is always at least squared distance 2, or from B to C, or vice versa.

So in terms of Euclidian distance, d min squared, I'm always going to get a squared distance of 2 at this point. And actually, any place you diverged in the trellis, because of the structure of the code, this is going to be true. I'm always going to get a squared distance of 2 when two trellis paths come together in the trellis. I always have a binary difference of 11, so by the same argument I'm always going to get a Euclidian squared distance of 2 when two trellis paths come together. And furthermore, I'm going to have at least one difference somewhere in between, which is going to, again, in terms of Euclidian distance, lead to a Euclidian distance of at least 1.

So the conclusion is that between any two sequences of constellation points that are labeled by two distinct code words here, I'm going to have a cumulative Euclidian squared distance of at least 5, by the same argument as I had for the binary code. But now it turns out that I'm still in a regime where binary distance, Hamming distance translates directly to Euclidian distance. This is what you saw.

Basically, what we're using here is this is Z-squared, you can think of as the image of the (2, 2) binary code. RZ-squared is the image of the (2, 1) binary code, and this 2Z-squared is the image of the (2, 0) binary code. You get the same partition structure for these codes and their co-sets. This is the set of all four two-tuples. We divided it into two subsets, each of minimum distance 2 here, and here we divide it into four subsets, each of minimum distance infinity. I'm not going to take the time to make this precise, but think of how that relation goes.

So I've assured myself of a Euclidian squared distance at least 5 between signal point sequences that correspond to two different code words. That's excellent. Is the minimum squared distance of the code then equal to 5, as it was in the binary case of this trellis code? No, it's not. And the problem is I could make this kind of error, which is completely invisible to the code. I can make a within subset error, and this only has minimum square distance of 4.

So the conclusion of this analysis is that this is what's called a four-state two-dimensional trellis code. This four-state two-dimensional trellis code has a minimum squared distance of 4, and it has a number of nearest neighbors per two dimension of 4. That's the number of nearest neighbors within 2Z-squared, which is one of these subsets. So the conclusion is that already, this has a legitimate 3 dB coding gain. Its coefficient is still just the QAM coefficient, and this very simple code has this performance curve moved over. So it's 3 dB all the way up the line. So that's pretty good. This is a trivial code that could be decoding-- by the way, it was, again, done by the Viterbi algorithm of a four-state trellis. You just take the sequences and it's exactly the same. You measure distance in terms of minimum squared distance.

The only difference is when you decode, you first find what the closest point is within the A, B, C and D subsets. That is what you're going to take as your representative for that subset. So you first make basically an uncoded decision, which is the closest point in A? Which is the closest point in B, which is the closest point in C, which is the closest point in D? You're going to get a metric for each of those. How far is each of those from the receive point? So then in trellis decoding, what the trellis is going to look like is something like this. A, D, A, D, B, C, and so forth. The trellis is going to look exactly the same, but here you get A, D, D, A, B, C, C, B.

So what metric are we going to use for each of these branches? First, we find the closest point within each subset, and then we use the metric for that point. Clearly, if we're going to choose A for this time interval, then we should use the closest point in A as our receive decision. We can get that survivor without any memory whatsoever when we make that decision, and then we take the sequences of representatives and find the best of those representatives as our best trellis-coded sequence. Any sequence of points that has labels that correspond to some path through this trellis is a valid point and vice versa. If and only if the sequence of labels is according to this trellis is it a legitimate trellis-coded point. So it's a nice amalgamation.

Great. With a simple four-state code, basically we've doubled the size the constellation. The constellation expansion is 2, the constellation expansion factor with twice as large a constellation, we've been able to achieve a coding gain of 3 dB. Not bad. So how do we extend this idea? First, extension would be to-- all right, let's go up to an 8 state, 16 state, whatever code. But is that going to improve our minimum distance using this block diagram? It's not, because we're still stuck with this within subset distance. So we can never get past d min squared equals 4 with a four-way partition. What we have to do is make one more level of partition and we need to make an eight way partition, at which point we're going to get a within subset distance of 8. I left out one step, I'm sorry. I'll come back to it.

We can get a code minimum squared distance up to 8, if we extend the minimum squared distance up to 8. The thing I didn't use, I didn't tell you before, was in computing the coding gain of this code-- let's just call it c-- what should it be? It should be the minimum squared distance of the code over the volume of the code per two dimensions. In this case, n is 2 so I'm just going to do it over the volume of the code. Now what is the volume of a convolutional code? This is actually an interesting problem, because the Voronoi region of a trellis code is actually an infinite dimensional object. A trellis code is really an infinite dimensional lattice. It's a lattice in infinite dimensional space. And if you look at the Voronoi region, it extends out into all these dimensions. It might not for such a simple code, but in principle it does.

However, you can find other fundamental regions. The way to think about this is instead of putting the volume here-- well, what have we cost ourselves in volume? We've doubled the size of the constellation. And in two dimensions, that means we've doubled the volume. So we've quadrupled the minimum distance at the cost of doubling the volume. We've covered twice as much space now because we have twice as many points. And so, that's where we get the-- this is equal to, for this code, that's 4. We have a factor of 2 loss in volume, and so that's where we get what I said was a coding gain of 2 or 3 dB.

Just to get you to the bottom line, we should replace the volume by two times-- I call it eta of the code-- where eta of the code is its redundancy per two dimensions. In this case, we introduce one redundant bit per two dimensions, and that's why the constellation has to expand by a factor of 2. You see, we get the right answer. The redundancy has increased by one bit, so this has to be 2 to the b plus 1, so the constellation has to be twice as large, so the volume is twice as large. I'll just hand-wave and tell you that's the factor you want down here representing volume -- 2 to the redundancy. And you can prove this by finding a region which you can show is a fundamental region of the trellis code and has this volume. But I won't go into that.

OK, so that's the calculation which I should've mentioned before, I skipped right over it. Back to what I was saying. To improve trellis codes, the first thing you can do is to, say, go to an eight-way partition. So now we're going to have eight subsets. So we're going to need three bits here. So let's use two bits here and use a rate 2/3 trellis code. We're going to have three bits coming out here to select subsets, and we're going to have b minus 2 bits down here to select a point still a 2b plus 1 point QAM constellation. So our redundancy per two dimensions is still one bit per two dimensions. We're still going to lose a factor of 2, but now we can go up to d min squared equals 8 and we can go up to a coding gain of 6 dB.

So in his original paper, Ungerboeck did this. He showed codes going up to 256 states in two dimensions that gets you up to a nominal 6 dB coding gain. And because they have low coefficients here, they get very close to 6 dB of effective coding gain. Already in Ungerboeck, we have codes-- I mean, eventually you can get back to just the within-subset coefficient again, which is always 4. You can get up to, say, 6 dB with maybe 512 states of effective coding gain, which is considerably better than the densest lattice we would think of using, and also quite reasonable to implement. Now we're going to need a 256, 512-state Viterbi algorithm decoder.

This was a stunning breakthrough. Immediately in 1993, everybody started building turbo codes. Back in 1982, everybody started building trellis codes. It happened to be the time of the V.32 modem, and everybody said boy, this is just what we need to get an additional 3 or 4 dB coding gain in the V.32 modem. What was actually done there is this was simply an eight-state convolutional code, so it was rate 2/3. And there was tweaking of this. It was made rotationally invariant, made nonlinear. And a lot of activity, which was very nice, stimulated by the standards effort. But that was basically the state of the art in the '80s.

The one other improvement that was made was the following idea. We really don't like doubling the size of the constellation, particularly for a large one, mostly for implementation reasons. There were some feelings against nonlinearities a larger constellation would cost you. You wanted to minimize the constellation expansion ratio. So how were we ever going to do that? You can't do it this way in two dimensions. The way to do that is to go up to four dimensions.

So let's do subset partitioning in four dimensions. So now think of four dimensional constellations. In practice, these are just built from the Cartesian product of two two-dimensional constellations. We start with Z4. The next one down is D4, which has a minimum square distance of 2. It's sometimes called a checkerboard constellation in four dimensions. It's just the red-black idea again. Then below that, we have a four-way partition into RZ4, which still has minimum distance 2. And one more level of partitioning here into something into RD4, which has minimum squared distance of 4.

So this is an eight-way partition of a four-dimensional constellation. This idea was introduced by Lee-Fang Wei. Again, this is an interesting column over here. You can think of this as being the (4, 4, 1) code. This is the Euclidian image of the (4, 3, 2) code. It kind of expanded in the lattice. We know we can't do any better than (4, 2, 2). Down here, this is the (4, 1, 4) code. So again, this is the same partition as we get for binary codes with the distances-- again, for binary codes and this level of partitioning, Hamming distance translates to Euclidian distance. It's really just the image of these codes as you go up to the lattices.

Now in this case, what's the best we can ever do? Coding gain. We still can't go up to a squared distance of greater than 4 because our within-subset distance is going to be within RD4. So this is still limited to 4. But now, what's our redundancy per two dimensions? Here, say, we're going to use still a rate 2/3 convolutional code. We'll now have an eight-subset selector in 4D, so everything is per 4D here. So this is going to be 2b minus two bits and this is going to be, it turns out, a constellation that is twice as large in four dimensions. We still have one redundant bit per four dimensions, so we need twice the size of the constellation in four dimensions. But that's only 1/2 a bit of redundancy per two dimensions. So this denominator becomes 2 to the 1/2. Waving hands furiously here.

And so this could go up to 2 to the 3/2, which is 4.5 dB. So there's actually a particularly nice eight-state code here that Wei came up with which, really, in terms of performance versus complexity is the only code that anyone has ever come up with that is distinctly better than any of Ungerboeck's original codes. All of Ungerboeck's original codes were in one dimension, two dimensions, or PSK codes. You could also do this on phase shift key constellations.

So with the Wei code, it gives you a very simple way to get 4 1/2 dB. Actually, you have to go up to 16 states to get 4 1/2 dB. The Wei code is only 4.2 dB. And this is basically what was adopted for standards in the early '90s. The V.34 modem standard, also the ADSL standard has the same code in it. If you have DSL at home, that's what it's using in it. Again, with a very simple code, the eight-state Viterbi algorithm, you can get 4.2 dB of coding gain. The error coefficient in this case, this has 24 nearest neighbors in D4, or 12 per two dimensions. So you lose 0.15 dB or something. I'm sorry, 0.3 dB. Anyway, this is a very nice code.

There are 32 and 64-state codes in V.34. They don't gain you very much. They each double the complexity and only gain you about 0.2 dB for each one. All these things tend to saturate. You can't get more than about 5 to 6 dB of coding gain with a reasonable size trellis code. So where are we now? We can get 5 or 6 dB of coding gain with, say, a 256-state trellis code. We can get another 1 to 1 and a 1/2 dB of shaping gain. So all together, we're up to 6 to 7.5 dB. From here, we're up somewhere into the 2 to 3 dB range. In other words, combining trellis code with, say, 256 states with some shaping scheme with maybe 1 dB of shaping gain, this gets you maybe 5 dB or 5.5 dB. We get 6 dB total effective gain, and we're still 3 dB or maybe 2 or 3 dB from Shannon limit.

So, close but not quite there yet. Five minutes to go. So in the notes, there's one page on higher performance schemes. There are higher performance schemes. There are turbo codes for the bandwidth-limited regime that get you to within 1 dB of the Shannon limit. Sequential decoding gets you to within about 1 and a 1/2 dB of the Shannon limit. The idea of multi-level codes -- you can regard each of these two-way partitions as being specified by one bit. So with three bits of labels, you can specify an eight-way partition.

Another idea is to code on each of these levels separately. Use a binary code on each of these levels separately. Each of these corresponds to an effective channel that has different characteristics. You've basically got to take the Gaussian distribution and alias it around in accord with whatever this partition actually is. But you can come up with an effective binary channel at each of these levels. It turns out to be a binary symmetric input channel, always in these cases. So you could think of taking a binary-- now, each one of these channels is going to have its own capacity. It turns out that information theoretically, the capacity of multi-level coding on each of these levels independently is the same as the aggregate capacity of coding it on the whole channel. It's a lovely separability result.

So in principle, if you could code at capacity on each of these binary channels, then you could get to the capacity of the aggregate channel, with a large enough constellation. So that's a good idea. We now know a lot of binary capacity-approaching code, so let's use a turbo code or a low-density parity-check code at each level, an appropriate code. It turns out, as you might expect, that the first level is the really hard level. In this case, you might have a rate 1/3 or 1/2 code and really work hard with your code. At the next level, the effective error probability is much better. Making hard decisions, you might have a probability of 1/10 or lower of making an error.

So these are really much easier problems. It might be a better idea at these levels just to use algebraic coding or something that works very well. The capacity is nearly one per bit at each level, so you want to code with very little redundancy at these next levels, but it's still capable of approaching capacity. Well, it turns out this is a good scenario for Reed-Solomon codes or some kind of algebraic code. So the moral is you only really need your capacity-approaching code at maybe the first or first and second levels. Down here, just use a very high rate-- 1,023, 1,003 Reed-Solomon code over gf of 2 to the 10th-- very little redundancy, and drive the error rate down as low as you like at each of these levels. So multi-level coding is a good way in principle.

I'm not sure if it's been done in practice. In fact, I don't know of any capacity-approaching code that has been done in practice, except there's a very crude technique, I would say, called Bit-Interleaved Coded Modulation, BICM, which is the following simple idea. Let's just take a single low-density parity-check code or some capacity-approaching code. It's going to have to be designed for a BICM channel. Let's totally interleave the bits and then let's use them as the bits that control each of these levels here. At the output of the channel, we know which level the bits came out of. So at the output of the channel, we know what the appropriate a posteriori probabilities are for each bit. They're going to be wildly different. A bit sent through here is going to be sent through a very highly reliable channel, a bit sent through here is going to be sent through a very low reliability channel.

But with all this interleaving and the independence approximation in the analysis, we're going to be able to design a code for this mixture channel, a mixture of high reliability and low reliability binary channels, that's going to get to the capacity of this mixture channel, which as I've already told you is equal to the capacity of the aggregate channel. I guess there's some elegant features of it. But that's, in practice, what people use. They use it more because they've already developed a chip for binary channels, and they want to reuse that chip for some large constellation channel, some bandwidth-limited channel. So this was originally advocated by Qualcomm as a, quote, pragmatic approach.

OK, we've got a chip that works for binary channels, so let's just interleave outputs and use it over this multi-level channel and it'll probably work well enough. And, in fact, that's true. And the information support for it, again, is the fact that you don't lose anything by coding bit by bit from a capacity point of view. I think that's the only one that's been used in practice. And on wireless cellular channels for instance, that tends to be what's used.

And of course, I know that perhaps a couple of you have an exam at 11:00, so I'll make a point of letting you out of here. I've really enjoyed teaching the course this term. It was the first time I was able to get to the analysis of capacity-approaching codes, so I enjoyed doing that. At this point, I think the subject of coding for the additive white Gaussian noise channel, the traditional channel, is pretty well wrapped up. I don't think we're going to see any great improvements.

There's still some very interesting theoretical questions about the convergence of these iterative decoding algorithms. What do they converge to? How fast do they converge? There are interesting connections to statistical physics and inference and so forth that people are still exploring. They're interested in graph theoretical properties. There's attempts to algebraically construct these capacity-approaching codes, so we can say a lot more about their parameters and their performance, particularly for lower probabilities like 10 to the minus 15.

So you go to the information theory symposium, you'll still hear a lot of papers about coding effectively for additive white Gaussian noise or memoryless binary symmetric input channels. But people ask me, is coding dead now? Are Fourier transforms dead? No, they're not dead. But they've become sort of part of the standard core of what you should know. I think this is a subject that anyone in communication should certainly learn, but is it an area where people are going to make further advances that are important from a practical point of view? Obviously not very much, in view of the Shannon limit. We are very close to the Shannon limit for these channels, and have been now for a few years. So it's important to know about these things. It's important to know when you're in the wireless course, for instance. You basically calculate mutual information and assume that you can go at the mutual information. This is how you do it.

So you should know how it's done, and that's why I think this course continues to be worth teaching. But from a point of view of research, it's become a rather mature area, and the action has gone off to other kinds of channels. Multi this, multi that, so forth. Anything to do with networks or multi-user is where the action in coding is. Data compression, of course, is a subject that we haven't talked about at all, which is also coding. So anyway, I think it's good for you to have all this under your belts. I hope that you find it useful in your future lives, and I wish you all well on the final exam. Thank you.