Lecture 11: Learning: Identification Trees, Disorder

• Download this transcript - PDF (English - US)

PATRICK WINSTON: Ladies and gentlemen, the Romanian national anthem.

I did not ask you to stand, because I didn't play it as a symbol of Romanian national identity.

But rather, to celebrate the end of the Cold War, which occurred about the time that you were born.

Before that, no one came to MIT from Eastern Europe.

But since that time, we've been blessed by having in our midst Lithuanians, Estonians, Poles, Czecs, Slovaks, Bulgarians, Romanians, Slovenians, Serbs, and all sorts of people from regions of the world formally excluded to us.

Believe me, you are all welcome in our house.

Almost all, that is to say.

Because you may recall that Romania is the traditional home of vampires.

And since the end of the Cold War, vampires have had new vectors for emerging from their traditional places and penetrating into the world at large.

You may have vampire in your suite, or on your floor.

And it's important to know how to recognize them, and take the necessary precautions.

So if you have this concern, I would expect that the first thing you would do would be to look at some data concerning the characteristics of vampires.

So there's a little database of samples of individuals who have been determined to be vampires and not vampires.

And our task today-- and what you'll understand how to do by the end of the hour-- is to use data like this to build a recognition mechanism that would help you to identify whether someone is a vampire or an ordinary person.

So this is a little different from the kind of problem we worked with neural nets.

Right?

So what's the most conspicuous difference between this data set and anything you could think to work on with nearest neighbors, which we studied last time.

Katie, do you have any thoughts about why it would be difficult to use nearest neighbors with data like this?

The question mark is there because this is MIT, and a lot of people are completely nocturnal.

So you can't tell whether they cast a shadow or not.

We want to take that into account.

So what's different about this from the electrical cover data set?

STUDENT: [INAUDIBLE] PATRICK WINSTON: Could you use the nearest neighbor technique to identify vampires with this data?

STUDENT: [INAUDIBLE] PATRICK WINSTON: So obviously-- Yes, Lana?

STUDENT: [INAUDIBLE] STUDENT: You cannot really quantify-- PATRICK WINSTON: Oh, that's the problem.

This is not numerical data.

This is symbolic.

So we're not saying that your ability to cast a shadow is 0.7.

You either cast a shadow, down cast a shadow, or we can't tell.

It's a symbolic result.

So problem number one we have to face with data of this kind is that it's not numeric.

And there are other characteristics, as well.

For example, it's not clear that all of these characteristics actually matter.

So some characteristics don't matter.

And a corollary to that is that some characteristics do matter, but they only matter part of the time.

And finally, there's the matter of cost.

Some of these tests may be more expensive to perform than others.

For example, if you wanted to determine whether someone casts a shadow, you'd have to go to the trouble of getting up during daylight.

That might be an expensive operation for you.

You'd have to go find some garlic and ask them to eat it.

That might be expensive.

So some of these tests might be expensive relative to other tests.

But once you realize that we are talking in terms of tests, and not a vector of real values, then what you do is clear.

You build yourself a little tree of tests.

So who knows how this problem will turn out?

But you can imagine a situation where you have one test up here which might have three outcomes.

And one but only one of those outcomes might require you to perform another test.

And only when you've created the tree of tests that look like this are you finished.

So given this set of tests and a set of samples, the question becomes, how do you arrange the tests in a tree like that so as to do the identification that you want to do?

So since we're talking about identification, it's not surprising that this kind of tree is called an identification tree.

And there's a tendency-- and I may slip into it myself-- to call this a decision tree.

But a decision tree is a label for something else.

This is an identification tree.

And the task is to create a good one.

So what is a good one versus a not so good one?

What characteristic would you like for a decision tree-- for an identification trade to have, if you're going to call it good identification tree?

What do you think, Krishna?

What would be a good characteristic?

STUDENT: Maybe the minimum number of levels?

PATRICK WINSTON: Yeah.

He said minimum number of levels.

What's another way you could say what a good one is?

Each test costs something, right?

So what's another way of thinking about what a good tree would look like?

STUDENT: Minimum cost.

PATRICK WINSTON: The minimum cost.

And if they all have the same cost, then it's the number of tests.

So overall, what you like is a small tree rather than a big one.

So you might be able to take your sample data and divide it up, so that at the bottom of the tree, at the leaves, all of the sets that are produced by the tests are uniform, homogeneous.

We'd like that tree to be the simplest possible tree you can find, not some big complicated one that also divides up all the data into uniform subsets.

By uniform subset-- at the bottom of the tree, you have all of the vampires together, and all the non-vampires together.

So you'd like a small tree.

So why not just go all the way and do British Museum, and calculate all possible trees?

Well, you can do that, but it's one of those NP problems.

And as you know, NP problems suck in general.

And so you don't want to do that.

You want to have some kind of heuristic mechanism for building a small tree.

And we want a small tree because-- Why do we want a small tree?

Because of the cost.

but there's another, more important reason why we want a small tree.

Let me give you a hint.

It's Occam's Razor.

The simplest explanation is often the best explanation.

So if you have a big, complicated explanation, that's probably less good than a simple, small explanation.

Occam's Razor.

Spelled so many ways it doesn't matter how I spell it.

And that's good, because I can't spell.

So how are we going to go about finding the best possible arrangement of those four tests in a tree like that?

Well, step one will be to see what each test does with the data.

And by the way, before I go a step further, you know and I know that this is a sample data set that's very small, suitable for classroom manipulation.

You'd never bet your life on a data set this small.

We use it only for classroom illustration.

But imagine that these rows are multiplied by 10.

So instead of eight samples, you've got 80.

Then you might begin to believe the results that are produced.

So I'm just going to pretend that each one of those represents 10 other samples that I haven't bothered to show.

But we can work with this one in the classroom, because it's pretty small.

And we can say, well, what does this shadow test do?

Well, the shadow test divides the sample population into three groups.

There's the I Don't Know group of people who are nocturnal.

There are the people who do cast the shadow, the Yes people.

And the people who do not cast a shadow, the No people.

So if I look at those rows up there and see which ones are vampires, it looks to me that if there's no shadow cast-- there's only one that doesn't cast a shadow-- and that is a vampire.

So that's a plus over there.

Vampire.

Now, if we look at the ones who do cast a shadow, all those are not vampires.

They're all OK.

And now there're 8.

Three are vampires.

So that means that two of these must be vampires.

And I've got three, four, five, six so far.

So there must be two left.

So that's the way the shadow test divides up the data.

Now let's do garlic.

Vampires traditionally don't eat garlic.

I don't know why.

So we look at the garlic test, and we see that all of the Nos-- well, there're three Yeses, and they all produce a No answer.

So if somebody eats garlic, they're not vampires.

That means the three vampires must be over here.

Then there are two left.

So that's what the garlic test does.

See what we're trying to do?

We're trying to look at all these tests to see which one we like best on the basis of how it divides up the data.

So now we've got complexion.

And there are three choices for this.

You can have an average complexion.

But a lot of vampires, in my experience, are rather pale.

So pale is a possibility.

And then the other option is that just after gorging themselves with blood, they tend to get a little red in the face.

So we'll have a ruddy over here.

Once again, we have to go back to our data set to see how this test divides things up.

So there are three ruddies, and one's a No, one's a No, and one's a Yes.

So two Nos and a Yes.

Two Nos and a Yes.

Now we can try for pale complexion people.

There are only two of those.

A No and a No.

That must mean that there are two pluses over here, because there are three vampires altogether.

Two, four, six, seven, eight, nine.

Eight, sorry.

Eight.

Only eight.

Just one more to go, and that's the accent.

Historically, vampires go to great length to protect their accent and not betray their origins.

But nevertheless, we can expect that if they've just arrived-- if they're just in from Transylvania, part of Romania-- they may still have an accent.

So there's a normal, some still have a heavy accent, and some persist in having odd accents.

So let's see.

Accent.

Four of them, right at the top, have no accent.

Two Nos and a Yes.

Heavy accent.

Three of those.

A Yes and two Nos.

That means we must have a plus here.

3, 6, plus and a minus.

So we can look at this data and say, well, what will be the best test to use?

And the best test to use would surely be the one that produces sets here, at the bottom of the branches, that correspond to the outcomes of the test.

We're looking for a test that produces homogeneous groups.

So just for the sake of illustration, I'm going to suppose that we're going to judge the quality of the test by how many sample individuals it put into a homogeneous set.

So ideally, we'd like a test that will put all the vampires in one group and all the ordinary people in another group right off the bat.

But there are no such tests.

But we can add up the number of sample individuals who are put in to at least homogeneous sets.

So when we do that, this guy has 3 in a homogeneous set here.

A fourth.

But these are not a homogeneous set.

So the overall score for this guy will be 4.

This one, well, not quite as good.

It only puts 3 individuals in a homogeneous set.

This one here, 2 individuals into a homogeneous set.

Everybody else is all mixed up with some other kind of person.

And over here, how many samples are in a homogeneous set?

0.

So on the basis of this analysis, you would conclude that the ordering of the test with respect to their quality is left to right.

So the best test must be the shadow test.

So let's pick the shadow test first, see what we can do with that.

If we pick the shadow test first, then we have this arrangement.

We have question mark, and we have Yes, casts a shadow, and No, doesn't.

We have 3 minuses here.

We have a plus here.

And unfortunately, over here, we have plus, plus, minus, minus.

So we need another test to divide that group up.

Yes.

STUDENT: How did you get the 4 on the shadow test again?

Why was it 4?

PATRICK WINSTON: Well, if I look at the data and I see who-- the question is, what about that shadow test?

If you look at the shadow test, and you say, well, there are 4 question marks.

And if we look and see what kind of people belong to those 4 question marks, there are 2 vampires and 2 non-vampires.

That's why it's 2 pluses and 2 minuses.

STUDENT: No, I understand that.

The question is, how did you get to the score of 4?

PATRICK WINSTON: Oh, yeah.

The question is how did I get this number 4?

It has nothing to do this, because this is a mixed set.

In fact, I've got three guys in a homogeneous set here, and one guy in a homogeneous set here, and I'm just adding them up.

STUDENT: OK.

PATRICK WINSTON: So very simple classroom illustration.

Wouldn't work in practice.

Yes.

STUDENT: How do you adjust this for larger data sets where it's unlikely you're going to have any [INAUDIBLE]?

PATRICK WINSTON: The question is, how do I adjust this for larger data sets?

You're one step ahead.

Trust me, I'll be doing large data sets in a moment.

I just want to get the idea across.

And I don't want there to be any thought that the method we use for larger data sets has got anything magic about it.

OK, so we're off and running.

And now we have to pick a test that will divide those four guys up.

So we're going to have to work this a little harder, and repeat the analysis we did there.

But at least it'll be simpler, because now we're only considering 4 samples, not 8.

Just the 4 samples that we still have to divide up that have come down that left branch.

So I have the shadow test.

It has 3 outcomes.

We have the garlic test.

It has 2 outcomes.

Yes and No.

We have the complexion test.

There's 3 outcomes.

Average, pale, and ruddy.

And we have finally the accent test.

And that comes out to be either normal, heavy, or odd.

And now, it's a little awkward to figure out what the results are for this data set as shown.

So let me just strike out.

The ones that we're no longer concerned with, and limit our analysis to the samples for which the outcome of the shadow test is a question mark.

This is exactly the four people we still need to separate, right?

So switching colors, keeping the color the same.

We actually don't want to do the shadow test anymore, right?

Because we've already done that.

There's no point in doing that again.

We don't have to look at that.

It's already done all the division of data that it can.

So the garlic test.

Well, let's see.

Garlic.

2 Yeses, 2 Nos.

The Yeses produce Nos and the Nos produce Yeses.

So if the person does eat garlic, they're OK.

And if they don't eat garlic, bad news-- they're vampires.

Well, that looks like a pretty good test.

But just for the sake of working it all out, let's try the others.

Complexion.

2 Ruddies, a Yes, and a No.

1 pale, and that's a No.

1 pale, and that's a No.

And we must have 1 average, and sure enough, that's a Yes.

Now we can do accent, the one on the far right, and look at how that measures up against the people who are still under consideration as samples.

Accent.

Let's see.

2 Nones, a Yes and a No.

No Heavies.

2 Odds, a Yes and a No.

All right.

So now we can do the same thing we did before, and just say, for sake of classroom illustration, how many individuals are put into a homogeneous sets.

And here we have 4.

And here we have 2.

And here we have 0.

So plainly, the garlic test is the test of choice.

So we go back over here, and we've completed the work that we needed to do.

So that's the garlic test.

And that produces 2 pluses.

Let's see.

Eats garlic, Yes.

Eats garlic, No.

I guess the pluses go over here like so.

And these are the two ordinary people.

And we're done with our task.

And now you can quickly run off and put this into your PDA, and forever be protected against the possibility that one of those vampires got out in the flood of people that came in from Eastern Europe.

Except what do we do a large data set?

Well, the trouble is, a large data set's not likely to produce-- if you have a large data set, no test is likely to put together any homogeneous set right off.

So you never get started.

Everything would be 0.

Every test would say, oh it doesn't put anybody into homogeneous sets.

So you're screwed.

You need some other, more sophisticated way of measuring how disordered this data is.

Or how disordered these sets are that you find at the bottom of the tree branches.

That's what you need.

You need a way of measuring disorder of these sets that you find at the bottom of these branches, so you can find a kind of overall quality to the test based on your measurement of disorder.

Now, the first heuristic of a good life is, when you have a problem to solve, ask somebody who knows the answer.

It's the least amount of work.

It's not even as hard going to Google.

So who would you ask about ways of measuring disorder in sets?

There are two possible answers.

STUDENT: You could just do entropy.

PATRICK WINSTON: What?

STUDENT: Find the entropy of the set.

PATRICK WINSTON: Who studies entropy?

STUDENT: Probability.

PATRICK WINSTON: What kind of classes?

STUDENT: Physics.

STUDENT: Thermodynamics.

PATRICK WINSTON: Thermodynamics!

The thermodynamicists are good at measuring disorder, because that's what thermodynamics is all about.

Entropy increasing over time, and all that sort of stuff.

There's another equally good answer.

STUDENT: Statisticians?

PATRICK WINSTON: Statisticians.

Perhaps, but it's not the second best answer.

It's actually not even the best answer.

That's the best answer.

What's your name?

STUDENT: Leo.

PATRICK WINSTON: Oh, yeah.

[LAUGHTER] PATRICK WINSTON: Leonardo has got his finger on it.

The information theorists are pretty good at measuring disorder, because that's what information is all about, too.

So we might as well borrow a mechanism for measuring the disorder of a set from those information theory guys.

So what we're going to do is exactly that.

Let's put it over here, so we'll have it handy when we want to try to measure those things.

The gospel according to information theorists is that the disorder, D, or some set is equal to-- now let's suppose that this is a set of binary values.

So we have positives and then we have negatives.

Pluses and minuses.

But pluses, they don't go very well in an algebraic equation, because they might be confused with adding.

So I'm going to say P and N. And then it'll be the total, which is P plus N. We only have two choices, positive and negative.

So the disorder of set, according those guys, is equal to minus the number of positives over the total number, times the log to the base 2 of the positives over the total, minus the negatives over the total, times the log 2 of the negatives over the total.

Those negatives look a little worrisome, because you think, well, maybe this thing can go negative.

But that's not going to be true, right?

Because these ratios are all less than 1, and the logarithm of something that's less than 1 is negative.

So we're OK.

So that's a lovely way of measuring disorder.

And then we ought to draw a graph of what that curve looks like.

And what we're going to graph it against is the ratio of positives to the total number.

So that's going to be an axis where we go from 0 to 1.

So let's just find a couple of useful values.

And by the way, it pays to pay attention to these curves, because if you pay attention to this stuff, you can work the quiz questions on this very rapidly.

Otherwise, we see people getting out their calculators and quickly becoming both lost and screwed.

OK so let's see.

Let's suppose that the number of positives is equal to the number of negatives.

So we've got a completely mixed-up set.

It has no bias in either direction.

So in that case, if P over T is equal to 1/2, then this is equal to minus 1/2, times the logarithm of 1/2.

And I guess, since they're both the same, we can multiply by two.

And what's that value?

[INAUDIBLE], what does that calculate out to?

STUDENT: Minus [INAUDIBLE] PATRICK WINSTON: Minus [INAUDIBLE].

Well, with a minus sign, you just turn the argument upside down, so it's log(2).

So what's log(2)?

Logarithm of base 2 of 2?

1!

So this whole thing is-- STUDENT: 1.

PATRICK WINSTON: 1.

So [INAUDIBLE], in her soft way, says, well, let's see.

2 times 1/2.

That cancels out.

The minus, that flips the arguments so it's log to the base 2 of 2, and that's 1.

So this whole thing, You work out the algebra, it gives you 1.

So that's cool.

So right here in the middle where they're equal, we get a value of 1.

Next thing we need to do is let's calculate what happens if P over T is equal to 1.

That is to say, everything is a positive.

Any guesses?

Maybe 10, 20, minus 15?

Let's work it out.

So if P over T equal 1, that would be minus 1 times the log to the base 2 of 1.

What's that?

STUDENT: [INAUDIBLE] PATRICK WINSTON: A 0?

Oh, yeah.

Because 2 raise to the 0 is one.

So this part is 0.

Now, what about this other part?

If everything's a P, then nothing's an N.

So we've got 0.

And we can quit already.

Well, not quite.

We ought to work it out.

Log 2 to the base 2 of 0.

What's that?

STUDENT: [INAUDIBLE] PATRICK WINSTON: Who?

Minus infinity?

Uh oh.

0 times minus infinity is What I didn't get that when I was in high school.

Finally, 1801 makes a difference.

Finally.

What's the answer.

We're interested in the limit as N over T goes to 0, right?

And when you have a deal like this, what do you do?

You use that famous rule, that we all mispronounce when we see it written, right?

We use the good old El Hospital's rule.

OK, it's L'Hopital.

L'Hopital's Rule.

You have to differentiate the-- I guess we differentiate this guy as a ratio or something, and see what happens when it goes to 0.

And what we get when we use L'Hopital's Rule is that, oh thank God, this is still zero.

So now we know that we have a point up there and a point down there.

So now we've got three points on the curve, and we can draw it.

It goes like that.

No, it doesn't go like that.

It's obviously a Gaussian, right?

Because everything in a nature is a Gaussian.

Can you put that laptop away, please?

Everything in nature is a Gaussian, so it looks like this.

That right?

No, actually, not everything in nature is a Gaussian.

And in particular, this one isn't a Gaussian either.

It looks more like one of those metal things they used to call quonset huts.

That's what it looks like.

Boom, like so.

So that is the curve of interest.

Now, did God say that using this way of measuring disorder was the best way?

No, Got has not indicated any choice here.

We use this because it's a convenient mechanism, it seems to make sense, but in contrast to the reason it's used information theory, it's not the result of some elegant mathematics.

It's just a borrowing of something that seems to work pretty well.

Any of those curves would work just about the same, because all we're doing with it is measuring how disordered a set is.

So one thing to note here is that in this situation, where we're dealing with two choices-- P and N, positives and negatives-- we get a curve that maxes out at one.

And notice that it kind of gets up there pretty fast.

In fact, if you're down here at 2/3, are you're up here, this is about 0.9.

So it gives you a large number for quite a bit of that area in the middle.

So that, unfortunately, still doesn't tell us everything we need to know.

That tells us how to measure a disorder in one of these sets.

But we want to know how to measure the quality of the test overall.

So we need some mechanism that says, OK, given that this test produces three different sets, and we now have a measure of the disorder in each of these sets, how do we measure the overall quality of the test?

Well, you could just add up the disorder.

Let's write that down, because that sounds good.

So you can say that the quality of a test is equal to some sum over the sets produced.

And what we're going to do is we're going to add up the disorder of each of those sets.

I'm almost home, except that this means we're going to give equal weight to a branch that has almost nothing down it-- we're going to give the same weight to that as a branch that has almost everything going down it.

So that doesn't seem that make sense.

So one final flourish is we're going to weight this sum according to the fraction of the samples that end up down that branch.

So it's, as usual, easier to write it down than to say it.

So we're going to multiply that times the number of samples in the set, divided by the number of samples handled by test.

So if half the samples go down a branch, and if that branch has a certain disorder, then we're going to multiply that disorder times 1/2.

All right.

So now let's see how it works with our sample problem.

Well, here is our sample data.

And we didn't need anything fancy for it.

But let's pretend it was a large data set.

Well, let's see.

What would we do?

Well, go down this way, there are 4 samples down that direction.

That's half of the total number of samples.

So whatever we find down there, we're going to multiply by 1/2.

This one we're going to multiply by 3/8.

And this one we're going to multiply by 1/8.

Now, what do we actually find at the bottom of these things?

Well, here's a homogeneous set.

Everything's the same.

So we go to that curve and say, what is the disorder of a homogeneous set?

It's zero.

Let's see, they're all the same.

I guess that means it's 0 over there.

So the disorder of this set of three samples is zero.

The disorder of this set of one sample, all the same, is zero.

The disorder of this set-- well, let's see.

Half of the samples there are plus, and half are minus, so we go over to our curve, and we say, what's the disorder of something with equal mixture of pluses and minuses?

And that's one.

So the disorder of this guy is one.

So now we've got 1/2 times 1, and 3/8 times 0, 1/8 times 0.

So the quality of this particular test, as determined by the disorder of the sets it produces, is 1/5.

0.5.

Let's do this one.

So we have 3/8 coming down this way, 5/8 coming down this way.

3/8 is multiplied by the disorder of a set of uniform things.

That's disorder 0.

So this guy over here, let's see.

That's 2/5 and 3/5 multiplied-- You know, this is one of those deals where if you look at the curve, you're pretty close to the middle.

And that curve goes all the way up to about 0.9 there.

So you can kind of just look at this, and eyeball it, and say, well, whatever it is, the overall, this is going to be something multiplied times 5/8.

Something like 0.9 times 5/8.

So let's just say, for the sake of discussion, that that's going to be about 0.6, which is within a hundredth, I think, of being right.

Just kind of guessing.

OK, well now we're on a roll.

Here, we have 3/8 coming down this branch, 3/8 coming down this branch, 1/4 coming down this branch.

This is 0.

And this is one of those deals where these two are about 0.9.

So it looks like it's going to be 3/8 plus 3/8 is 3/4.

Times about 0.9.

So that's going to turn out to be about 0.7.

So one last go here.

3/8, 3/8, and 1/4.

Oh, that's interesting.

Because these two are what we got contributed up to that 0.7.

This one is 0.4 times-- this is evenly divided, so that's going to have disorder of 1.

So that's going to be 0.25 bigger than the number we got over here.

So that's going to end up being about 0.95.

So thanks god our answer is the same as we got with our simple classroom measurement of disorder.

Except this is measuring how disordered stuff is, we want the small number, not the big number.

So once again, based on this analysis, you'll be sure to pick the shadow cast, because 0.5 is less than 0.6, which is less than 0.7, which is less than 0.95.

So that accent test is really horrible.

Don't use it.

Just because somebody has a heavy accent doesn't mean they're a vampire.

In fact, most vampires have worked very hard on their accent, as I mentioned before.

All right, so now we know that we're still going to pick the shadow test as our first go.

So that's good.

Now, let's see if we can repeat the exercise with our second selection, the one we have to have to pick those guys apart.

And this is going to be easier, because there are fewer things to work with.

Ooh, wow, look.

That's 0.

That's 0.

That's 1/2.

That's 1/2.

So the disorder of this guy is 0.0.

So this is 1/4, 1/4, 1/2, 0, 0.

1/2 times 1.

Ooh, that's 0.5.

That was easy.

How about this one?

Oh, he says 1.

Let's see.

That's 1.

That's 1.

That's 1/2.

That's 1/2.

Yeah, it is one.

So sure enough, the answer also comes out to be the same as before, when we did our just simple intuition exercise.

So I don't know.

Christopher, is this all about using information theory?

STUDENT: No.

PATRICK WINSTON: No, no, no.

See, it's not about the math.

It's about the intuition.

And the intuition is that you want to build a tree that's as simple as possible.

And you can build a tree that's as simple as possible if you look at the data, and say, well, which test does the best job of splitting things up?

Which test does the best job of building subsets underneath it that are as homogeneous as possible?

So all this information theory, all this entropy stuff, is just a convenient mechanism for doing something that is intuitionally sound.

OK?

It's not about information theory.

It's about a sound intuition.

Oh, by the way.

Does this kind of stuff ever get used in practice?

10s of thousands of times.

This is a winning mechanism that's used over and over again, even when the data is numeric.

How would it work if it's numeric data?

Well, let's think about that for a little bit.

So let's suppose that we have an opportunity.

We're an EMT or something, we work in the infirmary.

What do they call it these days?

Something else.

But anyhow, you work in that kind of area, and you have the opportunity to take people's temperature.

And so over time, you've accumulated some data on the temperature of people.

And maybe you've found that there's a vampire here at about 102.

There's a normal person here, about 98.6.

But then they're scattered around.

Some people have fevers when they come in.

So the question is, is there a way of using numerical data-- things that you can put real numbers-- is there a way of using that with this mechanism?

And the answer is yes.

You just say, is the temperature greater than or less than some threshold?

And that gives you a test, a binary test, just like any of these other tests.

[? Krishna? ?] Right?

But where would I put the threshold?

I suppose I could just put it at the average value.

But that might not be the place that does the best job of splitting the samples into homogeneous groups.

Christopher?

STUDENT: So you run this numerical analysis on different places with different thresholds.

PATRICK WINSTON: So you try different places, he says.

And he's right.

Because this is a computer, this is our slave.

We don't care how much it works to figure out the right threshold.

So what we do is we say, well, maybe the threshold's halfway between those two guys, or halfway between those two guys, or those two guys, or those two guys, or those two guys.

So we can try one less threshold than we have samples.

And we don't care if there are 10,000 samples, because this is a computer, and we don't care if it works all night.

So that's how you find the threshold for a numeric test.

By the way, I assured you earlier on you would never use the same test twice.

Is that true for this?

Yes, you would still never use the same test twice.

But what you might do is you might use a different threshold on the same measurement the next time around.

So when you start having numerical data, you may find yourself using the same test with the same axis but with a different value.

All right.

So now that we have this, then we can go back and compare how this method would look when we put it up against the sort of stuff we were talking about last time, with the electrical covers.

So with the electrical covers, we had a situation like this.

I don't know.

We had samples that were places like this, and we had a division of the space that look pretty much like that.

Not quite exactly in the right spots, but pretty close.

So these are the decision boundaries for the situation where we are using nearest neighbors to divide up the data.

What would the decision boundaries look like if these were four different kinds of things, and we were using this kind of mechanism?

And maybe there's a lot of samples all clustered around places like that.

What would the decision boundaries look like?

Would they be the same as this?

god, I hope not.

Why?

Because what we're going to do is we're going to use a threshold on each axis.

So therefore, the decision boundaries are going to be parallel to one axis or the other.

So we might decide, for example-- Oh, shoot.

I think I'll draw it again, because it'll get confused if I draw it over the other one.

So it looks like this.

And that's how nearest neighbors does it.

But a identification tree approach will pick a threshold along one axis or the other.

Let's say it's this axis.

It's only got one choice there.

So it's going to put a line there.

And now, what's the next thing it does?

Well, it still has these two different kinds of things to separate.

We're going to assume we've got four different kinds of things.

So it's going to say, oh!

I've Come down the negative side, so I need a threshold on the remaining data.

And these are the only two things that are now remaining.

So my only choice is to put a threshold in there.

Now I guarantee this, absolutely guaranteed-- on the quiz, somebody-- presumably somebody who doesn't go to lectures-- will draw that line all the way across.

And that's desperately wrong.

Because we've already divided this data set in half.

Now the choice of what we do over here is governed only by the remaining samples that we see, these two.

And so the threshold is going to go in there like that.

So that's what happens when you go back.

This is used 10s of thousands of times.

Always used.

What are the virtues of it?

Number one, you don't use all the tests.

You use only the test that seem to be doing some useful work for you.

So that means that you do a better job, because your measurement technique is simpler.

And it costs less, because you're not going to the expense of doing all of the testing.

So it's a real winner.

But you know what?

Some classes of people-- not scientists, but I mean people like doctors and stuff.

They don't like to look at these tress.

They're kind of rule-oriented.

So they look a tree like this for determining what kind of thyroid disease you have, and it would have maybe 20 or so tests in it of various kinds of hormones, like thyroxine and this and that.

And they say, ah, we can't deal with that.

So we have to work with them.

So what we do is we convert the tree into a set of rules.

How do we convert the tree into a set of rules?

Oops, wrong one.

Go away, go away.

Here's what I want.

Yeah, good.

How would we convert this tree into a set of rules?

It's straightforward.

[INAUDIBLE], what do we do?

STUDENT: You'd basically just look down each branch-- PATRICK WINSTON: You'd basically just go down each branch to a leaf.

So you say, for example, here's one rule.

If shadow equals question mark, and garlic equals oh, [INAUDIBLE] want to choose No.

Doesn't eat garlic.

No.

I think I'll say Yes.

Yes.

That changes the answer.

Then if it eats garlic, it's not a vampire, right?

That's one of four possible rules, because there are four leaf nodes.

Now, almost done.

We are done, except for one thing.

We can actually take these four rules, and start thinking about how to simplify them.

You can ask questions like, if I have a rule that tests both the shadow and the garlic, do I actually need both of those antecedents?

And the answer is, in many cases, no.

And in particular, in this case, no.

Because if we look at our data set, what we discover is that in the event that we're talking about a shadow question mark-- oh, I guess I had a better choice the other way.

Oh, no.

If you look at the garlic, all the garlics-- Yes, Yes, and Yes-- it turns out that the answer is no, independent of what the shadow condition is.

So we can look at the rules, and in some cases, we'll discover that our tree is a little bit more complicated than it needs to be.

We can actually get rid of some of the clauses.

So in the end, we can develop a very simple mechanism based on good old fashioned rule-based behavior, like you saw almost in the beginning of the subject, that does the job.

And now, without any royalty, you're all free to put this in your PDA and use it to protect yourself in the days to com, especially since Halloween's just around the corner.

Free Downloads

Video

• iTunes U (MP4 - 171MB)
• Internet Archive (MP4 - 171MB)

Subtitle

• English - US (SRT)