Lecture 14: Learning: Sparse Spaces, Phonology

• Download this transcript - PDF (English - US)

PATRICK WINSTON: So today we're gonna talk about a few miracles of learning in the context of the theme that we're developing here in the class.

We started off with a discussion of some basic methods.

We talked about nearest neighbors.

And we talked about identification trees.

And those are kind of basic things that have been around for a long time.

Still useful.

Still the right things to do when you're faced with a learning problem and you're not sure what method to try.

Then we went on to talk about some naive biological mimicry.

We talked about neural nets.

And we talked about genetic algorithms.

And you look at those things and you think and reflect back on what we talked about.

And you have to say to yourself, are these nugatory ideas?

Perhaps pistareens?

Or are they supererogatory ideas that deserve to be center stage?

Does anybody know what those words mean?

A pistareen?

Well, a pistareen is a Spanish coin.

It was so small.

It was of little worth.

These ideas like neural nets, genetic algorithms, I classify them as pistareens because getting them to do something is rather like getting a dog to walk on its hind legs.

You can make it happen, but they never do it very well.

And you have to think it took a lot of trickery and training to make it happen.

So not too personally high on those ideas.

But we teach them to you anyway because, of course, we only editorialize part of time and part of time we like to cover what's in the field.

Today we're starting a couple of discussions of mechanisms or ideas or things to know about that are quite different.

Because now we're going to focus on the problem rather than on the mechanism.

And then a later on we're going to talk about deep theory, FIOS, for its own sake.

But this week I want to talk about mechanisms that were devised.

I want to talk about research that was done.

Let me not say mechanisms.

Let me say research that was done to attempt an account of some of the things that we humans do well.

Sometimes without even knowing that we do it.

Now Krishna here tells me his first language was Telugu.

Telugu.

I once had another student whose first language was Telugu.

I said to him, that must be one of those obscure Indian languages.

And he said, yes.

It's spoken by 56 million people.

French is spoken by 52.

[LAUGHTER] PATRICK WINSTON: He's going to be our experimental subject.

Krishna, if I pluralize words-- you know what it means to pluralize a word.

So if I say for example, horse, then if I ask you for the plural you'll say horses.

So if I say dog, what's the plural?

STUDENT: Then dogs.

Or in my language?

PATRICK WINSTON: No, no, no.

In English.

STUDENT: Oh, dogs.

PATRICK WINSTON: Well, what about cat?

STUDENT: Cats.

PATRICK WINSTON: And he got it right.

Isn't that a miracle?

When did you start speaking English?

STUDENT: Second grade.

PATRICK WINSTON: Second grade.

But he still got it right.

But he never learned that he's actually pluralizing those words differently.

But he is.

So when you pluralize dog, what's the sound that comes after?

It's a z sound.

Zzzzzz.

Dogzzz.

If you stick your fingers up here you can probably feel your vocal cords vibrating.

If you stick a piece of paper in front of your mouth you'll see it vibrate.

But when you say cats, the pluralizing sound is sss, like that.

No vocalizing.

No vibration of the vocal cords.

And old Krishna here learned that rule, as did all of you other non-native speakers of English, effortlessly and without noticing it.

You learned it.

Buy you always get it right.

How can that possibly be?

Well, be the end of hour you'll know how that might be.

And you'll experience a case study in how questions of that sort can be approached with a sort of engineering point of view.

You can say, what if God were an engineer?

Or alternatively, what if I were God and I am an engineer?

Think about how it might happen that way.

So we want to understand how it might be that the machine could learn rules like that.

Phonological rules.

Not just that one, but all the phonological rules you'd acquire in a course on phonology.

That part of speaking that deals with those syllabic and sub-syllabic sounds.

The phones of the language.

So when Yip and Sussman undertook to solve this engineering problem, both being dedicated engineers, the first thing they did was learn the science.

So they went to sit at the foot of Morris Halle, who would develop-- was largely responsible for the development theories of so-called distinctive features.

And here's how all that works.

You start off with a person who wants to say something.

And out that person's mouth comes some sort of acoustic pressure wave.

And if I say, hello, George.

And you say hello, George.

Everybody will understand that we said the same thing.

But that acoustic waveform won't look anything alike.

It'll be very different for all of us.

So it's a miracle that words can be understood.

In any case, it goes into an ear.

And it's processed.

And out comes a sequence of distinctive feature.

Vectors.

A distinctive feature is a binary variable like is the phone voices or not.

That is to say, are your vocal cords vibrating when you say it?

If so, then that's plus voiced.

If not, it's minus voiced.

So according to the original distinctive feature theory and consistent with most of the theories that have been derived since the original one, there are on the order of 14 of these distinctive features that determine which phone you're saying.

So if you say ah, that's one combination of these binary features.

If you say tuh, that's another combination of these binary features.

14 of them.

So how many sounds does that mean, in principle, there could be in a language?

SEBASTIAN: 2 to the 14th.

PATRICK WINSTON: And what's 2 the 14th, Sebastian?

Well, it ought to be about 16,000, don't you think?

2 to the 10th is 1,000.

2 the fourth is 16.

So there are about 16,000 possible combination.

But no language on Earth has more than 100 phones.

That's strange, isn't it?

Because some of those choices are probably excluded on physical ground.

But most of them are not.

So we could have a lot more phones in our language than we actually do.

English is about 40.

So the sequence of distinctive features could be viewed as then producing meaning after, perhaps, a long series of operations.

But in the end, those operations feedback in here because many of the distinctive features are actually hallucinated.

We think we heard them, but they're not there.

Or they're not even in the acoustic waveform.

They're there for the convenience of the phonologist who make rules out of them.

It's remarkable how much of this feedback there is, and even injection from other modalities.

Many of you may have heard about the McGurk Effect.

Here's who the McGurk Effect works.

Look at me while I say ga, ga, ga, ga, ga, ga.

OK.

I said, g-a.

Now how about ba, ba, ba, ba.

OK.

I said ba like a sheep.

But if I take the sound I make when I say ba and play it while you're taking video of me saying ga, what do you think you hear?

You don't hear ba.

Some people report that they hear a d-a sound like da.

When I look at it, I can't make any sense out of it.

It looks like there's a disconnection between the speech and the video.

But it does not sound like ba.

But if I shut my eyes and say ba, ba, it's absolutely clear that it's b-a.

So what you see has a large influence on what you hear.

It's also interesting-- although a side issue-- it's also interesting to note that it's very difficult pronounced things correctly if you don't see the speaker.

So many people wonder when they learn foreign languages why they can't speak like a native.

And the answer is, they're not watching the mouth of the speaker.

I was talking to a German friend once and said, you know, I just can't say the damned umlaut right.

And he said, oh, the trouble with you Americans is you don't realize that American cows say moo but German cows say muu.

[LAUGHTER] PATRICK WINSTON: And, of course, I got instantly because I could see that the umlaut sounds are produced with protruding lips, which we don't have any sounds an English that require that.

Ah, but back to what we know from the phonologists about all this stuff.

If you talk to Morris Halle, he will tell you that over here-- I like to think of it as a marionette.

There are five pieces of meat down here.

And the combination of distinctive features that you're trying to utter are like the control of a marionette on those five pieces of meat.

So if you want to say an a sound, the marionette control goes into a position that produces that combination.

So let's see.

What does that distinctive feature sequence look like for typical word?

Well, here's a word.

A-e-p-l.

Apples.

And we can talk about what distinctive features are arrayed in that particular combination of phones.

So one of the features that they like to talk about is syllabic.

Syllabic.

That roughly means, can that sound form the sort of core of a syllable?

And the answer is a can, buy these can't.

So it's plus, minus, minus, minus.

Down here a little ways you'll run into the voiced feature.

And for the voiced feature, well, we can do the experiment ourselves.

Ahh.

Sounds like it's voices to me.

Pa.

No.

That's not voiced.

Oo.

Yep.

Zzz.

We already said that was voiced.

So that's the combination you see when you utter apples for the voiced feature.

Then another one is the continuent one.

That roughly says is your vocal apparatus open?

Is there no obstruction?

And so ahh plus pa is constricted.

Oo, open.

Zzz, open.

So that one happens to run right along with voiced in that particular word.

Oh, and there are 14 altogether.

But let me just write down one more.

The strident one.

That says, do you use your tongue to form a little jet of air?

So you don't on aa, pa, oo.

Buy you do on z.

So that gets a plus.

So that's a glimpse through a soda straw of what it would like to represent the word apples as a set of distinctive features all arranged in a sequence.

So it's a matrix of features.

Going down in the columns we have our distinctive features.

And going across we have time.

So as the first thing Sussman and Yip did in their effort to understand how phonological rules could be learned is to design a machine that would interpret words and sounds and things that you see so as to produce the sounds of the language.

So they imagined the following kind of machine.

The machine has some kind of mystery apparatus over here that looks out into the world and sees what's there.

So I'm looking out in the world and I see two apples.

So what this machine might do then is, at some point, decide that there are two apples out there.

Then, thinking in terms of these guys as computer engineers, they think in terms of a set of registers that hold values for concepts like noun and verb and plural.

And we've not done anything with the machine yet.

We've provided no input.

So those registers are all empty.

Then, up in here, we have a set of words.

And they're all kinds of words.

Apple is one of them.

And those words up there know about how the concept is rendered as a sequence of a phones, that is to say a sequence of distinct features.

Then, over here, most importantly, they have a set of constraints.

So we'll talk about a particular constrain, the plural constraint.

Plural constraint number one.

And it's going to reach around and connect itself to some other parts of the machine.

Finally, there's a buffer of phones to be uttered.

And they're going to flow out this way to the speaker's mouth and get translated into a acoustic wave form.

So those are the elements of the machine.

Now how are the elements connected together?

Well, the words are connected, of course, into the buffer that is used to generate the sound over here on the far left.

The plural register is connected to what you see in the world.

What you see in the world is connected not only to plural register, but to all of the objects in the word repertoire.

This plural constraint here deserves extra attention because it's going to be desirous of actuating itself in the event but the thing observed in the world is plural.

There are lots of them.

So it's going to be connected then to the plural port.

There's going to be a z sound port down here connecting to that file element in the buffer.

And finally, over here is going to be a plussed voiced port, which is going to be connected to the second phoneme in the sequence.

That's how the machine is going to be arranged.

An of course, this is just one of many constraints.

But it's a constraint that has a very peculiar property.

Information can flow through it in multiple ways.

So we think of most programs as having an input and an output.

But I try to be careful to draw circles here instead of arrows.

Because these are ports and information can flow in any direction along them.

What I want to do now is to show you how this machine would react if I suddenly present it with a pair of apples like so.

So the assumption is that the vision apparatus comes in and produces the notion, the concept, of two apples.

So once that has happened-- that's operation number one-- then information flows from that meaning register up here to the apple word.

So that's part of stage number two.

Another part of stage number two is information flows along this wire and marks that as plus plural.

So operation number one is the activity of the vision system.

Activity number two is the flow of information from that vision system into the word lexicon and into this plural register.

So far so good.

Here's activity number three.

This word is also connected to the registers.

And information flows along those wires so as to indicate that it's a noun but not a verb.

That's part of part number three.

At the same time, part number three, information flows down this wire and writes a-p-l into those are elements of the buffer.

Now this constraint up here, this box, says, well, I can now see some stuff in that buffer that wasn't there before.

So it says, do I see enough stuff on my ports to get excited about expressing values on other ports?

Well, let's see.

What has it got?

It's got the elements in this buffer.

Also up here in step three flow the plural thing.

So it know that the word is plural.

So it says, is this voiced?

P is pa.

That's not voiced.

Is this a z sound.

No, that's not as z sound.

So it sees what it likes on only one of its three ports.

So it says, I'm not going to do anything.

I'm [INAUDIBLE].

I'm not in this particular combat.

So far so good.

What happens next?

What happens next is that some time passes.

And the elements of the buffer flow to the left toward the speaker's mouth.

So we get an a, p, l.

Same as we had before, but shifted over.

Now what happens?

Now what happens is that the l is now in the penultimate position.

So information flows up here.

Item number four-- oh, I guess that's item number five.

Item number four is the leftward flow of the word.

So in phase number five, the p is witnessed by this constraint.

p is-- sorry, l is witnessed by this constraint.

We moved it over one.

L is lll.

L is voiced.

So we have some flow up here like that.

That's number five.

Now we have voiced and we have plural.

And we have nothing here.

So there's a great desire of this buffer to have something written into it.

So now there's a flow down in there, of z, as item number six.

So that's how the machine would work in expressing the idea that there are apples in the field of view.

Mmm.

Real apples.

Not plastic imitations.

So that's how the machine works.

But all those connections are reversible.

So if I hear apples then I get the machine running backwards and my visual apparatus can imagine that there are apples out there.

That's how it works.

That's just by way of background the machine that they could see it for using the phonological rules once they're learned.

All the phonological rules are expressed in these constraints.

But since these constraints are such that information can flow in any direction, they deserve to be called propagators.

And in the good old days when everyone took 6.001, they learned about propagators as a kind of architecture for building complex systems.

But in any event, there's the Sussman-Yip machine.

And now comes the big question.

How do you learn rule rules like that?

Well, what we need is we need some positive examples and some negative examples.

And for the simple classroom example I've chosen the same challenge that I presented to Krishna.

We're gonna have cats and dogs.

So we're gonna look at the distinctive features that are associated with those words.

Syllabic.

Voiced.

Continuent.

And strident.

Just four of the 14 features that are associated with each of the sounds on those words.

Could you close the laptop, please?

Just for the distinctive features that are arrayed in those words by way of illustration.

So here we have k-a-t-z.

Phonetically spelled.

And if we work that out, let's see.

What is syllabic?

That's not.

That is.

That is.

That's not.

Voiced?

Ka.

Nope.

Ah.

Yep.

T. Nope.

Z. Yes.

That can't be right.

Cats.

I misspelled it.

Because cats.

Sss.

His a hissing sound but there's no voicing.

So that's not as z sound.

That's an s sound.

So that's not plus voiced.

It's minused voiced.

Continuent.

Let's see.

Is my mouth open when I say k?

No.

Ah?

Yes.

T?

No.

S?

Yes.

And strident.

Minus, minus, minus, plus.

It's only with the s sound that I have that kind of jet forming with my tongue.

Now we can look at dogs.

And now we have the z sound as the pluralization.

We know that because when we say it, dogzz.

Yep.

There it comes out as a-- we're only gonna look at the last two columns because they're the only ones that are going to matter to us.

So that's plus.

And that's minus.

Gu, gu, gu, gu.

That's plussed.

And that's plussed.

They're both voiced.

Is that right?

Dogu?

Gu.

Gu.

Is g sound voiced?

Yeah, I didn't think so.

G sound is voiced?

Look-- oh.

Oh, it is voiced buy it's not a continuent.

Just like that.

Yeah.

Cat, dogu zz.

Yeah.

It is voiced.

And it has to be for my example to work out.

And that's minus, minus, minus, plus.

So what we're interested in is, how come one word gets an s sound and how come the other words gets a z sound?

Well, it's a pretty sparse space out there.

We've already decided that there are 14,000 possible phonemes and there are only 40 in the language.

So that's one thing we can consider.

The other thing that we can think is that, well, maybe this is a logical problem.

Like the kind of problem you'd face if you were designing a computer.

And so Sussman and Yip got stuck for three months thinking about the problem that way.

Couldn't make any progress whatsoever.

And that happens a lot when you're doing a search.

You think you've got a way of approaching it.

Try to make it work.

You stay up all night.

Stay up all night again.

Still can't make it work.

Eventually, you abandon ship and try something else.

So then they began to say, well, let's see.

All we care about is the stuff before the two ending sounds.

We care about that part of the matrix.

And we care about that part of the matrix.

And we can ask, in what ways are those things different?

And they're different all over the place.

That's why they're different words.

We can ask the question a little bit differently.

And we can say, what can we not care about?

And still retain enough of an understanding of how the words are different so as to put the proper plural ending on them.

And they worried about that for a long time.

Couldn't find a solution.

The search space was too big.

And then they said, maybe what we ought to do is we ought to think about generalizing this guy here so that we don't care about it.

So now we don't care about that guy.

And then he went down through here saying, well, let's see when we have to stop generalizing.

Because we've screwed everything up and we can no longer keep the z sound words separated from the s sound words.

So that eventually distilled itself down to the following algorithm.

First thing they did was to collect positive and negative examples.

And there's a positive example and a negative example.

That's not enough to do it right.

But that's enough to illustrate the idea.

So the next thing they did was something that's extremely common in learning anything.

And that is to pick a positive example to start from.

It's actually not a bad idea in learning anything to start with a positive example.

So they picked a positive example and they called that a seed.

So in our particular case, cats is going to be our seed.

And the question we're going to ask is, what are the words that get pluralized like cat?

So we've got a positive and negative example.

We've picked a seed.

And now, the next step is to generalize.

And what I mean by generalize is you pick some places in the phoneme matrix that you just don't care about.

So you may pick a positive example.

And you don't care about it.

So you change it to an asterisk or, as demonstrated in the program I'm about show you, a ball.

Or you pick one that's negative and you turn it to a ball.

Bo.

So cats, this seed, becomes a pattern.

And in order to pluralize the word this way, you have to match all the stuff in here.

But now what we're going to do is we're going to gradually turn some of those elements into don't care symbols until we get to a point where we've not cared about so much stuff that we think that we pluralize that one with an s sound too.

So we keep generalizing until we cover, that is to say we admit or match, a negative example.

So that's how it works.

So we generalize like crazy.

And as soon as we cover a negative example, we quit.

Otherwise, we just go back up here and generalize some more.

And now we've got to pick a search technique to decide which of these guys to actually generalize when.

We could pick one at random.

And they tried that.

It didn't work.

So what they decided is that the thing that influences the pluralization most is the adjacent phoneme.

And if that isn't the thing that solves the problem, it'll be the one next to that.

So in other words, the closer you are, the more likely you are to determine the outcome.

So these guys over here are least likely to matter.

And those are the ones that are generalized first.

So if we do that, what happens?

Looks like we're going to come in here and see that there's a big difference between the non-voiced t and the voiced g.

But that's only a guess because I've only shown you a fraction of the 14 distinctive features that are involved.

So I suppose you like to see a demonstration.

Yeah.

So there's our 14 features.

And that's our seed there, sitting prominently in the display with pluses and minuses indicating the values of the distinctive features for all three of the phones involved.

That funny left bracket isn't a mistake.

That's just one convention for rendering the ah sound in cat.

So it's pretty hard to tell from just that matrix what's going to be the determining feature that separates the positive examples from the negative examples.

You notice that there are actually two examples down here.

There's cat and duck.

Is ducks got an s sound?

Ducks?

Yep.

So dogs and ducks.

They both get pluralized with an s sound.

And then we have beach doesn't.

That's beaches.

Dog.

We know that's a z.

Gun.

Gunz.

So that's not in the group.

So we can run this experiment.

Now here we go.

We're generalizing like crazy.

Generalizing, generalizing, generalizing from left to right.

So nothing in the first two columns matters.

Now we get to the t.

Wow.

There it is.

So it looks like you pluralize with a s sound.

The sss.

If, and only if, you're not voiced and you're not strident in the second to the last-- in the last phone of the word that you're trying to pluralize.

So that's one phonological rule that the system has learned.

And guess what?

It's the same rule that's found in phonological textbooks.

So now we can try another experiment.

So this time we're trying to deal with dog and gun.

And our negatives are what was previously positive plus beach, which is still in there as a negative example.

So let's see how that one works.

Nothing matters except for the last column, the last phone.

And now we find out that if the last sound is voiced, then the pluralization gets the z sound, a voiced determinator.

And finally, just to deal with beaches.

That's beach in it's funny phonetic spelling.

So now, if the final sound in the word is strident, if its got this jetty sound-- beach.

Beach.

Then it gets the ea sound.

So let's go back to experiment number one.

Because I want to point out one small thing about the way this works.

You'll notice that it talks about coverage and excluded down here in the lower left-hand corner.

Excluded, well, there are three negative examples, so they better all be excluded.

You don't want to cover any of the negatives.

But it says coverage, two and two.

That's because it actually is doing-- and now we have the vocabulary to say it quickly-- it's doing a beam search through this space.

So it's not just doing a depth first search.

It's doing a beam search so as to reduce the possibility of overlooking a solution.

So it says, oh, the coverage.

Both of the beam search elements cover both of the positive examples.

And they, in fact, have converged to the same solution.

So that's how the Sussman and Yip thing worked.

And then the next question to ask is, of course, why did it work?

And so the answer, as articulated by Sussman and Yip-- or rather more by Sussman.

Or rather more by Yip and a little bit less by Sussman.

Yip thinks that it worked because it's a sparse space.

And when you have a high dimensional sparse space, it's easy to put a hyperplane into the space to separate one set of examples for another set of examples.

So let's consider the following situation.

Suppose we have a one-dimensional situation.

And we have two white examples and we have two purple examples.

Well, too bad for us you can't separate them.

Now suppose that this is actually the projection of a two-dimensional space that looks like this.

Here are the white examples down here.

And here are the purple examples up here.

Now it's easy to see that you can separate them with just a line that goes across like that.

Now let's take this one more step and suppose that this is actually a projection of a three-dimensional space.

It looks like this.

This will be dimension one.

This'll be two going back there.

And this will be three up here.

And suppose that the positive examples are right here on this line.

Let's say this is-- well, we're gonna draw a little old cube like so.

Those are purple examples that are up there.

How many ways are there of partitioning the space along those axes?

Well, now they're not even just two.

They're three.

So one way to separate the purple from the white is to draw a hyperplane-- or in this case it's a three dimension, so a plane-- through here on the number three axis.

You could also put a plane in on that axis.

Or you could do both.

So in one case your dividing line would be-- let's see.

On the first axis that would be 1/2.

And then the don't care.

Don't care.

Another solution that would be don't care.

And then we divide on the number 2 axis with a plane at 1/2 and don't care.

Or we could do it with 1/2, 1/2, and don't care.

So the higher the dimension of the space, the easier it is sometimes to put in a plane that separates the data.

That's why Sussman and Yip think that we use so little of possible phoneme space.

Because it makes the thing learnable.

That's one possibility.

So one explanation for sparse space is learnability.

There's another interesting possibility, and that is that if you have a sparse space, high dimensional space with 14 dimensions, and if the 40 points of your language are spread evenly throughout that space-- now let me say it the other way.

If they are placed at random in that space, then according to the central limit theorem, then they'll be about equally distant from each other.

So it ensures that the phonemes are easily separated when you speak.

But if you go to ask a linguist if that's true, they don't know.

Because they're not looking at it from a computational point of view.

Well, we can look at it from a computational point of view.

So I did that.

After Sussman and Yip published their paper.

And here's the result.

This is a diagram that shows all of the phonemes that are separated by exactly one distinctive feature.

So if you look over in this corner here, you'll see that the constants-- w and x-- are separated by exactly one distinctive feature.

So they're not exactly distant from each other in the space.

On the other hand, they are pretty easy to separate relative to the vowels.

Which are here in this part of the diagram.

Which are all tangled up and the vowels are all close to each other.

So guess what?

Vowels are much harder to separate than constants.

Not surprisingly, because there are many pairs of them that are different.

And only one distinctive feature.

All right.

So now you back up and you say, well, gosh.

That's all been sort of interesting.

But what does it teach us about how to do science and stuff?

And what it teaches us is-- this is an example.

Ow.

This is an example which we can use to illuminate some of thoughts of David Marr, who I spoke of in a previous lecture, connection with vision.

But here's Marr's catechism.

I can't spell very well so I won't try to respell it.

But this is Marr's catechism.

So what Marr said is, when you're dealing with an AI problem, first thing to do is to specify the problem.

Gee, that sounds awfully normal.

The next thing is to devise a representation suited to the problem.

The third thing to do, vocabulary varies, but it's something like determine an approach.

Sometimes thought of as a method.

And then four, pick a mechanism or devise an algorithm.

And, finally, five, experiment.

And of course, it never goes linearly like that.

You start with the problem and then you go through a lot of loops up here.

Sometimes even changing the problem.

But that's just the scientific method, right?

You start with the problem and you end up with the experiment.

But that's not what people in AI, over the bulk of its existence, have tended to do.

What they tended to do is to fall in love with particular mechanisms.

And then they attempt to apply those mechanisms to every problem.

So you might say, well, gee, neural nets are so cool.

I think all of human intelligence can be explained with a suitable neural net.

That's not the right way to do it.

Because that's mechanism envy.

You fall in love with mechanism.

You try to apply it where it isn't the right thing.

This is example starting with the problem and bringing to the problem the right representations, gosh, distinctive features.

Once we've got the right representation, then the constraints emerge, which enable us to devise an approach, write an algorithm, and do an experiment.

As they did.

So this Sussman-Yip thing is an example of doing AI stuff in a way that's congruent with the Marr's catechism.

Which I highly recommend.

They could have come in here and said, well, we're devotees of the idea of neural nets.

Let's see if we can make a machine that will properly pluralize words using a neural net.

That's a loser.

Because it doesn't match the problem to the mechanism.

It tries to force fit the mechanism into some Procrustean bed where it doesn't actually work very well.

So what this leaves open, of course, is the question of, well, what is a good representation?

And here's the other half Marr's catechism.

Characteristic number one is that it makes the right things explicit.

So in this particular case, it makes distinctive features explicit.

Another thing that Marr was noted for was stereo vision.

So in that particular world, discontinuities in the image, when you go across an edge with the things that were made explicit.

Once you've got to a representation that makes the right things explicit, you can say, does it also expose constraint?

And if you have a representation that exposes constraint, then you're off and running.

Because it's constraint that you need in order to do the processing that leads to a solution.

So don't have the right representation.

If it doesn't expose constraints, you're not going to be able to make a very good model out of it.

And finally, there's a kind of localness criteria.

If you have a representation in which you can see the right answer by looking at descriptions through soda straw, that's probably a better representation than one that's all spread out.

It's true with programs, right?

If you can see how they work by looking through a soda straw, you're in much better situation to understand something if you have to look here and there and on the next page and in the next file.

So all this is basically common sense.

But this is kind of common sense that makes you smarter as an engineer and scientist.

Especially as a scientist because if you go into a problem with mechanism envy, you're apt to study mechanisms in a naive way and never reach a solution that will be satisfactory.

Free Downloads

Video

• iTunes U (MP4 - 164MB)
• Internet Archive (MP4 - 164MB)

Subtitle

• English - US (SRT)