Lecture 7: Hashing, Hash Functions

Related Resources

Lecture Notes (PDF)
Assignments
Exams

• Download this transcript - PDF (English - US)

Today starts a two-lecture sequence on the topic of hashing, which is a really great technique that shows up in a lot of places. So we're going to introduce it through a problem that comes up often in compilers called the symbol table problem. And the idea is that we have a table S holding n records where each record, just to be a little more explicit here. So each record typically has a bunch of, this is record x. x is usually a pointer to the actual data. So when we talk about the record x, what it usually means some pointer to the data.

And in the data, in the record, so this is a record, there is a key called a key of x. In some languages it's key, it's x dot key or x arrow key, OK, are other ways that that will be denoted in some languages.

And there's usually some additional data called satellite data, which is carried around with the key.

This is also true in sorting, but usually you're sorting records. You're not sorting individual keys. And so the idea is that we have a bunch of operations that we would like to do on this data on this table. So we want to be able to insert an item x into the table, which just essentially means that we update the table by adding the element x.

We want to be able to delete an item from the table -- -- so removing the item x from the set and we want to be able to search for a given key. So this returns the value x such that key of x is equal to k, where it returns nil if there's no such x. So be able to insert items in, delete them and also look to see if there's an item that has a particular key. So notice that delete doesn't take a key. Delete takes a record.

OK, so if you want to delete something of a particular key and you don't happen to have a pointer to it, you have to say let me search for it and then delete it.

So these, whenever you have a set operations, where operations that change the set like in certain delete, we call it a dynamic set. So these two operations make the set dynamic. It changes over time.

Sometimes you want to build a fixed data structure.

It's going to be a static set. All you're going to do is do things like look it up and so forth.

But most often, it turns out that in programming and so forth, we want to have the set be dynamic. Want to be able to add elements to it, delete elements to it and so forth.

And there may be other operations that modify the set, modify membership in the set. So the simplest implementation for this is actually often overlooked.

I'm actually surprised how often people use more complicated data structures when this simple data structure will work. It's called a direct access table. Doesn't always work.

I'll give the conditions where it does.

So it works when the keys are drawn from our small distribution. So suppose the keys are drawn from a set U of m elements. OK, zero to m minus one.

And we're going to assume the keys are distinct.

So the way a direct access table works is that you set up an array T -- -- from zero to m minus one to represent the dynamic set S -- -- such that T of k is going to be equal to x if x is in the set and its key is k and nil otherwise.

So you just simply have an array and if you have a record whose key is some value k, the key is 15 say, then slot 15 if the element is there has the element.

And if it's not in the set, it's nil.

Very simple data structure. OK, insertion.

Just go to that location and set the value to the inserted value. For deletion, just remove it from there. And to look it up, you just index it and see what's in that slot.

OK, very simple data structure. All these operations, therefore, take constant time in the worst case.

But as a practical matter, the places you can use this strategy are pretty limited. What's the issue of limitation here? Yes.

OK, so that's a limitation surely.

But there's actually a more severe limitation.

Yeah. What does that mean, it's hard to draw?

No. Yeah. m minus one could be a huge number.

Like for example, suppose that I want to have my set drawn over 64 bit values. OK, the things that I'm storing in my table is a set of 64-bit numbers.

And so, maybe a small set. Maybe we only have a few thousand of these elements. But they're drawn from a 64-bit value. Then this strategy requires me to have an array that goes from zero to 2 to the 64th minus one.

How big is 2^64 minus one? Approximately?

It's like big. It's like 18 quintillion or something. I mean, it's zillions literally because it's like it's beyond the illions we normally use.

Not a billion or a trillion. It's 18 quintillion.

OK, so that's a really big number.

So, or even worse, suppose the keys were drawn from character strings, so people's names or something.

This would be an awful way to have to represent it.

Because most of the table would be empty for any reasonable set of values you would want to keep.

So the idea is we want to try to keep something that's going to keep the table small, while still preserving some of the properties. And that's where hashing comes in. So hashing is we use a hash function H which maps the keys randomly.

And I'm putting that in quotes because it's not quite at random. Into slots table T.

So we call each of the array indexes here a slot.

So you can just sort of think of it as a big table and you've got slots in the table where you're storing your values.

And so, we may have a big universe of keys.

Let's call that U. And we have our table over here that we've set up that has -- -- m slots.

And so we actually have then a set that we're actually going to try to represent S, which is presumably a very small piece of the universe. And what we'll do is we'll take an element from here and map it to let's say to there and take another one and we apply the hash function to the element.

And what the hash function is going to give us is it's going to give us a particular slot. Here's one that might go up here. Might have another one over here that goes down to there. And so, we get it to distribute the elements over the table. So what's the problem that's going to occur as we do this? So far, I've been a little bit lucky. What's the problem potentially going to be?

Yeah, when two things are in S, more specifically, get assigned to the same value. So I may have a guy here and he gets mapped to the same slot that somebody else has already been mapped to. And when this happens, we call that a collision.

So we're trying to map these things down into a small set but we could get unlucky in our mapping, particularly if we map enough of these guys. They're not going to fit.

So when a record -- -- to be inserted maps to an already occupied slot -- -- a collision occurs.

OK. So looks like this method's no good. But no, there's a pretty simple thing we can do. What should we do when two things map to the same slot? If we want to represent the whole set, but you can't lose any data, can't treat it like a cache. In a cache what you do is it uses a hashing scheme, but in a cache, you just kick it out because you don't care about representing a set precisely. But in a hash table you're programming, you often want to make sure that the values you have are exactly the values in the sets so you can tell whether something belongs to the set or not.

So what's a good strategy here? Yeah.

Create a list for each slot and just put all the elements that hash to the same slot into the list.

And that's called resolving collisions by chaining.

And the idea is to link records in the same slot -- -- into a list. So for example, imagine this is my hash table and this for example is slot i.

I may have several things that are, so I'm going to put the key value -- -- have several things that may have been inserted into this table that are elements of S. And what I'll do is just link them together. OK, so nil pointer here.

And this is the satellite data and these are the keys.

So if they're all linked together in slot i, then the hash function applied to 49 has got to be equal to the hash function of 86 is equal to the hash function of 52, which equals what?

There's only one thing I haven't.

i. Good.

Even if you don't understand it, your quizmanship should tell you. He didn't mention i.

That's equal to i. So the point is when I hash 49, the hash of 49 produces me some index in the table, say i, and everything that hashes to that same location is linked together into a list OK.

Every record. Any questions about what the mechanics of this. I hope that most of you have seen this, seen hashing, basic hashing in 6.001, right? They teach it in?

They used to teach it 6.001. Yeah.

OK. Some people are saying maybe.

They used to teach it. Good.

So let's analyze this strategy. The analysis.

We'll first do worst case.

So what happens in the worst case?

With hashing? Yeah, raise your hand so that I could call on you. Yeah.

Yeah, all hash keys, well all, all the keys in S.

I happen to pick a set S where my hash function happens to map them all to the same value. That would be bad.

So every key hashes to the same slot.

And so, therefore if that happens, then what I've essentially built is a fancy linked list for keeping this data structure. All this stuff with the tables, the hashing, etc., irrelevant.

All that matters is that I have a long linked list.

And then how long does an access take?

How long does it take me to insert something or well, more importantly, to search for something.

Find out whether something's in there.

In the worst case. Yeah, it takes order n time.

Because they're all just a link, we just have a linked list. So access takes data n time if as we assume the size of S is equal to n.

So from a worst case point of view, this doesn't look so attractive. And we will see data structures that in worst case do very well for this problem.

But they don't do as good as the average case of hashing.

So let's analyze the average case.

In order to analyze the average case, I have to, whenever you have averages, whenever you have probability, you have to state your assumptions.

You have to say what is the assumption about the behavior of the system. And it's very hard to do that because you don't know necessarily what the hash function is. Well, let's imagine an ideal hash function. What should an ideal hash function do?

Yeah, map the keys essentially at random to a slot.

Should really distribute them randomly.

So we call this the assumption -- -- of simple uniform hashing.

And what it means is that each key k in S is equally likely -- -- to be hashed to any slot in T and we're actually have to make an independence assumption. Independent of where other records, other keys are hashed.

So we're going to make this assumption and includes n an independence assumption. That if I have two keys the odds that they're hashed to the same place is therefore what?

What are the odds that two keys under this assumption are hashed to the same slot, if I have, say, m slots? One over m.

What are the odds that one key is hashed to slot 15?

One over m. Because they're being distributed, but the odds in particular two keys are hashed to the same slot, one over m.

So let's define. Is there a question?

No. OK.

The load factor of a hash table with n keys at m slots to be alpha which is equal to n over m, which is also if you think about it, just the average number of keys per slot.

So alpha is the average number of keys per, we call it the load factor of the table. OK.

How many on average keys do I have?

So the expected, we'll look first at unsuccessful search time. So by unsuccessful search, I mean I'm looking for something that's actually not in the table. It's going to return nil.

I look for a key that's not in the table.

It's going to be what? It's going to be order.

Well, I have to do a certain amount of work just to calculate the hash function and so forth. It's going to be order at least one plus, then I have to search the list and on average how much of the list do I have to search?

What's the cost of searching that list?

On average. If I'm searching at random.

If I'm searching for a key that's not in the table.

Whichever one it is, I got to search to the end of the list, right? So what's the average cost over all the slots in the table? Alpha.

Right? Alpha.

That's the average length of a list.

So this is essentially the cost of doing the hash and then accessing the slot and that is just the cost of searching the list.

So the expected unsuccessful search time is proportional essentially to alpha and if alpha's bigger than one, it's order alpha. If alpha's less than one, it's constant.

So when is the expected search time -- -- equal to order one?

So when is this order one?

Simple questions, by the way.

I only ask simple questions. Some guys ask hard questions.

Yeah. Or in terms first we'll get there in two steps, OK.

In terms of alpha, it's when?

When alpha is constant. If alpha in particular is.

Alpha doesn't have to be constant.

It could be less than constant. It's O of one, right. OK, or equivalently, which is what you said, if n is O of m.

OK, which is to say if the number of elements in the table is order, is upper bounded by a constant times n.

Then the search cost is constant.

So a lot of people will tell you oh, a hash table runs in constant search time. OK, that's actually wrong.

It depends upon the load factor of the hash table.

And people have made programming errors based on that misunderstanding of hash tables. Because they have a hash table that's too small for the number of elements they're putting in there. Doesn't help.

The number may in fact will grow with the, since this is one plus n over m, it actually grows with n.

So unless you make sure that m keeps up with n, this doesn't stay constant. Now it turns out for a successful search, it's also one plus alpha.

And for that you need to do a little bit more mathematics because you now have to condition on searching for the items in the table. But it turns out it's also one plus alpha and that you can read about in the book.

And also, there's a more rigorous proof of this.

I sort of have glossed over the expectation stuff here, doing sort of a more intuitive proof.

So both of those things you should look for in the book.

So this is one reason why hashing is such a popular method, is it basically lets you represent a dynamic set with order one cost per operation, constant cost per operation, inserting, deleting and so forth, as long as the table that you're keeping is not much smaller than the number of items that you're putting in there. And then all the operations end up being constant time. But it depends upon, strongly upon this assumption of simple uniform hashing.

And so no matter what hash function you pick, I can always find a set of elements that are going to hash, that that hash function is going to hash badly.

I just could generate a whole bunch of them and look to see where the hash function takes them and in the end pick a whole bunch that hash to the same place.

We're actually going to see a way of countering that, but in practice people understand that most programs that need to use things aren't really reverse engineering the hash function. And so, there's some very simple hash functions that seem to work fairly well in practice.

So in choosing a hash function -- -- we would like it to distribute -- keys uniformly into slots and we also would like that regularity in the key distributions -- -- should not affect uniformity.

For example, a regularity that you often see is that all the keys that are being inserted are even numbers.

Somebody happens to have that property of his data, that they're only inserting even numbers.

In fact, on many machines, since they use byte pointers, if they're sorting things that are for example, indexes to arrays or something like that, in fact, they're numbers that are typically divisible by four.

Or by eight. So you don't want regularity in the key distribution to affect the fact that you're distributing slots. So probably the most popular method that's used just for a quick hash function is what's called the division method.

And the idea here is that you simply let h of k for a key equal k modulo m, where m is the number of slots in your table.

And this works reasonably well in practice, but you want to be careful about your choice of modulus.

In other words, it turns out it doesn't work well for every possible size of table you might want to pick.

Fortunately when you're building hash tables, you don't usually care about the specific size of the table.

If you pick it around some size, that's probably fine because it's not going to affect their performance.

So there's no need to pick a specific value.

In particular, you don't want to pick -- -- m with a small divisor -- -- and let me illustrate why that's a bad idea for this particular hash function.

I should have said small divisor d.

So for example -- -- if D is two, in other words m is an even number, and it turns out that we have the situation I just mentioned, all keys are even, what happens to my usage of the hash table? So I have an even slot, even number of slots, and all the keys that the user of the hash table chooses to pick happen to be even numbers, what's going to happen in terms of my use of the hash table?

Well, in the worst case, they are always all going to point in the same slot no matter what hash function I pick.

But here, let's say that, in fact, my hash function does do a pretty good job of distributing, but I have this property. What's a property that's going to have no matter what set of keys I pick that satisfies this property? What's going to happen to the hash table? So, I have even number, mod an even number. What does that say about the hash function? It's even, right?

I have an even number mod. It's even.

So, what's going to happen to my use of the table?

Yeah, you're never going to hash anything to an odd-numbered slot. You wasted half your slots.

It doesn't matter what the key distribution is.

OK, as long as they're all even, OK, that means the odds slots are never used. OK, an extreme example, here's another example, imagine that m is equal to two to the r. In other words, all its factors are small divisors, OK?

In that case, if I think about taking k mod n, OK, the hash doesn't even depend on all the bits of k, OK? So, for example, suppose I had one..., and r equals six, OK, so m is two to the sixth. So, I take this binary number, mod two to the sixth, what's the hash value?

If I take something mod a power of two, what does it do?

So, I hash this function. This is k, OK, in binary. And I take it mod two to the sixth. Well, if I took it mod two, what's the answer? What's this number mod two?

Zero, right. OK, what's this number mod four? One zero.

What is it mod two to the sixth?

Yeah, it's just these last six bits.

This is H of k. OK, when you take something mod a power of two, all you're doing is taking its low order bits. OK, mod two to the r, you are taking its r low order bits.

So, the hash function doesn't even depend on what's up here.

So, that's a pretty bad situation because generally you would like a very common regularity that you'll see in data is that all the low order bits are the same, and all the high order bits differ, or vice versa.

So, this particular is not a very good one.

So, good heuristics for this is to pick m to be a prime, not too close to a power of two or ten because those are the two common bases that you see regularity in the world.

A prime is sometimes inconvenient, however. But generally, it's fairly easy to find primes.

And there's a lot of nice theorems about primes.

So, generally what you do, if you're just coding up something and you know what it is, you can pick a prime out of a textbook or look it up on the web or write a little program, or whatever, and pick a prime.

Not too close to a power of two or ten, and it will probably work pretty well. It will probably work pretty well. So, this is a very popular method, the division method. OK, but the next method we are going to see is actually usually superior.

The reason people do this is because they can write in-line in their code. OK, but it's not usually the best method. And the reason is because division, one of the reasons is division tends to take a lot of cycles to compute on most computers compared with multiplication or addition. OK, in fact, it's usually done with taking several multiplications.

So, the next method is actually generally better, but none of the hash function methods that we are talking about today are, in some sense, provably good hash functions. OK, so for the multiplication method, the nice thing about it is just essentially requires multiplication to do. And, for that is, also, we are going to assume that the number of slots is a power of two which is also often very convenient.

OK, and for this, we're going to assume that the computer has w bit words. So, it would be convenient on a computer with 32 bits, or 64 bits, for example.

OK, this would be very convenient.

So, the hash function is the following.

h of k is equal to A times k mod, two to the w, right shifted by w minus r. OK, so the key part of this is A, which has chosen to be an odd integer in the range between two to the w minus one and two to the w.

OK, so it's an odd integer that the full width of the computer word. OK, and what you do is multiply it by whatever your key is, by this funny integer.

And, then take it mod two to the w.

And then, you take the result and right shift it by this fixed amount, w minus r. So, this is a bit wise right shift. OK, so let's look at what this does. But first, let me just give you a couple of tips on how you pick, or what you don't pick for A. So, you don't pick A too close to a power of two. And, it's generally a pretty fast method because multiplication mod two to the w is faster than division. And the other thing is that a right shift is fast, especially because this is a known shift. OK, you know it before you are computing the hash function. Both w and r are known in advance. So, the compiler can often do tricks there to make it go even faster.

So, let's do an example to understand how this hash function works. So, we will have, in this case, a number of slots will be eight, which is two to the three.

And, we'll have a bizarre word size of seven bits.

Anybody know any seven bit computers out there?

OK, well, here's one. So, A is our fixed value that's used for hashing all our keys. And, in this case, let's say it's 1011001. So, that's A.

And, I take in some value for k that I'm going to multiply.

So, k is going to be 1101011. So, that's my k.

And, I multiply them. What I multiply two, each of these is the full word width.

You can view it as the full word width of the machine, in this case, seven bits.

So, in general, this would be like a 32 bit number, and my key, I'd be multiplying two 32 bit numbers, for example. OK, and so, when I multiply that out, I get a 2w bit answer. So, when you multiply two w bit numbers, you get a 2w bit answer.

In this case, it happens to be that number, OK? So, that's the product part, OK? And then we take it mod two to the w. Well, what mod two to the w says is that I'm just taking, ignoring the high order bits of this product. So, all of these are ignored, because, remember that if I take something, mod, a power of two, that's just the low order bits.

So, I just get these low order bits as being the mod.

And then, the right shift operation, and that's good also, by the way, because a lot of machines, when I multiply two 32 bit numbers, they'll have an instruction that gives you just the 32 lower bits. And, it's usually an instruction that's faster than the instruction that gives you the full 64 bit answer. OK, so, that's very convenient.

And, the second thing is, then, that I want just the, in this case, three bits that are the high order bits of this word. So, this ends up being my H of k. And these end up getting removed by right shifting this word over.

So, you just right shift that in, zeros come in, in a high order bit, and you end up getting that value of H of k. OK, so to understand what's going on here, why this is a pretty good method, or what's happening with it, you can imagine that one way to think about it is to think of A as being a binary fraction.

So, imagine that the decimal point is here, sorry, the binary point, OK, the radix point is here.

Then when I multiply things, I'm just taking, the binary point ends up being there.

OK, so if you just imagine that conceptually, we don't have to actually put this into the hardware because we just do what the hardware does.

But, I can imagine that it's there, and that it's here.

And so, what I'm really taking is the fractional part of this product if I treat A as a fraction of a number.

So, we can certainly look at that as sort of a modular wheel.

So, here I have a wheel where this is going to be, that I'm going to divide into eight parts, OK, where this point is zero. And then, I go around, and this point is then one. And, I go around, and this point is two, and so forth, so that all the integers, if I wrap it around this unit wheel, all the integers lined up at the zero point here, OK? And then, we can divide this into the fractional pieces. So, that's essentially the zero point. This is the one eighth, because we are dividing into eight, two, three, four, five, six, seven.

So, if I have one times A, in this case, I'm basically saying, well, one times A, if I multiply, is basically going around to about there, five and a half I think, right, because one times A is about five and a half, OK, or five halves of 5.5 eighths, essentially.

So, it takes me about to there. That's A.

And, if I do 2^A, that continues around, and takes me up to about, where?

About, a little past three, about to there.

So, that's 2^A. OK, and 3^A takes me, then, around to somewhere like about there.

So, each time I add another A, it's taking me another A's distance around. And, the idea is that if A is, for example, odd, and it's not too close to a power of two, then what's happening is sort of throwing it into another slot on a different thing.

So, if I now go around, if I have k being very big, then k times A is going around k times.

Where does it end up? It's like spinning a wheel of fortune or something. OK, it ends somewhere.

OK, and so that's basically the notion.

That's basically the notion, that it's going to end up in some place. So, you're basically looking at, where does ka end up? Well, it sort of whirls around, and ends up at some point. OK, and so that's why that tends to be a fairly good one. But, these are only heuristic methods for hashing, because for any hash function, you can always find a set of keys that's going to make it operate badly. So, the question is, well, what do you use in practice?

OK, the second topic that I want to tie it, so, we talked about resolving collisions by chaining.

OK, there's another way of resolving collisions, which is often useful, which is resolving collisions by what's called open addressing.

OK, and the idea is, in this method, is we have no storage for links.

So, when I result by chaining, I'd need an extra linked field in each record in order to be able to do that.

Now, that's not necessarily a big overhead, but for some applications, I don't want to have to touch those records at all. OK, and for those, open addressing is a useful way to resolve collisions.

So, the idea is, with open addressing, is if I hash to a given slot, and the slot is full, OK, what I do is I just hash again with a different hash function, with my second hash function.

I check that slot. OK, if that slot is full, OK, then I hash again. And, I keep this probe sequence, which hopefully is a permutation so that I'm not going back and checking things that I've already checked until I find a place to put it. And, if I got a good probe sequence that I will hopefully, then, find a place to put it fairly quickly. OK, and then to search, I just follow the same probe sequence.

So, the idea, here, is we probe the table systematically until an empty slot is found, OK? And so, we can extend that by looking as if the sequence of hash functions were, in fact, a hash function that took two arguments: a key and a probe step. In other words, is it the zero of one our first one?

It's the second one, etc.

So, it takes two arguments. So, H is then going to map our universe of keys cross, our probe number into a slot.

So, this is the universe of keys.

This is the probe number. And, this is going to be the slot. Now, as I mentioned, the probe sequence should be permutation.

In other words, it should just be the numbers from zero to n minus one in some fairly random order.

OK, it should just be rearranged.

And the other thing about open addressing is that you don't have to worry about n chaining is that the table may actually fill up. So, you have to have that the number of elements in the table is less than or equal to the table size, the number of slots because the table may fill up.

And, if it's full, you're going to probe everywhere. You are never going to get a place to put it. And, the final thing is that in this type of scheme, deletion is difficult.

It's not impossible. There are schemes for doing deletion. But, it's basically hard because the danger is that you remove a key out of the table, and now, somebody who's doing a probe sequence who would have hit that key and gone to find his element now finds that it's an empty slot. And he says, oh, the key I am looking for probably isn't there.

OK, so you have that issue to deal with.

So, you can delete things but keep them marked, and there's all kinds of schemes that people have for doing deletion. But it's difficult.

It's messy compared to chaining, where you can just remove the element out of the chain.

So, let's do an example -- -- just so that we make sure we're on the same page.

So, we'll insert a key. k is 496.

OK, so here's my table. And, I've got some values in it, 586, 133, 204, 481, etc.

So, the table looks like that; the other places are empty.

So, on my zero step, I probe H of 496, zero. OK, and let's say that takes me to the slot where there's 204. And so, I say, oh, there's something there. I have to probe again.

So then, I probe H of 496, one.

Maybe that maps me there, and I discover, oh, there's something there. So, now, I probe H of 496, two. Maybe that takes me to there.

It's empty. So, if I'm doing a search, I report nil. If I'm doing in the insert, I put it there. And then, if I'm looking for that value, if I put it there, then when I'm looking, I go through exactly the same sequence.

I'll find these things are busy, and then, eventually, I'll come up and discover the value.

OK, and there are various heuristics that people use, as well, like keeping track of the longest probe sequence because there's no point in probing beyond the largest number of probes that need to be done globally to do an insertion. OK, so if it took me 5, 5 is the maximum number of probes I ever did for an insertion. A search never has to look more than five, OK, and so sometimes hash tables will keep that auxiliary value so that it can quit rather than continuing to probe until it doesn't find something.

OK, so, search is the same probe sequence.

And, if it's successful, it finds the record.

And, if it's unsuccessful, you find a nil.

OK, so it's pretty straightforward.

So, once again, as with just hash functions to begin with, there are a lot of ideas about how you should form a probe sequence, ways of doing this effectively.

OK, so the simplest one is called linear probing, and what you do there is you have H of k comma i.

You just make that be some H prime of k, zero plus i mod m.

Sorry, no prime there. OK, so what happens is, so, the idea here is that all you are doing on the I'th probe is, on the zero'th probe, you look at H of k zero.

On probe one, you just look at the slot after that. Probe two, you look at the slot after that. So, you're just simply, rather than sort of jumping around like this, you probe there and then just find the next one that will fit in. OK, so you just scan down mod m. So, if you hit the bottom, you go to the top. OK, so the I'th one, so that's fairly easy to do because you don't have to recomputed a full hash function each time.

All you have to do is add one each time you go because the difference between this and the previous one is just one.

OK, so you just go down. Now, the problem with that is that you get a phenomenon of clustering.

If you get a few things in a given area, then suddenly everything, everybody has to keep searching to the end of those things. OK, so that turns out not to be one of the better schemes, although it's not bad if you just need to do something quick and dirty.

So, it suffers from primary clustering, where regions of the hash table get very full. And then, anything that hashes into that region has to look through all the stuff that's there. OK, so: long runs of filled slots. OK, there's also things like quadratic clustering, where you basically make this be, instead of adding one each time, you add i each time.

OK, but probably the most effective popular scheme is what's called double hashing. And, you can do statistical studies. People have done statistical studies to show that this is a good scheme, OK, where you let H of k, i, let me do it below here because I have for them. So, H of k, i is equal to an H_1 of k plus i times H_2 of k. So, you have two hash functions on m. You have two hash functions, H_1 of k and H_2 of k. OK, so you compute the two hash functions, and what you do is you start by just using H_1 of k for the zero probe, because here, i, then, will be zero. OK.

Then, for the probe number one, OK, you just add H_2 of k.

For probe number two, you just add that hash function amount again. You just keep adding H_2 of k for each successive probe you make.

So, it's fairly easy; you compute two hash functions up front, OK, or you can delay the second one, in case. But basically, you compute two up front, and then you just keep adding the second one in. You start at the location of the first one, and keep adding the second one, mod m, to determine your probe sequences.

So, this is an excellent method.

OK, it does a fine job, and you usually pick m to be a power of two here, OK, so that you're using, usually people use this with the multiplication method, for example, so that m is a power of two, and H_2 of k you force to be odd.

OK, so we don't use and even value there, because otherwise for any particular key, you'd be skipping over.

Once again, you would have the problem that everything could be even, or everything could be odd as you're going through.

But, if you make H_2 of k odd, and m is a power of two, you are guaranteed to hit every slot.

OK, so let's analyze this scheme.

This turns out to be a pretty interesting scheme to analyze.

It's got some nice math in it. So, once again, in the worst case, hashing is lousy.

So, we're going to analyze average case.

OK, and for this, we need a little bit stronger assumption than for chaining. And, we call it the assumption of uniform hashing, which says that each key is equally likely, OK, to have any one of the m factorial permutations as its probe sequence, independent of other keys.

And, the theorem we're going to prove is that the expected number of probes is, at most, one over one minus alpha if alpha is less than one, OK, that is, if the number of keys in the table is less than number of slots. OK, so we're going to show that the number of probes is one over one minus alpha.

So, alpha is the load factor, and of course, for open addressing, we want the load factor to be less than one because if we have more keys than slots, open addressing simply doesn't work, OK, because you've got to find a place for every key in the table.

So, the proof, we'll look at an unsuccessful search, OK? So, the first thing is that one probe is always necessary. OK, so if I have n over m, sorry, if I have n items stored in m slots, what's the probability that when I do that probe I get a collision with something that's already in the table?

What's the probability that I get a collision?

Yeah? Yeah, n over m, right? So, with probability, n over m, we have a collision because my table has got n things in there. I'm hashing, at random, to one of them. OK, so, what are the odds I hit something, n over m? And then, a second probe is necessary. OK, so then, I do a second probe. And, with what probability on the second probe do I get a collision?

So, we're going to make the assumption of uniform hashing.

Each key is equally likely to have any one of the m factorial permutations as its probe sequence.

So, what is the probability that on the second probe, OK, I get a collision?

Yeah? If it's a permutation, you're not, right? Something like that.

What is it exactly? So, that's the question.

OK, so you are not going to hit the same slot because it's going to be a permutation. Yeah?

That's exactly right. n minus one over m minus one because I'm now, I've essentially eliminated that slot that I hit the first time.

And so, I have, now, and there was a key there.

So, now I'm essentially looking, at random, into the remaining n minus one slots where there are aggregately n minus one keys in those slots.

OK, everybody got that? OK, so with that probability, I get a collision. That means that I need a third probe necessary, OK?

And, we keep going on. OK, so what is it going to be the next time? Yeah, it's going to be n minus two over m minus two. So, let's note, OK, that n minus i over m minus i is less than n over m, which equals alpha, OK?

So, n minus i over m minus i is less than n over m.

And, the way you can sort of reason that is that if n is less than m, I'm subtracting a larger fraction of n when I subtract i than I am subtracting a fraction of m.

OK, so therefore, n minus i over m minus i is going to be less than n over m. OK, so, or you can do the algebra. I think it's always helpful when you do algebra to sort of think about it sort of quantitatively as well, you know, qualitatively what's going on. So, the expected number of probes is, then, going to be equal to, it's going to be equal to because we're going to need some space, well, we have one which is forced because we've got to do one probe, plus with probability n over m, I have to do another probe plus with probability of n over m minus one I have to do another probe up until I do one plus one over m minus n. OK, so each one is cascading what's happened. In the book, there is a more rigorous proof of this using indicator random variables. I'm going to give you the short version. OK, so basically, this is my first probe. With probability n over m, I had to do a second one. And, the result of that is that with probability n minus one over m minus one, I have to do another. And, with probability n over two minus m over two, I have to do another, and so forth. So, that's how many probes I'm going to end up doing. So, this is less than or equal to one plus alpha. There's one plus alpha times one plus alpha times one plus alpha, OK, just using the fact that I had here. OK, and that is less than or equal to one plus I just multiply through here.

Alpha plus alpha squared plus alpha cubed plus k.

I can just take that out to infinity.

It's going to bound this. OK, does everybody see the math there? OK, and that is just the sum, I, equals zero to infinity, alpha to the I, which is equal to one over one minus alpha using your familiar geometric series bound. OK, and there's also, in the textbook, an analysis of the successful search, which, once again, is a little bit more technical because you have to worry about what the distribution is that you happen to have in the table when you are searching for something that's already in the table.

But, it turns out it's also bounded by one over one minus alpha. So, let's just look to see what that means. So, if alpha is less than one is a constant, it implies that it takes order one probes. OK, so if alpha is a constant, it takes order one probes. OK, but it's helpful to understand what's happening with the constant.

So, for example, if the table is 50% full, so alpha is a half, what's the expected number of probes by this analysis? Two, because one over one minus a half is two. If I let the table fill up to 90%, how many probes do I need on average?

Ten. So, you can see that as you fill up the table, the cost is going dramatically, OK? And so, typically, you don't let the table get too full.

OK, you don't want to be pushing 99.9% utilization.

Oh, I got this great hash table that's got full utilization.

It's like, yeah, and it's slow.

It's really, really slow, OK, because as alpha approaches one, the time is approaching and essentially m, or n.

Good. So, next time, we are going to address head-on in what was one of the most, I think, interesting ideas in algorithms.

We are going to talk about how you solve this problem that no matter what hash function you pick, there's a bad set of keys.

OK, so next time we're going to show that there are ways of confronting that problem, very clever ways.

And we use a lot of math for it so will be a really fun lecture.

Free Downloads

Video

• iTunes U (MP4 - 164MB)
• Internet Archive (MP4 - 320MB)

Free Streaming

• VideoLectures.net

Subtitle

• English - US (SRT)