Lecture 21: Cryptography: Hash Functions

• Download this transcript - PDF (English - US)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation, or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

SRINIVAS DEVADAS: All right, let's get started. Good morning everyone. I see a lot of tired faces. I'm not tired. Why are you tired?

[LAUGHTER]

 

I only lecture half the time. You guys take the class all the time. So today's lecture is about hash functions. And you may think that you know a lot about hash functions, and you probably do.

But what we're going to do today is talk about really a completely different application of hash functions, and a new set of properties that we're going to require of hash functions that I'll elaborate on. And we're going to see a bunch of different applications to things like password protection, checking the integrity of files, auctions, and so on and so forth. So a little bit of a different lecture.

Both today and on Thursday I'm going to be going to be doing cryptography and applications, not too much of algorithms. But we will do a little bit of analysis with respect to whether properties are satisfied, in this case by hash functions or not. So let's just dive right in.

You all know what hash functions are. There's no real change in the definition. But the kinds of hash functions that we're going to be looking at today are quite different from the simple hash functions, like taking a mod with a prime number that we've looked at in the past. And the notion of collisions is going to come up again, except that again we're going to raise the stakes a little bit.

So a hash function maps arbitrary strings-- let me do this right. So you're not making a statement about the length of the string. You will break it up, even if you had a string of length 512, or maybe it was 27, you do want to get a number out of it. In a specific range there's going to be a number of bits associated with our hash functions. And previously we had a number of slots associated with the output of the hash function. But the input could be arbitrary.

And these arbitrary strings of data are going to get mapped, as I just said, to a fixed length output. And we're going to think about this fixed length as being a number of bits today, as opposed to slots in the hash table. Because we really aren't going to be storing a dictionary or a hash table in the applications we're going to look at today. It's simply a question of computing a hash.

And because the fixed length output is going to be something on the order of 160-bits, or 256-bits, there's no way that you could store two arrays to 160 elements in a hash table, or even two arrays to 64 really. And so we're going to just assume that we're computing these hashes and using them for certain applications.

I just wrote output twice I guess. So map it to a fixed length output. We want to do this in a deterministic fashion.

So once we've computed the hash of a particular arbitrary string that is given to us, we want to be able to repeat that process to get the same hash every time. We want to do this in a public fashion. So everything is public. There's no secrecy.

There's keyed hash functions that we won't actually look at today, but maybe in passing I'll mention it next time. We're not looking at keyed hash functions here. There's no secrets in any of the descriptions of algorithms or techniques I'm going to be describing today.

And we want this to be random. We want it to look random. True randomness is going to be impossible to achieve, given our other constraints. But we're going to try and approximate it with pseudo-randomness. But we'd want it to look random, because we are interested-- as we were in the case of dictionaries and the regular application of hash functions-- we are interested in minimizing collisions.

And in fact we're going to raise the stakes really high with respect to collisions. We want it to be impossible for you, or anyone else, to discover collisions. And that's going to be an important property of collision resistance that obviously is going to require randomness. And those are the three things we want, deterministic, public, and random.

And so just from a function description standpoint you have 0, 1 star here, which implies that it's an arbitrary length strength. And we want to go to 0, 1 d. And this is a string of length d. So that means that you're getting d-bits out from your hash function. And here the length is greater than or equal to 0.

So that's it. Not a lot that's new here. But a few things that are going to be a little bit different. And there's some subtleties here that we'll get to.

I want to emphasize two things, one of which I just said. There's no secrecy, no secret keys here in the hash functions that we are describing. All operations are public. So just like you had your hash function, which was k mod p, and p was a prime and p was public and known to everyone who used the dictionary, everything here we are going to be talking about is public. So anyone can compute h.

And we're going to assume that this is poly-time computation-- not too surprising-- but I'm being quite flexible here. When you look at dictionaries, and you think about using dictionaries, and using it to implement efficient algorithms, what is the assumption we kind of implicitly made-- are perhaps explicitly in some cases-- with respect to computing the hash? Anybody? Yeah?

AUDIENCE: Constant time?

SRINIVAS DEVADAS: Constant time. We assumed-- so this is not necessarily order 1, right? So that's important. So we're going to-- I want to make sure you're watching. So you're going to raise the stakes even with respect to the complexity of the hash. And as you'll see, because of the desirable properties, we're going to have to do that. We're going to ask for really a lot with respect to these hash functions.

Nobody can find a collision, right? And if you have something as simple as k mod p, it's going to be trivial to find a collision. And so these order 1 hash functions that you're familiar with aren't going to make the grade with respect to any of the properties that we'll discuss in a few minutes. All right, so remember this is poly-time computation.

And there's lots of examples of these hash functions. And for those of you who are kind of into computer security and cryptography already, you might have heard of examples like MD4 and MD5. These are versions. MD stands for message digest. These were functions that were invented by Professor Rivest.

And they had d equals 128 way back when-- 1992, if I recall-- when they were proposed. And these algorithms have since been broken in the sense that it was conjectured that they had particular properties of collision resistance that it would take exponential time for anybody to find collisions. And it still kind of takes exponential time, but 2 raised to 37 is exponential at one level, but constant in another level. So you can kind of do it in a few seconds now.

So a little bit of history. I'm not going to spend a lot of time on this. MD5 was used to create what was called a secure hash algorithm. This is 160-bits. And this is not quite broken at this point. But that people consider it broken, or soon to be broken.

Right now the recommended algorithm is called SHA-3, secure hash algorithm version three. And there was a contest that ran for like 18 months, or maybe even longer, that eventually was won by what turned into the SHA-3. And they had a different name for it that I can't recall. But it turned into SHA-3.

And what happened along the way, as we went from MD4, MD5, SHA-1 to SHA-3, is that this amount of computation that you had to do increased. And the complexity of operations that you had to do in order to compute the hash of an arbitrary string increased to the point where-- you want to think about this as 100 rounds of computation. And certainly order d computation, where d is the number of bits. And perhaps even more. So it's definitely not order 1.

So as I said a little bit of context with respect to the things that are out there. At the end of the lecture I'll give you a sense for how these hash functions are built. We're not going to spend a lot of time on creating these hash functions. It's really a research topic onto itself and not really in the slope of 6.046. What is in the scope of 6.046, and what I think is more interesting, which is what we'll focus our energy and time on, is the properties of these hash functions. And why these properties are useful in a bunch of different apps.

And so what is it that we want? We want a random oracle. We want to essentially build something that looks like that, deterministic, public, random. And we're going to claim that what we want is this random oracle which has all of these wonderful properties that I'm going to describe. I'm going to describe the random oracle to you, and then I'm going to tell you about what the properties are.

And then unfortunately this is an ideal world and we can't build this in the real world. And so we're going to have to approximate it. And that's where the MD4's and the MD5's and the SHA-1's came in, OK? So this is not achievable in practice.

So what is this oracle? This oracle is on input x, belonging to 0,1 star. So that could be an arbitrary string. If x not in the book-- so there's this the book, all right? And there's this infinite capacity book that has all of the computations that were ever done prior. And they're always stored in the book.

And that's how we're going to get determinism. Because this book initially gets filled in. All of the entries in the book are filled in using pure randomness. So you flip a coin d times to determine h of x.

So that's basically it. And you just keep flipping. You have to flip d times. And so if x was 0, you flip d times, d was 160. You flipped a coin 160 times and got a string. If x were 1, flip 160 times, you get a different string with very high probability, obviously. And so on and so forth.

But what you do is you have this book. So you're going to record x h of x in the book, OK? So at some level your hash function is this giant look-up table in the sky, right? Actually not giant, infinite capacity look-up table in the sky. Because you can put arbitrary strings into this.

And if it's in the book-- this is obviously the important part that gives you determinism-- then you return y, where x and y are in the book, OK? So you get a random answer every time, except as required for consistency with previous answers. So the very first time you see a string, or-- and the whole world can create this book. It's public.

So if I created the book at first with a particular string, let's say Eric. I was the string. And I'm the one who put the entry-- x equals Eric, and h of x, h of Eric equals some random 160-bit string-- into the book, I get credit for that, right? But if you come a nanosecond later and ask for h of Eric, you should get exactly what got put into the book when I asked for h of Eric. And so on and so forth.

So this is true for everybody. So this is like-- I mean basically impossible to get. Because not only can anybody and everybody query, you have to have this ordering associated with people querying the book. And you have to have consistency. All right. So everyone convinced that we can't build this? All right.

If you took anything out of this lecture, that's what you should take. No, no. There's a lot more.

So we want to approximate the random oracle. And we're going to get to that. Obviously we're going to have to do this in poly-space as well.

So what's wrong with this? Of course this picture is I didn't actually say this, but you'd like things to be poly-time in terms of space. You don't want to store an infinite number-- this is worse than poly-time, worse than exponential time, because it's arbitrary strings that we're talking about here, right? So you can't possibly do that. So we have to do something better.

But before I get into how we'd actually build this, and give you a sense of how SHA-1 and MD5 were built-- and that's going to come a little bit later-- I want to spend a lot of time on the what is interesting, which are the desirable properties. Which you can kind of see using the random oracle. So what is cool about the random oracle is that it's a simple algorithm. You can understand it. You can't implement it. But now you can see what wonderful properties it gives you. And these properties are going to be important for our applications, OK?

And so let's get started with a bunch of different properties. And these are all properties that are going to be useful for verification or computer security applications. The first one, it's not ow, it's O, W. It's one-wayness, all right? So one-way, or one-wayness. And it's also called-- you're not going to call it this-- but perhaps this is a more technical term, a more precise term, pre-image resistance.

And so what does this mean? Well this is a very strong requirement. I mean a couple of other ones are also going to be perhaps stronger. But this is a pretty strong requirement which says it's infeasible, given y, which is in the-- it's basically a d-bit vector, to find any x such that h of x equals y. And so this is x is the pre-image of y.

So what does this say? It says that I want to create a hash function such that if I give you a specific-- we call it a 160-bit string, because we're talking SHA-1 here, and that's the hash-- I'm going to have, it's going to have to be impossible for me to discover an x that produced that 160-bit string, OK? Now if you go look at our random oracle, you realize that if you had a 160-bit string, and perhaps you have the entire book and you can read the entire book.

It's an infinite capacity book. It's got a bunch of stuff in it. And know that any time anyone queried the book the first time for a given x, that there was this random 160-bit number that was generated and put into the book. And there's a whole lot of these numbers, right?

So what's going to happen is, you're going to have to look through the entire book, this entire potentially infinite capacity book, in order to figure out if this particular y is in the book or not. And that's going to take a long time to do, potentially, OK? So in the case where you have a random oracle you'd have to go through and find-- looking at the output hash corresponding to each of the entries in the random oracle, you're going to start matching, match, match, match, match, it's going to take you exponential time. Well actually worse than that, given the infinite capacity of the book.

So this clearly gives you that. Now you may not be a completely satisfied with that answer because you say well, you can't implement that. But we'll talk a little bit, as I said, about how you could actually get this. But what's-- I should be clear-- is that the simple hash functions that we've looked at in the past just to build dictionaries do not satisfy this, right?

So suppose I had h of x equals x square mod p. Is this one-way, given a public p? No of course not, right? Because I'm going to be-- it's going to be easy for me to do something. Even though this is discrete arithmetic I could do something like, well, I know that what I have here-- actually let's do it with something that's simpler, and then I'll talk about the x squared.

If I had something as simple as x mod p, I mean that's trivially broken in terms of one-wayness. Because I know that h of x could be viewed as the remainder. So anything-- if this is h of x, and let's just call that y for a second, because that's what we had it out there. Something that's a multiple of y plus the remainder-- so I could have a-- is that right? Is that what I want? Yeah.

No, plus y. So I want a of-- well since I can't figure it out, why can't you? What do I need to put in there in order to discover an x that would produce a y? Can I write an equation? Yeah?

AUDIENCE: Could you just write y itself?

SRINIVAS DEVADAS: Just y itself. That's right. Good point. Just y itself in this case. Good. I knew you guys were smarter than me. This proves it.

So if you just take y-- and y remember is going to be something that's 0 to p minus 1, right? And that's it. It just goes through, right? So that's a trivial example, right?

Now if I put x squared in here, obviously it's not y, but I could start looking at-- what I have here is I'm going to get y that looks like x squared. But I could take the y that I have, take the square root of that, and then start looking for x's that give me the y that I have. Actually it's not a complicated process to try and figure out, through trial and error potentially, what an x is that produces a particular y for the kinds of hash functions that we've looked at, all right?

Now as you complicate this equation it gets harder. Because you have to invert this set of equations. And that's what the game is going to be when you go create one-way hash functions. The amount of computation that you do in order to compute the y is going to increase to the point where, as I mentioned, you have 80, 100 rounds of computation, things getting mixed in.

And the hope is that you create this circuit, if you will, that has all this computation in that. Going forwards is easy, because you've specified the multiplications and the mods and so on and so forth. But not all of these operations have simple inverses. And going backwards, which is what you need to do in order to break one-wayness, or discover the x given a y, is going to be harder and harder as the computations get more complex, OK?

So everyone have a sense of what one-wayness is? So that's one-wayness. There's four other properties, two of which are very related.

CR and TCR. So CR is collision resistance. It's infeasible to find x and x prime such that x not equal to x prime, and h of x equals h of x prime, which is of course a collision. OK? And that just says you have this crazy hash function where you can't discover collisions. Well it would be absolutely wonderful. In fact that's what we wanted when we built dictionaries.

But why don't we use SHA-3 in dictionaries? Why don't we use SHA-3 in dictionaries? Yeah?

AUDIENCE: Because it's more complicated than we need.

SRINIVAS DEVADAS: Yeah, it's horribly slow, right? It would take longer to compute the hash than access the dictionary, when you actually had a reasonable dictionary that maybe had some collisions. I mean you just go off and you have a linked list, you can afford a few collisions, what's the big deal, right? So it just doesn't make any sense to use this level of heavyweight hash function, even if it satisfies collision resistance-- which some of these are conjectured to do-- for the applications we've looked at.

But there'll be other apps where collision resistance is going to be important. So that's collision resistance. And then there's-- TCR is target collision resistance. It's a weaker form-- so sometimes people CR strong collision resistance, and TCR weak occlusion resistance. We'll use CR and TCR here.

And this says it's infeasible, given x-- so there's a specific x that you want to find a collision for, as opposed to just finding a pair that goes once to x and x prime. And any pair would suffice to break the collision resistance property. But TCR says is I'm going to give you a specific x. And I want you to find an x prime who's hash collides with the hash of x, OK? That's TCR.

OK that's TCR for you. And that just to be clear, I think you probably all got this, obviously we want this here because we have a deterministic hash function. And it's a trivial thing to say that if you had x, and you had x again, that you get the same hash back from it. That's a requirement, really. So we want two distinct x and x primes that are not equal that end up colliding. That's really what a collision is.

And so you see the difference between CR and TCR? Yup? Yeah?

AUDIENCE: Are we to assume that given an x it's very easy to get the h of x back?

SRINIVAS DEVADAS: So the question was, given an x, it's poly-time computation to get h of x. Absolutely. Public poly-time computation given an x to get h of x. So going this way is easy. Going this way-- I ran out of room-- hard. OK?

AUDIENCE: So does that mean that TCR is basically the same as 1?

SRINIVAS DEVADAS: No, no, no, absolutely not. TCR says it's OK. You can compute this. You can get x. And you can get h of x. So given x, you know that you can get h of x.

I didn't actually put that in the definition. And maybe I should have. So given x you can always get h of x. Remember that. It's easy to get h of x. So any time I say given x, you can always add it, saying given x and h of x.

So I'm given x. I'm given h of x. I obviously need to map-- I need to discover an x prime such that h of x prime equals h of x, OK? Now you have situations where for-- it may be the case that for particular x's you can actually do this. And that's enough to break TCR. So you have to have this strong property that you really don't want to find collisions are for some-- even if there's a constant fraction of x's that break the TCR property, you don't like your hash function, OK? Because you might end up picking those and go build security applications using those properties.

I want to talk a little bit about the relationship between OW, CR, and TCR. So I'm going to get back to that. And we're going to talking about hash functions that satisfy one property but don't satisfy the other. And I think your question will probably be answered better, OK? Thanks for the question.

So those are the main ones. And really quickly, if you want to spend a lot of time on this-- but I do want to put up-- I think I'll leave these properties up here for the duration. Because it's important for you to see these definitions as we look at the applications where we require these properties, or a subset of these properties.

But that we have pseudo randomness. And this is simply a function of the fact that-- so this is PRF-- we know we can't build a random oracle. And so we're going to have to do something that's pseudo-random. And basically what we're saying here is the behavior is indistinguishable from random.

So we're going to have to use non-linearity, things that are called non-linear feedback shift registers, to create pseudo-random functions. There's many ways that we can create pseudo-random functions. We won't really get into that. But obviously that's what we want.

And then the last one is a bit tricky. And we will have an app that requires this way at the end. But this is infeasible given h of x to produce h of x prime, where x and x prime are-- and it gets a little bit fuzzy here-- are related in some fashion, right? So a concrete example of this is, let's say that x prime is x plus 1. So this is a reasonable example of this.

So what this says is you're just given h of x. It doesn't actually say anything about one-wayness yet. But you could assume, for example, that if this was a one-way hash function, that it would be possible to get x from h of x, correct? And let's keep that though. Hold that thought, all right? We're going to get back to it.

So if I'm just given the hash through some computation, it may be possible for me to create another hash, h of x prime, such that there's some relationship that I can prove or argue for between the strings that created the hashes, namely x and x prime, OK? That's what malleability is, right? Now you might just go off and say here's an x, here's a y, here's h of x, and here's h of y. These look completely random. And you might go off-- I'm being facetious here-- I say that y is x's third cousin's roommate's brother-in-law or something, right? I mean just make something up, right?

So clearly there's got to be a strong, precise relationship between x and y. If in fact you could do this and get y equals x plus 1, that'd be a problem, right? But if you are-- and then you can do this sort of consistently for different x's and y's, that would absolutely be a problem, right? But what you're really asking for-- and typically when you want non-malleability-- it's things where you have auctions, for example, where you are to be careful about making sure that you don't want to expose your bid. And so maybe what you're doing is exposing h of x.

You don't want somebody to look at your h of x and figure out how they could beat your bid by just a little bit. Or in case of Vickrey auctions, where the second highest bidder wins, now just be a little bit below you, right? So that's the kind of thing that you want to think about when it comes to non-malleability, or malleability, where you want a strong relationship between two strings that are related in some ordered fashion, like x equals-- x prime equals x plus 1, or just x prime equals 2 times x.

And you don't want to be able to-- you don't want the adversary to be able to discover these new strings. Because that would be the system, all right? So any questions about properties? Are we all good on these properties? All right, because I'm going to start asking you how to use them for particular applications, or what properties are required for certain applications, OK?

One last thing before we get there. I promised a slightly more detailed analysis of the relationships between these properties. So let's do that. Now if your just look at it, eyeball it, and you look at collision resistance and TCR, what can I say about the relationship between CR and TCR? If h is CR, it's going to be TCR, right? It's got to be TCR. It's a strictly stronger requirement.

But not reverse. And you can actually give a concrete example of a particular hash function that is TCR. I'm not going to go there. It's actually a little more involved than you might think it is, where a TCR hash function is not collision resistant. But you can see that examples such as these should exist, simply because I have a more stringent property corresponding to collision resistance as opposed to TCR, right?

So if you're interested in that particular example, you're not responsible for it, get in touch with me and I'll point you to a, like a three-page description of an example. So I didn't really want to go in there. But what I do want to do is talk about one-wayness and collision resistance. Because I think that's actually much more interesting, all right?

So if h is one-way-- any conjectures as to what the question mark is in the middle? Can I make strong statements about the collision resistance of a hash function, if I'm guaranteed that the hash function I have is a one-way hash function, or vice versa? Another way of putting it is, can you give me an example of, just to start with, a hash function which is one-way but not TCR, not target collision resistant?

So I'm going to try and extract this out of you. This is somewhat subtle. But the way you want to think about this is, let's say that h of x is OW and TCR, OK? And so I have a bunch of inputs. And this is the output. And I get d-bits out. And I've got x1, x2, to xn, OK?

Now I've given this h-- I've been given this h which is one-way and TCR. It satisfies those properties that you have up there. In the case of one-way, I give you an arbitrary d-bit string. You can't go backwards and find a bunch of the xi's that produce exactly that d-bit string, all right? So it's going to be hard to get here.

But you're allowed now to give me an example. So this is some hash function that you can create, which may use h as well. And h is kind of nice because it has this one-way property. So let's say that we want to discover something where one-way does not imply TCR. So I want to cook up a hash function h prime such that h prime is one-way, but it's not TCR, OK?

The way you want to think about this is you want to add to h. And you want to add something to h such that it's still hard-- if you add h it's still hard to go from here to there. Because you've got to go deeper. If you add to, for example, the inputs of h. Or you could add to the outputs of h as well, or the outputs of the current h.

But you can basically go deeper, or need to go deeper in order to find the break one-wayness, in order to find an x, whatever you have, that produces the d-bit string that you have, right? So what's a simple way of creating an h prime such that it's going to be pretty easy to find targeted collisions even, not necessarily collisions, it's pretty easy to find targeted collisions, without breaking the one-way property of h? Yeah?

AUDIENCE: So if you have x sub i, if i odd then return h of x of i. So that's minus 1. So return the even group.

SRINIVAS DEVADAS: Sure. Yep.

AUDIENCE: Given x any x of i, you can usually find another x of i that was the same output? You can go backwards.

SRINIVAS DEVADAS: You can't go backwards. Yeah, that's good. That's good. I'm going to do something that's almost exactly what you said. But I'm going to draw it pictorially.

And what you can do, you can do a parity, like odd and even that was just described. And all I'll do is add a little [? XNOR ?] gate, which is a parity gate, to one of the inputs. So you have and b here. So I've taken x1, and I have a and b here. So I've added-- I can add as many inputs as I want to this function.

Oh I should mention by the way, h of x is working on arbitrary strings. And obviously I put in some number here that corresponds to n, which is a fixed number. So you might ask, what the heck happened here with respect to arbitrary strings? And there's two answers.

The first answer is, well, ignore arbitrary. And assume that you only have n-bit strings. And n this is really large number, right? And that may not be particularly satisfying.

The other answer is, which is more practical, which is what's used in practice, is that typically what happens is, you do have particular implementations of hash functions that obviously need to have fixed inputs, n, for example. And n is typically 512. It's usually the block size. And you chunk the input up into five 12-bit blocks.

And typically what you do is, you take the first five 12-bits, compute the hash for it. And then you can do it for the remaining blocks. And then you can hash all of them together, all right? So there's typically more invocations. I don't really want to get into it. But there's typically more invocations of h when the input would be 2 times n, or 3 times n, all right?

So we don't really need to go there for the purposes of this lecture. But keep that in mind. So we'll still stick with our arbitrary string requirement.

So having said that, take a look at this picture. And see what this picture implies. I have an h prime that I've constructed, right? Now if I look at h prime, and I give you an output for h prime-- so h prime now has, it's a function of a and b, and x2 all the way to xn, right? So it's got an extra input.

If I look at h prime, and I look at the output of h prime that is given to me, and I need to discover something that produces that, it is pretty clear that I need to figure out what these values are, all right? And I need to know what the parity of a and b is. And maybe I don't need to know exactly what a and b are, but I absolutely need to know what the parity of a and b are, because that's x1. And the one-way I'd break would require me to tell you what the value of x1 is, and the value of x2, and so on and so forth.

So it's pretty clear that h prime is one-way, right? Everybody buy that? h prime is one-way.

But you know what? I've got target collisions galore, right? All I have to do is flip-- I have a equals 1 and b equals 1. And I have a equals 0 and b equals 0. They're going to give me the same hash, right?

So trivial example, but that gets to the essence of the difference between collision resistance and one-wayness, target collision resistance and one-wayness, all right? So this is one-way but not TCR, simply because a equals 0, b equals 0 for arbitrary x's produce the same thing as a equals 1 and b equals 1, right? So those are collisions.

So admittedly contrived. But it's a counterexample. Counterexamples can be contrived. It's OK.

All right. So that was what happens with that. Let's look at one more interesting thing that corresponds to the other way, right? So what I want to show is that a TCR does not imply one-wayness. OK, so now I want an example where it is clear that I have target collision resistance, because I can just assume that. And we're going to use the same strategy.

I'm just going assume that I have an h that's target collision resistant. And I'm going to try and cook up an h prime that is not one-way. So I'm going to assume that in fact h is TCR and OW.

And I'm going to take away one of the properties. And if I take it one of the properties I have a counterexample, right? So think about how you could do this.

You have h as before. And I want to add some stuff around it such that it's going to be easy to discover-- for a large, for a constant fraction of hashes that I've given to me, not for any old hash. Because you can always claim that one-wayness is broken by saying I have x, I computed h of x, now I know what-- given h of x I know what x is. I mean you can't do that, right? So that's not breaking the one-wayness of it.

It's when you have an h of x and this is the first time you've seen it, you're trying to find what x is, right? So how would you-- how would you set it up so you break the one-wayness of h without necessarily breaking the target collision resistance of the overall hash function that you're creating? And you have to do something with the outputs, OK? You have to do something.

This is a little more involved. It's not as easy as this example. It's a little more involved. But any ideas? Yeah, go ahead.

AUDIENCE: So x is less than b returns x. If x is greater than b, return [INAUDIBLE].

SRINIVAS DEVADAS: Beautiful. Right. What color did you get last time?

AUDIENCE: Blue.

SRINIVAS DEVADAS: You got a blue last time? All right. Well you get a purple. You have a set. Actually we have these red ones that are precious, that are-- no, we don't. We chose not to do red. I don't know. There was some subliminal message I think with throwing red Frisbees that we didn't like.

But OK. So thank you. And h of x is simply something where I'm going to concatenate a zero to the x value and just put it out. And clearly this is breaking one-wayness because I'm just taking the input, I'm adding a zero to it, and shipping it out. So it's going to be easy to go backwards, right?

And this only happens if x is less than n, as the gentleman just said. Less than or equal to n in terms of the input length, OK? Otherwise I'm going to do h of x. So this is good news. Because I'm actually using the hash function in the case where I have a longer input string. This is bad news for one-wayness because I'm just piping out the input.

And so if I get an x, and I see what the x is out here, and let's just say for argument's sake that-- you could even say that n is going to be something that is less than d, which is the final output, which has d-bits. And so if you see something that h prime produces that's less than d-bits you instantly know that you can go backwards and discover what input produced that for the h prime, right? Because you just go off and you go backwards. This is what it tells you.

Now on the other hand if it's larger obviously you can't do that. But there's a whole lot of combinations that you can do that for. So this breaks one-wayness, OK?

Now you think about TCR. And what you want a show of course is that this maintains TCR. So that's the last thing that we have to show. We know that it breaks one-wayness. But if it broke TCR we don't quite have our example. So we want to show that it actually maintains TCR, which is kind of a weakish property that we need to maintain.

And the reason this maintains TCR is that there's really only two cases here obviously, corresponding to the if statement. And it's pretty clear that if x is less than or equal to n, clearly different x's produce different h prime x's, correct? Because I'm just passing along the x out to the output.

So if x is less than n I am going to get different hashes at the output. I'm just passing them out. So that's easy.

And for the other case, well I assume that h of x was CCR, correct? Because that was the original assumption, that I had h, which was CCR. So in both cases TCR is maintained because else h of x maintains TCR, all right? So again, a bit of a contrived example to show you the difference between these different properties so you know not to mix them up. You know what you want to ask for, what is required when you actually implement an application that depends on particular properties.

All right? Any questions so far about properties or any of these examples? We're going to dive in to using them.

OK. So start thinking computer security. Start thinking hackers, protecting yourself against the bad guys that are out there who are trying to discover your passwords, trying to corrupt your files, generally make your life miserable. And we'll start out with fairly simple examples, where the properties are somewhat obvious, and graduate to this auction bidding example which should be sort of the culmination of at least this part of the lecture.

And depending on how much time I have I'll tell you a little bit about how to implement hash functions. But I think these things are more important from a standpoint of giving you a sense of cryptographic hashes. All right. Password storage.

How many of you write your password in an unencrypted text file and store it in a readable location? There you go, man. Thank you for being honest. And I do worse. Not only do I do that, I use my first daughter's name for four passwords. I won't tell you what the name is.

So that's something that we'd like to fix, right? So what do real systems do? Real systems cannot protect against me using my first daughter's name as a password, right? So there's no way you can protect against that.

But if I had a reasonable password, which had reasonable entropy in it-- so let's assume here that we have reasonable entropy in the password. And you can just say 128-bits. And it's not a lot, right? 128-bits is 16 characters, OK? And you don't have to answer this-- how many of you have 16 characters in your password?

Oh I'm impressed. OK. So you've got 128-bits of entropy. But the rest of you, forget it. This is not going to help you, OK? But what I want, assuming you have significant entropy in your password-- because otherwise, if there's not enough entropy you can just enumerate all possible passwords of eight letters. And it's not that much. It's 2 raised to 50, what have you. And you can just go off. And none of these properties matter.

You just-- you have your h of x. It's public. We'll talk about how we use that in a second. But clearly if the domain is small you can just enumerate the domain. So keep that in mind.

I talked about h of x, and it's obviously going to be relevant here. But suppose I wanted to build a system, and this is how systems are built, ETC slash password file, assuming you have long passwords it does it this way, otherwise it needs something that's called a salt. But that's 6, 8, 57 and we won't go there. So we just assume a large entropy.

What is it that a system can do? What can it store in order to let you in, and only let you in when you type your password, and not let some bogus password into the system? Or somebody with a bogus password into the system. Yeah, go ahead.

AUDIENCE: If you capture the password when you enter it and compare it to what's stored--

SRINIVAS DEVADAS: Yes.

AUDIENCE: If it's a one-way hash you know you have what the correct password is.

SRINIVAS DEVADAS: That's exactly right. That's exactly right. So it's a really simple idea, a very powerful idea. It, as I said, assumed that the entropy-- and I'm belaboring the obvious now-- but it is important when you talk about security to state your assumptions. But you do not store password on your computer. And you store the hash of the password.

Now why do I store my password on the computer? Because this is so inconvenient, right? So this is what the system does for me. But the fact of the matter is, if I lose my password, this doesn't help me. Because what the system wants you to do is choose a password that is long enough, and the h is one-way. So anybody who discovers h of PW that is publicly readable cannot discover PW, all right? That's what's cool about this.

How do you let the person log in? Use h of PW to compare against h of PW prime, which is what is entered, where PW prime is the typed password.

And clearly what we need is the disclosure of h of PW should not reveal PW. So we definitely need one-wayness. What about-- what about collision resistance? Our target collision resistance?

Think practitioner now, right? Are we interested in this hash function being collision resistant? What does that mean in this case? Give me the context in this particular application? Yeah, go ahead.

AUDIENCE: It means that someone entering a different password will have the same hash [INAUDIBLE].

SRINIVAS DEVADAS: Exactly. So it means that what you have is a situation where you do not reveal-- and so what might happen is that h of PW prime equals h of PW. But h of PW equals h of PW prime. But PW is not equal to PW prime. What you have is a false positive.

Someone who didn't know your password but guessed right-- and this is a 128-bit value, and they guessed right-- is going to get it. You don't particularly care of the probability of this occurrence. It's really small. Typically you're going to have systems that lock you out if you try 10 tries that occurs one, two, wrong passwords, right?

So really in systems you do not require-- you do want to build systems that have minimal properties with respect to the perimeters that are used. So from a system building standpoint just require OW. Don't go overboard. Don't require collision resistance or TCR, OK?

Let's do a slightly different example. Also a bit of a warm-up for what's coming next, which is a file modification detector. So for each file F, I'm going to store h of F. And as securely. So you assume that this means that h of F cannot be modified by anybody, h of F itself.

And now we want to check if F is modified by re-computing h of F. Which could be, this could be modified. So this could actually be F prime. You don't know that.

You have a file. It's a gigabyte. And somebody might have tampered with one of the bits in the file. All you have is a d-bit digest that corresponds to h of F that you stored in a secure location. And you want to check to see, by re-computing h of F, the file that is given to you, and comparing it with what you've stored, the h of F that you've stored.

And so what property do we need in order to pull this off? Of hash functions. What precisely do we need to pull this off? What is the adversary trying to do? And what is a successful break?

A successful break is if an adversary can modify the file and keep h of F the same, right? That would be a successful break, right? Yup. Go ahead.

AUDIENCE: TCR.

SRINIVAS DEVADAS: TCR? Yeah, absolutely. You need TCR. So you want to modify the file. So you're given that the file-- the adversary is given the file, which is the input to the hash, and is going to try and modify-- modify the file, right?

So let's do a couple more. And we're going to advance our requirements here a little bit. So those two are basic properties. I want to leave this up there.

We're going to do something that corresponds to digital signatures. So digital signatures are this wonderful invention that came out of MIT in a computer science laboratory-- again, Ron Rivest and collaborators-- which are a way of digitally signing a document using a secret key, a private key. But anybody who has access to a public key, so it could be pretty much anybody, could verify the authenticity of that signature, right? So that's what a digital signature is.

So we're going to talk about public cryptography on Thursday, in terms of how you could build systems or encryption algorithms that are public key algorithms. But here I'll just tell you what we want out of them. Essentially what we have here in the case of signatures, we actually want to talk about encryption here, are-- there's two keys associated with a public key system.

Anybody and everybody in the system would have a public key that you can put on your website. And you also have a secret key-- that's like your password-- that you don't want to write down, you don't want to give away, because that's effectively your identity. And what digital signatures respond to are that you have two operations. You have signing and verification.

So signing means that you create a signature sigma that is the sign using your private key, your secret key, off a message M. So you're saying this is this message, it came from me, right? That's what signing means. You have this long message and you sign it at the bottom. You're taking responsibility for the contents of that message.

And then verification is you have M sigma and a public key. And this is simply going to output true or false. And so the public key should not reveal any information about the secret key. And that's the challenge of building PKI systems, that we'll talk about in some detail next time. But we don't need to think about that other than acknowledging it today.

So the public and private key are two distinct things, neither one of which reveals anything about the other. Think of them as completely distinct passwords. But they happen to be mathematically related. That's why this whole thing works. And that mathematical relationship we'll look at in some detail on Thursday.

But having said that, take a look at what this app is doing for us, right? This is a security application. And I haven't quite gotten to hash functions yet. But I'll get to it in just a minute.

But what I want to do is emphasize that there's two operations going on. One of which is a signature, which is a private signature, in the sense that it's private to me, if I'm Alice. Or private to Alice. And you're using secret information on this public message, M, because that's going to be publicized. And you're going to sign the public message.

And then anybody in the world who has access to Alice's public key is going to be able to say, oh I'm looking at the signature, which is a bunch of bits. I'm looking at the message, which is a whole lot of bits. And I have this public key, which is a bunch of bits. And I'm going to be able to tell for sure that either Alice signed this message, or Alice did not sign this message.

And the assumption here is that Alice kept her private key secret. And of course, what I just wrote there, that the public key does not reveal anything about the secret key, OK? So that's digital signatures for you, in a nutshell. And when you do MIT certificates you're using digital signatures a la Rivest-Shamir-Adleman, the RSA algorithm. So you're using this all the time, when you click on 6.046 links, for example.

And what happens is M is typically really large. I mean it could be a file, right? It could be a large file. And you don't necessarily want to compute these operations on large files. So for convenience, what happens is you end up hashing the file. And for large M it's easier to sign h of M. And so replace the M's that you see here with h of M, all right?

So now that we're given that we're going to be doing h of M in here, think about what we wanted to accomplish with M, right? I told you what we wanted to accomplish with M. There's a particular message. I'm Alice. I'm going to keep my secret key secret. But I want to commit to signing this message M, all right? And I want to make sure that nobody can pretend to be me who doesn't know my secret key. And nobody does.

So if I'm going to be signing the hash of the message, now it comes down to today's lecture. I'm signing the hash of the message h of M. What property do I require of h in order for this whole thing to work out? Yeah, go ahead.

AUDIENCE: Is it non-malleability?

SRINIVAS DEVADAS: Non-malleability, but even before that-- suppose-- absolutely, but non-malleability is kind of beyond one of these properties over on the right. You're on the right track, right? So do you want to give me a different answer? You can give me a different answer.

AUDIENCE: Oh, I'm not sure.

SRINIVAS DEVADAS: OK. What? Yeah, back there.

AUDIENCE: I think you wanted to one-way because otherwise you could take that signature and find another message that you could credit.

SRINIVAS DEVADAS: I can make M public. I can make M-- M can be public. And h of M is public. So one-wayness is not interesting for this example if M is public. And we can assume that M eventually gets public. Because that's the message I'm signing, right? I can also put M out.

So I want the relationship-- I want you to focus on the relationship between h of M and M and tell me what would break this system. And you're on the right track. Yeah, go ahead. Or way back there. Yeah, sorry about that.

AUDIENCE: TCR.

SRINIVAS DEVADAS: TCR. Why TCR?

AUDIENCE: [INAUDIBLE].

SRINIVAS DEVADAS: So I have M. So what happens here-- I should write this out. I'm given-- as an adversary I have M and h of M. It is bad if Alice signs h of M, but Bob claims Alice signed M prime. Because h of M equals h of M prime, right? That is bad.

So the M is public-- could you stand up? M is given. There's a specific M, and a specific h of M in particular, that has been exposed. And h of M is what was used for the signature. So you want to keep h of M the same. It's a specific one. So it's not collision resistance, it's target collision resistance, because that's given to you.

And you want to keep that the same. But you want to claim that oh, you promised me $10,000, not $20, right? If you can do that, you signed saying you want to pay $10,000, not $20, then you've got a problem.

So your thing is very close. It's just that it doesn't need to be a strong relationship between the 10,000 or the 20. I mean I give you a concrete example of that. But it could be more, it could be less. Anything that is different from what you signed, be it with the numerical relationship or not, would cause a problem and break this scheme, all right? Are we good?

All right, one last example, the most interesting one. And as I guessed I'm probably not going to get to saying very much about how cache functions are implemented. But maybe I'll spend a minute or two on it. So let's do this example that has to do with commitments.

Commitment is important, right? You want to commit to doing things. You want to keep your promises. And in this case we have a legal requirement that you want to be able to make people honor their commitments, and not weasel their way out of commitments, right? And we want to deal with this computationally.

And let's think about auctions. So Alice has value x, e.g. an auction bid. Alice computes what we're going to call C of x, which is a commitment of x, and cements it, right? C of x, C of x is-- let's assume that the auctioneer, and perhaps other auctionees as well, see C of x. You have to submit it to somebody, right? So you can assume that that's exposed.

And what is going to happen is, when bidding is over Alice is going to open-- so this is-- C of x can be thought of as sealing the bid. So that's the commitment. You're sealing the-- you're making a bid and you're sealing it in an envelope. You've committed to that. That's obviously, what happens in real life without cryptography, but we want to do this with cryptography, with hash functions.

And so now Alice opens C of x to reveal x. So she has to prove that in fact x was her bid. And that it matches what she sealed. When you open it up, think about it conceptually from a standpoint of what happens with paper, and then we have to think about this computationally and what this implies, right?

So again I'll do a little bit of set up. And then we have start talking about the properties that we want for this particular application. So there are a bunch of people who are doing bidding for this auction. I don't-- I want to be the first-- I don't want to spend a lot of money. But I want to win. All of us are like that, right?

If I know information about your bid, that is obviously a tremendous advantage. So clearly that can't happen, right? If I know one other person's bid I just do plus 1 on that. If I know everybody else's I just do plus 1 on the maximum. So clearly there's some secrecy that's required here, correct?

So C of x is going to have to do two things. It can't reveal x. Because then even maybe the auctioneer is bad. Or other people are looking at this. And you can just assume that C of x is-- the C of x's are all public.

But I also need a constraint that's associated with C of x that corresponds to making sure Alice is honest, correct? So I need to make Alice commit to something, right? So what are the different properties of the hash function that if I use h of x here, that I'd want h to satisfy in order for this whole process to work like it's supposed to work with paper and envelopes? Yeah, go ahead.

AUDIENCE: It has to be one-way [INAUDIBLE].

SRINIVAS DEVADAS: It has to be one-way. And explain to me-- so I want a description of it has to be one-way, because why?

AUDIENCE: Because you want all the c x's to be hidden from all the other options.

SRINIVAS DEVADAS: Right. C of x should not reveal x, all right? All right. That's good. Do you have more? It has to be collision resistant. OK. I guess.

A little bit more. You're getting there. What-- why is it collision resistant?

AUDIENCE: Because you want to make sure that Alice, when she makes a bid that she commits that bid. If she's not going to resist it then she could bid $100 and then find something else.

SRINIVAS DEVADAS: That's exactly right. So CR, because Alice should not be able to open this in multiple ways, right? And in this case it's not TCR in the sense that Alice controls what her bids are. And so she might find a pair of bids that collide, correct? She might realize that in this particular hash function, you know $10,000 and a billion dollars collide, right?

And so she figures depending on what happens, she's a billionaire, let's assume. She's going to open the right thing. She's a billionaire, but she doesn't necessarily want to spend the billion, OK? So that's that, right? But I want more. Go ahead.

AUDIENCE: You don't want it to be malleable. Assuming that the auctioneer is not honest because you don't want to accept a bribe from someone and then change everyone else's bid to square root of whatever they bid.

SRINIVAS DEVADAS: That's exactly right. Or plus 1, which is a great example, right? So there you go. I ran out of Frisbees. You can get one next time.

So yeah, I don't need this anymore. You're exactly right. There's another-- it turns out it's even more subtle than what you just described. And I think I might be able to point that out to you. But let me just first describe this answer, which gives us non-malleability.

So the claim is that you also want non-malleability in your hash function. And the simple reason is, given C of x-- and let's assume that this is public. It's certainly public to the auctioneer, and it could be public to the other bidders as well.

Because the notion of sealing is that you've sealed it using C of x. But people can see the outside of the envelope, which is C of x. So everyone can see C of x. You still want this to work, even though all other bidders can see C of x.

So given C of x, should not be possible to produce C of x plus 1. You don't know x is. But if you can produce C of x plus 1, you win, all right? And so that's the problem.

Now it turns out you now say OK, am I done? I want these three properties. And I'm done, right? There's a little subtlety here which these properties don't capture. So that's why there's more here. And I don't mean to titillate, because I'll tell you what is missing here.

But let's say that I have a hash function that looks like this. And this here is non-malleable. It is collision resistant. And it's one-way, all right? So h of x has all these wonderful properties, all right?

I'm creating an h prime x that looks like this, which is a concatenation of h of x, and giving away the most significant bit of x, which is my bid, right? I'm just giving that away, right? The problem here is that we haven't really made our properties broad enough to solve this particular application to the extent that there's contrived cases where these properties aren't enough, OK?

And the reason is simple. h prime x is arguably NM, CR, and OW. And I won't go into to each of those arguments. But you can think about it, right?

If I'm just giving you one bit, there's 159 others, there's a couple of hundred others, whatever it is that I have in the domain. It's not going to be invertible. h prime x is not going to be invertible if h of x is not invertible. h prime x is not going to be breakable in terms of collision resistance if h of x is not breakable, and so on and so forth.

But if I had a hash function like that, is it a good hash function for my commitment application? No, obviously not. Because if I publicize this hash function-- remember, everything is public here with respect to h and h prime-- you are giving away the most significant that corresponds to your bid in this particular hash function, right?

So you really need a little bit more than these for secrecy, for true secrecy. But in the context of this example, I mean it's common sense that you would not use the hash function like that, right? So it's not that there's anything profound here. It's just that I want to make sure that you understand the nuances of the properties that we're requiring.

We had all the requirements corresponding to the definitions of NM and CR and OW. And you need a little bit more for this example, where you have to say something, perhaps informally, like the bits of your auction are scrambled in the final hash output, which most hash functions should do anyway, and h of x will definitely do. But you kind of unscrambled it by adding this little thing in here, corresponding to the most significant thing, all right?

So I'll stop with that. Let me just say that the operation-- or sorry, the work involved in creating hash functions that are poly-time computable is research work. People put up hash functions and they get broken, like MD4 was put up in '92 and then got broken, SHA-1 and so on and so forth. And so I just encourage you to look up SHA-3 and just take a quick scan and what the complexity of SHA-3 is with respect to computing the hash given an arbitrary string, all right?

I'll stick around for questions.