Lecture 15: Nondeterministic Programming

• Download this transcript - PDF (English - US)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high-quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

Let's get going here. So this is a lecture that's actually appropriate for Halloween, because it's a scary topic. Non-deterministic programming. So we've been looking mostly at deterministic programs. So a program is deterministic on a given input if every memory location is updated with the same sequence of values in every execution.

So if you look at the memory of the machine, you can view that as, essentially, the state of the machine. And if you're always updating every memory location with exactly the same sequence of values, then the program is deterministic. Now it may be that two memory locations may be updated in a different order.

So you may have one location which is updated first in one execution, and another that's second, and then in a different execution, they may be a different order. That's OK, generally. The issue is whether or not every memory location sees the same order. And if they do, then it's for every execution, then it's a deterministic program.

So what's the advantage of having a deterministic program? Yeah?

AUDIENCE: It always runs the same way [INAUDIBLE].

PROFESSOR: It always runs the same way. So what? What's that good for?

AUDIENCE: So you can find bugs easier.

PROFESSOR: Yeah, debugging. It's really easy to find bugs if every time you run it it does the same thing. It's much harder to find bugs if, when you run it, it might do something different.

So that leads to our first major rule of thumb about determinism, which is you should always write deterministic programs. Don't write

non-deterministic programs. And the only problem is, boy is that poor quality there. So basically, it says, always write non-deterministic programs unless you can't. So sometimes, the only way to get performance is to do something non-deterministic. So this lecture is basically about some of the ways of doing non-deterministic programming. So it's appropriate that we say this is not for those who are faint of heart. We are treading into dangerous territory here. So the basic rule is, as I say, any time you can, make your program deterministic.

So we're going to talk about the number one way that people introduce non-determinism into programs, which is via mutual exclusion and mutexes, which are a type of lock, and then look at some of the anomalies that you get. Besides just things being non-deterministic, you can also get some very, very weird behavior sometimes for the execution.

So we'll start out with mutual exclusion. So let's take a look, for example, suppose I'm implementing a hash table as a set of bins. And I'm resolving collisions with chaining. So here, each slot of my hash table has a chain of all the values that resolve to that slot. And if I have a value x, let's say it has key 81, and I want to insert x into the table, I first compute a hash of x. And let's say it hashes to this particular list here.

And then what I do is I say, OK, let me insert x into the table. So I make the next pointer of x point to whatever is the head of the table. And then I make the table 0.2x. And that effectively inserts x into the hash table. Fairly straightforward piece of code. I would expect that most of you could write that even on an exam and get it right.

But what happens when we say, oh, let's have some concurrency. Let's have the ability to look up things in a hash table in different parallel branches of a parallel program. So here, we have a concurrent hash table now where I've got two values, and I'm going to have two different threads inserting x and y. So one of them is going to do this one, and one of them is going to do this one. So let's just see how this can screw up.

So first, we hash x, and it hashes to this particular slot. So then we do, just as we're doing before, making its next pointer point to the beginning of the array. Then y gets in the picture, and it decides oh, I'm going to hash. And oh, it hashes to exactly the same slot.

And then y makes its next pointer point to the same to the head of the list. And then it sets the head of the list to point to y. So now y is in the list. Whoops, now x puts itself in the list, effectively taking y out of the list. So rather than x and y both being in the list, we have a concurrency bug.

So this is clearly a race. So it's a determinacy race, because we have two parallel instructions accessing essentially the same location, at least one of which-- in this case both of them-- performing a store to that location. So that's a determinacy race. And how things are going to work out depends upon which one of these guys goes first. Notice, as with most race bugs, that if this code all executed before this code, we're OK. Or if this code all executed before this code, we're OK. So the bug occurs when they happen to execute at essentially the same time and their instructions interleave.

So this is a race bug. So one of the classic ways of fixing this kind of race bug is to insist on some kind of mutual exclusion. So a critical section is a piece of code that is going to access shared data that must not be executed by two threads at the same time. So it shouldn't be accessed by two threads at the same time. So it's mutual exclusion. So that's what a critical section is.

And we have a mechanism that operating systems typically provide-- as well as runtime systems, but you can build your own-- called "mutexes," or "mutex locks," or sometimes just "locks." So a mutex is an object that has a lock and unlock member function. And any attempt by a thread to lock an already locked mutex causes that thread to block.

And "block" is, by the way, a hugely overused word in computer science. In this case, by "block," they mean "wait." It waits until the mutex is unlocked. So whenever you have something that's locked, somebody else comes and tries to grab the lock. The mutex mechanism only allows one thread to access it. The other one waits until the lock is freed. Then this other one can go access it.

So what we can do is build a concurrent hash table by modifying each slot in the table to have both a mutex, L, and a pointer called "head" to the slot contents. And then the idea is that what we'll do is hash the value to a slot. But before we access the elements of the slot, we're going to grab the lock on the slot. So every slot in the table has a lock here. Now, I could have a lock on the whole table. What's the problem with that? Sure.

AUDIENCE: [INAUDIBLE] basically can't do anything. You can't read. You couldn't be reading from the table.

PROFESSOR: Yeah, so if you have a lock on the whole table--

AUDIENCE: You would defeat the purpose [INAUDIBLE]

PROFESSOR: You defeat the purpose of trying to have a concurrent hash table, right? Because only one thread can actually access the hash table at a time. So in this case, what we'll do is we'll lock each slot of the hash table. And there are actually mechanisms where you can lock each element of the hash table or a constant number of elements. But basically, what we're trying to do is make it so that the odds are that if you have a big enough table and relatively few processors you're running on, the odds that they'll conflict are going to be very low.

So what we do is we grab a lock on the slot, and then we play the same game of inserting ourselves at the head. And then we unlock the slot. So what that does is it means that only one of the two threads in the previous example can actually execute this code at a time. And so it guarantees that the two regions of code will either execute in this order or in this order, and you'll never get the instructions interleaved.

Now, this is introducing non-determinism. Why is this going to be non-deterministic? Yes?

AUDIENCE: [INAUDIBLE] lock first, it'll be [INAUDIBLE].

PROFESSOR: Yeah, depending upon which one gets the lock first, the length list in there will have the elements in a different order. So a program that depends on the order of that list is going to behave differently from run to run. So let's recall the definition of a determinacy race. It occurs when two logically parallel instructions access the same memory location, and at least one of the instructions performs a write.

So that is, we do have a determinacy race when we introduce locks. Locks are essentially, we're going to have an intentional determinacy race. So a program execution with no determinacy races means the program is deterministic on that input.

So if there are no determinacy races, then although individual locations may be updated in a different order in a parallel execution, every memory location will be updated by exactly the same thing at the same time. The order will be of update of operations on any given location will be the same always. So that's actually a theorem, which we're not going to prove.

But I think if you think about it, it's fairly straightforward. If you never have two guys in parallel that could possibly affect the same location, then the behavior always is going to be the same thing. Things are going to get written in the same order.

So the program in that case always behaves the same on that given input, no matter how it's scheduled and executed. We'll always have essentially the same behavior, even though it may get scheduled one way or another. And one of the nice things that we have in our race detection tool Cilkscreen is that if we do have determinacy races that exist in an ostensibly deterministic program-- that is, a program with no mutexes. If basically it just reads and writes on locations and so forth, then Cilkscreen guarantees to find such a race. So It's nice that we get a guarantee out of Cilkscreen.

So this is all beautiful, elegant, everything works out great if there are no determinacy races. But when we do something like a concurrent hash table, we're intentionally putting in a determinacy area. So that asks sort of a natural question. Why would I want to have a concurrent hash table? Why not make it so that my program is deterministic?

Why might a concurrent hash table be an advantageous thing to have in a program that you wanted to go fast? Some ideas? Where might you want to use it? Yeah?

AUDIENCE: Speed?

PROFESSOR: Yeah, speed. But I mean, what's an application? What's a use case, as the entrepreneurs would ask you? Where is it that you would really want to use a concurrent hash table to give you speed? Yeah?

AUDIENCE: If you started using it [INAUDIBLE] along with your system [INAUDIBLE] values.

PROFESSOR: Yeah, it could be that there's some sort of global table that you want a lot of people to be able to access at one time. So if you lock down and only had one thread accessing at a time, you reduce how much concurrency that you could have. That's a good one. Yeah?

AUDIENCE: Perhaps most of the time, people are just reading. So if you had something concurrent, your reading should be fine. So in that case, a lot more reading high performance [INAUDIBLE]

PROFESSOR: Yeah, so in fact, there's a type of lock called a reader-writer lock, which allows one writer to operate, but many readers. So that's another type of concurrency control. So just another place, a common place that you use it, is when you're memoizing. Meaning I do a computation, I want to remember the results so that if I see it again, I can look it up rather than having to compute it again from scratch.

So you might keep all those values in a hash table. Well, if I go in the hash table, now I'm going to have concurrent accesses to that hash table if I've got a parallel program that wants to do memorizing. And there are a bunch of other cases.

So we have determinacy races. And we have a great guarantee that if there is a race, we guarantee to find it. Now, there's another type of race, and in fact, you'll hear more about this type of race if you read the literature than you hear about determinacy races. So a data race occurs when you have two logically parallel instructions holding no locks in common. And they access the same location, and at least one of the instructions performs a write.

So this is saying that I've got accesses. And if they have no locks in common-- so it could be that you have a problem where one of them holds a lock L, and another one holds L prime. And then they access the location, that's going to be a data race, because they don't hold locks in common.

But if I have L and L being the locks that the two threads hold, and they access the same location, that's not a data race now. It is a determinacy race, because it's going to matter which order it is, but it's not a data race, because the locks, in some sense, are protecting access.

So Cilkscreen, in fact, understands locks and will not report a determinacy race unless it is also a data race. However, since codes that use locks are non-deterministic by intention, they actually weaken Cilkscreen's guarantee. And in particular, in its execution that it does, if it finds a data race, it's going to say, I'm going to ignore that data race.

But now it is only going to follow one of the two paths that might arise from that data race. In other words, it doesn't follow both paths. If you could think about it, when one of them wins-- so you have a race between two critical sections.

When one of them wins, you can imagine that's one possible outcome of the computation. When the other wins, it's another path. And what Cilkscreen does is it picks one path. In fact, it picks the path which is the one that would occur in the cereal execution. So there's a whole path there that you're not exploring. So Cilkscreen's guarantee is not going to be strong there.

However, if the critical sections, in fact, commute-- that is, they do exactly the same thing, no matter what the order. So for example, if they're both incrementing a value, then the result, after doing one versus after the other is the same value, then you get a guarantee out of Cilkscreen. So Cilkscreen could still be very helpful for finding bugs, because typically, when you organize your computation, if it occurs in this order, there's typically some execution or input where you can make things occur in the other order.

So you can actually cover more races than you might imagine on first blush. But it is a danger. But what we're talking about today is dangerous programming, non-deterministic programming. So when you start using mutexes, some of the guarantees and so forth get much dicier. Any questions about that?

Now, if you have no data races in your code, that doesn't mean that you have no bugs. So for example, here's a way somebody might fix that insertion code. So we hash the key, we grab a lock, we set x next to be whatever is the head of the list, and then we do an unlock. And now we lock it again. Now we follow the head to set x-- sorry, we set x to be the head of the list and then unlock again.

And now notice that in this case, technically, there is no data race if I have two concurrent threads trying to access these at a time, because all the axis I'm doing, I'm holding lock L. Nevertheless, I can get that same interleaving of code that causes the bug. So just because you don't have a data race doesn't mean that you don't have a bug in your code. As I say, this is dangerous programming.

However, typically, if you have mutexes and no data races, usually it means that you went through and thought about this code. And if you were thinking about this code, you would say, gee, really I'm trying to make these two instructions be the critical section. Why would I unlock and lock again? So most of the time, as a practical matter, if you don't have data races, it probably means you did the right thing in terms of identifying the critical sections that needed to be locked and not unlocking things in the middle of them.

So as a practical matter, no data races usually means it's unlikely you have bugs. But no guarantees. As I say, dangerous programming. Non-deterministic programming is dangerous program. Any questions about that? Anybody scared off yet? Yeah?

AUDIENCE: So what you can do is the opposite. You don't have any bugs, but you made the critical distinction to [INAUDIBLE]

PROFESSOR: Yes, so certainly from a performance point of view, one of the problems with locking is that-- and we'll talk about this a little bit later-- with locking is that if you have a large section that you decide to lock, it means other threads can't do work on that section. So they're spinning, wasting cycles. So generally, you want to try to lock things as small as possible.

The other problem is, it turns out that there's overhead associated with these locks. So if there's overhead associated with the locks, that's problematic as well, because now you may be slowing down. If this is in an inner loop, notice that we've now, even if I just have the lock and unlock without these two spurious ones here, we may be more than doubling the overhead. In fact, locking instructions tend to be much more expensive than register operations. They usually cost something on the order of going to L2 cache as a minimum. So it's not even L1 cache. It's like going out to L2 cache.

Now, it turns out there are some times where you have data races. So we say if there are no data races, then you have no guarantee there's no bugs. If there are data races, your program still may be correct. Here's an example of a code where you might want to allow a benign data race.

So here we have, let's say, an array A that has these elements in it. And we want to find, what is the set of digits in the array? So these are all going to be values between 0 and 9. And I want to know which ones are present of the digits 0 to 9. Which ones are not present?

So I can write a little code for that. Let me initialize an array called "digits" to have all-zero entries. And now let me go through all the elements of A and set digits of whatever the digit is to be 1. So set at 1 if that digit is present. And I can do that in parallel, even.

So what can happen here is I can have, if I've done this in parallel, this particular update of digits of 6 will be set to 1 when this one is being sent to 1. Is that a problem? In some sense, no. They're both being set to 1. Who cares? But there is a race there. There is a race, but it's a benign race. Well, it may or may not be benign.

So there's a gotcha on this one. So this code only works correctly if the hardware writes the array elements atomically. So for example, not on the x86-64 architecture we're using. But on some architectures, you cannot write a byte value. You cannot write a byte value as an atomic operation. It implements a right to a byte by reading a word, masking out things, changing the field, masking again, and then writing it back out.

So you can have a race on a byte value. In particular, even if I were going to do this with bits, I could have a race on bits, although C doesn't let me access bits directly. The smallest unit I can access is a byte.

So you have to worry about what's the level of atomicity provided by your architecture? So the x86 architecture, the grain size of atomic update is you can do a single-byte write, and it will do the right, proper thing-- do the right thing on the write. So we have both things. No bugs. No data races doesn't mean no bugs. Presence of data races doesn't mean you have bugs. But generally, they're fairly well overlapped.

Now, why would I not want to put in a lock and unlock here just to get rid of the race? If I run Cilkscreen on this, it's going to complain. It's going to say, you've got a race here. Why would I not want to put a lock on here, for example?

AUDIENCE: Because then we don't have parallelism anymore?

PROFESSOR: No, well, I'd have parallelism maybe up to 10, for example, right? Because I have 10 different things that could be going on at a time. But that's one reason. That is one reason. What's another reason why I might not want to put locks in here?

AUDIENCE: [INAUDIBLE]

PROFESSOR: It could be that all the numbers-- that's a case where it doesn't get me much speedup. But what's another reason I might want to do this?

AUDIENCE: [INAUDIBLE]

PROFESSOR: I think you're on the right track. Overhead. Yeah. Overhead. This is my inner loop. So if I'm locking and unlocking, all this is doing is just doing a memory [? dereference ?] and an assignment.

And that may be fairly cheap, whereas if I grab a lock and then release the lock, those operations may be much, much more expensive. So I may be slowing down the execution of this loop by more than I'm going to gain out of the parallelism of this. So I may say, I may reason, hey, there is a good reason why not have a data race there.

So I may want to have a data race, and I may want to say that's OK. And if that happens, however, you're now going to get warnings out of Cilkscreen. And I generally recommend that you have no warnings on Cilkscreen when you run your code. So the Cilk environment provides a mechanism called "fake locks." So a fake lock allows you to communicate to Cilkscreen that a race is intentional.

So what you then do is you put a fake lock in around this access here. And what happens is when Cilkscreen runs, it says, oh, you grabbed this lock, so I shouldn't report a race. But during execution, no lock is actually grabbed, because it's a fake one. So it doesn't slow you down it all at runtime, but Cilkscreen still thinks that a lock is being acquired.

Questions about that? So this is if you want to have an intentional race, this is a way you can quiet Cilkscreen. Of course, it's dangerous, right? It's yet another example of what's dangerous here. Because what happens if you did it wrong? What happens if there really is a bug there?

You're now telling it to ignore that bug. So one way that you can make your code-- if you put in fake locks everywhere, you could make it so, oh, Cilkscreen runs just great, and have your code full of race bugs. So if you use fake locks, you should document very carefully that you're doing so and why that's going to be a safe thing to do. Any questions about that?

By the way, one of the nice things about some of the concurrency platforms like Cilk is that they provide a layer of abstraction where generally, you don't have to do very much locking. If you program with Pthreads, for example, you're locking all the time. So you're writing non-deterministic programs all the time, and you're debugging non-deterministic programs all the time. Whereas Cilk provides a layer of programming where you can do most of your programming in a deterministic fashion.

And occasionally, you may want to have some non-determinism here or there. But hopefully you can manage that if you do it judiciously. Any questions about mutexes and uses for them and so forth? Good.

So let's talk about how they get implemented. Because as with all these things, we want to understand not just what the abstractions is but how it is that you actually implement these things so that you can reason about them more cogently. So there's typically three major properties of mutexes when you look at them.

And when you see documentation for mutexes, you should understand what the difference is of these things. The first is whether it's a yielding mutex or a spinning mutex. So a yielding mutex, when you spin, it returns control to the operating system. And why might you want to do that? Whereas a spinning one just consumes processor cycles. Why would you want to do that? Yeah.

AUDIENCE: [INAUDIBLE] allow other threads.

PROFESSOR: Yeah, it can allow other threads or other jobs that could be running to use the processor while you're waiting. What's the downside of that? To speak to the-- either one. Go ahead.

AUDIENCE: It might be possible that whatever you're trying to do is essential, and you really want to get that done [UNINTELLIGIBLE] everything else executes. So you really want [INAUDIBLE]

PROFESSOR: Yeah, context switching a thread out is a heavyweight operation. And It may be, if you end up context switching out, it may be you only had to wait for a half a dozen cycles and you'd have the lock. But instead, now you're going and you're doing a context switch and may not get access to the machine for another hundredth of a second or something. So it may be on the order of 10 to the 6th-- a million instructions before you get access again, rather than just a few.

The second property of mutexes is whether they're reentrant or not. So a reenttrant mutex allows a thread that's holding a lock to acquire it again. So I may hold the lock, and then I may try to acquire the lock again. Java is full of reentrant locks, reentrant mutexes.

So why is this a positive or negative? What are the pros and cons of this one? Why might reentrancy be a good thing to want? Why would I bother doing-- why would I grab a lock that I already have?

AUDIENCE: It'd be too easy to do a check [INAUDIBLE].

PROFESSOR: It lets you do what?

AUDIENCE: It lets you not have to worry about locking when you already have a lock.

PROFESSOR: It lets you not worry about it. That's right. But why is that valuable?

AUDIENCE: It saves you one line in an If statment to check if you have a lock or not.

PROFESSOR: That could be. Basically, the If statement is embedded in there. But why would I care? Why would I want to be acquiring something that I already have? In what programming situation might that arise? This seems kind of weird, right? Could be recursion. Yeah.

So usually, what it comes from is when you have objects, and you have several methods on the object. And what you'd like to do is, if somebody's calling the method from the outside, you would like to be able to execute that particular-- I guess in C++ they don't call them "methods." They call them "member functions." "Member functions," they call them. In Java, they call them "methods," and in C++, they call them "member functions." Doesn't matter. It's the same thing.

So when you access one of these, normally, from the outside, you want to make sure you grab the lock associated with the object. However, it may be that what you're doing inside the object is you want to be able-- one of the operations may be a more complex operation that wants to use one of its own implementations. So rather than implementing it twice-- once in the locked form, once without getting the lock-- you just implement it once, and you use reentrant locks. And that way, you don't have to worry about, in coding those things, whether or not you've already got it. So that's probably the most common place that I know that people want reentrant locks.

Naturally, to acquire a reentrant lock, you have to do some kind of If statement, which is a conditional. And as you know, if it's an unpredictable branch, that's going to be very expensive. So generally, there is a cost to making it reentrant.

The third property is whether the lock is fair or unfair. So a fair mutex puts block threads essentially into a FIFO queue. And the unlock operation unblocks the thread that has been waiting the longest. So it makes it so that if you try to acquire a lock, you don't have some other thread coming in and trying to access that lock and getting ahead of you. It puts you in a queue.

So an unfair mutex lets any blocked thread go next. So the cheapest thing to implement is a spinning, non-reentrant, unfair lock-- mutex. Those are the cheapest ones to implement. Very lightweight, very easy to use.

The heavyweight ones are a yielding, reentrant, fair lock. And of course, you can have combinations, because all of these have, as you can see, different properties in terms of convenience of use and so forth, as well as different overheads. So there's some cases where the overhead isn't a big deal because it's not in the inner loop of a program or a heavily executed statement.

So let's take a look at one of the simplest locks, which is a simple spinning mutex. This is the x86 code for how to acquire a lock. So let's run through this. So we start out at the top. And I check to see if the mutex is 0, which is basically, it's going to be 0 if it's free and 1 if it has been acquired.

So we compare it. If it's free, then I jump to try to get the mutex. Otherwise, I execute this PAUSE instruction, and this turns out to be a-- it's humorous. It's x86 hack to un-confuse the pipeline.

So it turns out that in this case, if you don't have a pause here-- which is no-op and does nothing-- x86 mispredicts something or whatever, and it's more time consuming than if it doesn't have that there. The manual explains very little about this hardware bug except to say, put in the pause. So if you didn't get it, then you jump to spin mutex, and try again, check to see if it's free.

Now, notice that we're going to spin until it's free, and then we're going to try to get it. Why not just try to get it first? Well, think about that while we go through how to get it, and then I'll ask it again. Think about why it is that you might want to get it first.

So if I want to get the mutex, I first get a value of 1 in my register. And then I compute this exchange operation, which exchanges the value of the mutex with the value of the-- with the one that I have. So it exchanges the memory location with the register.

Now, this is an expensive operation-- exchange-- because it's an atomic exchange, and it typically has to go at least out to L2 to do this. So it's an expensive operation, because it's a read-modify-write operation. I'm swapping my register value with a value that's in the mutex.

So it turns out that if it's 0, then it means I got it. So I compare it with 0, and if it's equal to 0, I go onto the critical section. When I'm done with the critical section, I release the mutex by basically storing 0 in there, because I'm the only one who accesses the mutex at this point.

If I didn't get it, if the value is 1, notice that because I'm swapping a 1 in, even though the 1 got swapped in, well, there was a 1 there before. So it basically did not affect the value of the mutex. But I discover, oh, I don't have it. Then we go all the way back up there to spin mutex.

So here's the question. Why do I need all this preamble code? Why not just go straight to Get_Mutex, make the spin mutex here be a jump to Get_Mutex? Yeah?

AUDIENCE: Maybe it's because the exchange is expensive.

PROFESSOR: Excuse me?

AUDIENCE: The exchange is--

PROFESSOR: Yeah, because the exchange is expensive. Exactly. So this code here, I can compare. And as long as nobody's touching anything, this becomes just L1 memory accesses. Whereas here, it's going to be at least L2 to do the exchange operation. So rather than doing that-- moreover, this one actually changes the value.

So what happens when I change the value of the mutex? Even though I change it to the same value, what happens in order to do that exchange? Remember from several lectures ago. What's going to happen when I make an exchange there? What does the hardware have to do? What's the hardware going to do on any store to a shared memory location, to a memory location in shared memory that is actually shared? Yeah?

AUDIENCE: [INAUDIBLE]

PROFESSOR: Yeah, it's got to invalidate all the other copies. So if everybody spinning here-- imagine that you have five guys spinning, doing exchanges-- they're all creating all this traffic of invalidations, what's called an "invalidation storm." So they create an invalidation storm as they all are invalidating each other so that they can get access to it so that they can change the value themselves.

But up here, all I'm doing is looking at the value. All I'm doing is looking at the value to see if it's free. And it's not until the guy actually frees the value that it actually-- actually, this is interesting. I think I wrote this with Intel syntax, rather than AT&T, didn't I? The MOV mutex, 0 moves 0 into the mutex, which is Intel syntax.

I probably should have converted this to AT&T, because that's what we're generally using in the class. I'll fix that before I put the slides up. Basically, I pulled this out of the Intel manual.

So any questions about this code? Everybody see how it works? It relies on this atomic exchange operation. And I'm going to end up sitting here spinning until maybe I can get access to it. When I have a chance to get access to it, I try to get it. If I don't get it, I go back to spinning.

How do I convert this to a yielding mutex?

AUDIENCE: Instead of having that spinning mutex, you should-- you shouldn't have that. You should just have something that allows you to just [INAUDIBLE].

PROFESSOR: Yeah, so actually, the way you do it is you replace the PAUSE instruction. Exactly what you're saying. You've got the right place in the code. We basically call a yield. And you can use, for example, pthread_yield. What it tells the operating system is, give up on this quantum. You can schedule me out. Somebody else can be scheduled. Now, if nobody else is there to be scheduled, often you'll just get control back, and you'll jump again and give the operating system another time.

Now, one of the things I've seen in computer benchmarks that use locking is that they all use spin locks. They never use the yielding, because if you yield, then when the lock comes free, you're not going to be ready to come back in. You may be switched out.

So a common thing that all these companies do when they're vying for who's got the fastest on this benchmark or fastest on that benchmark is they go through and they convert all their yielding mutexes into spinning mutexes, then take their measurements, when in fact, as a practical matter, they can't actually ship code that way. So you'll see this kind of game played where people try to get the best performance they can in some kind of laboratory setting. It's not the same as when you're actually doing a real thing.

So you have a choice here. There's kind of a tension here. You'd like to claim the mutex soon after it's released. And you're not going to get that if you yield. At the same time, you want to behave nicely and waste few cycles. So what's the strategy for being able to accomplish both of these goals?

So one of these goals is the spinning mutex does a great job of claiming the mutex soon after it's released. The yielding mutex behaves nicely and wastes few cycles. Is there the best of both worlds? There's certainly the worst of both worlds, right?

What's the best of both worlds? How might we accomplish both of these goals with small modification to the locking code? So it turns out you can get within a factor of two of optimal. How might you do that while wasting few cycles?

So here's the idea. Spin for a little while, and then, if after a little while you didn't manage to access the mutex, then yield. So that if the new mutex was right there available to be accessed, you could access it, but you don't spin for an indefinite amount of time. So the question is, how long do you spin? So we're going to spin for a little while and then yield. Yeah?

AUDIENCE: [INAUDIBLE].

PROFESSOR: Yeah, exactly. So what you do is you spin for basically as long as a context switch takes. So if you spin for as long as it takes to do a context switch and then do a context switch, if the mutex became immediately available, well, you're only going to wait double what you would have waited. And if in the meantime during that first part where you're spinning it becomes available, you're not waiting at all any longer than you actually have to.

So in both cases, you're waiting at most a factor of two. In one case, you're waiting exactly the right. The other, you can actually wait a factor of two. So this is a classic amortized kind of argument, that you can amortize the cost of the spinning to the context switch.

So spin until you spend as much time as it would cost for a context switch. Then do the context switch. Yet another voodoo parameter. Yeah, so if the mutex is released while spinning, that's optimal. If the mutex is released after the yield, you're within twice optimal.

Turns out that 2 is not the optimal value. There's a randomized algorithm that makes it e over e minus 1 competitive where e is the base of the natural logarithm. So 2.7 divided by 1.7, which is what? Who's got a calculator? 2.7 divided by 1.7 is-- I should have calculated this out.

AUDIENCE: [INAUDIBLE]

PROFESSOR: It's about 1.6. Good. So it's better than 2. People analyze these things, right? So any questions about implementation of locks? There are many other ways of implementing locks. There are other instructions that people use. They do things like compare-and-swap is another operation that's used.

There are some machines have an operation called load-linked/store-conditional, which is not on the x86 architecture, but it is on other architectures. You'll see a lot of other things of doing some kind of atomic operation to implement a lock. Uniformly, they're expensive compared to register operations in particular or even L1 accesses, typically, in particular. Any questions?

So now that we've decided that we're going to use mutexes, and we understand we're writing non-deterministic code and so forth, well, it turns out there are a host of other system anomalies that occur. So locks are like, they're this really evil mechanism that works really well. It feels so good that nobody wants to stop using it, even though-- but nobody has better ideas.

One of the most interesting ideas in recent memory is the idea of using what's called "transactional memory," which is basically where memory operates like a database transaction. And it's allowed to abort, in which case you roll it back and retry it. Yet, transactional memory has a host of issues with it, and still people use locks.

So let's talk about some of the bad things that happen when you start doing locks. I'm going to talk about three of them, deadlock, convoying, and contention. So deadlock is probably the most important one, because it has to do with correctness. So you can have coded-- in fact, I've seen people with very fast code that has deadlock potential in it. It's like, if you deadlock, then your average running time is infinite if there's a possibility of a deadlock, right? Because you're averaging infinity with everything else that you might run.

So it's not good to have deadlock in your code, regardless. It's kind of like your code seg faulting. No decent code should seg fault. It should always catch its own errors and terminate gracefully. It shouldn't just seg fault in some circumstance. Similarly, your code should not deadlock.

So here's sort of a classical instance of deadlock. And deadlock typically occurs when you hold more than one lock at a time. So here, this guy is going to grab a lock A, going to grab a lock B, then unlock B, unlock A, and in there do a critical section. Why might I grab two locks? What's the circumstance where I might have code that looked very similar to this? Use case.

AUDIENCE: Two objects?

PROFESSOR: sorry?

AUDIENCE: You need two objects.

PROFESSOR: You need two objects. When might that occur?

AUDIENCE: Account transactions.

PROFESSOR: Yeah, account transactions. That's the classic one. You want to move something from this bank account to that bank account. And you want to make sure that as you're updating it, nothing else is occurring. Another place this comes up all the time is when you do graph algorithms. You always want to grab the edge and have the two vertices on each end of the edge not move while you do something across the edge. So lots of cases there.

It turns out the order in which you unlock things doesn't matter, because you can always unlock something. You never hold up for unlocking. The problem with deadlock is generally how you acquire locks. So in this example, Thread 2 grabs Lock B, then grabs Lock A. So it might be, for example, that you have some random process that's at the node of a graph.

And now it's going to grab a lock on the other end of an edge. But you might have the guy at the other end grabbing that vertex and then grabbing the one on your end. And that's basically the situation. So what happens is Thread 1 acquires a lock here. Thread 2 acquires that lock. And now which one can go? Neither of them. You've got a deadlock. Ultimate loss of performance.

So it's really a correctness issue. But you can view it, if you really say, oh, correctness, that's for sissies. We do performance. Well, it's still a performance issue, because it's the ultimate loss of performance. In fact, that's probably true of any correctness issue. No, that's not true. Sometimes you just get the wrong number. Here is a correctness issue that your code stops operating.

So there are three conditions that are usually pointed to that you need for deadlock. The first is mutual exclusion. Each thread claims exclusive control over the resources that it holds, in this case, the resources being the locks. So there's got to be some resource that you're grabbing, and that you're the only one who gets to have it. So in this case, it would be the locks.

The second is non-preemption. You don't let go of your resources until you complete your use of them. So that means you can't let go of a lock in a situation. If you're actually able to preempt-- so this piece of code over there has grabbed locks, and now I can come in and take them away, then you may not have a deadlock potential. You may have other issues, but you won't have a deadlock potential.

And the third one is circular waiting. You have a cycle of threads in which each thread is blocked waiting for resources held by the next thread in the cycle. So let me illustrate this with a very famous story that some of you may have seen, because it is so famous. It's the dining philosophers problem.

It's an illustrative story a deadlock that was originally told by Tony Hoare, based on an examination question by Edsger Dijkstra. And the story has been embellished over the years by many retellers. It's one of these things that if you're a computer scientist, you should know this story just because everybody knows this story.

So here's how the story goes, at least my version of it. I get to retell it now. So each of n philosophers needs the two chopsticks on either side of his or her plate to eat the noodles on the plate. So they're not worried about germs here, by the way.

So you have five philosophers in this case sitting around the table. There are five chopsticks between them. In order to eat, they need to grab the two chopsticks on either side. Then they can eat. Then they put them down. So here's what philosopher i does. So in an infinite loop, the philosopher does thinking, because that's what philosophers do.

Then it grabs the lock of chopstick i and grabs the lock of chopstick i plus 1. That's the 1. So if we index them, say, to the left of the plate, this is grabbing the chopstick to the left of your plate. This is grabbing the chopstick to the right of your plate. Then you can eat. Then you release your two chopsticks.

So here, that's the code. And then you go back to thinking. I guess they have no other bodily functions. So the problem is, one day they all pick up their left chopsticks simultaneously. Now they go to look for their right chopstick. It's not there. So what happens? They starve because their code doesn't let them release-- there's no preemption, so they can't release the chopstick they've already got.

And we have a circular waiting. They have mutual exclusion. Only one of them can have a chopstick at a time. And we have a circular waiting thing, because everyone is waiting for the philosopher on the right. Is that clear to everybody? That's the dining philosophers problem.

How do you fix this problem? What are solutions to fixing this problem? The problem being that you'd like them to eat indefinitely.

AUDIENCE: You can index the chopstick and say that [INAUDIBLE].

PROFESSOR: Yeah, you can pick the smaller index first. So in general, that means everybody would grab the one on their left, then the one on their right, except for the guy who's going between 0 and n minus 1. They would do n minus 1 and then 0. They would do n minus 1 first, and then 0. Sorry. They would do 0 first, and then n minus 1. [INAUDIBLE] Let me say that more precisely.

So this is a classic way to prevent deadlock. Suppose that we can linearly order the mutexes in some order so that whenever a thread that holds a mutex Li and attempts to lock another mutex Lj, we have it that Li goes before Lj in the ordering. Then no deadlock can occur.

So always grab the resource so if they can all order the resources-- so they're always grabbing them in some subsequence of this order, so they're always grabbing one that's larger and larger and larger, and you're never going back and grabbing one smaller, than you have no deadlock. Here's why. Suppose you have a cycle of waiting. You have a deadlock has occurred.

Let's look at the thread in the cycle that holds the largest mutex that's called Lmax in the ordering. So whatever is in the ordering. And suppose that it's waiting on a mutex L held by the next threat in the cycle. That's the condition.

Well, then it must be that Lmax falls before L, because we're gathering them always in an increasing order. But that contradicts the fact that Lmax is the largest. So a deadlock cannot occur. Questions? Is this clear?

Who's seen this before? A few people. OK. Is this clear? So if you grab them in increasing order, then there's always some guy that has the largest one, and nobody is holding one larger. So he can always grab the next one.

So in this case of the dining philosophers, what we can do is grab the minimum of i and i plus 1 mod n and then the maximum of i and i plus 1 mod n. That gives us the same two chopsticks. And in fact, for most of the philosophers, it's exactly the same order. But for one guy, it's a different order. It ends up being the guy who would normally have done n minus 1 and 0.

Instead, he does 0, n minus 1. So in some sense, it's like having a left-handed person at the table. You grab your left, then your right, except for one guy does right and then left. And that fixes it, OK? That fixes it. Good.

So that's basically the dining philosophers problem. That's one way of fixing it. There are actually other ways of doing it. One of the problems with this particular solution is you still can have a long chain of waiting.

So there are other schemes that you can use where, for example, if every other one grabs left and then right and then right and then left and then left and then right and then right and left and so forth, you can end up making it so that nobody has to wait to go all the way around the circle. Yeah?

AUDIENCE: [INAUDIBLE]

PROFESSOR: Well, that would be a preemption type of thing, where I grab one, and if I didn't get it in time, I release it and then try again. When you have something like that, there's an issue. It's, how do you set the timeout amount? And the second issue that you get into when you do timeouts is, how do you know you don't then repeat exactly the same thing and convert a deadlock situation into a livelock situation?

So a livelock situation is where they're not making progress, but they're all busily working, thinking they're making progress. So you timeout. Let's try again. What makes you think that the guys that are deadlocking aren't going to do exactly the same thing.

AUDIENCE: [INAUDIBLE]

PROFESSOR: And exactly. And in fact, that's actually a workable scheme. And there are schemes that do it. Now, that's much more complicated. Sometimes has more overhead, especially because things become available. And it's like, no, you're busy raiding some random amount of time before you try again.

So this is, by the way, the protocol that is used on the Ethernet for doing contention resolution. It's what's called "exponential backoff." And various backoff schemes are used in order to allow multiple things acquire mutually-exclusive access to something without having to have a definite ordering. So there are solutions, but they definitely get more heavyweight. It's not lightweight.

Whereas if you can prevent deadlock, that's really good, because you just simply do the natural thing. And that tends to be pretty quick. But yeah, all I'm doing is sort of covering the introduction to all these things. There are books written on this type of subject. Any other questions about dining philosophers and deadlock and so forth?

Now let me tell you how to deadlock Cilk++. So here's a code that will deadlock Cilk++, or has the potential. You might run it a bunch of times, it looks fine. Here's what we've done is main routine spawns foo. Here's foo down here. All foo does is grab a lock and then unlocks it. Empty critical section. It could do something in there. It doesn't matter.

Then the main grabs a lock, does a cilk_sync and then unlocks it. So what can go wrong here? Notice, by the way, this is only one lock, L. There's not two locks. So you can deadlock Cilk by just introducing one lock. So here's sort of what's going on. Let's let this be the main thread and this be foo. And this will represent a lock acquire, and this is a lock release.

So what happens is we perform the lock acquire here in the parent. First, we spawned here, then we acquire the lock here. And now foo tries to get access to the lock, and it can't because why? The main routine has the lock.

Now what happens? The main routine proceeds to the sync, and what does it do at the sync? It waits for all children to be done. And notice now we've created a cycle of waiting, even though we didn't use a lock. Main waits, but foo is never going to complete, because it's waiting for the main thread to release it, the main strand here to release it, the main function here. Is that clear?

So you can deadlock Cilk too by doing non-deterministic programming. So here's the methodology that will help you not do that. So what's bad here? What's bad is holding the lock across the sync. That's bad. So don't do that. Doctor, my head hurts. Well, stop hitting it. So don't hold mutexes across Cilk syncs. Hold mutexes only within strands, only with serially-executing pieces of code.

Now, it turns out that you can hold it across syncs and so forth, but you have to be careful. And I'm not going to get into the details of how you can do that. If you want to figure that out on your own, that's fine. And then you're welcome to try to do that without deadlocking something. Turns out, basically, if you grab the lock before you do any spawns, and then released it after the Cilk sync, you're OK. You're generally, in that case, OK.

So as always, try to avoid using mutexes, but that's not always possible. In other words, try to do deterministic programming. That helps too. And on your homework, you had an example of where it is that deterministic programming can actually do a pretty good job.

The next anomaly I want to talk about is convoying. Once again, another thing that can happen. This one is actually quite an embarrassment, because the original MIT Cilk system that we built had this bug in it. So we had this bug. So let me show you what it is. So here's the idea. We're using random work-stealing where each thief grabs a mutex on its victim's deck.

So in order to steal from a victim, it grabs a mutex on the victim. And now, once it's got the mutex, it now is in a position to migrate the work that's on that victim to actually steal the work. And you want to do that atomically. You don't want two guys getting in there trying to steal from each other.

So if the victim's deck is empty, the thief releases the mutex and tries again at random. That makes sense. If there's nothing there to be stolen, then just released the mutex and move on. If the victim's deck contains work, the thief then steals the topmost frame and then releases the mutex. Where's the performance bug here?

AUDIENCE: [INAUDIBLE] trying to steal from each other. Like A steals from B, B steals from C, C steals from D, and they all have locks on each other, and then--

PROFESSOR: No, because in that case, they'll each grab the deck from each other, discover it's empty, and release it. OK, let me show the bug. It is very subtle. As I say, we didn't realize we had this bug until we noticed some codes on which we weren't getting the speedups we were expecting. Let me show you where this bug comes from.

Here's the problem. At the startup, most thieves will quickly converge on the worker P0 containing the initial strand, creating a convoy. So let me show you how that happens. So here we have the startup of our Cilk system where one guy has work, and all these are workers that have no work to do. So what happens? They all try to steal at random.

In this case, we have this guy tries to steal from this fellow, this guy tries to steal from this fellow, et cetera. So of these, this guy, this guy, and that guy all are going to discover there's nothing there to be stolen, and they're going to repeat the process. This guy and this guy, there's going to be some arbitration. And one of them is going to get the lock. Let's assume it's this one here.

So what happens is, this guy gets the lock. What does this guy do? He's going to wait. Because he's trying to acquire the lock. He can't acquire the lock, so he waits.

So then what happens? This guy now wants to steal the work from this fellow. So he steals a little bit of work. Then these guys now, what do they do? They try again. So this guy tries to steal from there, this guy tries to steal from there, this one happens to try to steal there. This one sees there's work there to be done, so what does it do? It waits.

But these guys then try again. Maybe a little bit more stuff is moved. They try again. A little bit more stuff. They try again. But every time one tries and gets stuck on P0 while we're doing that whole transfer, they all are ending up getting stuck waiting for this guy to finish.

And now, we've got work over here, but how many guys are going to be trying to steal from this guy? None. They're all going to be trying to steal from this one, because they all have done a lock acquisition, and they're sitting there waiting. So this is called convoying, where they all pile up on one thing, and now resolving that convoy.

So this was a bug in startup. Why wasn't Cilk starting up fast? Initially, we just thought, oh, there's system kinds of things going on there. So the work now gets distributed very slowly, because each one is going to serially try to get this work, and they're not going to try to get the work from each other. What you want is that on the second phase, half the guys might start hitting this one.

So you get some kind of exponential distribution of the work in kind of a tree fashion. And that's what theory says would happen. But the theory is usually done without worrying about what happens in the implementation of the lock. What's the fix for this? Yeah?

AUDIENCE: Can you just basically shove-- when you're transferring, you should also say, I have work, so that people [INAUDIBLE] waiting for that guy to [INAUDIBLE].

PROFESSOR: You could do that, but in the meantime, it could be that the attempt to steal goes so much faster than the actual getting of the work, you're still going to get half the guys locked up on this one. And the other half might be locked up on this one. Good idea. What other things can we do?

AUDIENCE: Can you check how many people are waiting on the--

PROFESSOR: Yeah, so the idea is we don't want to use a lock operation. So here's the idea. We use a non-blocking function that's usually called "try_lock," rather than "lock." try_lock attempts to acquire the mutex. If it succeeds, great. It's got it. If it fails, it doesn't go to spin. It simply returns and say, I failed.

It doesn't go to spin or to yield or anything. It just says, oh, I failed, and tells that back to the user. But it doesn't attempt to block. So with try_lock now, what can these other processors do? They do a try_lock-- yeah?

AUDIENCE: [INAUDIBLE]

PROFESSOR: Exactly. Instead of waiting there on the guy that they fail on, they pick another random one to steal from. So they'll just continually try to get it. If they get it, then they can do their operation. If they don't get it, they just look elsewhere for work. So that's what it does. It just tries to steal again at random, rather than blocking.

And that gets rid of this convoying problem. As I say, dangerous programming, because we didn't even know we had a problem. Just our code was slower than it could have been. Questions about convoying?

So try_lock is actually a very convenient thing to use. So in many cases, you may find that, hey, rather than waiting on something with nothing to do, let me go see if there's something else I can do in the meantime.

Contention. So here's an example of a code where I want to add up some function of the elements of some array. So here I've got a value of n, which is a million. And I have a type X. So we have a compute function, which takes a pointer to a-- did I do this right? To value V. So anyway, my C++ is not as good as my C, and for those who don't know, my C isn't very good.

So anyway, we have an array of type X of n elements. And what I do is I set result to be 0, and then I have a loop here which basically goes and adds into result the result of computing on each element of the array. And then it outputs the result. Does everybody understand what's going on in the code? It's basically compute on every element in the array, take the result, add all those results together.

We want to parallelize this. So let's parallelize that. What looks like the best opportunity for parallelizing? Yeah, we go after the for and make it be a cilk_for. Let's add all those guys up. And what's the problem with that? We get a race. What's the race on?

AUDIENCE: Result.

PROFESSOR: Result. They're all updating result in parallel. Oh, I know how to resolve a race. Let's just put a lock around it. So here we have the race. First, let's analyze this. So the work here is order n. What is the span?

AUDIENCE: Log n.

PROFESSOR: Yeah, the span is log n for the control of the stuff here, because this is all constant time. So the running time here is order n over P plus log n. If you remember the greedy scheduling, it's going to be something like this, because this is the work over P plus the span. So we expect that if n over P is big compared to log n, we're going to do pretty well, because we have parallelism over log n.

So let's fix this bug. So this is fast code, but it's incorrect code. So let's fix it by getting rid of this race. So what we'll do is we'll put a lock before. We'll introduce a mutex L, and we'll lock L before we add to the result, and then we'll unlock it.

So first of all, this is a bad way to do it, because what I really should do is first compute the result of my array and then lock, add it to the result, and then unlock so that we lessen the time that I'm holding the lock in each iteration. Nevertheless, this is still a lousy piece of code. Why's that?

AUDIENCE: It's still serialized.

PROFESSOR: Yeah, it's serialized. Every update to result here has to go on serially. They're n accesses. They're all going to go one at a time. So my running time, instead of being n over log n, is going to be something like order n.

Believe me, I have seen many people write code where they essentially do exactly this. They take something, they make it parallel, they have a race bug, they fix it with a mutex. Bad idea, because then we end up with contention on this mutex. What's the right way to parallelize this? Yeah?

AUDIENCE: Maybe you could have each [INAUDIBLE] have result as an array and have each [INAUDIBLE] one place [INAUDIBLE]. And then at the end, some of all the--

PROFESSOR: But won't that be n elements to sum up?

AUDIENCE: [INAUDIBLE]

AUDIENCE: So basically, have, say. Eight results, instead of having--

PROFESSOR: For each thread. Good. So that each one could keep it local to its own thread. Now, of course, that involves me knowing how many processors I'm running on. So now, if that number changes or whatever-- there's a way of doing it completely processor obliviously.

AUDIENCE: Divide and conquer.

PROFESSOR: Yeah, do divide and conquer. Add up recursively the first half of the elements, add up the second half of the elements, and add them together. Next time, we're going to see yet another mechanism for doing that, which gets the kind of performance that you're mentioning but without having to rewrite the For loop as divide and conquer. We'll see that next time.

So in this case, we have lock contention that takes away our parallelism. Unfortunately, very little is known about lock contention. The greedy scheduler, you can show that it achieves T1 over P plus T infinity plus B where B is the bondage, that is, if you add the total time of all critical sections. That's a lousy bound, because it says, even if they're locked by different locks, you still add up the total time of all the critical sections.

And generally, although you can improve this in special cases, the general theory for understanding contention is not understood very well. And this upper bound is weak, but little is known about lock contention. Very little is known about lock contention.

So to conclude, always write deterministic programs, unless you can't. Always write deterministic programs, unless you can't. Great.