Lecture 10: Dynamic Storage Allocation

• Download this transcript - PDF (English - US)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high-quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

CHARLES LEISERSON: Today we're going to talk about dynamic storage allocation, which is a hugely rich and interesting topic, and hopefully cover some of the basics of dynamic storage allocation today. I want to start out with perhaps the simplest storage allocation scheme, and that's stack allocation, which I think you're probably fairly familiar with. So stack allocation is just an array and a pointer into the array, an index or a pointer into the array. And when you have a stack in a computer memory, you're talking about the array of all memory, and the stack pointer pointing that's used, for example, for calls.

The reason stack allocation is neat is because it's so simple. If I want to allocate x bytes, then what I do, is I just simply increment my stack pointer by x bytes, and then I return the original position of the stack pointer as being the location of the storage that's just been allocated. So pretty simple. Typically if you write a stack allocator, the routines are so simple they get inlined, although it's of course advantageous to say that things are inlined, but most compilers these days will inline. And as you can see, it's just a couple of operations here to perform this calculation.

To deallocate the bytes, we basically-- oh, yeah, I should mention, this isn't checking for overflow. So technically, one should check for overflow. One way to do that is for the assert package, and that way, at run time, you don't have to pay the extra overhead of checking. And see, these operations are so cheap that actually checking for overflow is perhaps as expensive as doing the rest of the operations themselves. Checking to see whether you went over the right end of the array here.

So to free x bytes, you basically just simply decrement the stack pointer. And of course, technically, you should also check for underflow, make sure somebody didn't say to deallocate more bytes than were actually already allocated on the stack. And so course, check for stack underflow.

So this is pretty good, because allocating and freeing take just constant time. Very quick. But you're required to free memory consistent with the stack discipline, which is the last in, first out discipline. So you don't get to free something that you allocated long ago until you've freed everything else that's been allocated since. So therefore, it has limited applicability, but it's great when it works.

Now it turns out, sometimes you can actually use the call stack yourself to store stack values, using a function called alloca. But this function is now deprecated, meaning that people advise you not to use it in any modern programming. And part of the reason is that the compiler is more efficient with fixed size stack frames. You remember when we went through how the compiler actually lays out memory and does the calling convention? If you have a fixed size stack frame, then the frame pointer and the stack pointer are redundant, and so you can reclaim a register, namely, the frame pointer register, and do everything off the stack pointer. But if you use things like alloca, which are pushing the sides of the stack frame, then you actually have to keep both pointers there, and that's a little bit less efficient in terms of the register usage. OK?

So that's pretty much stack storage. Any questions about stacks? Yeah?

AUDIENCE: Is there anything that detects overflow or underflow without using inserts? Like, if you wanted to check--

CHARLES LEISERSON: Yeah, you couldn't check explicitly. Just say, you know, for this, is before you check that, make sure that SP plus x, is it bigger than the right end?

AUDIENCE: But how do you know where the right end is?

CHARLES LEISERSON: Well, because I'm assuming here that in this case, you are allocating it. Now, if you're talking about using the call stack, you mean? You'll never run out. We'll do that analysis in a little bit. But you'll never run out of stack space. Although as a practical matter, most compilers assume a stack of a megabyte, and so if you overflow that, your program will fail in any of a variety of ways.

AUDIENCE: You can do guard pages.

CHARLES LEISERSON: You can do guard pages, which don't always work. But let me explain guard pages, OK?

So what's sometimes done is, if I have my stack growing down in memory-- OK, so here's my stack, and it's growing this way-- and I want to make sure, for example, the thing that's typically growing up this way is heat, and I want to make sure that they don't collide somewhere in here.

So one thing I can do is put some pages in here that are protected. These are read-only pages. Or actually even better is no permissions. So I can put no permissions so that if you ever try to read or write that page, it automatically generates a fault. And so the runtime will sometimes do that.

The difficulty is, what happens if you increased your stack so much that you went beyond your guard pages? But as a practical matter most of the times that people exceed the guard pages is only by small amount. And so just putting a few guard pages in there will make sure that you can use the memory management hardware, which basically ends up being free from your point of view as a programmer, in order to catch a stack overflow problem.

And you can do that yourself. You can use the call mmap, which we haven't talked about, to set permissions on pages at user level.

AUDIENCE: [INAUDIBLE]

CHARLES LEISERSON: No. You can actually handle the error however you want. You can vector into a user to find routine. Now, what are you going to do, I don't know, but you can indeed catch the error yourself and decide that there's some way of recovering, as opposed to crashing and burning. So generally, it's considered poor form for your software to fail, other than in a controlled way.

So you should always have your software exiting or calling error, and it should never segment fault. Most people consider any program that segfaults to be poorly engineered. Is there any case where there's a segfaulting program that is not poorly engineered? I don't know. I think most people, it's probably an understatement. OK?

So this is stack memory. You can build it yourself out of an array and a pointer, or you can I use this the call stack. And when you have something that's using that kind of discipline, it's great, because it's really cheap.

Now in C and C++, and we're going to start moving and talking more about things like C++, we have what's called heap allocation. And in C, the routines are called malloc and free, and C++ provides a new directive and delete. So unlike Java and Python, C and C++ provide no garbage collector. So in other words, you're responsible as a programmer for freeing allocated storage yourself. If you fail to do so, you create a memory leak. So a memory leak means I allocated something, I no longer am in a position to use it, but I failed to free it, so it can't be reused.

And if you keep doing that, especially if you're in a loop, then you have a memory leak. Where if you look at a program with a memory leak, and you look using [? Top ?] or whatever, how big is my program, you watch as it goes and computes and computes, and it keeps getting bigger and bigger and bigger and bigger as it's running, and it just runs away, getting more and more storage.

Also you have to watch out for things like dangling pointers. That is, you deallocated something, but somebody still wanted to use it. Or double-freeing. You have two places in the code where you free the same storage. Very bad to try to free the same storage. Yeah?

AUDIENCE: [INAUDIBLE]

CHARLES LEISERSON: It's basically going to be sitting in the storage structure, which we'll talk about how those are organized in a few minutes-- going to be sitting in the storage structure, but the program that is going to look at that and not recognize that it's in the storage structure, and think that it's something that now all its pointers can be reset. And so you basically clobber all the invariants of the storage structure, and somewhere down the road, you get a segment fault or something. OK? Yeah, John?

AUDIENCE: [UNINTELLIGIBLE] that comes out of that is--

CHARLES LEISERSON: Just a second. Why don't you take this?

AUDIENCE: [INAUDIBLE] One thing that often happens in the real world is when you free something, it adds it to a free, list and if you free something twice, it will add two copies of that pointer to the free list. So later on, when you malloc, malloc might return the same pointer to you twice. So the same memory block might be allocated for two completely different functionalities, and attackers use that to do all sorts of interesting things to your program.

CHARLES LEISERSON: Yeah. OK?

So fortunately, there are some really good-- why don't you keep that down there, and then if you guys want to chime in, you'll have some power to do so. So fortunately, there are now some tools to make that easier to debug these kinds of really nasty bugs. The one we'll be using is Valgrind, and what you do is you just say, Valgrind minus minus leak check equals yes, and then run your program the way you would normally run it. And what it will do is, it will monitor the way that the program is doing allocations and frees, and give you a report as to whether everything that was allocated was free, and whether you tried to free something twice, and so forth.

Now, this works with the existing allocators. If you have your own allocator that you decide you're going to write for a little piece of your code, you say, gee, I'm going to do my own little allocation here, you can actually communicate to Valgrind what that allocator is, and it can check for proper use of that structure as well. So you can see valgrind.org for details. And are a lot of other programs on the market, and out there as open software, such as purify and a variety of others.

So let's talk about heap allocation. I want to start by talking about fixed size allocation. So suppose the world has only one size of objects that you care about. Then it turns out that doing an allocator is really simple, and you could program one yourself.

So here's basically how it works. It's called using a free list. And what you have is, in your array, say, that you're using for storing all of your values, and here they're basically this size, what you do is you have a certain number that are used. That means that the program is using them, not the storage allocator. And what you do is you take advantage of the fact that if it's an unused block of storage, then you can put a pointer in there.

And so what you do, is you string all these unblocked pieces of storage together into a free list. So we point to there, which points to there, which points to there, which points to there, which points nowhere. So you just simply linked together all of the things that are free in that list. And then whenever you need one to allocate an object, what you do is, you grab one off the free list. So you set the value that you're going to want to take off free list to free-- so here x gets a pointer to this object, the one that's at the start of the list. Then you update free with free arrow next, which gives you this guy. So that's the next guy in line. And then you basically return x. And that's all there is to it. You just grab the first element off the free list.

Now, there's a little bit of subtlety in that the thing that you're grabbing off has a garbage pointer in it, because we're not initializing storage. There are some storage allocators that will always set the value that's returned to the user to all zeros, for example. But typically malloc doesn't do that. It basically leaves that as uninitialized storage. And so you can actually go and see, what were the values that used to be stored in there? And so basically, you can end up with a garbage pointer there, but you really don't care about that, because it's no longer on the list.

So you can see, for example, if I then went and I-- well, let's do this, to take a look at what happens when a free an item. So if I want to free an object x, I take some object x here. Let's say it's this guy that's being freed. What I do is, I make it point to the first element of the list. So basically, next equals free, So he basically points there. And then I say free equals x, meaning move the free pointer up to now point to the new beginning of the list. So we're basically pushing and popping things off the front of the list. It's just the stack discipline, except we're using a linked list, rather than using an array. So we push to free, and we pop to allocate.

Now the tricky thing here-- you can sort of see what might go wrong if I then said, oh, let me free this thing again. Suppose you said, now that this is free, suppose you came in and say, oh, I'll free that thing again. Well, now you're going to go through the same code here, and you're going to see, you're going to completely screw up the free list. So that's why you don't want to do a double free. So you can see exactly what's going to go on in the free list implementation.

So this is pretty good and pretty cheap. Allocating and freeing take constant time. That's great, just as with stack pointer. It's got good temporal locality, and the reason is because you're tending to allocate the thing that you freed most recently. The free list operates as a last in, first out data structure. So from a temporal locality point of view, it's really very good. You're always using the most recently freed stuff whenever you allocate, and that means you have very good caching behavior from a temporal locality point of view.

However, it has poor spatial locality, and the reason is due to what's called external fragmentation, where you get your storage distributed across virtual memory. So if you think about this, as you start allocating and freeing in different orders, the order of that free list gets very jumbled up. And as it increases, and you end up with jumping from one thing to another, you basically can end up causing you to use a lot of the page table, because you're basically using a huge piece of it, rather than-- and in particular, the translation look aside buffer can also be a problem. So the translation look aside buffer typically works on a page basis. And if you recently referenced things on a given page, that's very good for the TLB, because it's already got the translation, the virtual address mapping of that page dealt with. So it basically has poor spatial locality, because it's getting all jumbled up. Compared to stack allocation, which is always very nice in terms of its behavior, because you're going across consecutive locations in memory, which means you're always allocating and freeing from the same page, unless you trip over a page boundary.

So here's some ways of mitigating external fragmentation. Of course, this makes the code for free list more complicated. And then you have to weigh, how much am I losing in having a more complicated allocator versus suffering the fragmentation may occur? Yes, question?

AUDIENCE: You specificed before that it was fixed size. Is that why we don't specify anything about the size?

CHARLES LEISERSON: That's right. We're just assuming that it's all whatever the unit size is.

So this is really good when you've created your own data structure, and let's say you're doing nodes in a graph. You know exactly what elements you want in the graph, for each vertex. And now what you want to do is basically be able to give yourself a note of a graph, free a note of a graph, and manage that storage yourself. Yes, question?

AUDIENCE: [INAUDIBLE] allocated across virtual memory [INAUDIBLE] one application?

CHARLES LEISERSON: Why is it allocated across virtual memory if it's all one application? Well, because the size that you may end up needing can end up being large. And I'll show you in just a minute how that happens.

So here's the idea. What you can do is keep a free list per disk page. So disk page, a traditional page is 4,000 bytes. These days they talk about super pages that are megabytes, and who knows what size pages various operating systems will support. I believe the clouds use 4k, as configured right now are using 4k, which leads to some strange anomalies where the TLB can address less stuff than the L3 cache. But so be it.

So basically, they just keep a free list per disk page, and basically always allocate from the free list for the page that's fullest, the page that you've allocated the most of the stuff off of. Rather than for a page that's less empty. So what this means is that when you come to allocate, you have to figure out, which page should I do the allocation off of? But we'll have a free list on each page.

Whenever you free of a block of storage, you have to free it to the page on which the block resides. Don't put into a free list of a different page. You put it into a free list all of that particular page.

Now if a page becomes empty, in other words, there's only the free list on it, there's no actually used elements by the programmer, the virtual memory system will page it out. And since the programmer's never referencing things on that page, it basically costs you nothing. You end up not having to have an entry in the TLB, and you may have written to it, but you will end up allocating things preferentially to the more full pages.

And the basic idea is, it's better to have the use of your pages balance 90/10 than 50/50. So this kind [? of fig ?] configuration is better than that. So why is that? If you have 90% of your storage on one page and only 10% on another-- yeah, let's say we only of two pages in our system, and 90% of the objects are on one page, and 10% of the objects are on the other page, versus if they're 50-50. Then what you can do is look at the probability that two accesses are going to go to the same page. Because if you're going to the same page, then your paging hardware, which is very much like the cache, your TLB, which is very much like a cache-- whenever you get a hit on the same page, that's good, it costs you nothing. If you hit on a different page, that's bad.

So what happens is, the probability that two hit the same page is, well, for the first one, it's, what's the probability that the first one was on this page, versus on the other? So it's going to be on the same page if they're both on this page or they're both on this page. And so that's going to be 0.9 times 0.9 plus 0.1 times 0.1. Whereas if it's 50/50, then the probability that you get hit on the same page is now this 0.5 times 0.5 plus 0.5 times 0.5, which is only 50%. So we get 82% hit rate versus the 50% hit rate. So it's much more likely, if I'm stacking everything out in the extreme, it's better to have everything on one page and nothing on the other. Then I get 100% hit rate, because whenever I'm accessing something, I always am hitting on the page I had before.

So this kind of heuristic helps it so that you're swinging your accesses to using fewer pages, and the fewer pages, the less likely it is that you're going to stress your virtual memory paging, disk swapping, you're going to stress your TLB and what have you. Good for that? We're going to get deeper into this, so.

So that's basically fixed size allocation. It's really nice, because you can basically use a very simple free list. But of course what most people need and want is variable sized allocation. The most popular way of doing variable sized allocation is a scheme called binned free lists, and of that, then there are a zillion variations on it. But mostly, people use binned free lists.

The idea is, we're going to leverage the efficiency of normal free lists. Normal free lists I can allocate and free in just one operation, in just a constant amount of time I make. And what we're going to do is accept some internal fragmentation. So by internal fragmentation I mean the following. When somebody asks for something of a given size, what we're going to do is actually give them something which is a little bit bigger, maybe. And so we'll waste some.

And in particular, what we'll do is, we'll organize our system so that all the blocks that we ever give out are always a size the power of two. In practice, you don't always give out exactly a power of two, because then you get cache alignment problems. But this is the basic scheme, is to give out things are powers of two. And that way, if somebody asks for something that has 13 bytes, you give them 16 bytes. For somebody who asked for 65 bytes, have to give them 128 bytes. So you might waste up to as much as a factor of two.

But now, for a given that range of values, there's only a logarithmic number of different sizes that we have to give out, because we're only giving them out in powers of two.

So here's the basic allocation scheme to allocate x bytes. So what I do is, if I'm allocating x bytes, what I do is, I look in the bin ceiling of log x, because that's where everything of size two to the ceiling of log x is, which is basically rounding x up to the next power of two. If it's non-empty, I just return a block as if it were an ordinary free list. And what I'll be doing is returning a block which is at most twice the size of the requested block.

But that's generally OK. Somebody wants at least that many bytes. The fact that it's a few more bytes won't matter to them. They don't know that there's more bytes hidden there. They only take advantage of the fact that there are many bytes as I've asked for.

Otherwise, suppose that that bin is empty. There's nothing in that bin at this time. Well, then what you do is you start looking to find a block in the next larger non-empty bin, and you split it up. So for example here, suppose that the request is x equals 3. I have to look in bin 2, which is blocks of size 4. Uh oh, bin 2 is empty. So what I do is, I start going down, looking in the next bin. Oops, that's empty too, that's empty-- and then finally, I get to one where there is a block in it.

So what I do then is I dice this up. I cut it in half, and then cut that one in half, until I get a piece that's exactly the size I want to return. OK? And then I distribute all of these guys up to the other bins, like this.

So I return one of this size, and I populate these other guys with one block each of the appropriate size. And as you can see, they're basically growing in a geometric fashion. And then I distribute that bin.

Now there's a caveat here. Suppose that no larger blocks exist in my storage structure. Well, then I have to ask the operating system for more bytes, to allocate x more bytes of virtual memory. Now in practice what you do is you don't actually ask for x, since you never ask the operating system for small values. You always ask the operating system for big values. Give me another page worth, or give me another five pages, or something. Because that's an expensive call to the operating system, to ask it to allocate more storage.

Let me explain how that works. And this gets back to the question you had before, which is, where's the stuff coming from anyway? What the operating system is allocating is actually not physical pages, but rather parts of virtual memory. So we looked at this before, about how virtual memory is laid out for a typical program.

So what we do is, on the very high addresses, we have the stack, and the stack grows downwards. This is the call stack. At the low address, we typically start out with what's called the text segment, which holds your code. Then there's a region called the data segment, and this consists of data that needs to be initialized. So the compiler typically distinguishes between data that needs initialization and data which is going to be zero, because data that needs initialization, it writes into the binary executable file and stores that on disk. So when you load in your program, it just loads as part of the text here, it loads that data segment, saying what all of the storage is.

The next segment is, for historical reasons, called the BSS segment. And what BSS consists of is all the variables that at the start of the program should be initialized to 0. It doesn't bother storing these in the file. Why not? Because it's easy enough to just bring it in, understand what that distance there is, and then do a memset of this whole region with zero, to zero it out, rather than trying to read in a whole bunch of zeroes. An obvious kind of data compression.

Then heap storage is allocated in the space above the BSS segment. So the idea is, there's a part of heap which has already been allocated stuff. And whenever you need more heap storage, you ask the operating system-- the operating system moves just as if it were a stack, moves the line for where heap is to allocate, to give you more pages of heap storage that you can now allocate out of. And what it's doing is allocating pieces of virtual memory.

Now, here's a good mind question, which I think gets directly to your point. So think about this. We have a 64-bit address space. If I try to write at the rate of 4 billion bytes per second, that's pretty fast, because that's the speed at which the registers get written. Not the speed that memory gets written. So really, it's something like 4 billion bytes per second. It takes over a century to write all of virtual memory.

So as a practical matter, that means I never run out of stack space. I never run out of heap space. They're never going to cross, no matter how much I'm allocating. Even if I allocate on every cycle, there's not going to be a program-- unless you're going to set a program running for a millennium sort of thing, you're not going to see that. Most programs run for a few minutes. Some may run for days. Some may run for years. Very few do people write to run for a century.

So why not just allocate out of free virtual memory and never bother freeing? Why am I worried about freeing stuff all the time? Why not just allocate it? It's virtual memory. It's virtual. I just allocate more storage as I need it. Yeah?

AUDIENCE: [INAUDIBLE] memory?

CHARLES LEISERSON: So it is tied to real memory, and that's part of the answer. It is tied to real memory, because anything that you write must find itself a place on disk. Because it's going to be written to pages, and those pages are going to be paged and then written to what's called the swap space on disk, which allows you to put things there.

Even so, though, writing to disk-- it's still not going to take very long. I mean, you're never going to be able to use up disk, because disk takes so long to write for. But that is part of the answer.

AUDIENCE: So you need to basically use [INAUDIBLE] software? And so--

CHARLES LEISERSON: And the page table.

AUDIENCE: [UNINTELLIGIBLE] on the disk.

CHARLES LEISERSON: So the issue is, this is where the external fragmentation would be horrendous. If you just kept writing across memory, and you only had a couple of words written on every page that were actually used, the rest is just garbage, then what you're saying is that in order to access a given datum, I'm not going to have it in memory with it a lot of other stuff. What I'm going to have in memory with it is going to be all garbage, this stuff that I would have freed. And so therefore, you end up having your program distributed across a large region of virtual memory, and that means as we're doing the paging, et cetera, we're getting very poor page locality as I'm accessing things.

And so I have a finite number of pages that I fit in memory, and now I may be blowing that out, and every access, rather than going to something nearby, is instead going to disk. And so that can cause disk thrashing, where every access causes a disk access. So how fast are disk accesses? Order of magnitude? How fast are disk accesses? What's a disk access cost? Fast disk?

AUDIENCE: [INAUDIBLE]

CHARLES LEISERSON: No, it's slower than that. Disk access. So good memory access, DRAM memory is, like, 60 nanoseconds for DRAM access. It's like 60 nanoseconds. So the processor is going somewhere around, these days, between 3 and 4 gigahertz. So memory is like 200 processor cycles away.

How fast is disk? An ordinary disk, not a solid state device. Ordinary disk.

10 milliseconds is a good ballpark to give. That's a good number to know. 10 milliseconds, not 10 microseconds, not 10 nanoseconds. 10 milliseconds. So you are, what? 6 to 7 orders of magnitude slower if you go to disk. Really slow. So one of the things in performance optimization-- we're focusing a lot on the CPU-- is often the case for database things. How do you minimize those disk accesses? And all the things that we're learning about, performance for CPUs, can be applied to disk accesses, as well.

So the idea here is we want to avoid-- that's why we don't want to have our stuff strewn across a whole bunch of pages where we only have a fixed amount of internal memory, because then we'll end up paging all the time. Every access will cause us to do a disk access, because it won't be very likely that we're getting stuff on the same page. So getting back, for example, to the fixed allocation, that's why it's a good idea to try to keep those pages full, so that we have a small thing.

So the goal of most storage allocators is therefore to use as little virtual memory as possible, and try to keep the used portions very compact. Because the more compact it is, the more likely it is that when you access one, you'll be able to access something nearby without causing a page fault. So that's the goal of almost all storage allocators. Try to keep things compact, try to keep things tight and in very little virtual memory space, even though virtual memory is huge.

Now, that doesn't mean that everything has to be contiguous. For example, in this organization as I showed before, actually, I have it just on the previous slide, so why don't I go back there-- these are very far away in memory, but there's only a couple of them, and they're tight and local in the regions that they're operating in. What I don't want to do is have it where I have just little stuff. It's OK to go into the middle of virtual memory and have a whole bunch of consecutive pages full of data. That's fine. What's bad is having just a little bit of data, a little bit of data, and a little bit of data sort of sprinkled all over. That's bad. But as long as I can put a whole bunch of data nearby, it doesn't matter where in virtual memory I put it, particularly. So the idea is, let's try to use as little virtual memory as possible. Any questions about that? OK.

So how do we do that? So the first thing is, binned free list turns out to do a really good job of using relatively little memory. So suppose that the user program is using, at most, m memory at any point during the execution. Maybe allocating, maybe freeing. But if I took the high watermark of everything he says that he needs, we call that m.

Then it turns out that if you use a bin free list allocator, the amount of virtual memory consumed is at most m times log n. Order m times log n. And why is that? Here's kind of a hand-wavy proof sketch. So whenever I make a request for a block of size x, it may consume at most twice x storage that's actually in use.

And so it turns out that the total amount of memory devoted to blocks of size 2 to the k is therefore at most 2m. So for any given size, I might, at any given point in time, have the program using all blocks of that given size. Let's say size 2 to the k. And since there are a logarithmic number of those bins, each bin of which could have a size at most m, the total is order m log m.

Now it turns out that there's actually a stronger property. In fact, you can show that binned free list is order one competitive. Who knows the term "competitive" in this sense? The algorithmic meaning of "competitive"? So what it means is, it's at most a constant factor slower than the optimal omniscient algorithm that could figure out exactly what's going on in every step. So you're within a constant factor competitive with the optimal allocator, assuming that the optimal allocator does not do what's called coalescing. If it simply is dividing up things, then it can subdivide, but it can never glue things back together again. It turns out that in fact, binned free list is really much better than that.

So let's talk a bit about coalescing, because many, many memory allocators use coalescing. So the idea is, I'm concerned about the case where I end up with a whole bunch of little small objects, and yet the allocations now come from big chunks. And so I have to go and use more of my virtual memory, or maybe I can glue together those little pieces into a big chunk, and not have to go off to memory. So that's what coalescing is all about.

So the idea is, if two things are adjacent, let me glue them together. And there are really clever systems for doing that, including a very famous one called the buddy system. But using any of these mechanisms for coalescing, you always have to trade off between the simplicity of not coalescing. Not coalescing so stick it in the free list, I'm done. Coalescing is, stick it in the free list, oh, am I next to somebody I can merge with? Now let me merge. So you always have to weigh, is that really going to be an advantageous thing?

It turns out there are no good theoretical bounds that I'm aware that prove the effectiveness of coalescing. People have not figured out how to analyze this, or what an appropriate model is for understanding whether coalescing is a good idea. However, it does seem to work in practice for some things, and I think the reason for that-- and this is just my own take on it-- is that storage tends to be deallocated as a stack. So you tend to go into a region of a program where you do a lot of heap allocation-- let's say I'm building up a graph, and I keep doing a whole bunch of mallocs. And when I'm done with the graph, what do I do? I free it all. So I tend to have these batch allocations and then deallocations, and they tend to work often as a stack, often in a batch. And that means that when I do that freeing, I can sometimes take advantage of that and do coalescing.

In some systems, people have what are called memory pools, where you have a region where when you allocate out of it, you know you're allocating out of your own particular region specifically so that you can do coalescing of things in that region. And so you may have multiple allocators, and you'll call the allocator on the region, say, for a graph, because let's imagine you're creating two graphs, and one of them you're going to keep around, and the other you're going to dump. Well, if I create them at the same time, the storage gets all intermixed, I'll deallocate one graph. The other one will be, I'll have fragmentation all over. These little spots appeared where the other graph used to be-- not so good.

But if I say, use this allocator for this graph, use that allocator for that graph, then when I'm done and I'm allocating those out of two different regions of memory, that when I'm done with the first graph and deallocate everything, now I've created a nice big piece of contiguous storage. So people will sometimes play with different memory pools for allocators, and C++ has a fairly rich set of libraries and so forth for that, to allow you to do that kind of thing.

So let's go on and talk about garbage collection, because that's something you've probably seen in Java and Python, and what's going on under the covers there. Because even though C and C++ do not have garbage collection, you can actually implement your own garbage collector. It's not that hard, except for some details. It's only hard when you want it to perform well.

So the idea is to free the programmer from freeing objects. The downside, by the way, is when you free the programmer from freeing objects, they tend to create lots of garbage. And so what happens in Java programs, for example, is you'll have these-- I remember once having a Java program where, when it was sitting, waiting for user input, it would periodically go, garbage collect. It was just sitting there, and then it would go, garbage collect. We weren't doing anything. It was just periodically, the way they had written the code, they were somehow generating garbage, doing allocations and frees, when you were sitting around doing nothing. And so it would basically use up all of the space and then do a garbage collection, use up all the space and do a garbage collection. Naturally that's kind of a slow system.

And so even in something like Java, where you have a garbage collector, you do have to worry about whether you're creating garbage or not, and you can generate faster code by not creating as much garbage, or by finding ways of recycling the garbage yourself. So even in Java, it's worthwhile saying, give me a big chunk of memory, let me manage it myself, now let me free it. I don't have to free it, but when I'm done with it, I'll be done with it. But in the meantime, let me manage it myself so that I don't have it continually generating more garbage.

So the garbage collector can be built in, or you can do it yourself. And mostly in this class, we do stuff ourselves, so we know how they work. So that even when you're doing it with built in, you know what's going on under the covers.

So here's some terminology. The roots in garbage collection are the objects that are directly accessible by the program. And those typically are things like globals, stacked variables, and so forth. Things that the program can directly address by name. The live objects are reachable from the roots by following pointers, and the debt objects are the ones that the programmer can never get to again. And so what we're interested in doing in garbage collection is finding all the dead objects and recycling them so we can use them again, so that they don't end up taking up space in our virtual memory and slowing us down by making our virtual memory space continue to grow and grow and grow across many disk pages.

So one of the questions, in order for one to be able to do garbage collection, is how do I know where pointers are in objects? How do I know what's accessible? And so typically, in order to do that, you have to have a very strong idea of where is a pointer and where is data? Because you don't want to follow data as if it's a pointer, you want to know which regions of a given block of memory are pointers. And so strong typing helps a lot with that, which is why mostly people do garbage collection in strongly typed languages, because then you can actually identify it.

It also requires you to prohibit pointer arithmetic. So the ability to take a pointer and then index off that pointer means I can get to some piece of storage that I can't access directly. And so typically in these kinds of systems, pointer arithmetic is illegal. So you can do array indexing, as long as you treat the array as a single object. You can't treat an array as the component pieces. You have to treat it as a single object. So pointer arithmetic.

Well, you folks, who's familiar with pointer arithmetic? At this point, almost all of you have done stuff with pointer arithmetic, if your code is any good. Almost all of you have done adding to pointers in order to move through an array or what have you as being somewhat cheaper than doing an array index every time. It saves you an extra memory reference, typically. So that's one of their restrictions when you want to do garbage collection in a general setting.

Now probably the simplest and most useful one as a programmer to use for garbage collection is what's called reference counting. I know many, many programs where what you do is you do reference counting because you can. But as you'll see, it has a limitation.

So the idea in reference counting is to keep a count of the number of pointers referencing each object. So here we have a whole bunch of different objects of different sizes, and whenever there's a pointer going to the object, I add one to the reference count shown in green at the top here. I hope I got the count right. Whenever the count drops to zero, you free the object. That means nobody's pointing to this object [UNINTELLIGIBLE]. Let's free it. So let's see how that goes.

So suppose that, for example, that pointer got moved to over here. Went away from there, got moved to there. Well, now there are two fields that have to be updated. The one that we took the pointer away from and the one that we put it to.

So we update those. Oops, that guy went to zero. That means this stuff is garbage. But I can't just immediately go and collect that piece of storage. Why not?

AUDIENCE: To save the value of the pointer, and [INAUDIBLE] pointer later?

CHARLES LEISERSON: Yeah, to save the value of the pointer--

AUDIENCE: To save the value of the pointer and then use it again later. The pointer isn't there, but you know where--

CHARLES LEISERSON: Not quite, not quite. Yes?

AUDIENCE: [INAUDIBLE] the reference counts for [INAUDIBLE], and also--

CHARLES LEISERSON: Yeah. If you're going to deallocate this guy, he's got pointers. They have to follow his pointers and see which ones need to be deallocated. Good.

So this guy now, we realize, is garbage. But before we can just free him, I've got to deallocate these pointers. So this guy also got decremented, but he didn't. So this guy, we have to decrement these two fields, because we took away those pointers because we're going to garbage collect that. And when we do that, it turns out this thing turns to garbage, as well. So when you do reference counting, you have to make sure that when you decrement things and decide you're going to free something, you also then go through and free the things, and you continue that process until everything that has been deallocated can be deallocated. And then this becomes recyclable, put back into the storage, into your free list or whatever scheme you have.

So this is a very simple way, when you're implementing data structures, of building your own dynamic storage. Because you basically can allocate stuff, you can free it, and whenever the reference count goes to 0, boom. You just have it deallocated. OK? But you can program that yourself [INAUDIBLE].

Now there's a problem with the scheme, though. What's the problem? Cycles. Yeah, cycles are the problem for reference counting. So here's an example. The problem is that a cycle is never garbage collected. Let's take a look at this structure here. So everybody's got a pointer from somewhere, but it turns out that there's a cycle here where, notice that they are pointing to each other, we even have a guy coming off here, but nothing points into the cycle.

So there's no way that, from the roots, I can ever access these four guys. They're garbage, but my reference counting scheme will never find it, because no one's ever going to decrement one of those counters here. So those are all garbage, but how would I know? And that's basically the disadvantage of reference counting. And uncollected garbage, as we all know, stinks. So we don't want that situation to occur.

So nevertheless, reference counting is great if you've got an acyclic structure. And the number of times acyclic structures come up where reference counting is the simple and easy way to do the storage management is huge when you get into doing any kind of interesting programming. Lots and lots of opportunity to use reference counting approach. So questions about reference counting? You have a question? Yeah?

AUDIENCE: So this means you actually have to do this [UNINTELLIGIBLE] ever do a [INAUDIBLE]? You can, like, save the work--

CHARLES LEISERSON: That's right. Whenever you move a pointer, if you're going to move a pointer, you first decrement the counter on the place you're moving it from, and then garbage collect if need be, and then you move the pointer-- that's typically what you do-- you move the pointer and increment the count at that location, which typically doesn't require any extra work to decrement, which is the trick thing. Sorry?

AUDIENCE: There's no good way to do this in idle time. Like I'll have some idle time later, let me do it--

CHARLES LEISERSON: So that's what you use true garbage collection for Yeah?

AUDIENCE: You could actually, there's techniques where you can defer the work of incrementing it. Like you basically put the increment and decrement operations that you'd like to do into a buffer, and then you can do them later. But it's sort of the same.

CHARLES LEISERSON: Yeah, you could log. So there are schemes where you log that, oh, I decremented this, better check it later, type thing. Or I decrement it to 0. Let me just put it in there, and I'll go actually do the stuff in batch mode at some later time. Yep. So programming is fun, because you can be clever with schemes like that. And that could be quite a reasonable thing to do, which is wait until you're waiting on the user for input, and at that point, collect your garbage.

Now, the thing you have to risk is, what happens if that takes a long time to get there? And then there's the complexity of your algorithms and so forth. So you get into a lot of issues of software maintenance and so forth. But that can be a very effective thing, is take it out of the critical path of the actual complication you're doing, if that's mission critical, and put it in some part of the computation where it isn't mission critical. Very good idea.

I'm sorry?

AUDIENCE: For thread safety. Like if you have shared reference accounts, like multiple threads updating the counts, you don't want to have them all in the same count at the same time. So they fill it like a thread local buffer, and--

CHARLES LEISERSON: I see what you're saying. Yes. So we're going to get into that as we deal with multi-threading and so forth, that if you've got two guys that are operating on the same data structure, you have to be very careful about the consistency, and that one doesn't do something while the other is doing something else, and so forth. So for that reason, it can be a good idea to postpone operation.

So general garbage collection is based on a graph abstraction. So let's move into the graph world. So I can view objects and pointers as forming a directed graph G equals VE, where V are the objects and E are the pointers from one object to another. The live objects are reachable from the roots. And now the thing is that finding all of the live objects is then like doing a graph search. The common ways of doing a graph search are depth for search and breadth-first search. It turns out we're going to use breadth-first search for reasons that I'll show you.

So the idea in breadth-first search, to find all the objects. So the idea is, what we're going to do is go through and find, where are all the objects in my system that are reachable from the root? And then anything that I didn't find, ah. That must be garbage.

So here's the idea. What we're going to use is FIFO queue called Q. So that's a first in, first out. It has a head and a tail. And the mechanism is very much like a stack. Whenever I need to enqueue something, I put it on the tail end and increment the tail pointer. Whenever I need to take something off the queue, I increment the head pointer. So I'm always doing head and tail, pushing on the tail and pulling off the head. Enqueue and dequeue.

So the way that this works is, I basically go through all my objects, and if they're a route, I put a mark on it. And I enqueue-- this is pseudocode, by the way, so don't hold me to my C programming standard here. And I market, and then I enqueue it on the queue Q. And otherwise, I just simply mark it as zero.

So being marked means whether we've discovered it's reachable from a route. So to begin with, what we're going to do is mark everything that's reachable from the routes, and we're going to enqueue them on the FIFO. And everything else, we're going to mark as unreachable, for now. And now what we're going to do is see which ones we discover.

So what we do is, while the queue is non-empty, we're going to take something off of the queue and then look at all of its neighbors. So all the edges UV, look at all the things that go from U to V And if it's not marked, then we're going to mark it and put it on the queue. That's it. If it's already marked, we don't have to do anything with it, but if it's unmarked, we'll mark it and do it. And then in the end, everything that's marked is going to be something that is live, and everything that is unmarked is going to be things that are dead. But it's going to be actually even sexier and more clever than that, as you'll see.

So here's an example. We start out with my queue being empty, with head and tail pointing to the same place. And what we do, is we go through and we first-- in this case, I have just one route R, right here. So we basically mark R and put it on, and that's how we get started. And everything else is unmarked up here. So I'm going to mark with green up here, and I'm going to show what's on the queue with the green down here, and dark green when I pop something off.

So the first thing I do is I say, OK. Let me dequeue something. And in this case, I'm dequeueing R, because it's the only thing on there. And what I do now is I visit all the neighbors. So I visit b, so I put that on, mark it and put that on, and then I visit c, so I visit and put that on. Then I'm done with R. I visited all its neighbors. So then I go and I dequeue B, and now visit all its neighbors. Well, in [? visit ?] c, there's nothing to be done there, because c is already marked. So I don't enqueue it again. I already have observed that. And there's nothing else adjacent to b, so basically, I then go on and dequeue c. So I have c here, and now we go visit its neighbours, and the neighbors are d and e. So we enqueue d, enqueue e, and now we are done dequeuing all his neighbors and marking them, so now we pop off d. And now d has no neighbors, so there's nothing to be done. And now I dequeue e, and basically, e has one neighbor f, so we mark f. Then we dequeue f, we mark g, then we dequeue g, and g has-- the only neighbor is e. And now my queue is empty, and so I'm done. And now you'll see, what did I do? I marked all the things that were reachable from r, and everything else is garbage.

Now, that's only half the story. So I've marked them, but more relevantly, it turns out for garbage collection, is look at what I've got in my queue. What do I have in my queue? I've got all the live objects, got put in the queue as we were going along. All the live objects are put in the queue. And not only that, but notice they're all adjacent to each other in the queue. They're compact in the queue.

And so the idea is, we're going to take advantage of that property. That when I do breadth-first search, that the queue, when I'm done, I put every object in there. So all the live vertices are placed in contiguous storage in queue.

So this is the principle behind the copying garbage collector, which is one of the more popular garbage collectors that's used. The idea is that as I'm executing, I'm going to have some live objects and some dead objects, and I'm going to have some place that I'm doing my next allocation at. And as I go through, I allocate more things in what's called the from space, and I keep allocating, and I may do a deallocation, and maybe some more allocation, and maybe another deallocation, and some more allocation, and eventually I run out of my memory that I've assigned for my heap storage.

So when I get to this point, now what I do is I'm going to do breadth-first search on the from space. And what I'm going to do is I'm going to copy using the live storage here, using [UNINTELLIGIBLE] to a TO space, where the TO space implements the FIFO queue. So here's the TO space, and when I go through and I walk all that, I've now copied all of the values, all of the objects down to this space over here. And now I have this amount of storage left. I make the two space exactly the same size as the from space. I have to make it at least that big, because I know that it may be that everything here is alive. Maybe nothing got deallocated. So I have to make sure that the TO space is large enough to handle everything that might be alive there, and it may be everything. So the TO space is generally allocated to be exactly the same size as the FROM space. So basically, the way I compacted is I copied things down using the breadth-first search algorithm.

Now there's one problem with this, and that is that we have pointers in these objects. And when I copy it down here, how did I make sure that all the pointers now point to the new locations? So here's how you do that. So since the FROM address of the object is not generally equal to the TO address, pointers must be updated. So I have to know, of course, where the pointers are. So the idea is that when an object is copied to the TO space, what we're going to do in FROM space is store a forwarding pointer in FROM space saying, I copy you to here. So that whenever I look at something in the FROM space that's already been copied, I can say, oh, here's where it's been copied to. It's been copied to this region in the TO space.

When an object is removed from the FIFO queue in the TO space, we're going to update all its pointers. I'm going to give you an example of this. So if you think about it, when I remove something from the FIFO queue, at that point, I know all of its adjacent vertices have been placed into the queue, into the TO space. And so at that point, I know where all the final destinations of pointers are.

So here's an example. So suppose that I'm doing the copy of these guys, and I've got all the pointers around here, and now some of these guys have already made it into the FIFO queue. Let's say this guy has made it in, because I've got forwarding pointer to there, and this guy has made it in, so I've got forwarding pointer to there. And this is what my queue currently holds.

So when I remove an item from the queue, I then look at this guy, and what I do is first of all, I visit all of the adjacent neighbors. So I visit this guy and I say, oh, he's already been copied, because I have a forwarding pointer. So nothing to be done there. But this guy, he hasn't been copied yet. So I have to make sure that that guy is copied by enqueuing the adjacent vertices.

So I copy my neighbor to there. And part of copying is, if I'm actually making a copy, I'm going to have a pointer from here. The pointer now has to point to this guy over here, because that's how I'm copying. Otherwise it isn't a copy. So in the copy, this pointer is the same as that one, is pointing to the same storage location, namely this place over here. So I do that for all my adjacent vertices. People follow so far?

Now I've copied all of the neighbors for this guy. Now this guy, all of his neighbors are in their final destinations in the TO space. So that means I can now update these pointers to be the pointers in the TO space. So I do that as follows. Oh, first of all, I forgot to place the forwarding pointer here to indicate that he's been copied. That's how I'm doing the marks. Sorry about that.

So I copied the values, and that's just a straight copy, because all the pointer values are now pointing into this space. Then I put a forwarding pointer in. Now I'm ready to do the update of the pointers here.

So I update the pointers to the removed item. So here's one pointer. I get rid of him there and I make him point to the forwarding thing. He used to point to here, now he's pointing to here. This one used to point to here, I want to make him point to there. There we go. So now he's pointing here. And now this guy here now has his pointers to his final destinations. And so the invariance that I'm maintaining is that everything before the head of the queue all have their final pointers in place in the TO space. Question?

AUDIENCE: So do you know where to put the pointers-- how do [UNINTELLIGIBLE] pointers go from the TO space to the TO space so that they're actually getting to their final destination? So my question is, [UNINTELLIGIBLE] by following [UNINTELLIGIBLE]. But how do you know what pointer to follow? How do you know that [INAUDIBLE]?

CHARLES LEISERSON: If I have a pointer to something in the FROM space, then I update it to be the forwarding pointer in the TO space.

AUDIENCE: Yeah, the question is, how do you know that [INAUDIBLE] it's being pointed to [? the ?] actually in the TO space? Can you check, is this in the TO space?

CHARLES LEISERSON: Well, yeah. I mean, you know what the range of the two spaces is, so you could check that. OK. Question?

AUDIENCE: So the reason you would be using [? BFS ?] instead of [? DFS ?] is because [INAUDIBLE].

CHARLES LEISERSON: The reason you're using BFS is because it has this nice property that the queue represents the copying of all the values. If I use DFS, I need a stack to do it, and then I'm overwriting the locations where the vertices would be stored in the data structure. By using DFS, there's this just clever thing that the queue, when I'm done with DFS, represents all of the visited vertices. It's very clever. Wish I'd thought of that. But I was, I think, only 10 years old or something when they invented this, and I wasn't that precocious. OK? So very, very clever.

So this is the scheme behind the Minsky, Fenichel, Yochelson garbage collector. So Marvin Minsky, you may have heard of, was one of the inventors of a scheme like this. One reason he's so famous.

Good. So this is really nice, because it takes me linear time to copy and update all of the vertices. It's just one pass of breadth-first search. Very fast to copy and update all the vertices. Sometimes, by the way, people call these forwarding pointers, they call them broken hearts. You'll see that in the literature. They call them a broken heart if it's-- I don't quite remember why that's the-- it's like, I went there, oh, you're not there, it's over there-- it's like, I don't know.

Now, the last thing you have to do-- and this is something that's not talked about a lot in the literature-- is how do you manage these from and to spaces in virtual memory? So here's the strategy that works well. So basically, typically what I'm doing is after I've done a copy, I've emptied everything out of the FROM space, and I've got a portion that's used in the TO space, and then an unused portion that I'm going to be able to do allocations out of for the next pass around.

So what I do at that point is I make the old TO become the new FROM. So here's the new FROM. This is available space. And now what I do is I extend this space, if necessary, to make the new heap space that I'm going to allocate out of equal in size to the used space. And the reason for doing this is to amortize the cost of going through all of the storage, so that the cost of going the garbage question can be amortized against what's used. If I didn't extend this, I might have only freed up a tiny little bit of space, and now when I run out of that space, I'll run the garbage collector going over all the elements, but I'll only have gotten a few elements [INAUDIBLE]. So what I want to do is make sure that I've done at least a certain number of allocations proportional to the number that I'm going to have to walk over in order to finish it. So that's why you make this be the same size.

Now when the space runs out, the new TO is allocated with the same size as FROM. So if you can allocate it back in the old FROM space here, that's great. If there's room here to allocate it at the front, super. That's where you allocate it. But as in this example, the FROM space here is bigger than the space that goes up to here. So therefore what I do is I allocate it at the end, and then I go through and copy to here.

Now if this ended up being used, then the next time around, I would be able to reuse this space at the front for the next time that I flip the garbage collection. The idea is, if things will fit as early in memory as possible, that's where I put them. But if they don't fit, I put them later. And you can prove, if you do this, that the virtual memory space used is order 1 times the optimal, where you're getting nearly the best virtual memory space used compared to the amount of items that are in use by the user at any time. So it keeps everything sort of compact towards the front end here.

So in this case, I had to allocate to here. If the use had been very small, I could have allocated just a very small amount for the heap. When I was done, the new TO space would have fit at the beginning. I would have allocated it at the beginning. And it's a nice little exercise, for those of you who are analytically inclined, to figure out what the constant is here, because it's a little bit of a game to figure out what the amortized constant, for those who have some algorithmic energy and our confident about working in this kind of thing instead of studying for the quiz.

So dynamic storage allocation. Hugely interesting area. Lots is known and lots is unknown. For example, I mentioned coalescing. We really don't understand coalescing. Disaster people have been doing it for 50 years or more. We do not understand coalescing.

There are a lot of strategies that people have studied, things like the buddy system, I mentioned. There are other types of garbage collectors called mark and sweep garbage collection. Those are typically inferior to the copying garbage collection, because they leave you with a memory that isn't compact. And so you end up using more and more of your virtual memory, because you're leaving things in place. On the other hand, you leave things in place so there's no need to move pointers. So that can be a valuable thing.

There's things called generational garbage collection. Generational garbage collectors realize that things that are allocated typically are freed very soon thereafter, and so therefore, if you especially take care of the things that are allocated and guess that they're going to be used again, and that things that haven't been deallocated for a long time are very unlikely to be allocated, you can get better performance out of a garbage collector.

There's real time garbage collection. So this is what I've explained, is what's called a stop and copy garbage collector. Meaning that when you run out of storage, oops! Stop the world, run the garbage collector. There are real time garbage collectors which, whenever you do an allocation, it goes and it does a few cycles of garbage collection, and is such that you can always keep up with the allocations and deallocations going on. By doing a few cycles, you eventually get to garbage collect everything, but you do it intermixed, so that you're slowing everything down by a constant amount, rather than running and suddenly, OK, your robot arm is going like this, oops, garbage collect, garbage collect, garbage collect, [CREAKING SOUND]. So you can have your robot arm move smoothly, because it's paying the price all the way. It's not stopping and going.

There is multi-threaded storage allocation. What happens when you have a multi-threaded environment where you have many processors that are all allocating out of the same pool? How do they organize it? You can have the interesting things going there where things are allocated on one processor, and then get deallocated on another. And if you're not careful, you have something like a memory leak going on, which is called memory drift, where the memory gets allocated on one, and one guy's busy creating it, and it keeps getting deallocated on the other, but never gets recycled back to the guy who needs it. And so how is it that you figure out to make sure that memory drift doesn't end up having you grow your memory? In fact, many, many garbage collectors that are out there have this memory drift problem for multi-threaded storage allocation.

There's also parallel garbage collection. How do you, in a parallel environment, have the garbage collector going on as a separate process while you have other things going on at the same time? So a huge, huge, huge area of storage. And for our next lab, lab 3, you'll be building your own storage allocator. So you have a lot of fun with this. [? Reid ?] says this is his favorite lab.

Great. Any questions? OK. Good luck on the quiz next time.