6: Sample Size and Power Calculations

Related Resources

Lecture slides (PDF)

• Download this transcript - PDF (English - US)

FEMALE SPEAKER: The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high-quality educational resources for free. To make a donation or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: OK, so my name is Ben Olken and we're going to be talking about how to think about sample size for randomized evaluations. And more generally that the point of this lecture is not just about sample size but we've spent a lot of time, like in last lecture, for example, thinking about the data we're going to collect. Then the question is, well, what are we going to do with that data? And so it's about sample size but also more generally, we're going to talk about how do we analyze data in the context of an experiment. OK.

So as I said, where we're going to end up at the end of this lecture is, how big a sample do we need? But in order to think about how big a sample we need, we need to understand a little more about how do we actually analyze this data. When we say, how large does a sample need to be to credibly detect a given treatment effect, we're going to need to be a little more precise about what we mean by credibly and particularly think a little bit about the statistics that are involved in thinking through-- evaluate-- understanding these experiments.

And particularly, when we say something that's credibly different, what we mean is that we can be reasonably sure, and I'll be a little bit more precise about what we mean by that, that the difference between the two different groups-- the treatment and control group-- didn't just occur by random chance, right? That there's really something that we'll call statistically significantly different between these two groups, OK? And when we think about randomizing, right?

So we've talked about which groups get the treatment and which get the control, that's going to mean that we expect the two groups to be similar if there was no treatment effect because the only difference between them is that they were randomized. But there's going to be some variation in the outcomes between the two different groups, OK?

And so randomization is going to remove the bias. It's going to mean that the groups-- we expect the two different groups to be the same, but there still could be noise. So in some sense, another way of thinking about this lecture is that this lecture is all about the noise. And how big a sample do we need for the noise to be sufficiently small for us to actually credibly detect the differences between the two different groups, OK?

So that's what we're going to talk about is basically, how large is large so we can get rid of the noise? And let me say, by the way, that we've got an hour and a half, but you should feel free to interrupt with questions or whatever if I say something that's not clear because there's a lot of material that we're going to be going through pretty quickly. OK.

So when we think about how big our sample means to be-- remember, the whole point is how big does our sample have to be remove the noise that's going to be in our data? And when we think about that, we think essentially about how noisy our data is, right? So how big a sample we need is going to be determined by how noisy is the data and also how big an effect we're looking for, right?

So if the data is really noisy but the effect is enormous, then we don't need as big of a sample. But if the effect we're looking for is really small relative to the noise in the data, we're going to need a bigger sample. So actually, sometimes it's the comparison between the effect size and how noisy the data is. It's the ratio between these things that's really important.

Other factors that we're going to talk about are, did we do a baseline survey before we started? Because a baseline can essentially help us reduce the noise in some sense. We're going to talk about whether individual responses are correlated with each other. So for example, if we were to randomize a whole group of people into a given treatment, that group might be similar in lots of other respects. So you can't really count that whole group as if they were all independent observations because they might be correlated. For example, you all just took my lecture. So if you all were put in the same treatment group, you all were exposed to the treatment but you also all were exposed to my lecture and so you're not necessarily independent events.

And there are some other issues in terms of the design of the experiment that we'll talk about that can help affect samples as well, like stratification, control variables, baseline data, et cetera, which we're going to talk about, OK?

So the way we're going to go in this lecture is, I'm going to start off with some basics about, what does it mean to test a hypothesis statistically? And then when we get into hypothesis testing, there are two different types of errors that we're going to talk about. They're helpfully named type I and type II errors. And you have to be careful not to make a type III error, which is to confuse a type I and a type II error. So we'll talk about what those are.

Then we'll talk about standard errors and significance, which is, how do we think about more formally what these different types of errors are? We'll talk about power. We'll talk about the effect size. And then, finally, the factors that influence power, OK? So this is all the stuff we're going to go through, all right?

So in order to understand the basic concepts of-- when we're talking about hypothesis testing, we need to think a little about probabilities, OK? Because all this comes down, essentially, to some basic analysis about probability. So for example, suppose you had a professional-- and the intuition here is that the more observations we get, the more we can understand the true probability that something occurred-- whether the true probability that something occurred was due to a real difference in the underlying process or whether it was just random chance.

So for example, consider the following example. So suppose you're faced with a professional gambler who told you that she could get heads most of the time. OK, so you might think this is a reasonable claim or an unreasonable claim, but this is what they're claiming and you want to see if this is true. So they toss the coin and they get heads, right? So can we learn anything from that? Well, probably not because anyone, even with a fair coin, 50% of the time, they would get heads if they tossed it. So we're really can't infer anything from this one.

What you saw that they did five times and they got heads, heads, tails, heads, heads. Well, can you infer anything about that? Well, maybe. You can start to say, well, this seems less likely to have occurred just by random chance. But you know there's only five tosses. What's the chance that someone with an even coin can get four heads? Well, we could calculate that if we knew the probabilities. And it's certainly not impossible that this could occur, right?

And now, what if they got 20 tosses, right? Well, now you're starting to get information, although in this particular example, it was closer to 50-50. So now you have 12 versus eight. Could that have occurred by random chance? Well, maybe it could have, right? Because it's pretty close to 50-50. And now, suppose you had 100 tosses or suppose you had 1,000 tosses with 609 heads and 391 tails, right?

So as you're getting more and more data, right, you're much more likely to say something is meaningful. So if you saw this data, for example, the odds that could occur by random chance are pretty high. But if you saw this data with 609 heads and 391 tails out of 1,000 tosses, it's actually pretty unlikely that this would occur just by random chance, OK?

And so this shows you, as you get more data you can actually say, how likely was this outcome to have occurred by random chance? And the more data you have, the more likely you're going to be able to conclude that actually, this difference you observed was actually due to something that the person was doing and not just due to what would happen randomly.

And in some sense, all of statistics is basically this intuition, which is, you take the data you observe and you calculate what is the chance that the data I observe could have occurred just by random chance. And if the chance that the data I observed could have happened just by random chance is really unlikely, then you say, well then it must've been that your program actually had an effect, OK? Does that make sense? That's the basic idea, essentially, of all of statistics is, what's the probability that this thing could have happened randomly? And if it's unlikely, then probably there was something else going on.

Here's another example. So what this example shows is, now suppose you have a second gambler who had 1,000 tosses and they had 530 heads and 470 tails. What this shows is that-- and that's really a lot of data. But in some sense, what we can learn about this data depends on what hypothesis we're interested in. So if the gambler claimed they obtained heads 70% of the time, we could probably say, no, I don't think so, right? This is enough data that the odds that you would get this data pattern if you had heads 70% of the time are really, really small, right? So we could say, I can reject this claim.

But suppose they said that they claim they could get heads 54% of the time, OK? And you observe they got heads 53% of the time. Well, you probably couldn't reject this claim, right? Because this is similar enough to this that if this was the truth, this could have occurred by random chance. So in some sense, what we can say based on the data depends on how far the data is from our hypothesis and how much data we have. Does that make sense as some basic intuition? OK.

So how do we apply this to an experiment? Well, at the end of the experiment, what we're going to do is we're going to compare the two different groups. We're going to compare the treatment and the control group. And we're going to say-- we're going to take a look at the average, just like we were doing in the gambling example. We'll compare the average in the treatment group and the average in the control, OK? And the difference is the effect size.

So for example, in this particular case, in the Panchayat case, you'd look at, for example, the mean number of wells you've got in the village with the female leaders versus the mean number of wells in the villages with the male leaders, OK? So that's in some sense our estimate of how big the difference is. And the question is going to be, how likely would we have been to observe this difference between the treatment and the control group if it was just due to random chance, OK? And that's what we need the statistics to figure out.

Now one of the reasons-- so where does the noise come from? In some sense, we're not going to observe an infinite number of villages. Or we're not going to observe all possible villages. In fact, even if we observe all the villages that exist, we're not going to observe, in some sense, all of the possible villages that could've hypothetically existed if the villages were replicated millions and millions of times. We're just going to observe some finite number of villages.

And so we're going to estimate this mean by computing the mean in the villages that we observed, OK? And if there are very few villages, that mean that we're going to calculate is going to be imprecise because if you took a different sample of villages, you would get a slightly different mean, OK? If you sample an infinite number of villages, you get the same thing every time.

But suppose you only sampled one village. Or suppose there was a million villages out there and you sampled two, right? And you took the average, OK? If you sampled a different two villages, just by random chance, you would get a different average. And sometimes that's where the part of the noise in our data is coming from.

So for example-- sorry. So in some sense, what we need to know is, we need to know if these two groups-- it sort of goes back to the same as before, if these two groups were the same and I sampled them, what are the chances I would get the difference that I observed by random chance? So for example, suppose you observed these two distributions, OK? So this is your control group and this is your treatment group. Now you can see there is some noise in the data, right? This one is a mean of 50 and this one is a mean of 60. And there's some-- these are histograms, right? So this is the distribution of the number of villages that you observed for each possible outcome.

So you can see here that there's some noise, right? It's not that everyone here was exactly 50 and everyone here was exactly 60. Some people were 45. Some were 55 or whatever. But if you look at these two distributions, you could say it's pretty unlikely that if these were actually drawn from the same distribution of villages, all of the blue ones would be over here and all the yellow ones would be over here. It's very unlikely that if these were actually the same and you draw randomly, you get this real bifurcation of these villages, OK?

And where are we basing that idea, that conclusion on? We're basing the conclusion on the fact that there's not a lot of overlap, in some sense, between these two groups. But now, what if you saw this picture, right? What would you be able to conclude? Well, it's a little less clear. The mean is still the same. All the yellows still have an average of 60 and all the blues have an average of 50. But there's a lot more overlap between them. Now if we look at this, we can sort of eyeball it and say, well, there's really a pretty big difference even relative to the distributions there. So maybe we could conclude that they were really different. Maybe not.

And what if we saw this, right? This is still the same means. The yellows have a mean of 60 and the blues have a mean of 50. But now they're so interspersed that is harder to know-- it's possible, if you saw pictures like this, you would say, well, yes, the yellows are higher, but maybe this was just due to random chance, OK?

So what the purpose of these graphs are, is to show you is that in order-- so in both cases, we the same difference in the mean outcomes. It was 60 versus 50 in all three cases, right? But when you saw this graph, it was quite clear that these two groups were really different. When you saw this graph, is was much harder to figure out if these two were really different or if this was just due to random chance, OK? Does that make sense of where we're going?

And so, just to come back to the same theme, all the statistics are going to do in our case is going to help us figure out, are these differences big enough, given the distribution of data we have, how likely is it that the difference we observed could have happened by random chance. And so intuitively, we can look at this one and say, definitely different. And this one, maybe not sure. But if we want to be a little more precise about that, that's where we need the added statistics.

AUDIENCE: Is the sample size the same in both examples?

PROFESSOR: Yeah, the sample size is the same. Yeah, sample size is exactly the same. So you can see that the numbers go down because it's more spread out. All right.

So in some sense, what are the ingredients that we've talked about in terms of thinking about whether you have a statistically significant difference? If you think back to the gambler example, we talked about the sample size matters, right? So if we saw 1,000 tosses, we had much more precision about our estimates than if we had 10 tosses or five tosses. The hypothesis you're testing matters, right? Because the smaller an effect size we're trying to detect, the more tosses we need in the gambler example. If you're trying to detect a really small difference, you need a ton of data, whereas if you're trying to detect really extreme differences, you can do it with less data, OK?

And the third thing we saw is the variability of the outcome matters, right? So the more noisy the outcome is, the harder it is to know whether the differences that we observe are due just to random chance or if they're really due some difference in the treatment versus the control group. OK, so does this makes sense? Before I go on, these are the three ingredients that we're going to be playing with. Do these make sense? Do you have questions on this? OK.

So you may have heard of a confidence interval. How many of you guys have heard of a confidence interval? OK. How many of you can state the definition of a confidence interval? Thanks, Dan. I'm glad that you can. So what do we mean when we say confidence interval? What we mean by a confidence interval-- so let's just go through what's on the slide and then we can talk about it a little more.

So we're going to measure, say, 100 people and we're going to come up with an average length of 53 centimeters. So we want to be able to say something about how precise our estimate is. So we say the average is 53 centimeters. How confident are we or how precise are we that it's 53%? And that's what a conference interval is trying to say.

And a confidence interval, essentially, tells us that with 95% probability-- so we have a confidence interval of 50-56 says that with 95% probability, the true average length lies between 50 and 56. And so the precise definition is that if you had a hypothesis that the true average length was in this range with-- no, I'm going to get it wrong. It says that if you had a hypothesis that the true average was in here, it's within 95% probability that you would get the data that you observe, OK?

A converse way of saying it is that the truth is somewhere in this range, right? You can be 95% certain that the truth is somewhere within this range, So if you did 20 of these tests, only one out of 20 times would the truth be outside your confidence interval, OK?

And so an approximate interpretation of a confidence interval is-- so we know that the point estimate of 43, we have some uncertainty about that estimate. We think the average is 53, but there's some uncertainty. And the confidence interval says, well, it's 95% likely that the true answer is between 50 and 56, if that was the confidence interval, OK?

So why is that useful for us? Well, our goal is to figure out-- we don't care, actually, what our estimate of the program's effect is. We care what the true effect of a program is, right? So we did some intervention. Like, for example, we had a female Panchayat leader instead of a male Panchayat leader and we want to figure out what the actual difference that that intervention made is in the world. We're going to observe some sample of Panchayats and we'll look at the difference in that sample. And we want to know how much can we learn about the true program effect from what we estimated. And the confidence interval basically tells us that with 95% probability, the true program effect is somewhere in the confidence interval, OK? Does that makes sense?

How many of you guys have heard of the standard error? OK. So a standard error is related to the confidence interval in that a standard error says that if we have some estimate, you could imagine that if we did the experiment again, essentially, with a new sample of people that looked like the original sample of people, we might get a slightly different point estimate because it's a different sample. The standard error basically says, what's the distribution of those possible estimates that you could get, OK? So it says that basically, if I did this experiment again, maybe i wouldn't get 53, I'd get 54. If I did it again, maybe I'd get 52. If I did it again, I might get 53. The standard error is essentially the standard deviation of those possible estimates that you could get.

What that means in practice is that-- well, in practice, the standard error is very related to the confidence interval. And basically, a good rule of thumb is that a 95% confidence interval is about two standard errors. So if you ever see an estimate of the standard error, you can calculate the confidence interval, essentially, by going up or down two standard errors from the point estimate, OK?

And the confidence interval and standard error, essentially, are capturing the same thing. They're both capturing-- when I said we need statistics to basically compute, how likely is it that we would get these differences by random chance, those are all coming out in the standard error and the confidence interval, right? They're computed by both looking at how noisy our data is, which is the variability of the outcome, and how big our sample is, right? Because from these two things, you can basically calculate how uncertain your estimate would be. This is a lot of terminology very quickly, but does this all make sense? Any questions on this? OK.

So for example. So suppose we saw the sampled women Pradhans had 7.13 years of education and the men had 9.92 years of education, OK? And you want to know, is the truth that men have more education than women or is this just a random artifact of our sample? So suppose you calculated that the difference was 2.59. That's easy to calculate. And the standard error was 0.54 and the standard error was going to be calculated based on both how much data you had and how noisy the data was. You would compute that the 95% confidence interval is between 1.53 and 3.64, OK?

So this means that with 95% probability, the true difference in education rates between men and women is between 1.53 and 3.64. So if you were interested in testing the hypothesis that, in fact, men and women are the same in education, you could say that I can reject that hypothesis. With 95% probability, the true difference is between 1.53 and 3.64-- so zero is not in this confidence interval, right? So we can reject the hypothesis that there's no difference between these two groups. Does that makes sense?

So doing another example. So in this example, suppose that we saw that control children had an average test score of 2.45 and the treatment had an average test score of 2.5. So we saw a difference of 0.05 and the standard error was 0.26, OK? So in this case, you would say well, the 95% confidence interval is minus 0.55. This is approximately two. It's not exactly two. Minus 0.55-- oh, no, it is exactly two in this example. Minus 0.55 to 0.46, OK?

And here, you would say that if we were introducing the hypothesis that the null hypothesis is that the treatment had no effect on test scores, you could not reject that null hypothesis, right? Because an effect of zero is within the confidence interval, OK? So that's basically how we use confidence intervals. Yeah.

AUDIENCE: Shouldn't the two points of that confidence interval be equidistant from 2.59?

PROFESSOR: From 0.05 you mean? Yeah.

AUDIENCE: [INAUDIBLE]

PROFESSOR: Yeah, I think-- oh, over here?

AUDIENCE: Yeah.

PROFESSOR: So they actually don't always have-- so you raise a good point. So there may be some math errors here. I think a more reasonable estimate, by the way, is that this would have to be minus 0.05 for you to get something like this.

AUDIENCE: But in the first example, if 2.59 is the mean, is the difference--

PROFESSOR: So it's approximately the same, isn't it?

AUDIENCE: I think it's a little skewed--

PROFESSOR: Yeah.

AUDIENCE: On that side, it is 2.64.

PROFESSOR: OK. Yeah, so you raise a good point. So when I said that a rule of thumb is two times the standard error, that's a rule of thumb. And in particular cases, you can sometimes get asymmetric confidence intervals. So you're right that usually they should be symmetric and probably, for simplicity, we should have put up symmetric ones, but it can occur that confidence intervals are asymmetric.

For example, if you had a-- yeah, depending on the estimation, if you have truncation at zero-- if you know for sure that there can never be an outcome below zero, for example, then you can get asymmetric confidence intervals.

AUDIENCE: When the distribution is not normal?

PROFESSOR: Yeah. Exactly. But for most things that you'll be investigating, usually they're going to be--

AUDIENCE: Normal.

PROFESSOR: Yeah, for outcomes that are zero. One [UNINTELLIGIBLE] get non-normal, but yes, in general, they are pretty symmetric. But they might not be exactly symmetric. OK.

So as I sort of was suggesting as we were going through these examples, we're often interested in testing the hypothesis that the effect size is equal to zero, right? The classic hypothesis the you typically want to know is, did my program do anything, right? And so, how do you test the hypothesis that my program-- so we want to know, did my program have any effect at all? And so what we technically want to do is we want to test what's called the null hypothesis, that the program had an effect of nothing, against an alternative hypothesis that the program had some effect. So this is the typical test that we want to do.

Now you could say, actually, I don't care about zero. I want to say that I know-- for example, this is the standard thing that we would do in most policy evaluations that we're going to be doing. It doesn't have to be zero. Suppose you were doing a drug trial and you knew that the best existing treatment out there already have an effect of one. And so instead of comparing to zero, you might be comparing to one. Is it actually better than the best existing treatment? In most cases, we're usually comparing to zero, OK?

And usually, we have the alternative hypothesis that the effect is just not zero. We're interested in anything other than zero. Sometimes you can specify other alternative hypotheses, that the effect is always positive or always negative, but usually this is the classic case, which is we're saying, we think this thing had-- the null is no effect. We want to say, did this program have an effect and we're interested in any possible effect, OK? And hypothesis testing says, when can I reject this null hypothesis in favor of this alternative, OK?

And as we saw, essentially, the confidence interval is giving you a way to do that. It's saying, if the null is outside the confidence interval, then I can reject the null. Yeah.

AUDIENCE: Surely, if we're trying to assess the impact of an intervention, we're always going to think it's positive. Or in general, because-- I gave someone some money to increase their income or not. We've got a pretty good idea it's going to be positive. The probability it's negative is pretty--

PROFESSOR: Why do you--

AUDIENCE: Yeah, why do we change our significance level-- [INTERPOSING VOICES]

PROFESSOR: You ask a great question. And I have to say this is a bit of a source of frustration of mine. Let me give you a couple different answers to that. Here's the thing. If you did that-- if I said I can commit, before I look at the data, that I only think it could be positive, that would mean that if it's negative, no matter how negative, you're going to say that was random chance, OK? So it would require a fair amount of commitment on you, on your part, as the experimenter to say, if I get a negative result, no matter how crazy that negative result is, I'm going to say that's random chance, OK? And typically, what often happens ex post is that people can't commit to actually doing that.

So suppose you did your program and you-- so I actually have a program right now that I'm working on in Indonesia that's supposed to improve health and education. And it seems to be making education worse. Now, we have no theory for why this program should be making education worse, OK? But it certainly seems to be there in the data.

Now, if we had adopted your approach, we wouldn't be entertaining the hypothesis that it made education worse. We would say, even though it's looking like this program is making education worse, that must be random noise in the data. We're not going to treat that as something potentially real. Ex post, though, you see this in the data and you're likely to say, gee, man, that's a really negative effect. Maybe the program was doing something that I didn't think about. And in our case, actually, we're starting to investigate and maybe it's because it was health and education and we're sort of sucking resources away from education into health.

So it requires a lot of commitment on your part, as the researcher, that if you get these negative effects, to treat them as random noise. And I think that, because most researchers, even though they would like to say they're going to do that, if it happens that they get a really negative effect, they're going to want to say, gee, that looks like a negative effect. We're going to want to investigate that, take that seriously. Because most people do that ex post, the convention is that in most cases, to say we're going to test against either hypothesis in either direction.

AUDIENCE: Except that the approach--

PROFESSOR: Does that makes sense?

AUDIENCE: Your issue is do I do this program or not. So it doesn't matter whether the impact of the program is zero or negative. Even if it's zero, you're saying that it's--

PROFESSOR: You're absolutely right. So if you were strict about it and said, I'm going to do it if it's positive and not if it's zero, then I think you were correct that, strictly speaking, a one-sided hypothesis test will be correct and it would give you some more power.

AUDIENCE: So it would give you power.

PROFESSOR: Yeah, it would give you more power.

AUDIENCE: [UNINTELLIGIBLE]

PROFESSOR: Right. And the reason it gives you power is, remember, how does hypothesis testing work? It says, well, what is the chance this outcome could have occurred 95-- what would have occurred by chance 95% of the time? When you do a two-sided test, you say, OK-- where's my chalkboard? Here. You imagine a normal distribution of outcomes. You're going to say, well, the 95% is in the middle and anything in the tails is the stuff that I'm going to [UNINTELLIGIBLE] by non-random chance.

Well, what you're doing with a one-sided test is you're going to say, I'm going to take that negative stuff-- way out there negative stuff-- and I'm going to say that's also random chance. So I'm going to pick my 95% all the way to the left. And that means that the 5% that's not random chance is a little more to the right. Do you see what I'm saying? But it requires that if-- you're committing to, even if you get really negative outcomes, asserting that they're random chance, which is really, often, kind of unbelievable.

The other thing is that, although this is technically the way hypothesis testing is set up, the norms and conventions are that we all use two-sided tests for these reasons I talked about. And so I can just tell you that, practically speaking, I think if you do a one-sided test, people are going to be skeptical because it may be that you, actually, would do that, but I think most of the time, people can't commit to do that. And so the standard has become two-sided tests. But I certainly agree with you. It's very frustrating because one should be able to articulate one-sided hypotheses. That's sort of a long answer, but does that make sense?

It's OK. OK, now, for those of you on this side of the board, you won't be able to see, but maybe if I need to write something on the board it will be better. OK.

So now we're going to talk about type I and type II errors, which, as I mentioned, are not helpfully named. OK. A type I error-- so this is all about probability, so nothing we can ever say for sure. We can always say that this is more or less likely. And there's two different types of errors we can make when we're doing these probabilities or doing these assessments.

The first error, and it's called type I error, is we can conclude that there was an effect when, in fact, there was no effect, OK? So when I said the 95% confidence interval, that 95% is coming from our choice about type 1 errors. So for example-- a significance level is the probability that you're going to falsely conclude the program had an effect when, in fact, there was no effect, OK? And that's related to when you say a 95% confidence interval, the remaining 5% is what we're talking about here. That's the probability of making a type I error, OK?

And why is that? Well, we said there's a 95% chance that it's going to be within this range. That means that just by random chance, there's some chance it could be outside that range, right? So if your confidence interval was over here and zero was over here, you would say, well, with 95% conference, I'm going to assume the program had an effect because zero is not within my confidence interval. However, 5% of the time, the true effect could be over here outside your confidence interval. That's what a 95% confidence interval means. So in some sense, that's what we mean by a-- so that's in some sense what a type I error is. A type I error is the probability that you're going to detect an effect when, in fact, there's not.

And so the typical levels that you may see are 5%, 1% or 10% significance levels. And the way to think about those significance levels is, if you see something that's significant at the 10% level, that means it 10% of the time, an effect of that size could've been just due to random chance. Might not actually be a true effect. And if you've heard of a p-value, a p-value is exactly this number. A p-value basically says, what is the probability that an effect this size or larger could have occurred just by random chance, OK?

So that's what's called a type I error. And typically, there's no deep reason why 5% is the normal level of type I errors that we use, but it's kind of the convention. It's what everyone else uses. If you use something different, people are going to look at you a little funny. So the conventions are we have 5%, 10%, and 1%, as these significance levels. And you might say, gee, 5% or 10% seems pretty low. Maybe I would want a bigger one. But on the other hand, if you start thinking about it, that means that if you use 10% significance, that means that one out of every 10 studies is going to be wrong. Or if you had 10 different outcomes in your data set, one out of every 10 would be significant even just by random chance.

So the other type of error is what's called, as I said, helpfully, a type II error. And a type II error says that you fail to reject that the program had no effect when, in fact, there was an effect, OK? So this is, the program did something, but I can't pick it up in the data, OK? And we talk about the power of a test. The power is basically the opposite of a type II error. A power just says, what's the probability that I will be able to find an effect given that the actual effect is there, OK?

So when we talk about how big a sample size we need, what we're basically talking about is, how much power are we going to have to detect an effect? Or what's the probability that given that a true effect is there, we're going to pick it up in the data, OK? So here's an example of how to think about power. If I ran the experiment 100 times-- not 100 samples, but if I ran the whole thing 100 times-- what percentage of the time and in how many these cases would I be able to say, reject the hypothesis that men and women have the same education at the 5% level if, in fact, they're different, OK?

So this is a helpful graph which basically plots the truth and what you're going to conclude based on your data, OK? So suppose the truth is that you had no effect and you conclude your no effect, OK? Then you're happy. If there was an effect and you conclude there was an effect, you're happy. So you want to be in one of these two boxes. And the two types of errors you can make-- so one type of error is over here, right? So if the truth is there was no effect, but you concluded there was an effect, that would be making a type I error, OK? And this is what we talked about size. So this one, we normally fix this one at 5%. So it's only 5% of the time-- if there's no effect, 5% of the time you're going to end up here and 95% of the time you're going to end up here. That's what a 95% confidence interval is telling you.

And the other thing is, suppose that the thing had an effect but you couldn't find it in the data, OK? That's what's called a type II error. And that's, when we design our experiments, we want to make sure that our samples are sufficiently large that the probability you end up in this box is not too big, OK? So that's a sense of what we mean by the different types of mistakes or errors you could make. Yeah.

AUDIENCE: It's kind of a stupid question. So power is the probability that you are not making a type II error?

PROFESSOR: Yes.

AUDIENCE: So then power is the probability that you're in the smiley face box, that you are-- [INTERPOSING VOICES]

PROFESSOR: Yes. Power is the probability you're over here. Yeah, we say power is related to type II errors. Power is over here. This is the power. Power is conditional on there being an effect. What's the probability you're in this box, not this box? Probably should say one minus power to be clearer. OK? Does that makes sense? All right.

So when we're designing experiments, we typically fix this at conventional levels. And we choose our sample size so that we get this, the power, or the probability that you're in the happy face box over here to a reasonable level given the effect size that we think we're likely to get, OK? OK.

Now, in some sense, the next two things, standard errors, are about this box, size. And power is about this box, or these boxes. Yeah.

AUDIENCE: Why is power not also the probability that you end up in the bottom right box as opposed to the bottom left?

PROFESSOR: Because that's size.

AUDIENCE: Isn't size also linked to-- or power also linked to--

PROFESSOR: No, they're all related, but we typically-- they're related in the following way. We assert a size because when we calculate our standard error-- our confidence intervals, we pick how big or small we want the confidence intervals to be. When we say a 95% confidence interval, we're picking the size, OK? So this one, we get to choose.

AUDIENCE: So it's not sample size, it's size of the confidence interval?

PROFESSOR: No. Yeah, this is size is a-- yeah, it's the size of the confidence interval. That's right. Sorry, it's not the sample size. That's right. It's called the size of the test in yet more confusing terminology. That's right. This is the size of the confidence interval, essentially. And this one you pick, and this one is determined by your data. OK? All right.

OK, so now let's talk about this part, which is standard errors and significance. It's all kind of related. All right, so we're going to estimate the effect of our program. And we typically call that beta, or beta hat. So the convention is that things that are estimated, we put a little hat over them, OK? So beta hat is going to be our estimate of the program's effectiveness. This is our best guess as to the difference between these two groups.

So for example, this is the average treatment test score minus the average control test score. And then we're also going to calculate our estimate of the standard error of beta hat, right? And remember that the confidence interval is about two times the standard error. So the standard error is going to say how precise our estimate of beta hat is, which is, remember, if we ran the experiment 100 times, what will be the distributions of beta hats that we would get, OK?

And this depends on the sample size and the noise in the data, right? And remember we went through this already that here, in this case, the standard error of how confident we would be-- so the beta hat, in this case, is going to be 10, and in this case, it's also going to be 10, right? But here, these two things are really precisely estimated, so our standard error of beta hat is going to be very small because we're going to say we have a very precise estimate of the difference between them. And so the confidence interval is also going to be very small. And here, there's lots of noise in the data, so our estimate of the standard error is going to be larger.

So in both cases, beta hat is the same. It's 10 in both cases. But the standard error is very big here and very small here, OK? Now, when we calculate the statistical significance, we use something called a t-ratio. And the t-ratio-- it's actually often called the student's t-ratio, which I thought was because students used it. But it's actually named after Mr. Student. It's the ratio of beta hat to the standard error of beta hat, OK? And the reason that we happen to use this ratio is that, if there is no effect, if beta hat is actually zero, we know that this thing has a normal distribution, so we can calculate the probabilities that this thing is really or really small, OK?

So we calculate this ratio of beta hat over the standard error of beta hat. It turns out that if t is greater than, in absolute value-- sorry, if the absolute value of t, I should say, is greater than 1.96-- so essentially, if it's bigger than 2 or less than minus 2, we're going to reject the hypothesis of a quality at a 5% significance level. And why is that? It's because it turns out, from statistics, that if the truth is zero, OK? So if we're in the no effect box and the truth is zero, this ratio, it turns out, will have a normal distribution. And it just turns out from a normal distribution that the probability that the 5% confidence interval is 1.96 away from zero if you have a normal distribution. That's just a fact about normal distributions, OK?

So if we calculate this ratio and we say it's greater in absolute value than 1.96, we're going to reject the hypothesis of a quality at the 5% level, OK? So we can reject zero. Zero is going to be outside of our confidence interval. And if it's less than 1.96, we're going to fail to reject it because zero is going to be inside our confidence interval, OK? So in this case, for example, the difference was 2.59. The standard error was 0.54. The t-ratio is about seven. No, it's about five. So we're definitely going to be able to reject in this case. So we have a t-ratio of about five, OK? So you may see this terminology and this is where it's coming from.

Now, there's an important point to note here, which will come up later when we talk about power calculations, which is, in some sense, that the power that we have is determined by this ratio of the point estimate to our standard error. And so this says, for example, that if we kind of look at this a little more, that if you have bigger betas, you can still detect effects for a given standard-- so if you fix the standard error but you made beta bigger, you're more likely to conclude there was a difference, right? So what's going to increase your being able to conclude there was a difference? Either your effect size is bigger or your standard error is smaller, mechanically. OK.

So that's how we are going to calculate being in this box. So how do we think about power, which is the probability that we're in this box? We had an effect and we're able to detect that-- sorry, power's in this box-- that we had an effect, OK?

So when we're planning an experiment, we can do some calculations to help us figure out what that power is. What's the probability, if the truth is a certain level, that we're going to be able to pick it up in the data? And what do we need to do that? We're going to have to specify a null hypothesis, which is usually zero. We're going to be testing that something's different than zero, the two groups are the same, for example. We're going to have to pick our significance level, our size. And that, we almost always pick at 5%. We're going to have to pick an effect size. And we'll talk about what exactly this means in a couple more slides. But when we calculate a power, a power is for a given effect size, OK? And then we'll calculate the power.

So for example, suppose that we did this and a power was 80%. That would mean that if we did this experiment 100 times-- not 100 times, but actually repeated the whole experiment 100 times, 80% of the times we did this experiment, if the hypothesis is, in fact, false, and instead, the truth is this, we would be able to reject the null and conclude there was a true effect 80% of the time, OK?

That's a little bit complicated, but does that make sense, what we're going to be trying to do with power? So we're going to fix the effect size. So remember, we fix the bottom box. When we calculate power, we have to speculate not just effect versus no effect. We have to postulate just how effective the program is. So we're going to say, suppose that the effect size is 5%. The truth is 0.2, right? How big a sample would we need to be in this box 80% of the time, OK? So when we say power, that's what we mean.

And when we calculate the size of the experiments, you have to make a judgment call of how big a power do you want. The typical powers that we use when we do power calculations, are either 80% or 90%. So what does this mean? This means-- suppose you did 80%. Or [UNINTELLIGIBLE] this. If you did 80%, that would mean that if you ran your experiment 100 times and the true effect was 0.2 in this case, you would be able to pick up an effect, statistically 80 out of those 100 times. 20 out of 100 times, you wouldn't. And the bigger your sample size, the larger your power is going to be, OK? Does that make sense so far? OK.

Suppose you wanted to calculate what our power is going to be. What are the things you would need to know? You would need to know your significance level of your size. And as I said, this, we just assume, OK? This is that bottom box. We're just going to assume that it's 5%. And the lower it is, the larger sample you're going to need. But this one is sort of picked for you. We almost always use 5% because that's the convention. That's what everyone uses, essentially.

The second thing you need to know is the mean and the variance of the outcome in the comparison group. So you need to know-- so remember, all this power calculation is going to depend on whether your sample looks like this, really tight, or looks like this and is very noisy. Because you obviously need a much bigger sample here than here. So in order to do a power calculation, you need to know, well, just what does the outcome look like, right? Does the outcome really have very narrow variance? Is everyone almost exactly the same, in which case it's going to be very easy to detect effects? Or is there are huge range of people, in which case you're going to need bigger effects.

Now, how do we get this? So this one, we just conventionally set. This one, we have to get somewhere. And we usually have to get it from some other survey. So we have to find someone that collected data in a similar population. Or sometimes we'll go and collect data ourselves in that same population. Just a very small survey just to get a sense of what this variable looks like, OK? And if the variability is big, we're going to need a really big sample. And if the variability is really small, we're going to need a small sample. And it's really important to do this because you don't want to spend all your time and money running an experiment only to turn out that there was no hope of ever finding an effect because the power was too small, right? Yeah.

AUDIENCE: And this is in the entire population, not just the comparison group, right? It says-- --

PROFESSOR: Yeah, but before you do your treatment, the comparison and the treatment are the same.

AUDIENCE: They are the same.

PROFESSOR: Doesn't matter.

AUDIENCE: So it's a baseline population.

PROFESSOR: Baseline would be fine. Yeah. Before you do your treatment, they're the same. So it doesn't matter, OK?

And the first thing you need is, you need to make an assumption about what effect size you want to detect. And this one-- sometimes you also have to supply this. And the best way to think about what effect size you want to put in here is you want to say, what's the smallest effect that would prompt a policy response, OK?

So one could think about this, for example, by doing a cost-benefit calculation, right? You could say that we do a cost-benefit calculation. This thing costs $100. If we don't get an effective 0.1, it's just not worth $100, right? So that would be a good way of coming up with how big an effect size you want here.

And the idea, then, is if the effect is any smaller than this, it's just not interesting to distinguish it from zero, right? Suppose that the thing had a true effect of 0.001, right? But if it was that small of an effect, it could be completely cost effective. So say the thing happens at an effect of 0.001. Who cares, right?

So you want to be thinking about, from a policy perspective is, what's the smallest effect size you want to know, from a policy perspective, in order to set your power calculations? Yeah.

AUDIENCE: I have a question back at the mean and variance thing.

PROFESSOR: Oh, here. Yeah.

AUDIENCE: Yeah. So in terms of the baseline thing that you would collect-- so I'm on the implementation side of this, right? So we do projects. We collect baseline data. Now, the case that I'm thinking of, the baseline data that we would collect might not be exactly the same kind of data that we are looking for in terms of our study. What kind of base-- how--

PROFESSOR: Right, OK. So when we say baseline, there's two different things we mean by baseline. For this case, this is not strictly a baseline. This is just something about what's your variable going to look like. Let me come back to that in a sec. We also sometimes talk about baselines that we are going to use of actually collecting the actual outcome variable before we start the intervention, right? Those are also useful, and we'll talk about those in a couple slides. And those, one wants them to be more similar, probably, to the actual variable you're going to use.

Now, for your case, we often don't-- the accuracy of your power calculation depends pretty critically on how close this mean and variance are to what you're going to actually get in your data. And when you start in the example that you guys are going to work on or that maybe you've already started working on, you're going to find that it's actually pretty sensitive. Turns out it's pretty sensitive. So getting these wrong is going to mean your power calculation is going to be wrong. So that's sort of an argument for saying you want this to be as good as possible.

Now the flip side of that, though, is you're going to find that these power calculations are fairly sensitive to what effect size you choose as well. So you're going to find that if you go from a effect size of 0.2 to an effect size of 0.1, you're going to need four times the sample. That's just the way the math works out. By which I'm going to mean that I think that these power calculations are useful for making sure you're in the right ballpark, but not necessarily going to nail an exact number for you.

All that's by way of saying that you want to get-- because these things are so sensitive, you want to get as close as possible to what's actually going to be there. On the other hand you're going to find the results are also so sensitive to the effect size you want to detect that if this was a little bit off, that might be a tradeoff you would be willing to live with in practice.

AUDIENCE: So, from my--

PROFESSOR: Does that make sense?

AUDIENCE: Yeah, but it seems like the effect size-- your estimate of your effect size is this kind of-- we've got all this science for the calculation and yet your estimate of your effect size is based on--

PROFESSOR: You're absolutely right.

AUDIENCE: --getting that--

PROFESSOR: Hold on, though. Let me back up a little bit, though. You're right, except the-- in some sense, the best way to get estimates for your effect size is to look at similar programs, OK? So now there are lots of programs in education, for example. And they tend to find effect-- I've now seen a bazillion things that work on improving test scores. And I can tell you that they tend to get-- standardized effect size is the effect size divided by the standard deviation. And they tend to get effect sizes in the 0.1, 0.15, 0.2 range, right?

So you can look at those and say, well, I think that most other comparable interventions are getting 0.1, so I'm going to use 0.1 as my effect size. So you're right if you're just trying to sit here introspectful-- what my effect size is going to be, it's very hard. But if you use comparable studies to get a sense, then you can get a sense.

And the other thing I mentioned is, you can do cost-benefit analysis and say, well, look-- which is sort of another way of saying it, If there are other things out there which cost $100 per kid and get 0.1, then my thing, presumably, has got to do at least as well as 0.1 for $100-- suppose the other thing also costs $100 a kid, I've got to do at last as well as 0.1. Otherwise, I'd rather do this other thing. So it's another way of getting at the effect size.

AUDIENCE: Could you, then, also look at existing data in the literature for the mean and variance thing, or do you have to--

PROFESSOR: You could, but this one is going to be more sensitive to your population.

AUDIENCE: So it would just have to be very well-matched to be able to use it.

PROFESSOR: Right. I mean, look, if you don't have it, you could do it to get a sense, but this is one where the different populations are going to be very different in terms of their mean and variance. In order to get an estimate of this, you need a much, much, much smaller sample size than you need to get an estimate of the overall treatment effect of the program. So you can often do a small survey-- much, much, smaller than your big survey, but a small survey just to get a sense of what these things look like. And that can often be a very worthwhile thing to do.

AUDIENCE: I have a related question. How often do you see--

PROFESSOR: Oh, sorry. I just wanted to do one other thing on this. I've had this come up in my own experience, where I've done this small survey, and found that the baseline situation was such that the whole experiment didn't make any sense. And we just canceled the experiment. And it can be really useful. If you say, if I do this and my power is 0.01, for reasonable effect sizes, this is pointless. So it can be worth it. Sorry. Go ahead.

AUDIENCE: So to estimate the effect size, have you seen people run small pilots in different populations than they're eventually going to do their impact evaluation to get a sense of what effect size are they seeing with that same intervention?

PROFESSOR: Not usually, because you can't do a small pilot to get the effect size, right?

AUDIENCE: You're going to see something--

PROFESSOR: You've got to do the whole thing.

AUDIENCE: Yeah, yeah.

PROFESSOR: Right? That's the whole point of the power calculations is, in order to detect an effect of that size, you need to do the whole sample. So a small pilot won't really do it.

AUDIENCE: OK.

PROFESSOR: So it's not really going to-- you could get a-- no, I guess you really can't get a sense because you would need the whole experiment to detect the effect size.

AUDIENCE: Don't you think that there should be a lot more conversation about effect size before things start? Because if you've got a treatment, if you've got a program, and you can't have a very-- and you've struggled to have a good conversation about what is actually going to happen to the kids or what's going to happen to the health or what's going to happen to the income as a result of this, it really may be quite telling that you really don't know what you're doing. That there isn't enough of a theory behind your-- or practice or science or anything behind what your program is. If people are not pretty sure, what--

PROFESSOR: I mean, yes and no--

AUDIENCE: And then, also, on the resource allocation. Resource allocation, it just seems to me, most of the time, if your ultimate client is really probably the government, right? Because the government is the one that's going to make the big resource allocations--

PROFESSOR: It depends on who you're working with. It could be an NGO, whoever. But yes.

AUDIENCE: No, but an NGO is doing something, usually, as a demonstration that, in fact, if it works, then the government should do it.

PROFESSOR: Not always, but there's someone who, presumably, is going to scale up.

AUDIENCE: Right. And yes, businesses, maybe, right? But I would say, 90% of the time, it's going to be, ultimately, the government needs to--

PROFESSOR: Often, it's the government. In India, for example, there are NGOs who are-- I don't know who's worked on the Pratham reading thing. They're trying to teach-- NGOs trying to teach millions of kids to read, as an NGO. So sometimes NGOs scale up too. But anyway, you're right that there's an ultimate client who's interested in this.

AUDIENCE: So then, having a conversation very early on about--

PROFESSOR: Yeah. Could be very useful. That's absolutely right. That's absolutely right.

AUDIENCE: Because--

PROFESSOR: Now, in terms of your point about theory, though, yes and no. So I can design an experiment that's supposed to teach kids how to read. I know the theory says it should affect reading but I have no idea how much. And so--

AUDIENCE: Wouldn't you say that a significant percentage of the time, if it's a good theory about reading, it actually should tell you?

PROFESSOR: Not always. I mean--

AUDIENCE: Well, then I'd say it's not such a great theory, right? Wouldn't you--

PROFESSOR: It's a little bit semantic, but I think that a lot of times, I can-- say I'm going to teach kids to read a paragraph or whatever. But what percentage of the kids is it going to work for? What percentage of the kids are going to be affected? I think that using theory to calculate how-- I think theory can tell you a lot what variables should be affected. And that's what we talked about in the last lecture. I think theory can tell you what the sign of those effects likely to be. I think it's often putting a lot of demands on your theory to have them tell you the magnitude. And that's why you want to do the experiment.

AUDIENCE: And you just told me that even beyond the theory, you say, well, but we did this in one school and we saw it had this great thing, but you're saying-- [INTERPOSING VOICES]

PROFESSOR: But your confidence interval is going to be-- well, it's not nothing. It's going to tell you something, but your confidence interval is going to be enormous.

AUDIENCE: Right, nothing that you could rely on to set a good-- [INTERPOSING VOICES]

PROFESSOR: Right, it gives you a data point, but it's going to have a huge conference interval.

AUDIENCE: I don't want to belabor this, but if you think about it in business terms, right? I want to go out and raise some money.

PROFESSOR: Yes, absolutely. [INTERPOSING VOICES]

AUDIENCE: --something. And so, in order to raise that money, I have to tell you that, in fact, you're going to make this much money.

PROFESSOR: Right.

AUDIENCE: And, of course, it could turn out to be wrong. But I have to tell you you're going to get a 25% return on your money. And that means I have to explain to you why this business is going to successful, how many people are going to buy it, how I'm going to manage my costs down. So it's always curious to me that, when you're talking about social interventions, that I'm not having to make that same argument with that same level of specificity, which means I've talked about the effect size. Because I can't raise money if I tell you, look, I might only make you 5% or we might shoot the moon and make 100%. You'll say, thank you very much. This person doesn't know what their business is. I'm not going to give them my money.

PROFESSOR: Right. So you actually hit on exactly what's on the next slide. Which is exactly what I was going to say, which is, what you want to think about with your effect size is exactly this thing. What's the cost of this program versus the benefit it brings? And sometimes, what's the cost vis-a-vis alternative uses of the money, right? And that's going to be a conversation you're going to have with your client, which is going to say, if the effect size was 0.1, I would do it. And then you say, OK, I'm going to design an experiment to see if it's 0.1 or bigger, right? So I'm totally on board with that. Because, as I was saying, if the effect size is smaller than that, it still could be positive, but if your client doesn't care, if it's not worth the money at that level, then why do we need to design a big experiment to pick that up?

It's also worth noting this is not your expected effect size, right? I could expect this thing to have an effect of 0.2 but even if it was as low as 0.1, it would still be worth doing, OK? And in that case, I might want to design an experiment of 0.1, right?

Conversely, you guys can all imagine the opposite, which is you could say, I expect this thing to be 0.1, but maybe it's 0.2. Maybe it's actually-- I'm not sure how good it is. I think it's OK. But maybe it could be really great. And if it was really great, I would want to adopt it, so I would design an experiment to 0.2. So it's not the expected effect size, it's what you would use to adopt the program.

When we talk about effect sizes, we often talk about them-- we talk about what we call standardized effect size, OK? As I mentioned, how large an effect you can detect depends on how variable your sample is. So if everyone's the same, it's very easy to pick up effects. And we often talk about standardized effects are the effect size divided by the standard deviation of the outcome, OK?

So standard deviation of outcome is the measure of how variable your outcome is. So we often express our effect sizes relative to the standard deviation of the outcome, OK? And so when I was talking about test scores, for example, test scores are usually normalized to have a standard deviation of one. So this is actually how we normally express things in terms of test scores, but we could do it for anything.

And so effect sizes of 0.1, 0.2 are small. 0.4 are medium. 0.5 are large. Now what do we mean by that? This is actually a very helpful way of thinking about what a standardized effect size is telling you. So a standardized effect size of 0.2, which is what we were saying was a modest one, means that the average person in the treatment group, the median or the mean person of the treatment group, had a better outcome than 58% of the people in the control group.

So remember, if it was zero, it would be 50-50. It would be 50%, right? If there was no effect, the distributions would line up and this person's in the treatment group-- the median person in the treatment group would be better than 50% of the people in the control group. So this is saying, instead of lining up at exactly 50-50, it's lining up 58%-50%, OK? If you get an effect size of 0.5, which we were saying was a large effect, that means that 69% of the people in the treatment group are going to be bigger than the median person in the control group. Sorry, it's the other way around. The average member of the intervention group is better than 69% of people in the control group.

So the distributions are still overlapping. But now there's-- the middle of the treatment distribution is at the 69th percentile of the control. And a large effect of 0.8 would mean that the median person in the treatment group is at the 79th percentile of the control. That just gives you a sense of when we're talking about standardized effect sizes, how big we're talking about. And so you can see that 0.2, is actually-- you can imagine is going to be pretty hard to detect, right? If the median person in the treatment group looks like the 58th percentile of the control group, that's going to be a case where those distributions have a lot of overlap, right? And so this is going to be much harder to detect than this case when the overlap is much smaller. Yeah.

AUDIENCE: So in your experience, what do most people think their effect size is? Where do they settle? They probably wouldn't settle at 0.2?

PROFESSOR: Actually, a lot of people in a lot of educational interventions--

AUDIENCE: That's enough for them?

PROFESSOR: Yeah. I would say the typical intervention that people study that I've seen in education, the effect size is in the 0.15, 0.2 range. It turns out it's really hard to move test scores.

AUDIENCE: Yeah.

PROFESSOR: So yeah, I would say a lot of-- but you'll see when you do the power calculations, that to detect 0.2, you often need a pretty big sample. Look, it depends a lot on what your intervention is, but I've seen a lot in that range. And I'm just trying to think of an experiment I did. I can't think of it off hand. But yeah, I would say a lot in this range.

AUDIENCE: So would the converse be true, that in fact, you don't see too many that have a real large effect size?

PROFESSOR: I would say it's pretty rare that I see interventions that are 0.8. Yeah.

AUDIENCE: Do you think it's valuable that just because you're setting a low effect size in designing your experiment, you're being conservative. You can still pick up a [UNINTELLIGIBLE] effect size--

PROFESSOR: Of course.

AUDIENCE: It's just in the design process-- [INTERPOSING VOICES]

PROFESSOR: Right. This is the minimum thing you could pick up. That's absolutely right. That's right. So right, if you design for 0.2 but, in fact, your thing is amazing and does 0.8, well, there's no problem at all. You'll have a p-value of 0.00 something. You'll have a very strong [INAUDIBLE]. It's a good point. OK.

So how do we actually calculate our power? So there's actually a very nice software package, which, have you guys started using this yet? Yeah? OK.

AUDIENCE: I have a question. Can you just clarify something before you go on?

PROFESSOR: Yeah.

AUDIENCE: So by rejecting a null hypothesis, you won't be able to say what the expected effect is, so you won't be able to necessarily quantify the impact.

PROFESSOR: No, that's not quite right.

AUDIENCE: OK.

PROFESSOR: So you're going to estimate your-- you run your experiment, you're going to get a beta, which is your estimate, And you're going to get a standard error. You reject the null, which means you say with 95% probability, I'm in my confidence interval. So you know you're somewhere in the confidence interval.

And then beyond that, you have an estimate of where in the confidence interval you are. And your best estimate for where you are on the confidence interval is your point estimate. Does that make sense?

So in terms of thinking through the cost-benefit or whatever, your best guess of the effect of the program is your point estimate, is your beta. If you wanted to be a little more precise about it, you could say-- so this is your estimate, this is your beta hat, this is your confidence interval, right? Zero is over here, so you can reject zero in this case. But, in fact, there's a distribution of where your estimates are likely to be.

And when we said it was 95% confidence interval, that's because the probability of being over here is 95%. But this says you're most likely to be right here, but there's some probability over here. You're more likely to be near beta then you are to be very-- it's not that you're equally likely to be anywhere in your confidence interval. You're most likely to be right near your point estimate. So, in fact, if you actually cared about the range, you could say, well, what's the probability I'm over here? And calculate that. What's the probability I'm over here? And you could average them to calculate the average benefit of your program.

Usually, though, we don't bother to do this and usually what we do is we say our best estimate is that you're right at beta hat. That is our best estimate and we calculate our estimate based on that. But in theory, you could use the whole distribution [INAUDIBLE]. OK.

OK, so suppose we want-- so how do we actually calculate some of these? So using the software helps get a sense, intuitively, of what these tradeoffs are going to look like. And I don't know that I'll have time to go through all this, but we'll go through most of it, OK?

So for example, so if you run the software and look at power versus number of clusters-- hold on. So how would you set this up in the software? So we'll talk about clustered effects in a sec. As we discussed, you have to pick a significance level. You have to pick a standardized effect size. That's what delta is in the software.

So we use 0.2, OK? In the software, it's always a standardized effect size. You just divide by your standard deviation of your outcome. That's why you know your actual outcome variable because you know-- but I think the actual effect is whatever-- people get one centimeter longer in order to get a standardized effect size, I need to know the standard deviation of my outcome variable. And the program is going to give you the power as a function of your sample size, OK?

And one of the things that you can see is that this is not necessarily a linear relationship, right? So for example, here, we've plotted a delta of-- effect size of 0.2 and here's an effect size of 0.4. So this says that with about 200 clusters, you're going to get to a power of 0.8 with the effect size of 0.4, but you're still going to be at a power of 0.2 with an effect size of 0.2. So the formulas are complicated. They're not necessarily a linear function of your power.

When we think about power, we've talked about a couple of things that influence our power in terms of the variance of our outcome, right? The variance of our outcome, how big our effect size is. And those are the basic things that are going to affect our power. But there are things that we can do in our experiment-- in the way we design our experiment that are also going to make our experiment more or less powerful. And here are some of the things that we can do.

One thing that we can do is we can think about having a cluster-- so whether we whether randomize at the individual level or in clusters, whether we have a baseline, whether we use control variables or stratification, and the type hypothesis being tested. All four of these are things that we're going to do that for a given outcome variable and a given effect size, in some sense, are going to affect how powerful our experiment is. In some sense-- given that I may not have time to finish everything, the one that I want to focus on is the clustering issue. This is the one that is the biggest for designing experiments, and it often makes a big difference.

So the intuition for clustering is that-- so what is clustering? Clustering is, instead of randomizing-- suppose I want to do an experiment on whether the J-PAL executive ed class improves your ability to-- whether you took this lecture improves your understanding of power calculation, OK? Suppose I randomly sampled this half of the room and gave you my lecture and this half was the control group. And I flipped a coin so I split you in halves down the middle and I said, OK, I'm going to flip a coin, which is control, which is treatment.

You guys, presumably you all sat with your friends, OK? So people on this side of the room are going to be more like each other then people on that side of the room, OK? So I didn't get an independent sample, right? This group, their outcomes are going to be correlated because some of you are friends and have similar backgrounds and skills. And this group is going to be correlated.

On the other hand, suppose I had gone through everyone and randomly flipped a coin for every person and said, treatment or control, treatment or control, treatment or control? In that case, I would've flipped the coin 60 times and there would be no correlation between who is in the control group and who is in the treatment group because I wouldn't have been randomizing you into the same groups together, OK? By doing the cluster design, splitting you in half and then randomizing treatment versus control or splitting you into groups of 10-- you five, you 10, you 10. You 10, you 10, you 10, and then flipping the coin. I have less variation, in some sense, than if I had flipped the coin in individual-- person by person-- because those groups are going to be correlated. They're going to have similar outcomes.

So the basic point is that your power is going to be-- the more times you flip the coin to randomize treatment and control, essentially, the more power you're going to have because the more your different groups are going to be independent, OK?

So to go through this again, suppose you wanted to know-- this is, in general, about clustering. Suppose you wanted to know how the outcome of the national elections are going to be. So you could either randomly sample 50 people from the entire Indian population, or you randomly pick five families and you ask 10 people per family what their opinions are. Clearly, this is going to give you more information than this is because those family members are going to be correlated, right? I have views like my wife and like my father, et cetera. So we're not getting independent views, whereas here, you're getting, really, 50 independent data points. And that's the same as what we were talking about with the class. So this approach is going to have more power than this approach because of the way you did the sample. Yeah.

AUDIENCE: So is the only reason that you would cluster, then, just because you had to because you had no choice--

PROFESSOR: Yes.

AUDIENCE: --for political reasons or just feasibility.

PROFESSOR: And cost.

AUDIENCE: And cost.

PROFESSOR: Yeah.

AUDIENCE: Well, and the level of intervention.

PROFESSOR: Exactly. And we'll talk about that. There are lots of reasons people-- given this issue, people have lots of good reasons for clustering, but the point is that there are negative tradeoffs for sample size.

AUDIENCE: About the clusters. If you flip the coin for all of the class and then after, you decide that you will select among the people that you have assigned, you will select those seated-- you will select half of those seated on the left. Will that solve the problem--

PROFESSOR: You select half of the ones seated on the left?

AUDIENCE: Yeah.

PROFESSOR: Well, it's a different issue. Suppose I first select the left and now I go one by one, flip a coin of the people on the left. I don't have the clustering issue because I flipped the coin per person. But I have a different issue, which is that the people I selected are not necessarily representative of the whole population because I didn't pick a representative sample. I picked the ones who happened to sit over here.

AUDIENCE: My question--

PROFESSOR: So there's two different issues. One is, essentially, how many times you flip a coin is how much power you have, how independent you're thing is. The other issue is, is this group here representative of the entire population? You might think that people who sit near the window like to look at the river are daydreamers and they're not as good at math as people who don't sit near the window. And so I would get the effect of my treatment on people who like to sit near the window and aren't as good at math. And that might be a different treatment effect than if I had done it over the whole room. So it's a different issue.

AUDIENCE: Yeah, but my question was you first draw a random number of people that you assign to the treatment or to the control. And after that, within that people, you now say, I will take half of those people-- I will take half that are seated on the left and half that are seated on the right.

PROFESSOR: I'm not sure-- let me come back to your question. I'm not sure I fully understand what you're saying. Maybe we can talk about it afterwards. I think what you're saying may be about stratification. Why don't we talk about it later? Because we're running a little short on time. In fact, can I borrow somone's handouts? Because I want to make sure I cover the most important stuff in the lecture. Let me just see where we are. OK.

AUDIENCE: And if you need to, you can take ten extra minutes.

PROFESSOR: I may do that. I was going to ask you, Mark, for permission. I just wanted to see what I had left. OK. So where were we? Right. OK. Right.

So as I was saying, when possible, it's better to run clustered design. And so a cluster randomized trial is one in which the units that are randomized are clusters of units rather than the individual units. So I randomized a whole cluster at a time rather than individual person by person. And there are lots of common examples of this. So the PROGRESA program, for example, in Mexico was a conditional cash transfer program. They randomized village. Some villages were in, some villages were out. If a village was in, everybody was in. In the panchayat case we talked about, it was basically a village. It was a panchayat. So the whole panchayat was in or the whole panchayat was not in.

In a lot of education experiments, we randomize at the level of a school. Either the whole school is in or the whole school is out. Sometimes you do it as a class. A whole class in a school is in [UNINTELLIGIBLE] is out. In this iron supplementation example, it was by the family. So there's lots of cases where you would do this kind of clustering.

And there are lots of good reasons, as I've mentioned, for doing clustering. So one reason is you're worried about contamination, right? So for example, when they're interested in deworming, worms are very easily-- there's a lot of cross-contamination. If one kid has worms, the next kid who's also in school with him is likely to get worms. So if I just deworm half the kids in the school, that's going to have very little effect because my control- they're going to get recontaminated by the kids who weren't dewormed, right?

Or it could be the other way around. It could be that if I deworm half the kids, that's enough to knock worms out of the population. The control group is also affected. So you need to choose a level of randomization where your treatment is going to affect the treatment group and not affect the control group. So that's a very important reason for cluster randomizing.

Another reason is this feasibility consideration. So it's often just for a variety of reasons not feasible to give some people the treatment and not others. Sometimes within a village, it's hard to make some people eligible for a program and others not. It's just sometimes hard to treat people in the same place differently. And so that's often a reason why we do cluster randomization. And some experiments naturally just occur at a cluster level.

So for example, if I want to do something that affects an entire classroom, like give out-- suppose I want to train a teacher, right? That obviously affects all the kids in the teacher's class. There's no way to have that only affect half the kids in the teacher's class. It's just a fact of life. So there are lots of good reasons why we do cluster randomized designs even though they have negative impacts on our power.

So as I mentioned, the reason the cluster has a negative impact on your power is because the groups are correlated. The outcomes for the individuals are correlated. So, for example, if all of the villagers are exposed to the same weather, right? All villagers are exposed to the same weather. So it could be that the weather was really bad in this village. So all those people are going to have a lower outcome, for example, than if the weather was good. And so, in some sense, even if there are 1,000 people in that village, they all got this common shock, which is the negative weather, you don't actually have 1,000 independent observations in that village because they have this common correlated component, OK?

And this common correlated component we denote by the Greek letter rho, which is the correlation of the units within the same cluster. So rho measures the correlation between units in the same cluster. If rho is zero, then people in the same cluster are just as if they were independent. There's no correlation. Just as if they had been not in the same cluster. If rho is one, they're perfectly correlated and it means it they all have exactly the same outcome, OK? So it's somewhere between zero and one. And the lower the rho is, the better you are if you're doing a cluster randomized design.

And why is that? It's because the problem within a clustered randomized design is, as I was saying, if people were all exposed to the same weather, it's not as if you had 1,000 independent people in that village. You effectively had fewer than 1,000 because they were correlated. And rho captures that effect-- how much smaller, effectively, is your sample, OK? And the bigger rho is, the smaller your effective sample size is, OK?

And once again, when you do the power calculations, you can play with this and you'll note that small differences in rho make very big differences in your power. And I'll show you the formula in a sec. So often it's low, but it can be substantial. So in some of these test score cases, for example, it's between 0.2 and 0.6, which, 0.6 means that most of the differences are coming between groups, not within groups. So the groups, really, are much closer to one object. Yeah.

AUDIENCE: What does the 0.5 mean? Are you saying that in Madagascar, the scores on math and language--

PROFESSOR: It's the correlation coefficient, which is the-- technically, I believe it's the between variation divided by the total variation. I think that's the formula. Dan's shaking his head. Good. Excellent. A for me. It's what share of the variation is coming between groups divided by the total share of variation. So 0.5 means that, in some sense, half of the variation in your sample is coming between groups.

AUDIENCE: Okay.

PROFESSOR: What?

AUDIENCE: Isn't it within [INAUDIBLE]? If a rho is--

PROFESSOR: No, it's between. Because if rho is one, then each group is one. Yeah, it's between. If it was zero, then they're independent and it's saying that it's all coming from within. Yeah.

AUDIENCE: But here it's between math and language scores of one kid or between math plus language scores of two kids in the same group.

AUDIENCE: Or is it math and language scores in Madagascar are explained by--

PROFESSOR: This says the following. This was in Madagascar, they sampled math and language schools by-- they took math and language scores for each kid by classroom-- or by school. I think it was by school in this particular case. Then they said, looking over the whole sample that they looked at in Madagascar, what percentage of the variation in test scores came between schools relative to within schools. And they're saying that half of the variation was between schools. OK.

So how much does this hurt us, essentially? So we need to adjust our standard errors, given the fact that these things are correlated. And this is the formula, which is that for a given total sample size, if we have clusters of size m-- so say we have 100 kids per school-- and intercorrelation coefficient should be a rho, the size of the smallest effect we can detect increases by this formula compared to a non-clustered design. So this shows you what this looks like, OK?

So suppose you had 100 kids per school, OK? Suppose you had 100 kids per school and you randomized at the school level rather the individual level. If your correlation coefficient was zero, it would be the same as if we randomized at the individual level because they're totally uncorrelated. Suppose your correlation coefficient was 0.1-- rho was 0.1. Then the smallest effect size you could have would be 3.3 times larger than if you had done an individual design. So does that make sense how to interpret this?

And so this illustrates that, even with very mild correlation coefficients-- and we saw examples of those math test scores that were like 0.5. This is only 0.1, but it already means, in some sense, that your experiment can detect things-- if you had been able to individually randomize, you would be able to detect things that were three times as small, right? Now that's a combination of the fact that you have the correlation coefficient and the number of people per cluster.

AUDIENCE: Then in the previous slide, 0.5 does not mean half?

PROFESSOR: No, 0.5 is the correlation-- it's rho.

AUDIENCE: No, in the

PROFESSOR: It's rho.

AUDIENCE: Then it does not mean half of the difference--

PROFESSOR: No, it's half of the variance. Let me move on. We can talk about the formula for that. OK.

So what this means is, if the experimental design is clustered, we now not only need to consider all the other factors we talked about before, we also need to consider this factor rho when doing our power calculations. And rho is yet another thing we can try to estimate based on our little survey of our population to get a sense of what this rho is likely to be. And given this clustering issue, it's very important not just that you have a big enough number of people involved in your experiment, but that you randomize across a big enough number of groups, right?

And the way I like to think about is, how many times did you flip the coin as to who should be treatment and who should be control? And, in fact, it's usually the case that the number of groups you have is often more important than the total number of individuals that you have because the individuals are correlated within a group, OK? So moving on. So I'm going to flip through this. This is mostly going over some of this if you were doing the exercise quickly. OK.

And so this chart-- in your exercise shows you some of the tradeoffs that you should think about when you're trying to decide how you should trade off the number of groups you have versus the number of people within a group, OK? So in this particular case, a group was a gram panchayat, and within a group there were villages, OK? And there were different costs involved in doing these different things, right?

So going to the place involved transportation costs to get to the gram panchayat. That, say, was a couple of days. And then it took, like, half a day, say, for every village you interviewed. So that said, there's some cost of adding a new gram panchayat, but also some marginal cost of adding additional village per gram panchayat, OK? So you could calculate, based on all your parameters and power of 80% and whatever the intercluster correlation is in this particular case, you could say, well, if we had this many villages per gram panchayat, how many gram panchayats would we need and how many villages would we need, OK?

So you can do this set of exercises and you can say that-- and you'll note, for example, that as we reduce the number of gram panchayats we go to-- another way, as we add more villages per gram panchayat, the total number of villages we need to survey goes up. And in this particular case, it doesn't go up by that much because the intercluster correlation is not that high. And you could actually do this type of calculation and you could say, well I know my costs are, right? I know what my costs of going this place are. And I can calculate which of these designs is the cheapest design given what I want to achieve.

The other thing is, in this case, the experiment was happening everywhere and they were just trying to design the survey. But often when we're doing this, we also need to pay for the intervention itself. And, at least in a lot of the cases that I've worked with, the cost of actually doing the intervention is much bigger than the cost of doing the survey. And so, in that case, if you always have to treat every village in the gram panchayat, you can actually save a ton of money by going down in the number of gram panchayats and surveying a lot more villages.

But the whole point is there are these tradeoffs and you need to, in deciding how you're going to structure your experiment and how you're going to structure your survey, you need to think through what these tradeoffs are, make sure you have enough power, given your estimates of your intercluster correlation and sort of do the cost minimizing thing.

OK, so in the last five minutes or so, let me just highlight a couple of the other issues that come up in thinking about power calculations. So as I mentioned, the cluster design is one of the most important ones. And the key thing is making sure you have enough independent groups, where you flip the coin to randomize betwee treatment and control enough times. Some other things that matter are baselines, control variables, and the hypothesis being tested. So one minute on each of those.

A baseline has two uses-- main uses. One use of a baseline is that it lets you check whether the treatment and control group look the same before you started. And if you randomized properly, we know they should look similar. But you want to make sure that your randomization was actually carried out the way it was supposed to be and that it wasn't the case that people were pulling out of the hat until they got a treatment or something, that they were actually randomizing the way they were supposed to. And having a baseline conducted before you start can allow you to test that your randomization is actually truly random and your groups look balanced.

The other thing is, the baseline can actually help reduce your survey size needed because, but it requires you to a survey before you start the intervention, right? And the reason it can reduce your sample size is that now, instead of just looking at, say, test scores across kids, I can look at the change in test scores from before versus after the experiment started. And if people are really persistent, like if the people who did really well on the test this year are likely to do really well on the test next year, that can essentially reduce the variance of your outcome. It can be that the variance of difference in test scores can be a lot lower than the variance in test scores. And having a baseline can help you for that reason.

And as this slide points out, your evaluation costs basically double because you have to do two surveys, not one survey, but the costs of the intervention go down because you can have a slightly smaller sample. So if your intervention is really expensive relative to your survey, this can make a lot of sense. If your survey is really expensive relative to your intervention, you might not want to do this.

And to figure out how this is going to affect your power, you need to know yet another fact, which is how correlated are people's outcomes over time, right? What's the correlation between how well I do on a test today and how well I do on a test tomorrow? And some things are really correlated and some things are not that correlated. And a baseline really helps you on things that are really correlated.

Another thing that can help you is stratification. So what stratification can do is, stratification says, suppose I-- in some ways, it's conceptually a little bit like a baseline, which is, suppose I know that all of the people who live in this village tend to have similar outcomes and all of the people who live in this village tend to have similar outcomes. And all of the people who live in this village tend to have similar outcomes. If I can then randomize by village, I can compare the people in each village to each other, OK? So if I'm looking within village, if people in villages tend to be similar and I can randomize within village, if I look within villages, the difference between the treatment and the control group is going to be less noisy.

So stratifying is basically a way of saying I'm going to make sure my sample is balanced across the treatment and control groups within certain subgroups of the population. And then I'm going to compare within those subgroups of the population when I do my analysis. And once again, we can think this as a way of reducing noise. That if people in similar villages tend to be similar, if I only compare treatmet and control within the same village, the noise there is going to be smaller. So in some sense, it's similar.

So some things we tend to stratify by, if we know the baseline value of the outcome, we can sometimes stratify by that because we know that the effects are going to be similar for people who have very similar baseline values. Or often, I think, geographically we often tend to do that. So basically, we think that people in certain areas tend to be similar so we're going to make sure our treatments and controls are balanced in those areas as a way of reducing noise.

And the final thing we want to mention is the hypothesis being tested, which is, the more things you want to test, the bigger your sample is going to need to be. So for example, are we interested in the difference between two treatments as well as the treatment versus control? If so, we need a much bigger sample because we not only need to be able to tell the treatment versus the control but we also need to be able to tell the two treatments from each other, right?

So suppose you have two different treatments. Are you interested in just the overall effect of each of the two treatments? Or are you interested in whether the treatments interact produces a different effect if they happen together, right? The more things you're interested in, the bigger your sample needs to be because you need to design your sample to be big enough to answer each of these different questions.

Another thing, for example, you were interested in testing whether the effect is different in different subpopulations. Do you just want to know the average effect of your program or do you want to know if it was different in rural areas versus urban areas? If you want to know if it's different in rural versus urban areas, you're going to need a big enough sample in rural areas and a big enough sample in urban areas that you can compare the difference between them.

So the more different things that you want to test, obviously, the bigger your experiment's going to need to be. And a lot of times, in actually designing the experiment, this is something that comes up all the time, that you will very quickly figure out that the number of questions you would like to answer is far bigger than the sample size you can afford. And one of the really important conversations you need to have as you're starting to design an experiment are, which are the really critical questions that I really need to know the answer to?

So for example, in a project I was recently doing in Indonesia, it turned out that the government really wanted to know whether this program would work differently in urban versus rural areas because they had a view that urban areas are really different. And they were willing to do different programs in urban versus rural areas. So we designed our whole sample to make sure we had enough sampled in urban areas and in rural areas that we could test those two things apart. That almost doubled the size of the experiment, but the government thought that was important enough that they really wanted to do that.

The point here is that-- that was the one they wanted to focus on. There was a million other things we could have done instead. And so it's really important to think about, before you design the experiment, what the few key things you want to test are because, as I said, net you're never going to have enough money to test all things you want. That's sort of a universal truth.

So just to conclude, we've talk about in this lecture-- going back to the basic statistics of how you're going to analyze the experiment, thinking about how noisy your outcome is going to be and how you're going to compute your confidence intervals, how big your effect size is going to be. That's all what goes into doing a power calculation. You also need to do some guess work, right? The power calculation is going to require you to estimate how big your sample is going to be-- sorry, how much variance there's going to be, what your effect size is going to be. You have to make some assumptions. And a little bit of pilot testing before the experiment begins can be really useful, I think, mostly for thinking about-- just collecting some data can be useful to estimate these variances.

The power calculations can help you think about this question of how many treatments you can afford to have, and can I afford to do three different versions of the program or do I really need to just pick one or two? How do I make this tradeoff of more clusters versus more observations per cluster? The power calculation can be very helpful here.

And the other thing, and in some sense, the place I find power calculations the most useful, especially because there is a bit of guesswork in power calculations and you get rough rules of thumb. You don't get precise answers because it depends on the assumptions. But what I find this is really useful for is whether this is feasible or not, right? Is this something where I'm kind of in the right range where I think I can get estimates, or where there's no chance, no matter how successful this program is, that I'm going to be able to pick it up in my data because the variable is just way too noisy.

And it's really important that you do the power calculation, particularly-- both for structuring how to design the experiment, but particularly to make sure you're not going to waste a lot of time and money doing something where you're going to have no hope of picking it up. Because a study which is underpowered is going to waste a lot of everyone's time and you're very frustrated. Very frustrating for everyone involved. So you want to make sure you do this right before you start because otherwise, you're going to end up spending a lot of time, money, and effort on an experiment and ending up not being able to conclude much of anything. OK. Thanks very much.