by Andrea Hutson
More and more of our clients are being asked to do power analyses for grant applications. This is good in a sense, but it’s also a barrier to those out there who are not statistics nerds like me. Don’t worry though – it’s not hard, and you can even DIY!
What is power? Power represents the percent chance that you will find a difference between groups if it is really there. If you have low power (anything below 80%), there’s a chance that you’ll go to all the trouble of conducting a study only to find…nothing. And, even worse, you won’t know if it was because there truly was no effect of the program you’re studying, or if it was just due to the study being ‘underpowered.’
How many participants do I need to have appropriate power?
Luckily, there’s an easy way to calculate this for a simple study that does not require MPLUS or STATA or any software. In fact, it’s available for free online (and it has the exact same results as the above programs).
It’s here: https://www.ai-therapy.com/psychology-statistics/sample-size-calculator
What to enter? When you open up the calculator, the bottom section will look like this.
Assuming that you are trying to see if two groups are different (the program group vs. the control group), the only thing you will need to change is the effect size. The effect size statistic describes the magnitude of the difference between groups. Luckily there is standard guidance for this: a small effect is a size of about 0.2, a medium effect is about 0.5, and a large effect is about 0.8. In education research, I would say that these numbers change slightly: small is still 0.2, medium is 0.4. and large is 0.6. You might be able to find the effect size from other studies in your field. You can also calculate it from your own data if you have the means from each group and the standard deviation.
So let’s say you want to detect a medium effect in educational research. Simply change the “Effect Size” field to 0.4 and click “Submit.” At the top of the calculator, the result appears:
That is – we need 100 students in the experimental group and 100 in the control group to have a reliable study where we can detect a medium effect of 0.4 or above.
Here’s how you might write about it on a grant application:
Our sample size calculation is based on examining the effect sizes of similar studies in the extant literature. [Insert citation here if you can.] With an expected 200 participants (N = 100 in each group), we will be able to detect an effect size of 0.40 with 80% power.
What effect size can I detect?
If, on the other hand, you already know how many participants you have, you can play around with the calculator to find out the effect size you can detect. Let’s say we know we have 254 participants in the experimental group and 236 in the control group. Just to be safe, let’s go with the lower number – 236 participants.
Here’s how to write about it:
Given our sample size of 490 participants (N = 236 control; N=254 experiemental), we will be able to detect an effect size of 0.26 or greater with 80% power.
See? Not so bad!
What about groups?
if your data are GROUPED – let’s say that you have 450 participants participating from 10 different schools, the calculation gets a little more complicated. You actually need slightly MORE participants to account for the fact that students from similar schools are going to share many characteristics — teachers, school environment, neighborhood – that make their data more similar to each other than students from nearby schools. Because these data are correlated, our power to detect differences goes down slightly.
You’ll need to, therefore, generate an effective sample size that is affected by these things:
- the number of participants per group
- the number of clusters/groups
- the Intraclass Correlation Coefficient, or ICC. This is probably the most challenging to obtain, but might be available from the literature. See this fantastic article for more on ICC (Killip, et. al, 2004): https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1466680/
Once you have all three pieces of data, simply use this formula:
numerator = (number of participants per group) * (number of clusters/groups)
denominator = 1 + (ICC * (number of participants per group – 1))
effective sample size = numerator / denominator
Example
Let’s say we have 10 schools with exactly 45 participants per school. Let’s say we know our ICC is 0.015.
numerator = (number of participants per group) * (number of clusters/groups) = 45 * 10 = 450
denominator = (number of participants per group) * (number of clusters/groups) = 1 + (0.015 * (45-1)) = 1 + (0.015 * 44) = 1.66
numerator / denominator = 450/1.66 = 271. That’s right, even though we have 450 total participants, because the data are clustered, we effectively only have 271 participants.
Using the calculator, we find that instead of being able to detect an effect of 0.19, we really can only detect an effect of about 0.25. This isn’t a huge difference, practically, but it could sink your grant application.