Month: November 2020

I’ve got the Power! How many participants do I need for my study? / What effect size can I detect? A DIY Guide.

by Andrea Hutson

Urban Missionary Blog » Blog Archive » I've Got The Power!

More and more of our clients are being asked to do power analyses for grant applications. This is good in a sense, but it’s also a barrier to those out there who are not statistics nerds like me. Don’t worry though – it’s not hard, and you can even DIY!

What is power? Power represents the percent chance that you will find a difference between groups if it is really there. If you have low power (anything below 80%), there’s a chance that you’ll go to all the trouble of conducting a study only to find…nothing. And, even worse, you won’t know if it was because there truly was no effect of the program you’re studying, or if it was just due to the study being ‘underpowered.’

How many participants do I need to have appropriate power?

Luckily, there’s an easy way to calculate this for a simple study that does not require MPLUS or STATA or any software. In fact, it’s available for free online (and it has the exact same results as the above programs).

It’s here: https://www.ai-therapy.com/psychology-statistics/sample-size-calculator

What to enter? When you open up the calculator, the bottom section will look like this.

Assuming that you are trying to see if two groups are different (the program group vs. the control group), the only thing you will need to change is the effect size. The effect size statistic describes the magnitude of the difference between groups. Luckily there is standard guidance for this: a small effect is a size of about 0.2, a medium effect is about 0.5, and a large effect is about 0.8. In education research, I would say that these numbers change slightly: small is still 0.2, medium is 0.4. and large is 0.6. You might be able to find the effect size from other studies in your field. You can also calculate it from your own data if you have the means from each group and the standard deviation.

So let’s say you want to detect a medium effect in educational research. Simply change the “Effect Size” field to 0.4 and click “Submit.” At the top of the calculator, the result appears:

That is – we need 100 students in the experimental group and 100 in the control group to have a reliable study where we can detect a medium effect of 0.4 or above.

Here’s how you might write about it on a grant application:

Our sample size calculation is based on examining the effect sizes of similar studies in the extant literature. [Insert citation here if you can.] With an expected 200 participants (N = 100 in each group), we will be able to detect an effect size of 0.40 with 80% power.

What effect size can I detect?

If, on the other hand, you already know how many participants you have, you can play around with the calculator to find out the effect size you can detect. Let’s say we know we have 254 participants in the experimental group and 236 in the control group. Just to be safe, let’s go with the lower number – 236 participants.

Here’s how to write about it:

Given our sample size of 490 participants (N = 236 control; N=254 experiemental), we will be able to detect an effect size of 0.26 or greater with 80% power.

See? Not so bad!

What about groups?

if your data are GROUPED – let’s say that you have 450 participants participating from 10 different schools, the calculation gets a little more complicated. You actually need slightly MORE participants to account for the fact that students from similar schools are going to share many characteristics — teachers, school environment, neighborhood – that make their data more similar to each other than students from nearby schools. Because these data are correlated, our power to detect differences goes down slightly.

You’ll need to, therefore, generate an effective sample size that is affected by these things:

  • the number of participants per group
  • the number of clusters/groups
  • the Intraclass Correlation Coefficient, or ICC. This is probably the most challenging to obtain, but might be available from the literature. See this fantastic article for more on ICC (Killip, et. al, 2004): https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1466680/

Once you have all three pieces of data, simply use this formula:

numerator = (number of participants per group) * (number of clusters/groups)

denominator = 1 + (ICC * (number of participants per group – 1))

effective sample size = numerator / denominator

Example

Let’s say we have 10 schools with exactly 45 participants per school. Let’s say we know our ICC is 0.015.

numerator = (number of participants per group) * (number of clusters/groups) = 45 * 10 = 450

denominator = (number of participants per group) * (number of clusters/groups) = 1 + (0.015 * (45-1)) = 1 + (0.015 * 44) = 1.66

numerator / denominator = 450/1.66 = 271. That’s right, even though we have 450 total participants, because the data are clustered, we effectively only have 271 participants.

Using the calculator, we find that instead of being able to detect an effect of 0.19, we really can only detect an effect of about 0.25. This isn’t a huge difference, practically, but it could sink your grant application.

Quick fix: Survey items not displaying properly in your visualization

By Lindsay Lamb

With the election and obsessively recent rise in Covid cases, I have been spending a lot of time obsessively reviewing data. That coupled with reviewing recent survey data from teachers and students regarding their experiences in school (both in person and online), has left me knee deep in data. No matter where I review data, I keep coming across the same problem (which happens to be one of my biggest pet peeves): survey questions not displaying properly in a visualization.

Survey questions do not display properly when the survey question is too long, and instead of being able to read the full sentence, you get this:

Source: EdWeek To be fair, the website where I found this visualization was interactive and displayed the entire survey question when you hovered on the bar. This got difficult as the bars got smaller.

In some cases, you can resize the entire graph which can sometimes help. However, if you have limited space to display your data, you need to make some changes. You can either shorten survey item in your figure (which should signal that you should shorten the item when you administer the survey), allow the ellipses (my eye is twitching just writing this), or only show some of the responses (maybe set up small multiples).

If you do nothing (as in the figure above), this can be problematic. As the consumer of the data, you have to guess what the survey question is before you can make sense of the data. This is difficult and ultimately takes away from the overall meaning of the data. Sometimes you don’t choice. Some data dashboard programs (like Vocalize) do not allow you to change the space allocated to survey item labels. If, however, you are up for some Excel Ninja hacking, keep reading!

If you are using excel, simply change the data label by creating a fake row (I learned this super cool trick from Stephanie Evergreen), adding a data label for the fake data, and then displaying the survey questions with each corresponding label. Here’s a quick tutorial:

  1. Create a fake column with fake data.
  1. Now, add data labels in the figure to your fake data bars. As you can see, my fake data column contains one entry of 100% and the rest are 0%. I did this so I could easily grab the bar corresponding with the 100% data entry and add a label to it and all entries in that series (see below):
  1. Now, change the label position so it is to the left your bar and add the survey question. If you are using a Mac, you have to type in each survey item. I know this is tedious, especially if you have a lot of survey questions, but it is worth it. You can clone data labels if you have found a position you like you can clone this in the label option menu. If you have a PC, in the label menu you can select label values from a range in your spreadsheet and voila! Done!
  1. Still seeing some ellipses? No problem! You can now manipulate the size and position of your survey labels. Yay!!!

Here is a revised version of the visualization:

Revised figure.

I should also note that I removed the axis Y-values and removed the horizontal tick marks to clean it up a bit.

This chart is still a lot to take in, so in reality I would recommend splitting it up, or changing the color of items to either highlight the most or least common practices… but that is for another day and another blog post 🙂

Breaking Away from EXCEL – Find & Replace in R

by Andrea Hutson

Note: This post may be for you even if you don’t regularly use R! You’ll learn a technique that is FASTER than Find & Replace in EXCEL and is more accurate to boot.

Raise your hand high if are in the Find & Replace club – you use it in EXCEL to clean up your data before starting analyses, no matter what software you eventually use to analyze data.

If you are editing your data in EXCEL before analyzing, you’re not alone.

Here’s one way we often use EXCEL – to recode text data into numeric data. Let’s imagine we have a dataset like the one below:

a table of values that need to be recoded ("Strongly Agree", "Somewhat Agree" etc.)

We need to recode these variables so that they’re numeric and then we can analyze them.

The typical recode method in EXCEL looks something like this:

Use of Find and Replace in EXCEL - Find "Strongly Agree" and replace with "5"

It takes a while, but it works, right? Well, mostly. Have you ever done something like this?

Whoops, I forgot to do “Strongly Agree” before Agree, or to check “Find entire cells only”

In the example above, I coded “Agree” first, which replaced all of the text for “Strongly Agree”, “Somewhat Agree”, etc. with “4”. Yes, there are ways to avoid this happening,but I often forget about them until I’ve made the mistake.

Have you ever confused yourself so much you’ve had to start COMPLETELY OVER?

(I can’t be the only one!)

I’m here to tell you something exciting – your copy and paste days can be TOTALLY OVER, even if you don’t normally use R.

STEP 1: Load Your EXCEL Data

In step 1, you need to load your EXCEL data into R. This is harder than it should be, and requires you to use the “openxlsx” library or convert your file to a .csv format as there’s no native EXCEL support in R. I’ll plan to do a post all about this in the future, as this was my first major roadblock to using R – actually loading the dang data.

Here’s the first step. For simplicity in the next steps, please name your dataset “myData”. 

 
setwd("~/Directory Of Your Files")   # set working directory
 
     # You can get to this in R studio by going to Session -> Set Working Directory 
 
library(openxlsx) # if this is your first time using openxlsx, you'll need to install first
 
    # Use this code first if so ->   install.packages("openxlsx") 
 
myData <- read.xlsx("Your EXCEL FILE.xlsx", sheet=1)

STEP 2: Recode

I’m going to skip explaining the ‘lapply’ function right now – just know that you don’t need to change anything here but the values for your scales. That is, you can change “Strongly Agree” to “Very Often” if that’s the scale you’re using, or change the value for Strongly Agree to a different number. You can add more values, too, just make sure that:

  • each value is separated by a semicolon (;)
  • your text values are in SINGLE quotes,
  • the final bit of code has an end double quote and a closing parenthesis. 
# --> Load the library that has the recode function (car) - this one generally comes with R so you shouldn't need to install
 
   library (car)
 
# --> Now do a batch recode!  The code below works for your typical 1-5 "Strongly Disagree -> Strongly Agree" scale but you can use 
 
myData <- lapply(myData, FUN = function(x) recode(x, "'Strongly Agree' = 5; 'Agree' = 4; 
                                              'Neither Agree nor Disagree' = 3; 'Disagree' = 2;
                                              'Strongly Disagree'=1; 'not applicable' = NA")

STEP 3: Convert back to a table

Almost done. The last step to find/replace is to convert this new object you’ve created back into a data frame, which only uses one line of code.

 
# --> Change object back into data frame
 
   myData <- as.data.frame(myData)

STEP 4: Send back to EXCEL

Many of us that use R for data analysis will skip this step, and just start analyzing here. But you can absolutely just send this right back to EXCEL if you’d like. 

# --&gt; Write to EXCEL
 
   write.xlsx(myData, 'Data recoded.xlsx')

Boom – you’re done! Now, I know, some of you are thinking, ‘But Andrea, in EXCEL find/replace is just one step, and you’ve just given us FOUR steps to follow. I’m going to ignore this post and keep doing things as I’ve always done.’

But here’s why this way is better:

  • It is NOT just one step in EXCEL! You’ve got to Find/Replace each individual value…and it’s more time consuming than you think.
  • Once you have the base code written down you can reuse it over and over!
  • Similarly, if you save the R file, you can recode this particular data set over and over again. If you get a survey update with 10 more students, you don’t have to spend an hour finding/replacing – you literally just use the same exact code you’ve already written.
  • You’re not going to make silly mistakes.

Try it out! I’ve got the entire code below for your copy/pasting needs. The items that you will need to change should appear in RED. You should be able to leave all of the rest of the code as is.

 
# --> Set the directory where your files are : you can get to this in R studio by going to Session -> Set Working Directory 
 
setwd("~/Directory Of Your Files")   # set working directory
 
 
library(openxlsx) # if this is your first time using openxlsx, you'll need to install first
 
    # Use this code first if so ->   install.packages("openxlsx") 
 
myData <- read.xlsx("Your EXCEL FILE.xlsx", sheet=1)
 
# --> Load the library that has the recode function (car) - this one generally comes with R so you shouldn't need to install
 
   library (car)
 
# --> Now do a batch recode!  The code below works for your typical 1-5 "Strongly Disagree -> Strongly Agree" scale but you can use 
 
myData <- lapply(myData, FUN = function(x) recode(x, "'Strongly Agree' = 5; 'Agree' = 4; 
                                              'Neither Agree nor Disagree' = 3; 'Disagree' = 2;
                                              'Strongly Disagree'=1; 'not applicable' = NA")
 
# --> Change object back into data frame
 
   myData <- as.data.frame(myData) 
 
 
# --> Write to EXCEL
 
   write.xlsx(myData, 'Data recoded.xlsx')

Try it out and let me know how it goes! I promise, after you’ve saved yourself an hour of tedious finding and replacing, you’ll be a convert!