Month: March 2021

One year later…

By Lindsay Lamb

What to Write in a Blog That People Will Want to Read
Image soruce: lifewire.com

We all know the story. One year ago, life as we knew it changed forever. I will never forget it. Andrea and I were in the office on Thursday and were getting ready to leave for the day. We checked in about what projects we had coming up, what meetings we had on Friday to see which ones we both needed to attend, and where we could divide and conquer. We left and said, “See you tomorrow!”

That night, Andrea sent me an email saying there was a case of COVID-19 in Austin.

The next day, everything shut down. I sat outside and listened to the birds while holding my then 18-month-old in my arms and crying.

The next week was Spring Break and Andrea and I decided to work anyway. We had our first conference calls and Zoom meetings. Andrea and I checked in with each other almost daily to see how the other was doing mentally, physically, and emotionally. Some days were hard, and some days were harder. My husband and I set up a schedule to juggle work and childcare. I took the early work shift, getting up at 5 AM so I could work for a few hours before the kids got up. Then my husband would work while I took the kids outside and did some home-schooling activities with them.

In the midst of all of this, Andrea suggested I start writing a blog for our website. Um… what??? What on earth would I talk about???

Andrea is very business savvy, so I trusted her. She figured if we added some trainings and stories on our website someone might read them and seek out our help. At first, we both wrote mostly about what was going on with COVID-19, both in the world and in our homes. Eventually, we branched out and I started writing more and more. Potential clients reached out to us because of our blog posts. My dad, who started his career as a journalist, started a blog of his own!

As someone who takes the time to build relationships with others and collaborate in my work, this year has been hard… but the blog helped. Sure, I couldn’t physically reach out and talk to potential clients, share my experiences with peers or connect with other researchers in the field, but it helped me feel like I had a purpose in a very challenging time.

One year later, I am still writing this blog. I volunteered at a mass vaccination site and helped some of our most vulnerable population get their vaccine. I took my kids to visit my fully vaccinated parents who we had not seen since Christmas 2019. I am feeling hopeful.

Andrea and I weathered the storm (literally and figuratively!) and I honestly feel like our business is stronger than ever. Who knows what one year from now will look like, but I know after going through everything we have gone through this year, I can handle it… and I’ll probably tell you all about it too 🙂

Using R to recode variables and easily find errors

By Lindsay Lamb

Recently, Andrea wrote a post encouraging folks to break away from Excel and recode variables in R. I used this post (along with a lot of tutoring and hand-holding from Andrea) to begin to learn R. At the beginning of the year, I wrote about finally achieving my goal of learning R. To be sure, I still have a LONG way to go, but I am comfortable enough to stumble along, look up code, and find and fix problems on my own without bugging Andrea all the time. As one of my former colleagues told me, “You can’t break R, R is already broken. Our job is to fix it.”

In my previous role as an evaluation coordinator for the Austin Independent School District, we often talked about how much of what we did was figure out problems with data. Why did a student have multiple records? What did a code of F mean? Why did some students have missing data? How should we recode the data in order to get it into the right format for our analyses? We reveled in figuring out these logic puzzles.

The work Andrea and I engage in with our clients at Agile Analytics is pretty much the same… except it is a little more complicated. We are often pulling together disparate files from various data sources and often without a data dictionary. We have to do a lot of sleuthing to figure out if we received the data necessary for our analyses, what our data represent, where the data came from, and which students were included (or not included).

One thing I always do to help me answer these questions is to compute the mean, range of values, and standard deviation of values. When I trained new hires and graduate students in data analysis, I always emphasized the importance of computing measures of central tendency before running any analyses. It is even more important to conduct this first step in data analysis when you do not know the range of your data, how variables were coded, if values are character or numeric, if there is systematic missing data, how missing data are coded (or not), and if any variables are out of range.

Recently, I needed to review the data from one of our client’s previous evaluations that was not conducted by Agile Analytics. The program manager did not know much about the data, but knew it was messy and couldn’t tell me anything more about it. The data were coded in two ways that I typically don’t use: scores ranged from -1 to +1 or -2 to +2 on a 5 point Likert-scale. Our purpose in examining the data was to compute the mean and standard deviation for each variable and compute a factor analysis to help our client think through which items to trim on their survey. Because of this, I thought it was appropriate to recode the data to a standard 1 to 5 scale. This would also make it easier for our clients to understand the measures of central tendency I presented to them as we talked through which items to keep on their survey.

As I was recoding the data, I discovered that several variables included errors in their coding. These errors were super easy for me to identify because in my process of inspecting the data using R. First, I looked at the minimum and maximum of each variable in a table. I could easily see the codes assigned to each variable, and used this to help recode to a 1 to 5 scale. As a reminder, variables were either coded on a -2 to +2 scale or a -1 to +1. As you can see in my code below, I added a .5 even though this variable ranged from -2 to +2. That is because I noticed an error after I computed the table (see below):

Snapshot of code I used to examine data and recode data. I first made a table of the variable to examine the score range, then recoded the data, then made a table again.
When I computed the ables, I noticed some values were out of range (e.g., there was a score of 0.5) that had to be recoded. I confirmed my recoding when I computed the second table and could easily see what the original values were and how they corresponded with the new values.
Note: some of the data names have been redacted to keep our client anonymous.

Had I been recoding in Excel, I would have completely missed these errors. Sure, I would have found them eventually when I brought the data back in to SPSS, SAS or whatever stats package I used, but then I would have to go back to Excel and recode the data… with no record of what I had done. In R, you save your script, so it is easy to show your clients where the errors were and how you fixed them. In Excel, you have to save different iterations of your recoding because there is no script to save. You can also save projects and some code in SAS and SPSS, but R has everything all in one place, or printout. You can see the tables and code you used to recode the data in a single glance rather than having to walk someone through your code first and the output second. Someone else can also open your project (if data are in a shared location) and run it or tweak it to see if you get the same result. Notes can easily be added to the script to ensure everyone is on the same page.

Like I said, I am still an R neophyte, but I’m getting better at fixing it and sharing what I have learned along the way with the rest of you all 😉

How do you know which students were actually served by your program?

By Lindsay Lamb

We have all been there: working long hours on tight deadlines on a grant. Establishing measures, outputs and outcomes that are informative, meaningful and tied to a logic model. And inevitably, you come to one that should be easy, but often is not: “number of students served by a program.”

Recently, Andrea and I were working on a grant proposal with one of our clients and we were reminded of just how tricky this metric is – particularly during the time of COVID. Oftentimes, programs count a student if they attended one of their sessions. Some count attendance using a threshold for students meet in order to count (say 50%), while others count a student as being served by a program if a student attended a specific session (e.g., FAFSA training), and others count a student if they participated in a set number of programming hours integral to the program.

In the grand scheme of things, these all “count” (pun intended) – but are they measuring the same thing? How were these thresholds set? What does it mean if a student attends one 3-hour session and another student participating in the same program attends a 1-hour session? Where is the tipping point? It might be that attending 3 one-hour sessions spread out over a few weeks is more meaningful to students compared to one 3-hour session. It might be that some students lose focus in a 3-hour session and cannot retain all of the information they learned whereas other students might forget what they learned in-between sessions if they are spread out.

How do you know where that sweet spot is, and in terms of grant compliance does it really matter? Let’s leave the existential dilemma out of this, and all agree that these counts matter.

Now that we have that out of the way, how do you count students served by your program?

If you have any data to draw on – attendance records from each session, session duration, session type – you are way ahead! First, figure out the total number of classes/meetings/sessions that are offered each semester or year. Then determine the length of time students spend in each session. What does 100% participation look like? Now, walk back from 100% participation to set realistic targets. For example, use these data to determine the average number of sessions your students attend, the average number of hours of programming students engage in, the most common type of programming students participate in, and more.

Once you have averages you can use this information to set a threshold for participation and include expected gains in students served by your program that is more robust than simply counting bodies. Perhaps your threshold is one standard deviation above the average number of hours of programming or number of sessions, or a percent increase in number of hours of participation.

Some of you may have heard of this referred to as dosage. Having this information is extremely powerful when you tie it to your program outcomes.

For example, if you know the average number of hours in which students engage in your programming, and the type of programming they engage in, you can say things like, “Students who engaged in 8 hours of programming related to growth mindset experienced greater improvements in their academics than did their peers who engaged in fewer hours of programming.” Below is a graph from an analysis Andrea created a while back using this type of data:

Students who participated in more hours of programming experienced a greater percent change in behavioral improvement than did their peers who participated in fewer hours of programming.

This information will also help you identify the types of programming most related to positive outcomes. For example, if your program focuses on helping students achieve post-secondary success, maybe students who attended a session helping them complete the FAFSA is most directly tied to enrolling in college. Or perhaps it is attending a session with a guest speaker who attended community college that was immensely helpful.

Having the right data can help you – and your program staff – gain more insight into your program and focus your efforts on what matters most to students.

How should you track this information? Some organizations use Google Sheets, some use various survey platforms, and some use Excel. The point is to get the data and start building your database. From there, you can build dashboards in Excel – yes Excel! – and start tracking this information and make more informed decisions when it comes to including this type of information in grant proposals and grant reporting.