Recently, I was working with one of our clients in a board meeting and we discussed how to share results of our client’s work over the past year. I respectfully listened and eventually piped up, “How about a one-pager for donors?”
You all know Andrea have touted the importance of creating a one-pager... there are many ways one-pagers can work for you. They can help promote your findings, highlight opportunities for growth, showcase participants’ experiences with your program, and so much more.
In a separate meeting, I discussed that a one-pager for donors would not just be any one pager, it would be a one pager that would promote our client’s work and showcase the opportunities for growth. It had to be short, it had to be catchy, it had to show what our clients accomplished and also where they needed additional support without coming off as needy.
The way we tell the story to donors matters, and this was something I had not done before. I asked in my evaluation community if anyone had worked on a one-pager for donors before. I was surprised that most of my colleagues had not worked on something like this before. I decided to search some online resources. I found a few examples that some of my former colleagues created as well as from some organizations we work with, and the materials were too long, or not specific enough. I was hitting a dead end… there had to be some resources out there, right?
Finally, I found a great resource from spitfire strategies that made perfect sense.
The way to design an effective one-pager can focus on either on persuading the audience to donate or informing the audience about the great work the organization has engaged in over the past year. From there, the outline is pretty similar:
Some approaches on how to organize your one-pager with donors in mind based on recommendations from https://spitfirestrategies.com/
I shared this outline with our client and they were super exited. Their one-pager will actually be a combination of the two approaches with the idea to create a longer report for a few key donors and share the shorter version with all donors that highlights their program’s major successes over the past year as well as target opportunities for growth. Hopefully it will be a win win for all!
We have all been there: working long hours on tight deadlines on a grant. Establishing measures, outputs and outcomes that are informative, meaningful and tied to a logic model. And inevitably, you come to one that should be easy, but often is not: “number of students served by a program.”
Recently, Andrea and I were working on a grant proposal with one of our clients and we were reminded of just how tricky this metric is – particularly during the time of COVID. Oftentimes, programs count a student if they attended one of their sessions. Some count attendance using a threshold for students meet in order to count (say 50%), while others count a student as being served by a program if a student attended a specific session (e.g., FAFSA training), and others count a student if they participated in a set number of programming hours integral to the program.
In the grand scheme of things, these all “count” (pun intended) – but are they measuring the same thing? How were these thresholds set? What does it mean if a student attends one 3-hour session and another student participating in the same program attends a 1-hour session? Where is the tipping point? It might be that attending 3 one-hour sessions spread out over a few weeks is more meaningful to students compared to one 3-hour session. It might be that some students lose focus in a 3-hour session and cannot retain all of the information they learned whereas other students might forget what they learned in-between sessions if they are spread out.
How do you know where that sweet spot is, and in terms of grant compliance does it really matter? Let’s leave the existential dilemma out of this, and all agree that these counts matter.
Now that we have that out of the way, how do you count students served by your program?
If you have any data to draw on – attendance records from each session, session duration, session type – you are way ahead! First, figure out the total number of classes/meetings/sessions that are offered each semester or year. Then determine the length of time students spend in each session. What does 100% participation look like? Now, walk back from 100% participation to set realistic targets. For example, use these data to determine the average number of sessions your students attend, the average number of hours of programming students engage in, the most common type of programming students participate in, and more.
Once you have averages you can use this information to set a threshold for participation and include expected gains in students served by your program that is more robust than simply counting bodies. Perhaps your threshold is one standard deviation above the average number of hours of programming or number of sessions, or a percent increase in number of hours of participation.
Some of you may have heard of this referred to as dosage. Having this information is extremely powerful when you tie it to your program outcomes.
For example, if you know the average number of hours in which students engage in your programming, and the type of programming they engage in, you can say things like, “Students who engaged in 8 hours of programming related to growth mindset experienced greater improvements in their academics than did their peers who engaged in fewer hours of programming.” Below is a graph from an analysis Andrea created a while back using this type of data:
Students who participated in more hours of programming experienced a greater percent change in behavioral improvement than did their peers who participated in fewer hours of programming.
This information will also help you identify the types of programming most related to positive outcomes. For example, if your program focuses on helping students achieve post-secondary success, maybe students who attended a session helping them complete the FAFSA is most directly tied to enrolling in college. Or perhaps it is attending a session with a guest speaker who attended community college that was immensely helpful.
Having the right data can help you – and your program staff – gain more insight into your program and focus your efforts on what matters most to students.
How should you track this information? Some organizations use Google Sheets, some use various survey platforms, and some use Excel. The point is to get the data and start building your database. From there, you can build dashboards in Excel – yes Excel! – and start tracking this information and make more informed decisions when it comes to including this type of information in grant proposals and grant reporting.
This week brought us some very cold, very wintery weather here in Austin. We all spent a lot of time refreshing our various weather apps, checking the news, and listening to meteorologists’ weather reports on the radio. The reports shifted by the day, and sometimes even by the hour. I get it, this was an unprecedented weather storm which hasn’t been experienced in Austin in nearly 30 years. Many of us went without power, heat, water and food. Other than completely redoing the state of Texas’ power grid, offering some more detailed insights into the weather, power and water issues may have helped.
Let’s start with the weather. One thing that would have made knowing when the cold weather was coming, and what type of cold weather was likely to hit our area easier were error bars. Okay, okay, I know this wouldn’t work in a radio spot, but I think it could work in data visualizations on the news, and on apps or in a newspaper. Now, I’m not trying to upstage meteorologists (they have a hard job!) but perhaps some of their visualizations could use a little freshening up.
Here are some examples of what we typically see:
Source. The Weather Channel app on my iPhone
This is pretty a pretty typical experience that people have with navigating the weather. It is definitely informative, and you can click on the day and look at the forecast by hour, but there are still several unknowns. For example, it is always difficult interpreting (let alone explaining to an excited 2.5 and 6 year old) that a 35% chance of snow actually means there is a 35% chance that there will be snow in the greater Austin area, not that there is a 35% chance of snow at our house.
Here are some additional examples from the Weather Channel website and the Weather Underground website:
Source. The Weather Channel and The Weather Underground websites
The Weather Underground’s website contains much richer information such as time of day when the highs and lows will peak (which is helpful in Texas when sometimes the daily high occurs first thing in the morning), and what the temperature actually feels like versus what the reported temperature is.
Now, wouldn’t it be nice to also see something that includes a range of values, and the likelihood of wintry conditions happening? Here are two simple ideas. One is a traditional box and whisker’s plot, the other is a simple line graph. Obviously, having lots of these wouldn’t be helpful, but for one day it would be super helpful to see that the low temperature could drop below freezing, so you should probably take steps to protect your pipes.
Note. These are some simple ways to show what the projected temperature is (gray circle, or midline), and what the high or low ends might be based on the standard deviation of potential values.
After I created these charts, I watched the news. I was happy to see that one weather reporter showed an image with sliding bars indicating the likelihood of freezing rain, snow or sleet. This was incredibly helpful! Sure, we were going to get some snow, but the thing we were most likely to get was freezing rain. Had he not showed these sliders, we just would have heard 90% chance of a wintry mix. Not helpful.
So, speaking of freezing rain, that is what caused power outages in my neighborhood. Several trees were downed by the heavy ice causing additional power outages. Now there is no way a data visualization could have helped us on the front end here. The only thing that could have helped would have been the city trimming trees near power lines and Austin Energy winterizing equipment. Where data visualizations did come in and help is on the back end. Here is The New York Time’s take on explaining what happened:
This visualization shows us when the winter storm hit, so we can clearly see what normal power generation was, and how all sources of energy plummeted due to the storm. This made a bad situation worse as this storm affected nearly every Texan.
Another great data visualization came from Austin water:
These data visualizations are helpful because they show how much water people normally use, and how the reservoir was nearly depleted. The other image is a heat map by zip code indicating where there were water issues as of 2/19/21.
The good news is that as of this writing (which took a big pause in the middle due to power outages and spending all of my energy keeping my family safe), most people have power, water and food. My daughter’s school survived the storm, and we will get back to pandemic life soon. I am thankful to all of our neighbors who came together to help out during this challenging time – in a pandemic no less! We came together to make sure people had warmth, water and food. In a time where we have been learning about resiliency, wants vs. needs, I am constantly amazed by how resilient my children – and all children – have been this year.
This past summer, we had the opportunity to collaborate with Ignite MindShift as they partnered with San Antonio Works to provide high school students with an amazing virtual internship experience. We saw this as an opportunity to learn from these students, mentors, and non-profits who embarked on a new learning experience in the time of Covid. We are pleased to share with you the culmination of our learning in our Best Practices report.
We urge educators and non-profits to review the work and share with us your best practices as we continue to navigate online learning in the semesters – and years – to come. Here is a link to Ignite MindShift and their post, which includes a .pdf of our report:
As a reminder, we shared several posts related to this report (see below) and are excited to share the full report with you now. Our relevant posts are linked below:
As we near the end of what was a year like no other, I wanted to reflect on what I have learned and shared with you all while writing this blog.
Front and center for me are all of the innovative, telling, and nuanced data visualizations that have come from modeling and documenting Covid-19 cases. Initially, my go-to source was the John Hopkins Coivd-19 tracker. This interactive website was critical when the virus was beginning to spread worldwide and across the US. It painted a realistic picture and helped us mentally prepare for what was to come. We could drill down to countries, states, and counties with alarming precision. As the pandemic progressed, I found myself turning more and more to the visualizations presented in the New York Times. To this day, a visualization is included front and center and it can be filtered based on all sorts of criteria. Additionally, there are usually several other visualizations showing trends based on different populations, trends intersecting with state policies, and more. I have really enjoyed digging into these visualizations (okay, I guess enjoyed is the wrong word as I am constantly looking up Travis County (where I live), Dona Ana County where my folks live, and Orange County in California where my brother and his family live). In a weird way, I find comfort in looking at the trends in the data. It helps me mentally prepare for what is to come and to take comfort in knowing that we are all in this together. I also think they are really thought-provoking and have made folks who are not usually comfortable with data become more so.
I have also enjoyed the infographics the CDC have created. Some are pretty alarming, such as this one:
Source: CDC.gov Note: I would argue that all of dots should be in some variation of red as no one should swallow hand sanitizer, no matter who says it is a good idea, but other than that it is an effective infographic.
To be fair, I have not spent a lot of time reviewing the CDC’s infographics prior to the pandemic, so maybe they have always produced high quality work. Regardless, I have enjoyed their infographics and have gotten a lot of great ideas from them.
On a more personal level, one lesson I learned this year, okay re-learned, is to let your data tell the story. It is okay to have complicated data, let it tell its own story. I wrote a post about this issue earlier in the year when Andrea and I took data from a line graph and changed it into a lollipop graph. I also wrote about it in reworking some Covid data displays, which can get confusing. I think we all remember my extreme Covid makeover post 🙂 As a rule of thumb, if data cannot be explained on their own, consider breaking it up, making an infographic, or creating a one-pager.
Finally, as I recently discussed, simplify your presentations/reports/data visualizations and then simplify again. PowerPoint is a great tool to use in creating reports, and obviously presentations, and sometimes can get too busy with all the bells and whistles. As I wrote back in March, simpler is better.
This year was a challenge, to say the least. I spent a lot of time looking inward and forcing myself to work even when the world around me felt like it was falling apart. Having a blog was a good outlet for me and allowed me to share my experiences, some personal and some less so, in a way that also taught me important lessons. My hope is that at least one of you out there learned something or enjoyed my musings this year. I honestly believe that you are never too old/young/experienced/unexperienced/exhausted/stressed to learn something new.
So, my friends, go forth and visualize your data and remember my biggest lesson of this year: have grace and patience.
During the times of Covid-19, we have seen a boom for data visualizations. In my opinion, this is great for everyone (one of the few benefits of this awful pandemic)! Sure, there have been some misses (well, maybe some were more than misses) but people who never talked about or heard the term data visualization are now talking about data. Data visualizations are making it into mainstream media – TV news channels, newspapers, school district dashboards… everywhere.
We have written blogs about Covid-related data visualizations in the past, and as the end of the year is approaching — and with it, hopes of a vaccine — I thought it would be a nice time to share how folks are depicting vaccine timelines.
I came across this one on the Orange County Register that depicts how long it took other vaccines to get approved and stop the spread of a particular virus.
When I first saw it, I thought the syringe was a cannon. (Perhaps that says something about my current mental state.) I like the idea of using an icon (you know how I love icons!), but perhaps the shot icon is just too big. I do like the overall concept of the visualization — how you can see the timeline of when the disease emerged and when the vaccine was created. Simplifying could make this graphic so much more clear.
Here is a simpler version from A Learning a Day that averages the number of years it has taken to produce a vaccine and compares it to the rapid timeline for the Covid-19 vaccine. You can easily see that most vaccines take about 5 years to develop with several years devoted to the trial process. I like this graphic much better, but I also think it would be good to also include number of years across the top so you could see that the Covid-19 vaccine will be distributed in about a year and a half compared to 5 years. That information can be inferred, but it takes some previous knowledge.
The grey-ed out blocks indicate the normal timeline for producing a vaccine, which is also helpful in showcasing how truncated the timeline is for the Covid-19 vaccine. I also like this visualization because it shows the projected year when each component of the vaccine timeline is typically completed, which really hammers home how quickly and unprecedented the Covid-19 vaccination timeline has been.
As you can see, there are several ways to depict timeline data. When making a timeline visualization, the important rule is to choose what works best for the data. Let the data tell their story and build a visualization around that story. You might start with an idea in mind, but after messing around with your visualization realize you need to change plans. That is totally ok! Unsure what to do? Ask a colleague, friend, or family member (or all three)! Ideally, all of these folks should be able to read your visualization and understand it without having you explain it to them.
I hope you have found this era of data visualizations as inspiring as I have. Now get out there and share some data stories!
With the election and obsessively recent rise in Covid cases, I have been spending a lot of time obsessively reviewing data. That coupled with reviewing recent survey data from teachers and students regarding their experiences in school (both in person and online), has left me knee deep in data. No matter where I review data, I keep coming across the same problem (which happens to be one of my biggest pet peeves): survey questions not displaying properly in a visualization.
Survey questions do not display properly when the survey question is too long, and instead of being able to read the full sentence, you get this:
Source: EdWeek To be fair, the website where I found this visualization was interactive and displayed the entire survey question when you hovered on the bar. This got difficult as the bars got smaller.
In some cases, you can resize the entire graph which can sometimes help. However, if you have limited space to display your data, you need to make some changes. You can either shorten survey item in your figure (which should signal that you should shorten the item when you administer the survey), allow the ellipses (my eye is twitching just writing this), or only show some of the responses (maybe set up small multiples).
If you do nothing (as in the figure above), this can be problematic. As the consumer of the data, you have to guess what the survey question is before you can make sense of the data. This is difficult and ultimately takes away from the overall meaning of the data. Sometimes you don’t choice. Some data dashboard programs (like Vocalize) do not allow you to change the space allocated to survey item labels. If, however, you are up for some Excel Ninja hacking, keep reading!
If you are using excel, simply change the data label by creating a fake row (I learned this super cool trick from Stephanie Evergreen), adding a data label for the fake data, and then displaying the survey questions with each corresponding label. Here’s a quick tutorial:
Create a fake column with fake data.
Now, add data labels in the figure to your fake data bars. As you can see, my fake data column contains one entry of 100% and the rest are 0%. I did this so I could easily grab the bar corresponding with the 100% data entry and add a label to it and all entries in that series (see below):
Now, change the label position so it is to the left your bar and add the survey question. If you are using a Mac, you have to type in each survey item. I know this is tedious, especially if you have a lot of survey questions, but it is worth it. You can clone data labels if you have found a position you like you can clone this in the label option menu. If you have a PC, in the label menu you can select label values from a range in your spreadsheet and voila! Done!
Still seeing some ellipses? No problem! You can now manipulate the size and position of your survey labels. Yay!!!
Here is a revised version of the visualization:
Revised figure.
I should also note that I removed the axis Y-values and removed the horizontal tick marks to clean it up a bit.
This chart is still a lot to take in, so in reality I would recommend splitting it up, or changing the color of items to either highlight the most or least common practices… but that is for another day and another blog post 🙂
Recently, Andrea and I were speaking at a conference. Our session was entitled, Survey Design 101. We were going back to the basics with our audience. We walked them through the importance of designing good survey questions. We gave examples and cautioned them about being mindful of the wording of questions and selecting the right response options. Our goal in the session was to provide participants with a few strategies to make sure they are actually getting the answers to their main evaluation questions.
As we were talking about how to avoid common pitfalls of bad survey design (e.g., keeping surveys short, avoiding double-barreled questions, using easy to understand language), one of our attendees asked how many response options should be included for each question.
Our answer? It depends.
I know, I know. That is probably not the answer you want to hear, but honestly and truly, it does depend! Here are just some of the things it depends on:
It depends on the age of your participants.
Are your participants younger? If your participants are younger (think under 3rd grade), you probably only want 3 response options. Think along the lines of Always, Sometimes, Never as good response options. Responses need to be clear and easily distinguishable from each other. Kids at this age think in very black and white terms, so design survey questions and responses accordingly.
Are your participants older? If your participants are adults or in 3rd grade or above, you can consider using 5 to 7 response options. Some researchers use 9 response options. Sometimes using seven or more response options is exactly what you need. Perhaps responses tend to stack up on the extremes, and you want more spread in their responses so you can understand the subtle gradations in participants’ perceptions. Personally, I find it difficult to distinguish between some response options if there are 7 or more, but maybe that is just me. On the other hand, you can also err on including too few response options (e.g., Yes and No only). Sometimes asking too few questions limits the information you could get from participants. Maybe most participants’ viewpoints are not at either extreme (always or never) but are instead somewhere in the middle. Knowing this information will help program staff identify which elements or concepts are most favorable or least favorable. Unsure on how many responses to include? Test out your survey!
It depends on whether or not you want a mid-range response option. In some cases having a mid-range response option (e.g., sometimes) will result in most people selecting this as an option. Other times, this will push people to either extreme. My advice? Know your audience. Younger participants tend to select the mid-range options while older students do not. Unsure? Test out your survey!
It depends on the type of response.
If your response options involve frequency, make sure the frequencies are sequential and can be easily recalled and make sense based on the question. Make sure the responses capture the frequency and are timelines or frequencies that are realistic. This should help you determine the number of response options.
If you are unsure you are capturing the right responses, allow participants to write in their own response with an “other” and text-box option. This is particularly effective when you are piloting a survey and can give you insight into responses you may never have considered otherwise.
Regardless of how many response options you use, I strongly recommend using Likert scales. Likert scales help anchor response options and allow them to be more easily be analyzed.
One more thing to consider as you are designing your survey: if you have the time and bandwidth, test your survey! You can test your survey with a sub-sample of your larger survey population, you can test your survey with a sample of similar participants who are not in your study, test your survey with program staff, or test your survey with family and friends (probably the easiest, and likely the most unbiased!). After you test your survey, ask participants some follow-up questions about the response options and survey questions. They should be able to tell you if the responses match the questions, if there are too many survey responses, too few responses, or if the responses are just right.
These are just some of our internal rules of thumb, but I am sure there are so many more. Do you have some rules of thumb you would like to share with us? Drop us a line and let us know!
Recently, Andrea and I were asked to create a mini-course on survey design for a virtual conference. Given our years of experience creating, researching, administering, and analyzing survey data, this seemed easy enough. We decided to call it Survey design 101. We were excited –this presentation was going to be fun! (Okay, that is nerdy, but we have to find things that motivate us these days!)
I got to work on the PowerPoint, and boy did I have a lot to say! I remembered all I have learned from my data viz heroes Tufte, Stephanie Evergreen (who just posted a great blog on changes to make to your powerpoint to take it form looking great in-person to looking great in a webinar), and Ann Emery. I used color purposefully, only included points I considered integral to the presentation and trimmed my content. Then I trimmed again.
I was also motivated to keep the presentation short. I wanted to ensure there was ample time for questions, and wanted to have some time for interaction – at least interactions over chat. I sent a draft over to our colleague to review and her feedback was clear, and also a little unexpected: remove bullet points (maybe have one or two per slide, but no more), and if bullet points are important, make them their own slide. Oh, and add more images and icons.
The icons and image piece was fine (you all know how much Andrea and I love icons and images) but I was worried about making my presentation too long by pulling apart the bullet points. It felt a little weird only having one sentence or thought on a slide, but what the heck. As always, Andrea helped by adding some flair to our slides.
Here is how we took one slide and pulled it apart.
Here is what our slide looked like before
…And here is how some of our slides looked after the revamp.
Here are some slides containing the same information, but each point is presented separately making our points clearer.
Importantly, all of the information is the same. The only difference is that the information in the first slide is pulled apart so each point can stand alone. If the point couldn’t stand alone, we removed it. We kept the ‘title’ of the slide (Pitfall #1 of surveys – Too long!) the same for consistency in all 3 slides.
Although I thought adding more slides to the presentation would make the presentation itself longer, the amount of time we spent discussing each point was the same.
As a final note, pulling apart information into multiple slides is even more important now as people are passively watching webinars rather than actively participating in-person during conferences. Unfortunately, our attention spans are even worse now that we are working from home (and if you are anything like me these days, you are likely watching a presentation while also helping your 5 year old with school, keeping your two year old from turning your dog into a Jackson Pollock painting, and ensuring everyone is quiet while your husband talks with state leaders), so keeping your presentation moving at a relatively fast clip will help with this issue… Maybe 😉
One tool I have in my evaluator’s tool box might surprise you – it is the SWOT analysis (Strengths, Weaknesses, Opportunities, and Threats). What is the SWOT analysis? It is a method you can use to walk through a recent project, proposal, report, or any problem that could use some collaborative processing and innovation. It is helpful to conduct this analysis before, during or after the culmination of the project. Honestly, SWOT analyses can be used at any decision point, it just depends on the project, and the project timeline.
The quadrants below can help you think through each component of the analysis:
Quadrants of a SWOT analysis
What were the Strengths of your recent project/proposal/report? What was something internal to you/your organization that helped in this project/proposal/report? If you achieved your goal, why do you think you achieved your goal? If you did not achieve your goal, take the time to identify some strengths of your process. Find those kernels of positivity, trust me, they are there! Maybe it was setting strong meeting agendas, redesigning your report template, collaborating with project managers, or trusting your gut.
What were some Weaknesses internal to you/your organization you faced while working on this project/proposal/report? If you did not achieve your goal, what were some things that may have contributed this result? If you did achieve your goal, what were some obstacles you had to overcome in order to achieve them? Could communication between you and program staff and/or other collaborators been better? Perhaps the project didn’t quite align with your strengths and overall vision, or maybe the timeline was unreasonable.
What were the Opportunities you found in your recent project/proposal/report – even if you did not achieve your goal? What was something external to you/your organization that helped you in this project/proposal/report? Maybe you made a new connection with someone while working on this project. Can you leverage this project/report into new opportunities?
What were the Threats associated with this project/proposal/report? What was something external to you/your organization that threatened your project/proposal/report? Perhaps funding got cut, or there program staff experienced turnover, or the organization you were working with pivoted their goals midstream.
I have used this analysis to collect my thoughts after applying for a grants (both that we received and those we have not received), responding to RFPs (again for those that we did and did not receive), after a large data collection procedure, and after working on a large comprehensive report for a client involving multiple consultants. In all of these cases, engaging in a SWOT analysis has helped us recognize the strengths of our work, so we can remember to draw on these skills in future endeavors.
If you are truly reflective about your work, SWOT analyses can you help identify aspects of your work where you can grow. Doing so will help you learn from these experiences because chances are they will likely come up again. Maybe you realize that you need to be better about scheduling regular meetings with clients and/or other consultants, take the lead on a project when others fall short – even if you are not the most experienced person on the job, or take a class in a new statistical method.
SWOT analyses also provide you with an opportunity to celebrate your hard work and remember what helped you achieve our goals. When was the last time you celebrated an accomplishment? Take the time to do so now, you deserve it!
However you decide to use a SWOT analysis, I have always found this process to be helpful, regardless of the topic or the project outcome. Setting aside time to reflect on our work is always a good thing.
How can we expand on the SWOT analysis? Something I have not done yet but think would be excellent to do is to conduct a SWOT analysis with a client after the project is complete, or data have been collected, or the report has been finalized. The SWOT analysis process is a great way to foster conversation and reflect together. Who knows, you might even discover a new opportunity through this process!