Evaluation Tools

By Lindsay Lamb

Recently, I was in a meeting with a client presenting data from a quick survey check in. We were reviewing the data visualizations I made based on some earlier feedback. Sometimes when I share survey data, I collapse categories to show an overall level of agreement, or to indicate the overall percentage of favorable responses.

Since we were examining responses from students who were engaging in a virtual internship in the middle of a pandemic, we figured most responses would generate a negatively skewed distribution (see graph below, and if you’re interested in learning more about skewness, read this!).

Indeed, preliminary examinations of the data suggested that students’ responses were skewed such that there were few responses at the low end of the scale on our various survey questions (e.g., disagree, strongly disagree, not at all likely, etc.). We decided to examine movement in students’ responses at the top end of the distribution (e.g., strongly agree, agree, extremely likely). Based on this decision, I crated a visualization only showing the most favorable responses (presenting only the percentage of strongly agree or extremely likely responses, for example).

So, back tot the meeting.

All was going well. We were having a lively discussion of the data I presented in my one-pager when the client made a casual comment about being bummed about some of the responses. They had spent so much time pivoting their instructional material to be engaging in this new virtual environment, and only a third of responses indicated students felt the lessons were extremely interesting.

Ugh. My heart sank.

Fortunately, I knew there was more to the data. I pivoted. I had the data up in an interactive dashboard and shared my screen with our client. I recomputed responses to include extremely interesting and interesting.

“Actually, nearly all of your participants were interested in the lessons and found them relevant!”

Sure, a little over a third found the lessons extremely interesting, but the bigger story is that most students were engaged and found the content relevant to their lives. This was huge! Our client had pivoted their entire learning platform to a virtual environment, and students were engaged.

What did I learn?

• Sometimes it makes sense to have a plan, but you need to adapt. We are program evaluators and there isn’t something that we love more than a good evaluation plan (except maybe a logic model!). Sometimes we can get rigid and only want to look at the data in a specific way, thinking that we will know what the data will tell us. Sometimes, however, we need to let the data tell its own story.

• Don’t be afraid to pivot. Recomputing the data not only showed our agility as researchers and consultants, but also helped our client see their data in a new way. It helped our client see the value of their work and trust us in our services.

• The power of one-pagers. Having a one-pager with data visualizations and short text explanations helped our client easily see the data and understand the story. It also helped us recognize where we needed to go in our analysis. Having the data in a digestible way facilitated our conversation and helped us see where we needed to go. It was empowering for me and empowering for our client.

• Have your data handy. In addition to having a user-friendly one-pager, having the data accessible in a sharable format helped us see where we needed to go in our analyses. It was empowering for me and empowering for our client.

by Lindsay Lamb

Something that I have often struggled with in producing a succinct and powerful one-pager is where to put technical information, a brief description of data used in the study, analyses conducted in the study, study design, survey questions, etc.

In a typical research paper, that information is generally presented first. Most of us come from an academic background, where we learned to write reports with Introduction, then Methods, then Results (which oftentimes are full of technicalities), then, finally the Conclusion – where the actual findings and interpretation lie.

The problem? It’s BOOOOORING. You’ll lose all but the most dedicated reader with reports that are designed that way, and in a one pager, it’s a killer. The conclusions need to be front and center.

But what about that technical info about how the study works?

Sidebar to the rescue

This is where the sidebar comes in. You still need to be succinct, but it is a way to break up the layout of your one-pager and go more deeply into a specific topic that is essential to your one-pager but does not need to be part of the overall one-pager narrative.

Here is the template we’ve shared with you in the past:

Still not sure what to use a side bar for? The sidebar can also include photos, data visualizations, description of program elements, survey questions, survey response rates, quotes from participants, anything that your audience should know, but can stand alone in its own section. Here are just a couple of examples:

Here are some key takeaways to help you become the master of the sidebar:

1. Put information that is necessary to understand your report in the sidebar.
2. Add icons, photos, or a data visualization that will break up the layout of your report.
3. Keep it simple! Details belong in the report (or the appendix!)

by Lindsay Lamb

Have you ever been in a situation where someone needs a one-pager – like yesterday? You’re just sitting at your desk, and all of a sudden, your boss comes by and says, “Hey Lindsay, the superintendent [program manager, director, business analyst] needs a one-pager on that research [analysis, presentation, etc.] you’ve been working on. Can you get it to me in two hours?”

Internally, if you’re anything like me, you’re probably thinking: “AHHHHHH!!! HOW AM I GOING TO DO THIS??? THIS IS CRAZY!!!”

Externally, you remained poised and self-confident : “Of course, I’m happy to help!”

So, what can you do to make this happen? Take a deep breath and follow these simple steps.

1. Use what you’ve got. Before you get started, do you have a longer report, outline, or presentation? If yes, then this process just got A LOT easier. If not, don’t worry, I’m sure much of that information exists in your head. Go ahead and sketch out a rough outline either using word or on paper – whatever you need to do to get those creative juices flowing.
2. Use One-Pager Building Blocks: Now that you have a rough idea of what you want to say, use these four building blocks to help you get started building your report:

If you have a larger report, presentation, or outline, what are the main headings and/or major findings? Do you already have a description of the study? Trim that down and drop it in at the beginning or in the sidebar. If not, quickly jot down the things from your report/presentation/analysis that stand out to you. Whittle them down to the top four ideas and rank them so you know which one to add depending on how much space you have in your document. Keep in mind that when you create your report, you want to include as much white space as possible to make it easier for your audience to read you report. Don’t muddle it with too much information.

Now that you have a better idea of what you want to say, it’s time to start thinking about the layout. As a general rule, a one-pager can be broken down into three or four major sections per page: The introduction, a major finding, a visualization and a photo. Sidebars are a great way to visually break up the page and add some of this information. Here is a generic template that can give you a rough idea of what your one-pager might look like.

Here are the building blocks of a one-pager using our generic template. Remember, your one-pager can be two pages (one-page front and back):

3. Have you created data visualizations? Can you use those? Can you combine visualizations (small multiples, maybe)? If not, think of ways to create data visualizations quickly. Most people respond well to percentages and simple figures. If time permits, create a visualization using that type of information. Remember, as an evaluator, you most likely won’t be the one sharing the information from this report, so it is important to ensure the data visualization can stand on its own without a lot of explanation. Don’t have time to create a visualization? Use icons or photos to break up the text and add something visually appealing for your reader.
4. Summarize key takeaways. What do you want you audience to remember? Are there important strategies they need to consider? Program elements that should be removed? Funding sources that should be added or eliminated? Share them, and preferably using bullets 😉

Whew. That was quick, but I hope you have found it useful. If, like me, you find yourself in tis situation, remember to use the following strategies to help you create a one-pager in just a couple of hours:

1. Identify the major findings
2. Break up the page into three or four sections. Use a sidebar to include additional information relevant to your analysis such as a description of the study, data used, research questions, project timeline, etc.
3. Create effective and efficient data visualizations that correspond with your major findings. Don’t have time for visualizations? That’s okay, you can use icons or photos.
4. Highlight the key takeaways from your analysis. What do people need to know to make informed decisions?

That’s it!

Want to learn more? Our upcoming mini-course will take you deeper into creating an effective one-pager – stay tuned!

by Lindsay Lamb

We have talked a lot recently about tools for your evaluator’s toolbox. Today I have another essential tool to for you: consistent branding. Clear and consistent branding can help you more quickly and efficiently create short and long reports. Doing so will ensure that when people look at a report, they will instantly know that it came from you. I have found that consistent branding takes some of the guess work out of report writing. I no longer have to think through color choices, layout, font, or headers. That leg work has already been completed. I’m guessing you have a little free time now, so you might as well spend some time thinking through your branding. Trust me, you will thank me later.

Enough of my pep talk.

I’m sure you are wondering how to create a clean, consistent look for your reports, one-pagers, and blogs. Follow these three simple steps:

Logo

Make sure you have a unique logo that lets people know what your business/firm/program is about.

You can use a logo generator to help you out! We used logogenie.net.

Colors

If you have a logo, use colors from your logo in your reports.

Remember the color wheel? Use it! Make sure you incorporate complimentary colors (colors on opposite side of the arrows), and colors in a shade that is easy to see on screen, in print, and for those with color blindness. A good test for any report is to print it out in black and white. If your colors are all grey, try using gradations within the same color to convey meaning.

Use colors consistently such that all headers are the same color, etc.

Caveat – if you are generating a report for a client with clear branding, you might want to use their branding and simply insert your logo. Check with program staff, your supervisor, a colleague, a friend, or even reach out to us if you have questions and want some feedback!

Font/layout

Choose a font that is unique and easy to read

Stephanie Evergreen goes into great detail on the importance of choosing fonts in her book, Presenting Data Effectively.

You can select two fonts, or fonts that are bold and light to use to signal to the reader level of importance. There are a plethora of fonts out there, download something unique online.

The layout of your reports should also be consistent. Check out our recent blog post on one-pagers. The idea is that people should pick up a report and instantly know that it came from your organization/consulting firm.

One final tip – try out your branding! If you have a few ideas, try them out and see what your clients, friends, colleagues think. Send your ideas to us and we will look them over. Your brand is the first thing that people will see when experiencing a report you have created, so you want to make sure it is right.

When visiting with organizations about the importance of evaluating their programs, we generally see a lot of enthusiasm. Nonprofits, school districts, and others realize that evaluations:

• Let organizations know what’s working and what’s not
• Give stakeholders a way to provide valuable feedback
• Can give a sense of the impact the program is making
• Save time and money in the long run

Why then, aren’t organizations doing evaluations for EVERY program, EVERY time?

Two things are missing – time & money. Evaluations are often perceived as (and can be) massive behemoths that take hours and hours to conduct and cost quite a bit of money.

It doesn’t have to be this way.

We have recently been working on a method to produce a comprehensive, focused evaluation with a polished final report or professional presentation in just 24 hours.

Of course, you could spend the longest day ever on the evaluation and get (some types) done in one day. But you could also spread those hours out and get a high quality evaluation completed in a week, with 4-6 hours of work a day.

The 24 hour evaluation has 3 phases.

1. Phase One: Planning (4 h, Day 1).  In this phase, you’ll brainstorm ideas for your evaluation, create a mini-theory of change, select your evaluation questions, and decide on your strategy for data collection.
2. Phase Two: Data Collection & Analysis (12 h, Days 2 & 3).  In the next phase, you’ll collect data (whether new or existing), clean it up, and analyze it.  This is the longest phase, for obvious reasons, and if you choose to collect new data, it will necessarily take longer than one 24-hour day to complete the process.
3. Phase Three: Reporting (8 h, Days 4 & 5). In the final phase, you’ll pick a report type (either an evaluation brief or a presentation), write your content, edit, and publish or present.

A few caveats:

• The number of evaluation questions asked will be small and targeted. This may look like one deep question or several small questions.
• If you are planning on writing a report, the report will be a brief report of 5 pages with an additional one-pager summary. There’s not time to write a 30 page report in 24 hours! An evaluation brief, when written carefully, can actually be more effective than a long report because people are more likely to actually pick it up and read it.
• If you are planning to do interviews and focus groups, you will use a rapid analysis method – no time to transcribe tapes and upload themes into nVivo or some other analysis software. Instead, you’ll be pulling out the big and consistent messages that you’ve heard.

With this method, 90% of what your organization needs to do to make real, meaningful change could be done in one work week spread over 24 hours.

Ready to get started?

Fill out the form below to get the 24-hour evaluation schedule for FREE.

We absolutely 100% pinky swear never ever to send you spam.

By: Lindsay Lamb

Have you ever gotten frustrated adding an image into your report, especially in word? The text boxes don’t line up quite right, images jump around, things get anchored in a weird place, you can’t resize anything… and you want to throw your computer against the wall. Well, maybe that’s just me.

Although I have been writing reports for nearly a decade, it was only recently that I learned this cool trick that would have saved me lots of grief and frustration.

Recently, Andrea and I were working in tandem on a report. We each took a section, set a timer to draft it out, and got busy. I was in the zone. Chugging away, racing to get as much done as I possibly could before the timer went off. I decided not to get bogged down in adding figures, it was just too frustrating for me to add them in word. Instead, I created a couple of figures in excel, and added placeholders for them in word. I would deal with them later. I glanced over at what Andrea was doing. Cool and confident, Andrea was two pages in AND HAD FIGURES!!! She saw my eyes widen and said,

“Hey, have I ever shown you this cool way to add figures in Word?”

“No…”

“You just add a three-row table in word and drop the figure in. Make the top row the title, the middle row the image, and the bottom row for the note or source.”

My jaw dropped.

GENIUS!!!

Here is a step-by-step process for inserting a figure in word (or PowerPoint) using a table.

STEP 1: Insert a 3-row table

(Insert->Table)

Note: You can add columns if you are inserting a series of figures that all have the same title; the columns can be the same width and the rows can be the same height, ensuring that your figures are all the same size.

STEP 2: Drop in your figure, add your title, and your source/note.

Place the figure in the second row, the caption in the top row, and the note in the bottom row.

I just copy and pasted my figure from excel into word. I can easily make changes to the figure in excel and it will update in word. I can also make minor edits to the figure in word (like color or font) as well.

Now these three elements – title, figure, and note – are all grouped together. You can change the size of the figure as one unit, move the figure around without throwing off anchoring. Text in the remainder of the report can wrap around the figure in multiple ways, depending on the report layout.

Wait! Before you move on to your next figure, there is one final step. Obviously, you do not want the table borders to show up, but it is helpful to see the borders while you are laying out your report and copy editing. Let’s get rid of them.

STEP 3: Make the table border invisible for the final report, but visible while editing.

Select table, then right click and choose “No Border”

Now you’re done!

Here’s the final result:

A similar process exists in Microsoft publisher, but that software is not available on a Mac, and since many of us are working from home, we might not have access to our regular office setup. Also, sometimes, you just need to use word because not everyone has publisher.

In any case, if you are using publisher, you can insert an image placeholder and add text boxes for the title and note/source and then group those elements together. The difficulty with that process, however, is that you have to resize the figure in excel prior to inserting it into publisher or your figure might get wonky. Translation: you need to know the overall layout of your report so that you know how much space you have for your image prior to inserting it into your document. In word, the image resizes appropriately with the rest of the elements in the table.

Hope this trick helps you as much as it helped me!