Month: May 2020

Entering the Evaluation Job Market During a Recession: Advice for Recent Graduates

by Lindsay Lamb

I have been thinking a lot about students graduating lately. Graduating high school, college, graduate school. My daughter is graduating from Pre-K and will start Kindergarten in the fall. I have friends and family with kids graduating high school, excited to celebrate their accomplishments and start off on their adventure. Now, thanks to the pandemic, those dreams have a much different reality. There is no prom, there is no graduation, there is no party to celebrate with friends and family.

As if reading my mind, I was asked to provide recent doctoral graduates with advice they might find helpful as they enter what is sure to be a tough labor market. What could I possibly say to them?

Fortunately (maybe?) I graduated during the 2008-2010 recession. When I first entered the job market, post-doctoral funding was frozen, full-time professor positions were being transitioned into more adjunct faculty positions, the private and public sectors were beginning to enact hiring freezes.

So, back to that advice.

The prompt I got was, Based on your experience in 2008-2010 on the job market:

  1. What strategies or resources helped you get that job offer?

I made connections. I built relationships. I put myself out there and honed some skills. I essentially created my own internship with several educational non-profits in the area. Sure, this didn’t make me money right away, but it paid off big time in the end. This is the only reason why I got my job in AISD.

I was already on the fence about an academic role. With a degree in educational psychology, I wanted to have more of an effect on students, staff, and administrators I was researching. I wanted to make a difference in students’ lives. My experiences during this time reaffirmed my beliefs: I wanted out of academia.

I started contacting educational non-profits in the Austin area to see if they needed help. Surely, I could create a survey, conduct observations, do a quick focus group, anything to help them out. I had a lot of interest, but not any pay. On the plus side, I was still enrolled in graduate school, and was getting paid to be a teaching assistant and was teaching a class at a local university so I could make it work.

Because of this work, I finally got connected with the Director of Research and Evaluation at the Austin Independent School District. Luckily for me, they had a job opening. I spent 10 amazing years there conducting meaningful program evaluations and learning from my colleagues every day.

  1. What was the most challenging aspect of finding a job during the recession?

All of the uncertainty and frustration I faced. Several of the jobs that I applied for suddenly lost funding or were put on a hiring freeze. These unknowns exist today, and unfortunately, there isn’t anything we can do about them, but we can keep applying for jobs, learning new skills, and making new connections. Anything you can do to keep you going one day at a time.

  1. What advice would you share with your younger self and/or graduate students who are just now looking for jobs?

Don’t give up! I know this sounds hokey, but the only way things will not work out is if you stop looking for a job, stop honing your skills, and stop building connections. Find some free (or even not free, but cheap) mini-courses on data visualizations, creating one-pagers, learn R, start a blog! Also remember that you have learned amazing skills during graduate school that can be applied to many fields, not just academic jobs. Businesses are looking for skilled workers who can analyze data, create effective data visualizations, share results in an easy to understand way. If you take a job that is below your skill level, raise it up! Bring in your skills to show that you can do more and be more, maybe you can create an opportunity for yourself by showing all of your skills.

I know this is a challenging time, but we are all in this together. As cliché as it sounds, I truly believe things will work out one way or another. It might not be exactly what you had planned, but it is how we deal with setbacks and adversity that define use. These experiences force us to look inward and grow, and In the end, I truly believe that we will come out stronger than we were before.

Good luck!

Brand Your Documents in Three Easy Steps

by Lindsay Lamb

We have talked a lot recently about tools for your evaluator’s toolbox. Today I have another essential tool to for you: consistent branding. Clear and consistent branding can help you more quickly and efficiently create short and long reports. Doing so will ensure that when people look at a report, they will instantly know that it came from you. I have found that consistent branding takes some of the guess work out of report writing. I no longer have to think through color choices, layout, font, or headers. That leg work has already been completed. I’m guessing you have a little free time now, so you might as well spend some time thinking through your branding. Trust me, you will thank me later.

Enough of my pep talk.

I’m sure you are wondering how to create a clean, consistent look for your reports, one-pagers, and blogs. Follow these three simple steps:

Logo

Make sure you have a unique logo that lets people know what your business/firm/program is about.

You can use a logo generator to help you out! We used logogenie.net.

Colors

If you have a logo, use colors from your logo in your reports.

Remember the color wheel? Use it! Make sure you incorporate complimentary colors (colors on opposite side of the arrows), and colors in a shade that is easy to see on screen, in print, and for those with color blindness. A good test for any report is to print it out in black and white. If your colors are all grey, try using gradations within the same color to convey meaning.

A color wheel, complimentary colors are located diagonally across from each other.

Use colors consistently such that all headers are the same color, etc.

Caveat – if you are generating a report for a client with clear branding, you might want to use their branding and simply insert your logo. Check with program staff, your supervisor, a colleague, a friend, or even reach out to us if you have questions and want some feedback!

Here is our style guide, and a link to the logo generator we used.

Font/layout

Choose a font that is unique and easy to read

Stephanie Evergreen goes into great detail on the importance of choosing fonts in her book, Presenting Data Effectively.

You can select two fonts, or fonts that are bold and light to use to signal to the reader level of importance. There are a plethora of fonts out there, download something unique online.

The layout of your reports should also be consistent. Check out our recent blog post on one-pagers. The idea is that people should pick up a report and instantly know that it came from your organization/consulting firm.

Our typical layout scheme.

One final tip – try out your branding! If you have a few ideas, try them out and see what your clients, friends, colleagues think. Send your ideas to us and we will look them over. Your brand is the first thing that people will see when experiencing a report you have created, so you want to make sure it is right.

The Secret Reason Leaders Aren’t Reading Your Evaluation Reports

by Andrea Hutson

Evaluators: be honest: have you ever worked hard on an evaluation, found some really exciting findings, wrote an amazing report, and turned it in expecting….

  • praise from program leadership!
  • noticeable, positive program changes!
  • maybe a promotion? (or extended contract?)

And instead, got… crickets?

Let me raise my hand the highest. It’s definitely happened to me, and one time in particular is burned into my mind.

I was in a final report meeting with a large agency. The evaluation we were reporting on was a multi-year, multi-site evaluation with mixed methods, requiring multiple staff, travel, a ton of time with SPSS and R, and a very long time writing and editing the report.

There were about 10 people in the that final meeting, including the relatively new program director (sidenote: one interesting thing about long evaluations – in some organizations, change is so constant that often the only people who are with the program from start to finish are the evaluators).

Our final report was over 100 pages, with appendices, and lots of technical detail.

After I presented all of the results from the study, the program director looked at me and smiled. “Wow! These are really interesting results!” I smiled back.

“But…”, she started… and I felt my heart sink.

Then she dropped a bomb.

“But, you know no one will ever read this.”

Oh my goodness, so painful to hear – but I knew she was right.

It’s not that the content was bad (of course it wasn’t! I wrote it!), it’s just that it was too dang long.

Decision makers don’t have time to read.

The average 30-page evaluation report takes about an hour to carefully read and process (and at over 100 pages, my report would have taken 3+ hours)

The problem is, leaders – especially the higher you get up the food chain – don’t have an hour. 

There was a fantastic study of CEOs conducted in 2018 by the Harvard Business Review (https://hbr.org/2018/07/the-leaders-calendar) that explains why. (Although the report only looked at CEOs from the business world, non-profit and governmental leadership likely have very similar constraints.)

According to the study, CEOs spend the vast majority of their time (about 72%) in meetings, either with their own staff, leadership, or clients. They have very little time alone, and most of that time is in small chunks (i.e., not conducive to report reading).

Check out this time graph of one CEO’s alone time, arranged in 15 minute green blocks.

chart of ceo's time arranged in blocks
This CEO has very little time to himself. Source: https://hbr.org/2018/07/the-leaders-calendar

Even when we include Saturday and Sunday, this CEO only had 8 hours of one-hour or more alone time for the entire week, even though he was working a TON of hours (note that he starts working before 6 AM most days).

The study noted that the CEO spent the majority of that alone time (55%) reading and responding to emails. After we remove that time, he had only had about 3 or so hour-long chunks for all other unscheduled tasks.  FOR THE WEEK.

And that’s the reason that long reports don’t get widely read.

There’s just no time!

What can you do?

Now beyond going past the existential crisis about why we write reports at all, there are a few solutions. Here are three ways you can maximize your impact.

  1. Don’t write a report; make a presentation instead. This method is getting more popular over time; UX Evaluators typically present results in one final PowerPoint. If you noticed above, we added on a presentation, which was the product that got us traction with the program leadership. That’s also a possibility, if a report is absolutely required (although the product we had worked for hundreds of hours on became essentially a doorstop).
  2. Write a brief report instead of a long one. We’ve had tremendous success with 4 or 5 page evaluation briefs. When carefully written, these briefs can include the vast majority of findings, details, and recommendations that people need, in a format that’s digestible and a relatively quick read.
  3. Add on a powerful one-pager. One pagers delete all but the most essential content, giving just the highest level findings from a study. The best thing about one-pagers is that they can be added on to a brief report, or even a very long one. Stephanie Evergreen has a great post on the 1-3-25 model, (https://stephanieevergreen.com/the-1-3-25-reporting-model/) in which a one-pager is tacked on to an evaluation brief which is tacked on to a longer report (but notice: the ‘longer report’ is only 25 pages).

Want to learn more about option 3?

Lindsay and I are currently working on a course on one-pagers that’s designed for anyone who wants to make more of an impact with their reports. This course will provide easy steps to turn a long report into a one page document that engages readers and packs a punch. And you’ll do it without needing graphic design skills or any fancy software. We’ll be sharing some of it on the blog, but if you’d like to be notified when the course goes live. subscribe below.



Teaching your kids about data

1 Comment

by Lindsay Lamb

Recently, I read a great article in the New York Times, Turn Your Demanding Child Into a Productive Co-Worker. As mentioned in an earlier blog post, I have taken on a majority of childcare responsibilities for my two children during the pandemic. Needless to say, the first few sentences of Michaeleen Doucleff’s article resonated with me: “At 10 a.m. on a Tuesday morning, I lock myself in our bedroom, slide down the back of the door and begin to cry. “But Mama, but Mama,” my 4-year-old daughter screams from the hallway.”

I found a kindred spirit. I had to read on.

After venting about her current situation (which we all need to do, regardless of whether or not we have young children to look over right now), Doucleff went on to describe how in her work she has observed families from cooperative cultures sharing work with children. She suggested that we stop endlessly entertaining our children (which is exhausting), and instead show them our work and share in the process with them.

Easier said than done.

How could I share my work with my kids?

I could teach Hannah (who is 5) to sit calmly next to me and “work,” but I seriously doubt I could get my destructive Godzilla-esque 20-moth old to sit calmly while I write without him a) destroying my computer, b) destroying whatever Hannah was working on, or c) injuring himself.

After pondering a bit, I came up with an idea that would both use data and get the kids outside.

Me: “Hey Hannah, do you want to do a data gathering nature walk? At the end we can make a graph and show Daddy what we found!”

Hannah: “YES!!!”

Hayes: “Mama!!! Outside, outside!!!”

Hannah and I discussed what we would search for on our nature walk. I explained that we would count how many of each thing we saw during our walk and use that information, or data, to make a picture.

Me: “What things do you think we might see on our nature walk?”

Hannah: “Rabbits!”

Me: “That would be really cool if we saw a rabbit! What is something that you know we will see just in case we don’t see a rabbit?”

Hannah: “Birds, maybe a cardinal, and squirrels! Oh, and a blue jay.”

Me: “What would Hayes like to see?”

Hannah and I: “TRAINS!”

Me: “Yes, let’s add trains.”

Hannah: “And butterflies.”

I loaded up the kids in the double-wide stroller, and gathered crayons, a notebook, an envelope in which to carry the crayons and notebook, and snacks (you always need snacks). Ten hours later (well, maybe 20 minutes), we hit the trail for our fun-filled adventure. Since the weather was nice, I decided to run. Because I felt like a pack mule, I was more than happy to stop whenever Hannah excitedly screamed that she saw something from her list.

Like any good scientist conducting an observational field study, Hannah added some animals to her list based on what we saw during our run (and what she wanted me to run by). For example, we saw a parrot fly by, so we added parrot to the list. She also realized that she forgot to add cats, so we added cats. Then she wanted to go by the house with goats in our neighborhood, so we added goats to the list.

To keep a record of what she saw, she created a list of each animal (or train or butterfly) and added a check (also in a box) next to each animal. This way she could not only keep track of what she saw, but also how many of each animal (or train or butterfly) she saw during our walk. Here is her completed check list:

Hannah’s completed nature walk check list.

In case you can’t read her handwriting (though I did help her a bit), we searched for rabbits, cardinals, squirrels, butterflies, trains, blue jays, cats, parrots, and goats.

Hannah had so much fun seeing each item and adding a check box to her list. It turned our daily run on the same streets into a new adventure. We saw things we had never seen before and took the time to really explore our neighborhood.

When we got home, she couldn’t wait to make a graph. She ran upstairs and grabbed some construction paper, crayons, markers, and tape.

We sat down at the dining table and got to work. I got out some paper and showed her what a graph of the data we collected could look like. I explained that you can show data in lots of different ways. In the end, she wanted me to make two different bar graphs. She used one graph that I created and gave the other one to Hayes.

I said that a lot of times I use different colors to indicate that it represents a different category of something. With our data, a different color would represent a different animal (or train or butterfly). She started coloring the bars using a color that represented each animal we saw. We added numbers to the bars so you could easily see how many of each animal we saw. Then she decided to draw a picture of each animal next to each bar. This was music to my ears! We have a data viz pro on our hands!

Hannah working on her graph book.

Me: “What did we see the most of?”

Hayes: “Pink! Pink marker!”

Hannah: “Squirrels… and butterflies! I wish we would have seen a rabbit.”

Me: “I know, and maybe next time we will! Hannah, I love your graph. Did you know that graphs have a title, like the title of a book? What do you think the title of your graph should be?”

Hannah: “Today I saw a lot of things in nature, but I didn’t see any rabbits.”

In the end, she made a pretty great one-pager…

Hannah’s one-pager
Okay, Hannah’s graph book 🙂

Hayes’ creation could use a little help, but he had fun creating it (and seeing TWO TRAINS!) and that is all that matters.

Hayes’ graph.

For me, the best part about our data nature walk was talking – albeit abstractly – about what I do in my job and sharing in that experience with my daughter. Maybe it wasn’t exactly what Michaeleen Doucleff had in mind when she wrote her article, but it got us through another day and we all had fun.

The 24-Hour Evaluation

When visiting with organizations about the importance of evaluating their programs, we generally see a lot of enthusiasm. Nonprofits, school districts, and others realize that evaluations:

  • Let organizations know what’s working and what’s not
  • Give stakeholders a way to provide valuable feedback
  • Can give a sense of the impact the program is making
  • Save time and money in the long run

Why then, aren’t organizations doing evaluations for EVERY program, EVERY time?

Two things are missing – time & money. Evaluations are often perceived as (and can be) massive behemoths that take hours and hours to conduct and cost quite a bit of money.

It doesn’t have to be this way.

We have recently been working on a method to produce a comprehensive, focused evaluation with a polished final report or professional presentation in just 24 hours.

Of course, you could spend the longest day ever on the evaluation and get (some types) done in one day. But you could also spread those hours out and get a high quality evaluation completed in a week, with 4-6 hours of work a day.

The 24 hour evaluation has 3 phases.

  1. Phase One: Planning (4 h, Day 1).  In this phase, you’ll brainstorm ideas for your evaluation, create a mini-theory of change, select your evaluation questions, and decide on your strategy for data collection.
  2. Phase Two: Data Collection & Analysis (12 h, Days 2 & 3).  In the next phase, you’ll collect data (whether new or existing), clean it up, and analyze it.  This is the longest phase, for obvious reasons, and if you choose to collect new data, it will necessarily take longer than one 24-hour day to complete the process.
  3. Phase Three: Reporting (8 h, Days 4 & 5). In the final phase, you’ll pick a report type (either an evaluation brief or a presentation), write your content, edit, and publish or present.

A few caveats:

  • The number of evaluation questions asked will be small and targeted. This may look like one deep question or several small questions.
  • If you are planning on writing a report, the report will be a brief report of 5 pages with an additional one-pager summary. There’s not time to write a 30 page report in 24 hours! An evaluation brief, when written carefully, can actually be more effective than a long report because people are more likely to actually pick it up and read it.
  • If you are planning to do interviews and focus groups, you will use a rapid analysis method – no time to transcribe tapes and upload themes into nVivo or some other analysis software. Instead, you’ll be pulling out the big and consistent messages that you’ve heard.

With this method, 90% of what your organization needs to do to make real, meaningful change could be done in one work week spread over 24 hours.

Ready to get started?

Fill out the form below to get the 24-hour evaluation schedule for FREE.

We absolutely 100% pinky swear never ever to send you spam.