Year: 2020

Building relationships in a virtual classroom

By Lindsay Lamb

Source: chicagoparent.com

As we move to online learning this fall, we are documenting best practices for educators to use in a virtual environment. We gathered insight from interviews with educators, reviewed recent literature, attended virtual summits and webinars, and interviewed students and families.

As we discussed in our previous post, providing students with social and emotional learning is critical at this time. That being said, as important as fostering social and emotional skill development and creating engaging content are, they will fall flat if educators do not build relationships with their students. As all educators know, building relationships is critical to successful student engagement, both in person and virtually. It is also a major challenge for the virtual environment. To build relationships virtually, we urge educators to consider embedding the following practices into their lessons:

  • Share personal stories
  • Set norms
  • Set expectations of the online learning environment
  • Foster small group discussions
  • Routinely check in with students and families

Share personal stories. Sharing experiences and being open and vulnerable was a common theme that ran throughout the internship. During an observation of a mentor circle call, one mentor described how she handled a difficult work situation, showing her vulnerability in order to help her interns relate to the material.

Set norms. Educators can begin to build community by setting norms at the beginning of the school year (or program) to convey clear expectations and promote mutual trust and respect. These norms should include guidance on when cameras should be turned on and off, when and how to mute or unmute yourself, how much participation is required, when emails can be sent, what students can do if they have a question, and how students should communicate with others. It is important to revisit these norms during each class/group meeting to make sure they are still relevant and to check in to see if new norms should be added. To ensure that norms are internalized and adopted, consider co-constructing the norms with students. Doing so will increase buy-in, ensure students’ voices are heard, and help make the norms constructive and positive.

Set expectations for the online learning environment. As part of the norm setting process, it would be wise for educators to spend some time discussing expectations for the online lessons. In particular, younger students will require additional time learning how to engage in online learning. It is unrealistic to begin school online in the way teachers normally would in an in-person setting. Much more time must be spent describing the virtual format and what is expected of students, answering students’ (and families) questions, and providing ample time for students to get used to online learning. Many students do not know how to interact in an online environment and need help learning how to do so (e.g., difficulty for teachers to see all students, when the teacher or another student is talking, do not talk over them; raise your hand if you have a question; know how to mute yourself). Even older students can use guidelines to help set norms to foster positive interactions with each other online.

Foster small group discussions. Only offering Zoom meetings for 20 or more students will not effectively build relationships with students. Instead, educators can use the breakout session feature in Zoom to create small groups for students to engage and connect with each other in a more meaningful way. This allows students to connect with each other and build relationships, talk to each other without the conversation getting lost, and relate to each other outside of the larger group setting. This will be difficult for younger students, so family help will be required. Another option is for educators to meet one-on-one or in small groups with each student as frequently as possible. This could be done weekly, bi-weekly, or monthly and is particularly important for younger students. All students need to know that they matter, that they are cared for, especially now. While this might be more time consuming for some educators, the payoff is far reaching.

Routinely check in with students and families. Finally, we know online learning will have its ups and downs. Not everything educators planned out will work, and some things will work well for some students and families while others will not. The key to navigating this conundrum is constantly checking in with students and families. Our advice is to have daily check ins regarding the technology access, duration of lessons, content of lessons, and how the student is feeling. This will also build relationships with each student and their family allowing them to feel more comfortable advocating for their/their child’s needs.

Online learning is not perfect, particularly for young children. Helping educators identify best practices to use while engaging with children online will help make online learning a more positive experience for everyone. Stay tuned for more best practices and the launch of our report in partnership with Ignite MindShift soon!

Best practices as we prepare for online learning

By Lindsay Lamb

If you are like me and have school age children, you probably spent the summer worrying about everything related to your child, your child’s learning, your child’s social and emotional wellbeing, your child’s teachers, and education in general.

Perhaps as a way of coping with this, and also as part of a larger project we were working on with one of our clients, we have been developing a Best Practices document for online learning (keep your eye out for it, we’ll let you know when it is available!). Our goal was to find answers to the following questions:

  • What are some effective strategies teachers use to engage students in online learning?
  • What strategies from the in-person environment can pivot to an online format?
  • How do teachers build relationships with their students?
  • How can we best support students and families during these challenging times?

To explore these questions, we spent the summer interviewing teachers, participating in webinars, reading timely research articles, and speaking with experts in the field. Based on all of these data, we identified six broad categories we believe will help educators navigate teaching and learning this fall:

  • Supporting students’ social and emotional needs
  • Building relationships
  • Creating engaging content
  • Providing consistent routines and procedures
  • Expertly using technology
  • Knowing when to pivot

Supporting students’ social and emotional needs.

While all of these practices are important, the theme that holds them all together and ran through our interviews, research, and learnings was providing social and emotional support for students, families, and teachers. While evaluating Austin Independent School District’s (AISD) social and emotional learning (SEL) program, I often heard program staff say SEL is not one other thing to add to teachers’ plates, it IS the plate. Now, I feel like SEL is the soil from which all learnings will grow. Without the ability to talk about emotions, build relationships, foster a safe learning community, and share experiences, teachers will be unable to teach, and students will be unable to learn. As one of my daughter’s teachers said in a meeting, we all need to give each other some grace. We all need to be okay with things not being perfect or exactly how we envisioned learning to be this school year. We are in the midst of a pandemic, and what matters most is keeping each other safe, happy, and healthy.

Recently, one of my friends and former colleagues wrote about her experiences teaching music in an online learning environment. The core of her teaching? Social and emotional learning. Was teaching music online perfect? No. Did she and her students experience technology problems? Yes. Did students benefit? Absolutely. Students need the arts, especially now, and social and emotional learning is such a perfect fit for arts instruction. Students need something to help them connect with each other, express their feelings, focus on something outside of the stresses they are experiencing because of the pandemic and political unrest. They need to feel part of something bigger than themselves, and just get the chance to be a kid. Is teaching the arts – or any subject – online an ideal way to experience learning? No, but there are great teachers out there making the best of this situation, and best practices we can learn from them.

I just want to close by reminding everyone that so many educators are going above and beyond to be there for our children while also caring for their own families. I cannot thank them enough. On a personal note, my daughter got through the summer by continuing to participate in her dance class over Zoom. She relished in being able to just be a kid, to wear a costume, to dance with her brother, and be silly. Her teachers always made time to listen to her stories, do a quick check in with everyone and see how they were feeling, and build a community. Rather than bemoaning online learning, let’s embrace it for what it is since we cannot do anything to change the situation. Let’s look for the joy and find ways to connect, reflect, and just help your kid to be a kid.

SWOT analysis: Not just for Silicon Valley

By Lindsay Lamb

Photo courtesy swotanalysis.com and HBO

One tool I have in my evaluator’s tool box might surprise you – it is the SWOT analysis (Strengths, Weaknesses, Opportunities, and Threats). What is the SWOT analysis? It is a method you can use to walk through a recent project, proposal, report, or any problem that could use some collaborative processing and innovation. It is helpful to conduct this analysis before, during or after the culmination of the project. Honestly, SWOT analyses can be used at any decision point, it just depends on the project, and the project timeline.

The quadrants below can help you think through each component of the analysis:

Quadrants of a SWOT analysis
  • What were the Strengths of your recent project/proposal/report? What was something internal to you/your organization that helped in this project/proposal/report? If you achieved your goal, why do you think you achieved your goal? If you did not achieve your goal, take the time to identify some strengths of your process. Find those kernels of positivity, trust me, they are there! Maybe it was setting strong meeting agendas, redesigning your report template, collaborating with project managers, or trusting your gut.
  • What were some Weaknesses internal to you/your organization you faced while working on this project/proposal/report? If you did not achieve your goal, what were some things that may have contributed this result? If you did achieve your goal, what were some obstacles you had to overcome in order to achieve them? Could communication between you and program staff and/or other collaborators been better? Perhaps the project didn’t quite align with your strengths and overall vision, or maybe the timeline was unreasonable.
  • What were the Opportunities you found in your recent project/proposal/report – even if you did not achieve your goal? What was something external to you/your organization that helped you in this project/proposal/report? Maybe you made a new connection with someone while working on this project. Can you leverage this project/report into new opportunities?
  • What were the Threats associated with this project/proposal/report? What was something external to you/your organization that threatened your project/proposal/report? Perhaps funding got cut, or there program staff experienced turnover, or the organization you were working with pivoted their goals midstream.

I have used this analysis to collect my thoughts after applying for a grants (both that we received and those we have not received), responding to RFPs (again for those that we did and did not receive), after a large data collection procedure, and after working on a large comprehensive report for a client involving multiple consultants. In all of these cases, engaging in a SWOT analysis has helped us recognize the strengths of our work, so we can remember to draw on these skills in future endeavors.

If you are truly reflective about your work, SWOT analyses can you help identify aspects of your work where you can grow. Doing so will help you learn from these experiences because chances are they will likely come up again. Maybe you realize that you need to be better about scheduling regular meetings with clients and/or other consultants, take the lead on a project when others fall short – even if you are not the most experienced person on the job, or take a class in a new statistical method.

SWOT analyses also provide you with an opportunity to celebrate your hard work and remember what helped you achieve our goals. When was the last time you celebrated an accomplishment? Take the time to do so now, you deserve it!

However you decide to use a SWOT analysis, I have always found this process to be helpful, regardless of the topic or the project outcome. Setting aside time to reflect on our work is always a good thing.

How can we expand on the SWOT analysis? Something I have not done yet but think would be excellent to do is to conduct a SWOT analysis with a client after the project is complete, or data have been collected, or the report has been finalized. The SWOT analysis process is a great way to foster conversation and reflect together. Who knows, you might even discover a new opportunity through this process!

Knowing when to pivot

By Lindsay Lamb

Recently, I was in a meeting with a client presenting data from a quick survey check in. We were reviewing the data visualizations I made based on some earlier feedback. Sometimes when I share survey data, I collapse categories to show an overall level of agreement, or to indicate the overall percentage of favorable responses.

Since we were examining responses from students who were engaging in a virtual internship in the middle of a pandemic, we figured most responses would generate a negatively skewed distribution (see graph below, and if you’re interested in learning more about skewness, read this!).

A negatively skewed distribution (meaning most responses are positive, or favorable)

Indeed, preliminary examinations of the data suggested that students’ responses were skewed such that there were few responses at the low end of the scale on our various survey questions (e.g., disagree, strongly disagree, not at all likely, etc.). We decided to examine movement in students’ responses at the top end of the distribution (e.g., strongly agree, agree, extremely likely). Based on this decision, I crated a visualization only showing the most favorable responses (presenting only the percentage of strongly agree or extremely likely responses, for example).

So, back tot the meeting.

All was going well. We were having a lively discussion of the data I presented in my one-pager when the client made a casual comment about being bummed about some of the responses. They had spent so much time pivoting their instructional material to be engaging in this new virtual environment, and only a third of responses indicated students felt the lessons were extremely interesting.

Slightly over a third of students’ found the lessons extremely interesting, just over half found the lessons or extremely relevant to their lives.

Ugh. My heart sank.

Fortunately, I knew there was more to the data. I pivoted. I had the data up in an interactive dashboard and shared my screen with our client. I recomputed responses to include extremely interesting and interesting.

“Actually, nearly all of your participants were interested in the lessons and found them relevant!”

Pivoting and recomputing the responses told our client a more positive story.

Sure, a little over a third found the lessons extremely interesting, but the bigger story is that most students were engaged and found the content relevant to their lives. This was huge! Our client had pivoted their entire learning platform to a virtual environment, and students were engaged.

What did I learn?

  • Sometimes it makes sense to have a plan, but you need to adapt. We are program evaluators and there isn’t something that we love more than a good evaluation plan (except maybe a logic model!). Sometimes we can get rigid and only want to look at the data in a specific way, thinking that we will know what the data will tell us. Sometimes, however, we need to let the data tell its own story.
  • Don’t be afraid to pivot. Recomputing the data not only showed our agility as researchers and consultants, but also helped our client see their data in a new way. It helped our client see the value of their work and trust us in our services.
  • The power of one-pagers. Having a one-pager with data visualizations and short text explanations helped our client easily see the data and understand the story. It also helped us recognize where we needed to go in our analysis. Having the data in a digestible way facilitated our conversation and helped us see where we needed to go. It was empowering for me and empowering for our client.
  • Have your data handy. In addition to having a user-friendly one-pager, having the data accessible in a sharable format helped us see where we needed to go in our analyses. It was empowering for me and empowering for our client.

School Goes Virtual (Again)

by Lindsay Lamb (with a few random comments by Andrea Hutson)

Due to the recent increase in COVID-19 cases nationally, it is likely that if you have a school age child, they will not go back to school in a normal manner (regularly, in person) this fall.

As I have mentioned before, I have two children that I have been doing my best to keep educated, entertained, and healthy while we have all been sheltering in place these past four months. My eldest is slated to start kindergarten this fall, and what should normally be an exciting, happy time is now riddled with fears. Going back to school in person could mean infecting teachers, our entire family getting infected, and a disruption in routine if schools are unexpectedly closed again. Virtual learning provides its own set of challenges, including me continuing to have to juggle work and school time as well as social and emotional trauma that my daughter has experienced during this time of isolation.

I am not alone. The long and short of it is that closing schools has placed an impossible burden upon families, particularly families coming from low-income communities. But given the recent spike in COVID-19 cases, what do we do? School leaders are placed in an impossible position as are teachers and families. This affects the entire community. I do not envy leaders at this time.

[Sidenote from Andrea: I just received this question from a friend – “Would you rather facilitate a PTA meeting around schools reopening in the fall, or do a trust fall into a pit of crocodiles?” The fact that the crocodiles are an appealing option shows how charged this issue is.]

Where does this leave us?

Instead of getting caught in an endless loop of ranting and venting (which is hard), let’s find some positive things we can build on as we begin to navigate virtual student learning in the Fall.

Fortunately, we have some more time to plan than we did in the spring. We can draw on the virtual teaching and learning experiences from the spring and learn from them, because chances are, no matter what our plans are in the fall, we are going to need these best practices in our arsenal until we have a COVID-19 vaccine.

What do we know about virtual learning?

1. Not all virtual learning environments are bad!

Take for example, one of our clients who is helping interns in a summer work program learn essential social and emotional skills necessary for workplace success. These students come from all different backgrounds and are eager to learn. Our client expertly pivoted their course content, creating relevant and timely material that the students are eager to learn. We are working with them to develop a list of best practices to share with other educators in the coming months.

Some research has shown that some children, particularly older children, are thriving in the online environment. Anecdotally, another program that we work with measures the anxiety and depression levels of its teen participants, and their levels of both actually went down after lockdown began.

Online education works best when it is:

  • high quality (obviously)
  • structured, with points for real connection
  • a mix of formats (sorry, but a 7-hour long Zoom meeting is not going to be effective)
  • done with older children & teenagers

2. Teach parents to be co-educators. Parent involvement is even more critical.

Parents in this time know that, even if it is unfair, they have been thrust into the role of co-teacher for their students. To the extent possible, leaning into this can produce benefits that will last long after COVID is over.

One client offers reading tutoring services to at risk students in grades K-2. Although there were many challenges with continuing their program virtually, some tutors reported a strong benefit of having the ability to connect with students’ families in a way that had not been possible before. Parents are able to watch the tutoring sessions and get a better idea of where and how their children are struggling, and are able to learn from the tutors some ways to reach their children.

Schools and organizations can help by:

  • Engaging with parents early in the process
  • Understanding that parents will have different needs and levels of availability
  • Providing materials to teach the parents as well as the students

3. Don’t forget Social Emotional Learning (SEL), and be creative

Virtual learning leaves out the hugs, the high fives, the social and emotional connections and learning students need now more than ever. And who else needs this support? Teachers and other school staff. Andrea recently sat in on a meeting about teacher and student SEL during this time. Some takeaways:

  • SEL makes a meaningful difference for both teachers and students
  • Teachers must fully embrace these concepts before being able to teach them to students
  • Time must be set aside for SEL – it can’t be ‘one more thing’ added to already full plates
  • SEL can fall flat for older children if not approached carefully. Many elementary schools are now teaching SEL skills, and when kids get to middle/high school, they think, “Been there, done that.” When we are teaching SEL skills to our middle and high school students, instead of trying to use the same content aimed at younger children, we need to think of how we’d teach adults SEL concepts and adapt that way.

4. We need a better way to reach students who are disengaged

Raise Your Hand Texas interviewed five Texas teachers about their teaching experiences this Spring. These teachers faced a number of challenges, and hearing their stories is inspiring. The teachers got super creative, conducting zoom dance parties, scavenger hunts, virtual field trips, and more. They show that even in the time of COVID, school can be fun and creative, and that many teachers are doing everything they can think of to engage students.

However – As these teachers described, despite their best efforts, some students just stopped participating. The official tally is about 10% statewide, but we think the ‘real’ number is higher – and it’s not evenly spread across communities. LA, for example, lost about 1/3 of their students in the spring semester [there’s a wonderful NY Times article about this issue] Anecdotally, we’ve seen some organizations talk about losing 1/2, or even 2/3 of their student participants. This is a major issue that cannot be ignored by the local, state and federal governments. If we want true equity in education, districts need a plan now for how to reach out to these families and find and connect them to the resources they need to move forward.

Which students are more likely to drop out?

  • Younger students, particularly in PreK, K, and 1st grade
  • Students from low income families
  • Children with poor internet access
  • Children who do not have an adult in the home who can supervise learning
  • Students with special needs

I can speak to my own experience with younger students. Both of my children (5 and almost 2) were offered short zoom gather times (~30 minutes each) throughout the spring. By the end of my daughter’s meetings, her class of 12 was down to 2-3 participants, as was my son’s. Many parents and children clearly did not feel it was worth it to continue, despite the fact that the meetings were engaging and even involved small, fun crafts. We continued because:

  • it brought a sense of routine to our day
  • it was a nice break for me
  • I had the luxury of a flexible job
  • We had reliable internet
  • We had multiple computers/devices
  • We had resources

And even despite these things, it was sometimes a struggle to make it through one 30-minute zoom meeting daily. Going all virtual this Fall seems like a challenging prospect, to put it mildly.

[Another random note from Andrea: my 4-year old’s Montessori school also did 20 minute virtual meetings daily, and that experience made me thrilled that we can sit out this school year. Good luck, Lindsay, and other parents of kindergarteners.]

What can schools do?

Schools need to gather whatever information they have from the virtual and online learning that happened in the spring as they prepare to help their students in the Fall. Did some teachers try paper packets? Was that helpful or not so much? For all students or some students? What kind of lessons were most engaging? How are students feeling socially and emotionally? How can students connect socially while remaining six feet apart, or on Zoom? How are schools addressing these needs (which likely will be huge in the fall)? What resources do parents need to help engage in their child’s learning?

The best time to get that information would probably have been at the end of the 2019-20 school year, and no doubt many districts did that. But almost all districts were scrambling, and COVID cases started to dwindle at that time, leading to hope we’d be able to operate somewhat normally this fall.

Since that’s not the case, districts and organizations serving students should take time now to conduct parent, teacher, and student focus groups/interviews/surveys and distribute any information about potential pitfalls and/or best practices to their staff members right away. Districts need a plan to involve parents. A plan to support SEL. And most importantly, a plan to reach as many students as possible.

The one-pager’s sidekick, the sidebar!

by Lindsay Lamb

Something that I have often struggled with in producing a succinct and powerful one-pager is where to put technical information, a brief description of data used in the study, analyses conducted in the study, study design, survey questions, etc.

In a typical research paper, that information is generally presented first. Most of us come from an academic background, where we learned to write reports with Introduction, then Methods, then Results (which oftentimes are full of technicalities), then, finally the Conclusion – where the actual findings and interpretation lie.

The problem? It’s BOOOOORING. You’ll lose all but the most dedicated reader with reports that are designed that way, and in a one pager, it’s a killer. The conclusions need to be front and center.

But what about that technical info about how the study works?

Sidebar to the rescue

This is where the sidebar comes in. You still need to be succinct, but it is a way to break up the layout of your one-pager and go more deeply into a specific topic that is essential to your one-pager but does not need to be part of the overall one-pager narrative.

Here is the template we’ve shared with you in the past:

An overview of our one-pager template.

Still not sure what to use a side bar for? The sidebar can also include photos, data visualizations, description of program elements, survey questions, survey response rates, quotes from participants, anything that your audience should know, but can stand alone in its own section. Here are just a couple of examples:

Here are a couple of examples of sidebars we have used in recent reports.
Here is another sidebar example using data visualizations.

Here are some key takeaways to help you become the master of the sidebar:

  1. Put information that is necessary to understand your report in the sidebar.
  2. Add icons, photos, or a data visualization that will break up the layout of your report.
  3. Keep it simple! Details belong in the report (or the appendix!)

Create a One Pager In One Hour

by Lindsay Lamb

Have you ever been in a situation where someone needs a one-pager – like yesterday? You’re just sitting at your desk, and all of a sudden, your boss comes by and says, “Hey Lindsay, the superintendent [program manager, director, business analyst] needs a one-pager on that research [analysis, presentation, etc.] you’ve been working on. Can you get it to me in two hours?”

Internally, if you’re anything like me, you’re probably thinking: “AHHHHHH!!! HOW AM I GOING TO DO THIS??? THIS IS CRAZY!!!”

Externally, you remained poised and self-confident : “Of course, I’m happy to help!”

So, what can you do to make this happen? Take a deep breath and follow these simple steps.

  1. Use what you’ve got. Before you get started, do you have a longer report, outline, or presentation? If yes, then this process just got A LOT easier. If not, don’t worry, I’m sure much of that information exists in your head. Go ahead and sketch out a rough outline either using word or on paper – whatever you need to do to get those creative juices flowing.
  2. Use One-Pager Building Blocks: Now that you have a rough idea of what you want to say, use these four building blocks to help you get started building your report:
Use these building blocks to design your one-pager. You can even use them to segment the page (or pages as a one-pager can be two-pages front and back).

If you have a larger report, presentation, or outline, what are the main headings and/or major findings? Do you already have a description of the study? Trim that down and drop it in at the beginning or in the sidebar. If not, quickly jot down the things from your report/presentation/analysis that stand out to you. Whittle them down to the top four ideas and rank them so you know which one to add depending on how much space you have in your document. Keep in mind that when you create your report, you want to include as much white space as possible to make it easier for your audience to read you report. Don’t muddle it with too much information.

Now that you have a better idea of what you want to say, it’s time to start thinking about the layout. As a general rule, a one-pager can be broken down into three or four major sections per page: The introduction, a major finding, a visualization and a photo. Sidebars are a great way to visually break up the page and add some of this information. Here is a generic template that can give you a rough idea of what your one-pager might look like.

Here are the building blocks of a one-pager using our generic template. Remember, your one-pager can be two pages (one-page front and back):

Here’s a snapshot of our generic template. There are three sections represented here comprised of finding 1, finding 2, and a sidebar.
  1. Have you created data visualizations? Can you use those? Can you combine visualizations (small multiples, maybe)? If not, think of ways to create data visualizations quickly. Most people respond well to percentages and simple figures. If time permits, create a visualization using that type of information. Remember, as an evaluator, you most likely won’t be the one sharing the information from this report, so it is important to ensure the data visualization can stand on its own without a lot of explanation. Don’t have time to create a visualization? Use icons or photos to break up the text and add something visually appealing for your reader.
  2. Summarize key takeaways. What do you want you audience to remember? Are there important strategies they need to consider? Program elements that should be removed? Funding sources that should be added or eliminated? Share them, and preferably using bullets 😉

Whew. That was quick, but I hope you have found it useful. If, like me, you find yourself in tis situation, remember to use the following strategies to help you create a one-pager in just a couple of hours:

  1. Identify the major findings
  2. Break up the page into three or four sections. Use a sidebar to include additional information relevant to your analysis such as a description of the study, data used, research questions, project timeline, etc.
  3. Create effective and efficient data visualizations that correspond with your major findings. Don’t have time for visualizations? That’s okay, you can use icons or photos.
  4. Highlight the key takeaways from your analysis. What do people need to know to make informed decisions?

That’s it!

Want to learn more? Our upcoming mini-course will take you deeper into creating an effective one-pager – stay tuned!

Entering the Evaluation Job Market During a Recession: Advice for Recent Graduates

by Lindsay Lamb

I have been thinking a lot about students graduating lately. Graduating high school, college, graduate school. My daughter is graduating from Pre-K and will start Kindergarten in the fall. I have friends and family with kids graduating high school, excited to celebrate their accomplishments and start off on their adventure. Now, thanks to the pandemic, those dreams have a much different reality. There is no prom, there is no graduation, there is no party to celebrate with friends and family.

As if reading my mind, I was asked to provide recent doctoral graduates with advice they might find helpful as they enter what is sure to be a tough labor market. What could I possibly say to them?

Fortunately (maybe?) I graduated during the 2008-2010 recession. When I first entered the job market, post-doctoral funding was frozen, full-time professor positions were being transitioned into more adjunct faculty positions, the private and public sectors were beginning to enact hiring freezes.

So, back to that advice.

The prompt I got was, Based on your experience in 2008-2010 on the job market:

  1. What strategies or resources helped you get that job offer?

I made connections. I built relationships. I put myself out there and honed some skills. I essentially created my own internship with several educational non-profits in the area. Sure, this didn’t make me money right away, but it paid off big time in the end. This is the only reason why I got my job in AISD.

I was already on the fence about an academic role. With a degree in educational psychology, I wanted to have more of an effect on students, staff, and administrators I was researching. I wanted to make a difference in students’ lives. My experiences during this time reaffirmed my beliefs: I wanted out of academia.

I started contacting educational non-profits in the Austin area to see if they needed help. Surely, I could create a survey, conduct observations, do a quick focus group, anything to help them out. I had a lot of interest, but not any pay. On the plus side, I was still enrolled in graduate school, and was getting paid to be a teaching assistant and was teaching a class at a local university so I could make it work.

Because of this work, I finally got connected with the Director of Research and Evaluation at the Austin Independent School District. Luckily for me, they had a job opening. I spent 10 amazing years there conducting meaningful program evaluations and learning from my colleagues every day.

  1. What was the most challenging aspect of finding a job during the recession?

All of the uncertainty and frustration I faced. Several of the jobs that I applied for suddenly lost funding or were put on a hiring freeze. These unknowns exist today, and unfortunately, there isn’t anything we can do about them, but we can keep applying for jobs, learning new skills, and making new connections. Anything you can do to keep you going one day at a time.

  1. What advice would you share with your younger self and/or graduate students who are just now looking for jobs?

Don’t give up! I know this sounds hokey, but the only way things will not work out is if you stop looking for a job, stop honing your skills, and stop building connections. Find some free (or even not free, but cheap) mini-courses on data visualizations, creating one-pagers, learn R, start a blog! Also remember that you have learned amazing skills during graduate school that can be applied to many fields, not just academic jobs. Businesses are looking for skilled workers who can analyze data, create effective data visualizations, share results in an easy to understand way. If you take a job that is below your skill level, raise it up! Bring in your skills to show that you can do more and be more, maybe you can create an opportunity for yourself by showing all of your skills.

I know this is a challenging time, but we are all in this together. As cliché as it sounds, I truly believe things will work out one way or another. It might not be exactly what you had planned, but it is how we deal with setbacks and adversity that define use. These experiences force us to look inward and grow, and In the end, I truly believe that we will come out stronger than we were before.

Good luck!

Brand Your Documents in Three Easy Steps

by Lindsay Lamb

We have talked a lot recently about tools for your evaluator’s toolbox. Today I have another essential tool to for you: consistent branding. Clear and consistent branding can help you more quickly and efficiently create short and long reports. Doing so will ensure that when people look at a report, they will instantly know that it came from you. I have found that consistent branding takes some of the guess work out of report writing. I no longer have to think through color choices, layout, font, or headers. That leg work has already been completed. I’m guessing you have a little free time now, so you might as well spend some time thinking through your branding. Trust me, you will thank me later.

Enough of my pep talk.

I’m sure you are wondering how to create a clean, consistent look for your reports, one-pagers, and blogs. Follow these three simple steps:

Logo

Make sure you have a unique logo that lets people know what your business/firm/program is about.

You can use a logo generator to help you out! We used logogenie.net.

Colors

If you have a logo, use colors from your logo in your reports.

Remember the color wheel? Use it! Make sure you incorporate complimentary colors (colors on opposite side of the arrows), and colors in a shade that is easy to see on screen, in print, and for those with color blindness. A good test for any report is to print it out in black and white. If your colors are all grey, try using gradations within the same color to convey meaning.

A color wheel, complimentary colors are located diagonally across from each other.

Use colors consistently such that all headers are the same color, etc.

Caveat – if you are generating a report for a client with clear branding, you might want to use their branding and simply insert your logo. Check with program staff, your supervisor, a colleague, a friend, or even reach out to us if you have questions and want some feedback!

Here is our style guide, and a link to the logo generator we used.

Font/layout

Choose a font that is unique and easy to read

Stephanie Evergreen goes into great detail on the importance of choosing fonts in her book, Presenting Data Effectively.

You can select two fonts, or fonts that are bold and light to use to signal to the reader level of importance. There are a plethora of fonts out there, download something unique online.

The layout of your reports should also be consistent. Check out our recent blog post on one-pagers. The idea is that people should pick up a report and instantly know that it came from your organization/consulting firm.

Our typical layout scheme.

One final tip – try out your branding! If you have a few ideas, try them out and see what your clients, friends, colleagues think. Send your ideas to us and we will look them over. Your brand is the first thing that people will see when experiencing a report you have created, so you want to make sure it is right.

The Secret Reason Leaders Aren’t Reading Your Evaluation Reports

by Andrea Hutson

Evaluators: be honest: have you ever worked hard on an evaluation, found some really exciting findings, wrote an amazing report, and turned it in expecting….

  • praise from program leadership!
  • noticeable, positive program changes!
  • maybe a promotion? (or extended contract?)

And instead, got… crickets?

Let me raise my hand the highest. It’s definitely happened to me, and one time in particular is burned into my mind.


I was in a final report meeting with a large agency. The evaluation we were reporting on was a multi-year, multi-site evaluation with mixed methods, requiring multiple staff, travel, a ton of time with SPSS and R, and a very long time writing and editing the report.

There were about 10 people in the that final meeting, including the relatively new program director (sidenote: one interesting thing about long evaluations – in some organizations, change is so constant that often the only people who are with the program from start to finish are the evaluators).

Our final report was over 100 pages, with appendices, and lots of technical detail.

After I presented all of the results from the study, the program director looked at me and smiled. “Wow! These are really interesting results!” I smiled back.

“But…”, she started… and I felt my heart sink.

Then she dropped a bomb.

“But, you know no one will ever read this.”

Oh my goodness, so painful to hear – but I knew she was right.

It’s not that the content was bad (of course it wasn’t! I wrote it!), it’s just that it was too dang long.

Decision makers don’t have time to read.

The average 30-page evaluation report takes about an hour to carefully read and process (and at over 100 pages, my report would have taken 3+ hours)

The problem is, leaders – especially the higher you get up the food chain – don’t have an hour. 

There was a fantastic study of CEOs conducted in 2018 by the Harvard Business Review (https://hbr.org/2018/07/the-leaders-calendar) that explains why. (Although the report only looked at CEOs from the business world, non-profit and governmental leadership likely have very similar constraints.)

According to the study, CEOs spend the vast majority of their time (about 72%) in meetings, either with their own staff, leadership, or clients. They have very little time alone, and most of that time is in small chunks (i.e., not conducive to report reading).

Check out this time graph of one CEO’s alone time, arranged in 15 minute green blocks.

chart of ceo's time arranged in blocks
This CEO has very little time to himself. Source: https://hbr.org/2018/07/the-leaders-calendar

Even when we include Saturday and Sunday, this CEO only had 8 hours of one-hour or more alone time for the entire week, even though he was working a TON of hours (note that he starts working before 6 AM most days).

The study noted that the CEO spent the majority of that alone time (55%) reading and responding to emails. After we remove that time, he had only had about 3 or so hour-long chunks for all other unscheduled tasks.  FOR THE WEEK.

And that’s the reason that long reports don’t get widely read.

There’s just no time!

What can you do?

Now beyond going past the existential crisis about why we write reports at all, there are a few solutions. Here are three ways you can maximize your impact.

  1. Don’t write a report; make a presentation instead. This method is getting more popular over time; UX Evaluators typically present results in one final PowerPoint. If you noticed above, we added on a presentation, which was the product that got us traction with the program leadership. That’s also a possibility, if a report is absolutely required (although the product we had worked for hundreds of hours on became essentially a doorstop).
  2. Write a brief report instead of a long one. We’ve had tremendous success with 4 or 5 page evaluation briefs. When carefully written, these briefs can include the vast majority of findings, details, and recommendations that people need, in a format that’s digestible and a relatively quick read.
  3. Add on a powerful one-pager. One pagers delete all but the most essential content, giving just the highest level findings from a study. The best thing about one-pagers is that they can be added on to a brief report, or even a very long one. Stephanie Evergreen has a great post on the 1-3-25 model, (https://stephanieevergreen.com/the-1-3-25-reporting-model/) in which a one-pager is tacked on to an evaluation brief which is tacked on to a longer report (but notice: the ‘longer report’ is only 25 pages).

Want to learn more about option 3?

Lindsay and I are currently working on a course on one-pagers that’s designed for anyone who wants to make more of an impact with their reports. This course will provide easy steps to turn a long report into a one page document that engages readers and packs a punch. And you’ll do it without needing graphic design skills or any fancy software. We’ll be sharing some of it on the blog, but if you’d like to be notified when the course goes live. subscribe below.