Guest: Dr. Alaina Szlachta

In this episode of the Become an IDOL Podcast, Dr. Alaina Szlachta, author of the book "Measurement and Evaluation on a Shoestring," shares her insights on simplifying measurement and evaluation strategies for learning and development professionals. Tune in to hear:

1. How Dr. Szlachta's research on the challenges of measurement and evaluation led to the writing of her book.

2. Strategies for getting buy-in from stakeholders and being a strategic business partner when it comes to measurement.

3. Ways to leverage AI and technology to save time and make measurement and evaluation more efficient.

Connect with Alaina Szlachta on LinkedIn

Connect with Dr. Robin Sargent on LinkedIn

Discover how to kickstart your instructional design career in 2025—watch Dr. Robin Sargent's free webinar here

Listen to this episode below: 

Subscribe to Become an IDOL Podcast: Apple Podcast | Spotify | Google Podcast 

Enjoy the Episode Transcript below:  

Dr. Robin Sargent  

Welcome to Become an IDOL. I'm Dr. Robin Sargent, owner of IDOL courses. This is the place where newbies come to learn and veterans share their knowledge.

I have here with me today, Dr. Alaina Szlachta, and she is, well, you're a lot of things, but the thing that we're going to focus on today is that you are the author of the book, 'Measurement and Evaluation on a Shoestring'.

Dr. Alaina Szlachta

That's right

Dr. Robin Sargent  

but Dr Alaina, would you do a better job of introducing yourself and what it is that you do?

Dr. Alaina Szlachta

Of course, thank you for asking. Interestingly. As a side note, I put as a skill, I'm curious if you would agree with this or not, I put as a skill, published author in like my list of skills in the world, and it was actually a really fond moment for me, because we write all the time. You have a PhD, I have a PhD. We both have written. But to be like a published author with a book, it's something like I'm still settling into, like that first time author, feeling accomplished and proud of that.

In any event, that's not what you asked me to talk about. Who am I? In addition to being now a published author, I like to tell the world that I'm a data nerd and to be so truthful, that goes back to when I was a child and I was an athlete and I got feedback from coaches and peers on how to be a better athlete. That was data. And I think I fell in love with data and feedback because I saw its utility early on in life, and have kind of just like followed that trajectory until where I am today, which is essentially helping people use data to improve, whatever it is they want to improve. And my favorite way of supporting people, it's actually the niche that I offer is I create bespoke assessment tools to evaluate performance and capability growth. I feel like it's a really big blind spot, especially in the world of adult learning, where we have performance data from like operational tools, but we don't have data on like behaviors and thinking and attitudes that roll up to understanding where we are today and where we want to be.

And so I really enjoy creating those tools for mainly leadership development companies, but also leaders of learning teams can really benefit from those assessment tools too. 

 Dr. Robin Sargent  

Okay, so what actually inspired you to write your book then, right? So was there a defining moment a certain client, or you just wanted to get it out to the masses? Or what?

Dr. Alaina Szlachta

Actually, it was more like serendipity. So everybody has a COVID story of like what they did during COVID, and what I did during COVID is I was in India, and I did original research, because around that time, I just started my business, and I wanted to better understand why do people struggle with measurement so much because I had this lofty idea that when I left the corporate world and I started my own business, my clients would be so excited to incorporate measurement and data collection into their learning programs, and that they would just be so willing to follow my Best Practices advice, and they weren't even with my own clients who were hiring me for my expertise to build learning programs that could, like make a difference in the world.

They still pushed back on wanting to incorporate data collection and measurement strategy into their programming. And I thought to myself, Why? Why? Why is this such a pervasive challenge? So I did an original research I interviewed 40 people around the world during COVID and I asked the question like, 'What are your challenges when it comes to measurement and evaluation?' And if you could wave a magic wand and have the world be ideal, and everybody would be embedding measurement into the workforce and your daily practices, like, what would that look like and what would it take to get there?

So the book is really because I did a conference presentation at an ATD Core Four conference in 2021 I presented the results of my original research and the best practices to help L&D people improve their measurement practice. There was an editor from ATD press in the room. They reached out to me, and they said, Hey, we're looking for an author for this book. Would you consider submitting a proposal? And I said, Yes, and the rest is history. 

Dr. Robin Sargent  

Oh, and so that was. Yes, so you've been working on it since 2021

Dr. Alaina Szlachta

Well, it was like November of 2021 that I was invited to this opportunity. So it was the end of the year. I submitted the proposal for the book in 2022 in January, and got accepted pretty close to around that same time. So I was preparing to present the original research that I had done, along with an extensive literature review I had already the assets put together because I wanted to publish my findings in like a peer reviewed journal article, because that's what PhDs do. And so writing the book was actually relatively easy because I had all the assets. I had my original research. I had done an insane literature review on all the various models of measurement and ROI and evaluation strategy. I didn't get all of them. There's tons of models out in the world, especially when you think about like in the interdisciplinary sense, like social work cares about evaluating education, K12 cares about evaluating education, higher ed, human performance, improvement. There's a ton of different lenses that all have their unique models and methods around measuring and evaluating programming. So I did my best to look at the different models and perspectives across disciplines, but all of them touched adult learning and education in some way.

And so I wrote the book in probably four months, because I had all the stuff ready and raring to go, and I will say I had the help of a developmental editor who was insanely helpful, because I had this flow for like how every chapter would feel, which starts with a story, a real story. I wrote the book to make it feel like you're sitting next to me having coffee, because, let's be honest, most people don't find data and measurement all that interesting or accessible, and so I wanted to make it accessible by adding human story, real conversation format into the book, so that it was at least a good read, if not also insanely helpful. So my developmental editor, there was one chapter that was unlike the rest, and Jack was so helpful, and he's like, let's break out this one chapter into and infuse it into the rest of the chapters. And so once we got over that hurdle, the rest of the writing and getting it into the copy editing stages was was really pretty quick.

Dr. Robin Sargent  

So I bet a lot of I mean, and probably in your research, you found that just kind of like what you said, that a lot of people struggle with incorporating data, and I mean measurements, specifically evaluation. And of course, we are known to have limited resources. So how does your book help them break through that barrier, like what kind of things are in there?

Dr. Alaina Szlachta

The big, big message that I want people to take from the book is that in order to manage our resources wisely, we have to be hyper strategic. So I believe that the problem, especially in the corporate learning and development world, is that we try and pick a tactical approach to measurement, like let's use the Kirkpatric model, or let's do an ROI study, or let's follow the LTM Methodology. We pick a tactical approach without having a clear understanding of what are we really measuring for? What are we going to do with the data? How is the data going to help us improve or inform and so without being hyper strategic, we sometimes over complicate things. We make them more complex, time intensive and expensive than they actually need to be for what it is we're trying to accomplish.

So I'm model agnostic. I think Kirkpatrick and ROI and LTEM and the Measurement Map. And there are like 50 different models that you could pursue to help you better measure, and I talk about them in my book, they're all great, and they have their place in time. So our most important action that we could do to not only simplify measurement, but also to make it less time intensive and expensive, is to be really clear on why are we measuring? How are we going to use the data? What is the outcome that we want to get from our measurement practice? And then we can decide what resources do we want to invest? And for example, sometimes we just need to do a return on expectations measurement strategy. Which there's a few people that write about return on expectations. There's not like one way to do it, but the idea is that we just want to give our stakeholders a return for what they expect should come about in the program.

And so in the book, I talk about some questions that you can ask and what stakeholders to include in your return on expectations evaluation approach. It's actually really easy. It's just a conversational approach picking out the right metrics as a result. It's not complicated, it's not time intensive or expensive, and in some cases, just having a return on expectations approach is actually going to get our foot in the door to build relationships with stakeholders and start having a better understanding of what's happening with learning and how we can improve its effectiveness and impact.

Dr. Robin Sargent  

That actually kind of like is a nice setup for my next question, which is this lack of leadership support that we experience. What are some of these strategies that you would recommend so that people can get the buy in for measurement and evaluation? It sounds like you kind of hinted --

Dr. Alaina Szlachta

Yeah. And I have actually a whole chapter in the book devoted to, 'How do we get buy-in?' And I want to tell you one story. It is a story that I feature in that chapter about getting stakeholder buy-in my experience, and the whole book is written from my experience, from one education and L&D practitioner to another. What's worked for me that I think can work for you? And I've pretty much been in a situation in my whole career where I've been with limited resources, whether I was a team of one or maybe two, whether we were limited in budget or we didn't have the time. Sometimes we didn't know what we were doing. We had to figure it out along the way.

So in the story about how we build better relationships with stakeholders, I talk about one of the organizations that I worked with, and I needed access to data, and I was getting pushback. So this is a familiar challenge for many people. I was getting pushback on getting the data that I needed to help not only evaluate the effectiveness of our like big flagship program that we did in the company, which we did almost on a monthly basis, I wanted to know was it really having the effect and impact that I hoped it would, and more importantly, I wanted to uncover, how do I make it better? And I needed data and insights Well, our management team, by grant requirements, needed to listen in to the phone calls of our staff and evaluate the effectiveness of those calls based on a framework that every employee was trained to use to facilitate a conversation. So that's incredibly rich data that would really help me understand how effective is our training. Because if our training works, then people would show up, and it may be 70 to 80% effectiveness in phone conversations, and then, you know, working toward that 100% effectiveness over time, I couldn't get access to that manager data. So what did I do about it? I did a little investigation, and I just started having conversations with people that I already had a relationship with in the organization, people who I had built trust with they knew my intentions were good. I wasn't trying to stir up politics or make problems. I was just trying to make sure that we were being, you know, great with our use of time and resources and training.

So it turns out, through my questioning and conversations that the person who had access to the data was so overwhelmed, she was like three months behind in taking the data from managers and inputting it into a spreadsheet that was then given over to the chief operations, person that then got sent over to our funders to validate that we were doing what we said we would do as an organization, the one that there was only one person whose job it was to translate the data from managers into a spreadsheet and give it over to the powers that be. She was three months behind because she was one person trying to do way too many things for 40 hour work week. So when I discovered this, I proposed, hey, we have a new staff member on our team that's trying to learn the organization. How great would it be if she helped to translate the data into this system?

And what ended up happening is, not only did we support this person who was drowning in work, and that gave us a lot of kudos with the powers that be inside the organization, because I helped to offset management of resources, train our new person, and then that person who was working to get our backlog of data up to snuff. She made a suggestion of how we could change the system to make it more time efficient. So in the end, it was me just kind of asking people who I had relationships with to try and understand what was the real problem that was preventing me to get access from the data. In the end, it turned out that I was able to solve the problem, create a better system, and along the way, got an incredible sense of trust and rapport with people all throughout the organization. And so the thinking is really about being a strategic business partner, which Jess Omley has got a whole book that is coming out soon. She talks a lot about that, but the idea is that my job isn't just about putting out learning content. It's about helping the organization and going above and beyond to try and make sure that we're using our resources wisely.

And it's that thinking that builds trust and rapport with stakeholders, because you're demonstrating that you care about the success of the organization no matter what it takes, even if it is outside of your expertise or comfort and learning, but it still helps the organization in the long run. So that kind of thinking and some other exercises in the book are really where I believe people can make a huge difference with the relationships they have with stakeholders.

Dr. Robin Sargent  

I'm sure that you've kind of covered this a little bit about getting that radical specificity about what it is that you want to measure, and that kind of removes the overwhelm. But are there? Is there anything else you would tell to, like a new IDOL, about how they can start thinking about measurement and evaluation now, even if they are like just a baby in their journey in instructional design.

Dr. Alaina Szlachta

Yeah, I think our industry has a bad habit of, hey, just focus on vanity metrics, because that's what's accessible. It's better to give vanity metrics back to your stakeholders, because at least it's something to give. Vanity metrics being completion rates, participant satisfaction scores, and then even how much of the program like the things people clicked on versus the things they didn't. Right? So there's a lot of data we have access to with our learning management system. All of that's vanity data. It doesn't really help us to understand the outcomes or impact of the programs that we're creating.

And so I think when we are figuring out highly strategically, what do we invest in terms of measurement? Sometimes it's not a good use of time to invest anything in vanity metrics, because those vanity metrics don't necessarily help us understand, like I said, the bigger picture of did we do what we said we were going to do? Instead, I suggest that whenever you're starting a program that you're really clear on what problem is this program going to solve for the organization? And use the ROI lens of if you can even figure out what that problem is costing the organization, because the ROI Institute always says you can't ever calculate an ROI unless you know in advance what is the problem costing the business. So that we can then do that cost benefit analysis of, if we invest in a solution within learning and development, does the cost of that have the chance of outweighing the cost, or in this case, under costing the organization in terms of the cost of the program.

So try as a new person, I know it's going to seem overwhelming, and it might even be contrary to best practices of what you're seeing in the organizations that you're working with, but if we don't understand the problem that our solution is meant to solve, then we aren't going to get really valid data to give back to know if we solve the problem or not. And I think that's the missing piece with most of what's happening in L&D, is that it's not a problem solution focus. It's more about pumping out content because somebody asked for it, which isn't going to empower us to have as much of an impact as it could if we shifted the focus away from just delivering on a request versus 'Hey, actually we are here to help you solve problems.' Let's understand that problem a little bit better, because that's where the rich metrics come. Is like, Oh, this problem is actually costing the organization this much money because people aren't showing up to work on time. Okay, well, then how can a learning intervention help to support that? And then you take that conversation from there. 

Dr. Robin Sargent  

Okay, so I think you did a really good job of giving examples of vanity metrics, and I love what you said about how we need to measure, are you actually solving the problem? Could you just give them, like, a couple of examples by how a measurement correlates to measuring, like, if you solve the actual problem, just like couple examples.

Dr. Alaina Szlachta

Yeah, yeah. So one of my favorite ones to talk about, because this is a real story, and it's also in the book. So one of the projects I worked with on a with a client, with one of my first clients, actually, they had a problem with new managers. So new managers were being promoted from within, which was one of the values in the company. They wanted to prioritize hiring from within, and they were great at that. The problem was that once people were hired from within, they didn't stick around in their jobs for very long, and they left. So it was ultimately costing the organization more money, because then they had to hire a new person, either from within or from an external source, and they wanted to figure out what is going on here. So the problem was ultimately that specifically managers hired from within, they were still doing their individual contributor work that they were so good at, which is why they got hired in the first place. They were still doing a significant amount of their individual contributor work while taking on new managerial responsibilities.

So in the end, what they used to have as like a 40ish hour work week was turning into a 50 or 60 hour work week, and managers were saying to their peers and to anybody that would listen, if I had known that my job was going to make me work 50 to 60 hours a week, I would have never taken it. So the problem was really that new managers didn't know how to delegate, because in many cases, they were working with a team of peers, and now they're the boss.

So this is probably a story that's really familiar in many people's organizations, and so once we realized that the problem with turnover might be solved by giving people delegation skills and frameworks to help them delegate successfully and then manage the tasks that they had delegated successfully, once we got clear on that sort of intervention, well, how can training help with this problem of people working so many hours. Training could help by building delegation skills.

So then going back to your question, what are the metrics inside of that? If we know that the problem is that managers are leaving because they're stressed out, they're not feeling optimistic about their job. They're working way too many hours, and they're spending too much time on their individual contributor work versus strategic managerial work. Those are all metrics. Those are all things that we can track before a program begins. We can monitor those same metrics throughout the program, and we can continue monitoring those metrics long after the program is over. It's a hypothesis, if we teach managers how to be better delegators, and we give them the skills, the practice and the frameworks to delegate better, well, then the problem of turnover should be solved. If we can get our managers to feel more optimistic about their work. If we can get managers to spend more time in their managerial tasks and less time in their contributor tasks, they would actually work less hours.

So when we dig into the problem, we get metrics that help us to track if we solve the problem or not through the learning intervention. So it's that in depth exploration of what's the problem and what's contributing to it that really helps us to understand possible metrics to then track and see if we've solved that problem over time.

Dr. Robin Sargent  

Okay, that is a great example Dr Alaina. So all right, so I think one of my last questions we have to bring up AI and technology and how that is reshaping the way we do everything, but specifically measurement and evaluation. So what are some of the simple ways that you've started integrating AI or technology into, I mean, basically using --

Dr. Alaina Szlachta

Anything, anything and everything, right? Yeah, if the biggest problem, and this is reported consistently, and every conference presentation or workshop that I do, and in the research that I had done all those years ago, and in reports from ATD and Brandon Hall that talk about measurement and challenges, what we see time and time again is people say we don't have time. So going back to my book, and it's about, how do we still measure within limited resources? Well, the biggest, most precious resource is time. So if we could figure out the time conundrum. And then the hypothesis is we would be measuring more or we would be measuring more strategically, and then getting better outcomes of our measurement practice. So AI can really help us with the time conundrum.

So there's two ways that I write about in the book and practice myself. So AI can be a tool that helps us to save time in our ridiculously mundane tasks. And Robin, this is where you are, Queen of this. And I've actually adopted some of the things I've learned from you in my own practice. AI can help us to streamline and take the ridiculously dumb things that we have to do and, you know, delegate them to a technology tool. Well, in setting that up now, we have more time back to focus on more important things. And so the idea is, if you invest a little bit of time in automations or leveraging AI to take some of the rote tasks off your plate, now you've got more time to learn about measurement take on a new measurement project or analyze the data you already have. So that's one of the suggestions, is use AI to give yourself some time back so that you can dip your toe a little more in depth into the measurement world.

The other way that you can use AI tools is actually in saving time, in your analysis, your data cleanup, your working with big data sets. There are some wonderful AI tools that can help make that a little bit easier. So here's one really unexpected way that I've discovered AI can be super useful. So there are times when we aren't able to use a digital tool in the classroom.

For example, there's a prospective client, fingers crossed. I might work with them one day. They do training programs in prison systems. They are not allowed to use any kind of technology. I mean, maybe they can use overhead projectors, but they might be doing flip charts and whiteboards or chalkboards or whatever the training resources that are in the space. And so their conundrum is, well, how do we measure growth in a way that's not insanely time consuming with like giving people paper pencil exams or other things. And again, I'm the person that really focuses on capability evaluation and performance evaluation, which really isn't about knowledge growth, it's really about behavior change and changes in attitude and beliefs that also influence behavior change.

So one of the activities that I've tested with AI, I did this actually at an ATD international conference presentation. So I created posters that had a statement, and I asked people to take stickers red, yellow and green, where Red was disagree. Yellow was neutral. Green is agree. And I had them go around the room, look at each statement which was capturing their sentiment, and I had them use the stickers to then respond to the sentiment. And I took pictures of every one of those statements with the red, green, yellow stickers, and I uploaded them to an AI tool, and I gave it context. I said, here's a picture with a statement on the picture. Are stickers. Red means disagree, yellow means neutral, green means agree. Count the number of red, green, yellow stickers for the statement, and then translate that into a bar graph that helps me see how this group resonates with these different statements that became a baseline measure that I then did the same activity at the end of the workshop to then compare how are people feeling differently with the same statements and the same activity at the end of the workshop.

So this is something where it can be considered an icebreaker. You could do a ton of things, those of us who are facilitators, what can you imagine doing with that red, green, yellow activity? You could have people who said that they were green and they're super strong in something. You could have them paired with people who were red and have them exchange conversation. There's a lot of stuff that we can do with that activity that's valuable, but it's also a way of collecting data and then using AI to save time so that you don't have to count the red, yellow, green stickers, and you don't have to put that in an Excel spreadsheet.

You can have AI do that for you. So that's just one sort of out of the box way that we can save time using AI to do the stuff that we would normally do, counting stickers, tallying things, making them into some kind of Excel spreadsheet or bar graph. AI can do all of that with just a few prompts for us.

Dr. Robin Sargent  

This conversation has been so illuminating, I imagine that your book is just a treasure trove. So where can our listeners find your book, connect with you and learn more about you and your work.

Dr. Alaina Szlachta

Thank you for asking. So on my website, dralaina.com I have a book page. It's dralaina.com/book. There is a free download of the first chapter in the book, so you can get a feel for I talk a lot about the hypothesis framework. It is not anything new, but how we apply it in measurement might be useful. It's my, one of my favorite chapters in the book. So you can go download that there, and I am, of course, on LinkedIn, and that's where you'll find me. It's my preferred social media. How do you say it? I am monogamous to LinkedIn? Let's, let's put it that way.

Dr. Robin Sargent  

Yeah that's good. That's good. And that's your, those are your main channels, right? Your website, dr., not it's just Alaina, or is it dralainaszlachta.com?

Dr. Alaina Szlachta

I mean, I have Dr Alaina Szlachta, but nobody can spell my last name. Don't worry about trying. dralaina.com is another option.

Dr. Robin Sargent  

Okay. A, l, a, i, n, a, that is correct. Well, thank you so much, Dr Alaina, for joining us. Thank you for sharing all of your insights, and obviously your book, I'm sure, is already a success, but my hope that it continues to grow, because this is something that we all need to learn and do better. So thank you for the work that you've contributed to our industry.

Dr. Alaina Szlachta

Thank you, Robin. I appreciate having this conversation. I look forward to making sure that your IDOL folks are more prepared for measurement in the future.

If you are a new listener to Become an IDOL, we would love to hear from you.  Please visit our Contact Page and let us know how we can help you today!

Subscribe & Review in iTunes

Are you subscribed to my podcast? If you’re not, I want to encourage you to do that today. I don’t want you to miss an episode. I'll be publishing a new episode every two weeks and if you’re not subscribed there’s a good chance you’ll miss out on those. Click here to subscribe in iTunes!

Now if you’re feeling extra loving, I would be really grateful if you left me a review over on iTunes, too. Those reviews help other people find my podcast and they’re also fun for me to go in and read. Just click here to review, select “Ratings and Reviews” and “Write a Review” and let me know what your favorite part of the podcast is. Thank you!

Want more tips on how to become an IDOL? Get my free guide here

Join the Conversation!

We have a wonderful and friendly community of expert and novice IDOLs in the Become an IDOL Facebook Group. We would love to have you join us in the group or chat with IDOL courses on of our social media channels.