I take evaluation very seriously. Before we run theLearningLoop programme or indeed any programmes, I undertake to outline, with the client, what measures we will have in place to see if the programme has been a success.
I have just completed two cohorts with a new client and it is too soon to measure all the outcomes but we know that confidence has increased and ways in which they can improve. In a few months time we will be able to see the learning applied into action but meanwhile things are looking good!
Looking at just four of the evaluation forms, you may see why I am feeling confident we will achieve our desired outcomes of increasing confidence and capability in the subject matter experts. There are some hard business measures too for the learning they will be facilitating! Meanwhile take a look at the evaluation forms:
This title has been taken from the Towards Maturity report published in August of 2018. I was immediately drawn to the title and anyone who knows what a maths geek I am will understand why.
I always loved numbers and even just playing with them, multiplying numbers by themselves repeatedly just for fun! Yes I know it’s not normal and I also appreciate that not everyone else has the same love of numbers, in fact quite the opposite. I have several friends who will admit that numbers are almost a phobia.
Reading this report ,it is quite evident that we, in L&D are not great at collecting and using data to its best advantage. Some of the figures that struck me were:
Of those aspiring data to affect change only 2/5 were able to say that it helped the, demonstrate business impact (40%)
Also only 3/5 were successful in using data to help solve business problems (60%)
Bear in mind that was from a sample size of 700+ and the two figures above were those people who were really trying to use data affectively. This means in reality that there will also be a number of people not even trying so the 40% and 60% are likely to be very optimistic figures.
The most likely reasons cited were:
Data and it’s analysis is complicated
Lack of L&D skills in this area
If I look at the second point first. Why are we not addressing this lack of skills? Is it this phobia of numbers? A fear of what to do once you start collecting? An expectation that things have to change once you start collecting data effectively? Maybe it’s a combination of all three? Or maybe a misconception around what it means to collect and analyse data?
For me it is quite simple (and this may address the first point). In L&D we need to get nosey. When someone asks us to deliver a leadership programme, we need to ask why and how will you know it has been successful? If the first person who asks you doesn’t know, then ask someone else. Is it a real need or a perceived need?
The perceived need may be something like employee engagement scores being low. What we really need to determine is what effect that is having on the performance of the business:
High recruitment costs?
Lack of agility in the marketplace because there is a high attrition rate, staff not as familiar as they should be about products?
Poor customer service because the tools they use have had little investment?
So when you look at these examples, you can start to see it really is not about data analysis, but curiosity, perseverance and a healthy dose of skepticism. If you can pinpoint what the problem is and it is a real business need, then what you need to measure will be very obvious:
Reduction in recruitment costs
Reduction in time to market with new products
Range of new products and uptake
Customer satisfaction scores
These are not L&D statistics these are business measures and having highlighted the purpose of your L&D focus, the business will also want to measure it. That is not to say that at times you are not needed to do some data analysis and collection but I think we are over complicating it and not getting to the nub of the problem.
In my book “How not To Waste Your Money On Training” I show people simply how to “find the story in the data”. Using a simple example of a scoring grid, I show how you can, using a spreadsheet and playing with different graph types, discover little parts of the truth about what is going on. It takes a click and a small amount of curiosity. If you want to try it out then just use this example, using Excel to play around with different types of charts:
For each format ask yourself “what do I see now?”. Using this approach of curiosity and play I discovered:
A bar chart gives a good comparison one person against another for each part of their role
A spider diagram shows how well-rounded each team member is in their own right. Some are not rounded at all! Tracy seems the most well rounded.
Stacked column shows the teams strengths and weaknesses:
Who is the strongest in sales skills?
Who is the weakest in product knowledge and working independently (why might this be? Manager poor at delegating?)
So I would urge you L&D, before spending a lot of money on data analytics experts, get nosey and do some detective work yourselves. Keep it simple and dig into what is going on beneath the surface. Don’t just take one persons viewpoint or use just one method, mix it up and start finding the story in the data!
My conclusions from the report and my own anecdotal research suggests that:
L&D does not have the skills required for data analysis (I had better get that book finished!)
It is not as complicated as you think
It is about asking the right questions and finding the story in the data
We don’t always need data analytics experts to do this!
I have had no issue with the Kirkpatrick model, as it has always made sense to me. Just recently I have been challenged to reconsider my position. Last week running the Learning Loop, one of the participants said they have started to use Will Thalheimers’ Learning Transfer Evaluation Model, which made me curious about what that model had to offer, that was different to Kirkpatricks.
This week I read a blog from Work Learning Research saying how levels 3 and 4 of Kirkpatrick are misunderstood by most L&D people, that they use learner reactions as if they were valid level 3 and 4 measures. The blog got me ruffled a bit, particularly because the survey results:
1) use a sample size of TWO for vendors
2) there is no visible link with those 250 surveyed and whether they are actually even measuring at levels 3 or 4 (if they are not, then could they know the answer to the question?)
This makes any conclusions, drawn from the research, in my opinion tenuous to say the least.
What made me curious enough to look at Will Thalheimers’ model was any suggestion that it had something new to offer. What I did like was the focus on learning transfer, but then I thought – shouldn’t the focus be on performance? Will has a lot to say and criticise the Kirkpatrick model for, but I am looking for a deeper and better way to do evaluation. What I would like to offer is a deeper explanation of Kirkpatrick’s model and I hope you will make up your own mind how you might apply it.
“Begin with the end in mind” said Steven Covey. I agree wholeheartedly. If you do a thorough analysis, engage with stakeholders and line managers about the outcomes (in performance) required, then evaluation should be straight-forward. But please note if the analysis part is skipped over or done on a superficial level, THE EVALUATION WILL BE DIFFICULT WHICHEVER MODEL YOU USE.
Level 1 – learner reactions – still an important part of seeing if you have had sufficient engagement and struck the right chord with the objectives (Business focused and learner-centred according to the 5 Secrets of Accelerated Learning.
Level 2 – learning achieved – did they learn what they were supposed to and have they met those objectives? This is important to know for L&D and their line managers
Level 3 – impact on performance – this is what is then observed on the job and in my opinion should not be L&D’s sole remit. If they have analysed the needs correctly and determined the correct outcomes, the line managers should be engaged enough to help imbed the learning as well as measure performance improvements.
Level 4 – impact on the business- if the stakeholders have been engaged and the outcomes are focused on the business, the stakeholders will be interested in measuring the business impact.
This question was raised in the Creativity Zone at Learning Live 2017 by a number of people. From observation, I have noted that a great many L&D professionals and teams find evaluation tricky. From Towards Maturity report “Driving the New Learning Organisation”, the “Top Deck” are twice as likely to identify metrics they want to improve through learning. That sounds so simple and yet there are many organisations not doing this. This may be for a number of reasons:
Lack of clarity about who to talk to about the important metrics
Lack of knowledge in how this data could be captured
Lack of confidence in an approach that might work
Our Learning Loop Approach gets people thinking about the end before they rush into a solution. Care is taken to engage with stakeholders. Objectives are set rather than woolly aims. Performance objectives are used to drive better performance. Learning objectives are leveraged to help improve performance and a culture of social and self-reliant learning is encouraged.
So what might we advise you to do to start a fresh approach to ROI:
Measuring stuff in L&D is good and I am an advocate of using data to inform your decision making as well as demonstrating your worth. So is it a case of just measuring everything and it’s bound to be useful? For anyone who knows us and our approach, you know the answer to that already!
Here is what measurement I know happens already, in many places:
Number of people completed training, either face to face or online
Test scores from online quizzes
Amount of time taken for elearning and “engagement” during learning
Number of “no-shows” on courses
Number of training hours per year company wide
My question is “Does any of this help to improve performance?” The answer may be that sometimes training is not a performance issue but a compliance issue. The training has to be done, so we need to know how many hours we complete per year and do it in the most efficient way. Fair enough! If you have to do the training, make sure it’s effective and done in the most efficient way to not waste money.
What about measuring all these things when it’s related to performance? I can see value in this only if a good analysis has been done beforehand to:
Rule out issues that cannot be solved by training (poor systems, processes or lack of resources etc)
Identify the right stakeholders to work with, who will support you and put measures in place to measure the effectiveness
Determine the organisational outcomes that need to be met, with the appropriate stakeholders providing resources and support
Define learning outcomes that are geared towards improving performance and are both observable and measurable
Put in place follow up by line managers before the learning starts and performers know what they are going to get out of the learning before they attend
This sort of “joined-up” L&D, makes learning everyone’s responsibility and it also means that measurement is not just L&D’s responsibility. It means that “learners” are transformed into “performers”. It means that those measures listed at the top of the page could be used to inform which methods have had the most engagement (not that that is always an indicator for success!)It would be wrong, I believe to suggest that good engagement with one successful cohort will guarantee the same success in another cohort.
To my thinking, any suggestion that there could be some sort of permanent link between engagement and performance misses the point entirely. In a closed system like a chemical plant, where putting in the same chemicals at the same rate with the same process can produce expected, achievable results. Learning is not however a closed system: the variables are always changing as are the participants and the influences on their behaviour. The true measure, in my opinion has to be what measurable differences the learning has made to the performance of an individual. How have customer complaints reduced? Income increased? Sales boosted? A one time analysis is not the answer because things change. So analyse, plan, implement and measure, then round again to make sure you are capturing the “now” and not the “yesterday”.
So am I in L&D Narnia, expecting the impossible? Measuring the unmeasurable? Quite simply, I believe that before investing in learning and merely measuring “engagement” dig, dig and dig deeper to find out what is missing and what you will need to learn and do to make changes. Keep asking why? Until you have some sensible answer other than “why not?”
I am always thinking about how we can collaborate, but am mindful of those people who are not on social media, not having a voice at times. So we had a discussion on “Tricider” with a group of people from different places on:
“If L&D could do the analysis piece well, evaluation would be easy?”