Learner Survey Design PDF

Performance Focused Learner Surveys Presentation Based on Book Sonos Confidential 3 Four Pillars of Training Effectiveness Measure thing that matter, that only correspond to the 4 pillars. 1. Understanding: Do the learners understand? 2. Remembering: Will t...

Performance Focused Learner Surveys Presentation Based on Book Sonos Confidential 3 Four Pillars of Training Effectiveness Measure thing that matter, that only correspond to the 4 pillars. 1. Understanding: Do the learners understand? 2. Remembering: Will they remember? 3. Motivation to apply: Are they motivated to apply? 4. After training supports: Are there after training supports in place? Sonos Confidential 4 Rules of Survey Design 1. Remind learners of the learning experience. Unless the learning event is very short and learner surveys are presented soon after learning, learners should be reminded of the topics presented and the learning methods employed. 2. Increase learner attention. Learner surveys should not be too long or too short. Use descriptive answer choices; not likert scale, nor numeric scale. Likert and numeric scaled just don’t feel very important 3. Avoid biasing questions. Care must be given not to lead learners in one direction or another with the question stems or with the answer choices. For example, avoid affirmations such as “My skills and abilities improved as a result of this training.” These types of affirmations push learners to answer in the affirmative. Sonos Confidential 5 Rules of Survey Design 4. Ask clear and relevant questions. One of the rules of measurement is to never ask a question whose answer can’t or won’t be used to actually make some kind of improvement. 5. Ask questions that learners can answer knowledgeably. Example of question that learners are unlikely to answer with insight: “My learning was enhanced by the course structure.” 6. Provide descriptive and easily understood answer choices. Learners will make better decisions if we use descriptive answers rather than Likert-like response choices. 7. Provide follow-up learner surveys. These surveys are valuable in determining if learners have implemented the new material. Sonos Confidential 6 Distinguish Between Different Levels of Success Learner survey results should clearly differentiate between success and failure of the training and different levels of success. As learning professionals, we won’t know what to do with the learner-survey results. Sonos Confidential 7 Distinguish Between Different Levels of Success Delineate standards for each answer choice during question design. For example: Question Proposed Standards Now that you’ve completed the learning Not shown to learners experience, how well do you feel you understand the concepts taught? CHOOSE ONE. A. I am still at least SOMEWHAT Alarming CONFUSED about the concepts. B. I am now SOMEWHAT FAMILIAR with Unacceptable the concepts. C. I have a SOLID UNDERSTANDING of the Acceptable concepts. D. I AM FULLY READY TO USE the Superior concepts in my work. E. I have an EXPERT-LEVEL ABILITY to Superior/ Overconfident? use the concepts. Sonos Confidential 8 Design Surveys to Send Appropriate Stealth Messages to Our Key Stakeholders Stealth messages are messages conveyed during our routine practices and procedures. Learner surveys can also directly drive thoughts and actions. After Training Support After the course, when you begin to apply your new knowledge at your worksite, Proposed standards which of the following supports are likely to be in place for you? Not shown to learners MY MANAGER WILL ACTIVELY SUPPORT ME with key supports like time, Acceptable resources, advice, and/or encouragement. I will use a COACH OR MENTOR to guide me in applying the learning to my Acceptable work. I will regularly receive support from a COURSE INSTRUCTOR to help me apply Acceptable the learning to my work. I will be given JOB AIDS like checklists, search tools, or reference materials to Acceptable guide me in applying the learning to my work. Through LEARNING APP or other means, I will be PERIODICALLY REMINDED Acceptable of key concepts and skills that were taught. I will NOT get much direct support but will rely on my own initiative. Unacceptable Sonos Confidential 9 Design Surveys to Send Appropriate Stealth Messages to Our Key Stakeholders The question also sends the following messages, all of which are aligned with the research on learning transfer: Learners’ managers can provide important after-training supports. Coaches or mentors can provide helpful guidance. Instructors should not just disappear after training but can provide important supports in helping learners apply what they’ve learned. Job aids can provide important support in helping learners in their work. Reminders sent to learners after training can produce benefits. Sonos Confidential 10 Exercise: Evaluating Learner Survey Questions Sonos Confidential 11 Question 1 I learned new knowledge and skills from this training. A. Strongly Disagree B. Disagree C. Slightly Disagree D. Neutral E. Slightly Agree F. Agree G. Strongly Agree Sonos Confidential 12 Question 2 The training will have a significant impact on (check all that apply): A. Increase Quality B. Decrease Costs C. Decreasing Cycle Time D. Increasing Productivity E. Increasing Sales F. Decreasing Risk G. Increasing Employee Satisfaction H. Increasing Customer Satisfaction Sonos Confidential 13 Question 3 Regarding the topics taught, how motivated WILL YOU BE TO USE these concepts/skills in your work? CHOOSE ONE. A. I will NOT MAKE THIS A PRIORITY when I get back to my day-to-day job. B. I will make this a PRIORITY - BUT A LOW PRIORITY when I get back to my day-to-day job. C. I will make this a MODERATE PRIORITY when I get back to my day-to-day job. D. I will make this a HIGH PRIORITY when I get back to my day-to-day job. E. I will make this ONE OF MY HIGHEST PRIORITIES when I get back to my day-to-day job. Sonos Confidential 14 Question 4 Using job aids, checklists, or other prompts can be an effective way to ensure you properly apply skills to your job. Which of the following are true? SELECT ALL that apply. A. We did NOT RECEIVE ANY WORTHWHILE job aids, checklists, or similar prompts to direct our on-the-job actions. B. We RECEIVED ONE OR MORE WORTHWHILE job aids, checklists, or similar prompts. C. During the course, we USED ONE OR MORE WORTHWHILE job aids, checklists, or similar prompts in REALISTIC PRACTICE EXERCISES. D. Between sessions of the course, IN OUR WORKSITES, WE UTILIZED one or more WORTHWHILE job aids, checklists, or similar prompts IN A REAL-WORLD JOB TASK. Sonos Confidential 15 Question 5 Overall, how satisfied are you with this learning experience? A. Not at all satisfied. B. Not very satisfied. C. Somewhat satisfied. D. Very satisfied. E. Extremely satisfied. Sonos Confidential 16 Question 6 How likely is it that you would recommend this training course to a friend or colleague? A. NOT AT ALL LIKELY to recommend B. VERY UNLIKELY to recommend C. SOMEWHAT UNLIKELY to recommend D. SOMEWHAT LIKELY to recommend E. VERY LIKELY to recommend F. EXTREMELY LIKELY to recommend Sonos Confidential 17 Learner-Survey Training-Effectiveness Taxonomy Learner survey primary goal: Training Effectiveness Secondary Goal: Understanding ○ Tertiary Goal: Learners Engaged ○ Tertiary Goal: Cognitive Supports Effective Secondary Goal: Remembering ○ Tertiary Goal: Realistic Retrieval Practice ○ Tertiary Goal: Spaced Repetitions ○ Tertiary Goal: Situation-Action Triggering Secondary Goal: Motivation to Apply ○ Tertiary Goal: Belief in the Value of Concepts ○ Tertiary Goal: Self-Efficacy ○ Tertiary Goal: Resilience Secondary Goal: After-Training Follow-Through ○ Tertiary Goal: Reminding Mechanisms ○ Tertiary Goal: Job Aids ○ Tertiary Goal: Supervisor Follow-Up Sonos Confidential 18 Example Learner Survey Questions Review example learner survey questions Sonos Confidential 19 Standard Learner survey Draft Standard learner survey draft Sonos Confidential 20