Podcast
Questions and Answers
What aspect of learning is often overlooked according to the session introduction?
What aspect of learning is often overlooked according to the session introduction?
What is the significance of data collection mentioned in the session?
What is the significance of data collection mentioned in the session?
What do participants in the session reflect on?
What do participants in the session reflect on?
What is a discussed topic regarding learning design in the session?
What is a discussed topic regarding learning design in the session?
Signup and view all the answers
What is highlighted as a need for data by learning professionals?
What is highlighted as a need for data by learning professionals?
Signup and view all the answers
What primary challenge is associated with traditional learning evaluation methods?
What primary challenge is associated with traditional learning evaluation methods?
Signup and view all the answers
What aspect of evaluation does the Performance Focused Smile Sheet emphasize?
What aspect of evaluation does the Performance Focused Smile Sheet emphasize?
Signup and view all the answers
What significant finding was reported in the 2007 survey regarding Learning and Development (L&D) leaders?
What significant finding was reported in the 2007 survey regarding Learning and Development (L&D) leaders?
Signup and view all the answers
According to meta-analyses, what is the correlation between smile sheet feedback and actual learning success?
According to meta-analyses, what is the correlation between smile sheet feedback and actual learning success?
Signup and view all the answers
What was a common mistake in evaluations emphasized in the content?
What was a common mistake in evaluations emphasized in the content?
Signup and view all the answers
What approach did Hendrick, the new Chief Learning Officer, take regarding evaluation?
What approach did Hendrick, the new Chief Learning Officer, take regarding evaluation?
Signup and view all the answers
What criticism is directed at Likert scales in the context of learning evaluations?
What criticism is directed at Likert scales in the context of learning evaluations?
Signup and view all the answers
What has been a historical challenge in training evaluation noted in the content?
What has been a historical challenge in training evaluation noted in the content?
Signup and view all the answers
Study Notes
Session Introduction
- Stella Collins chairs a session focusing on learning evaluation.
- Willful Homer, a proponent of evidence-based learning, will present and introduce his initiatives, including the Debunkers Club.
- Participants encouraged to reflect on their expectations and motivations for attending the session.
Learning Evaluation Insights
- Learning evaluation is often seen as a neglected aspect of the field of learning and development.
- Complexity arises from assessing human learning, which encompasses comprehension, motivation, application, and success rates.
- Emphasis on the need for effective evaluation to make informed decisions.
Goals of the Session
- Aim to pique curiosity and provoke thoughts on enhancing learning evaluation practices.
- Provision of resources and URLs for deeper exploration post-presentation.
The Importance of Data
- Data collection is critical for effective decision-making across various fields.
- Learning professionals should pursue data that is accurate, valid, relevant, predictive, and cost-effective.
- The presentation relates agricultural data practices to the data needs of learning professionals.
Speaker Background
- Willful Homer runs a research and consulting firm focusing on learning evaluation.
- Experience includes consulting for various organizations such as the Navy SEALs, CDC, Oxfam, and educational institutions like MIT.
- Focus on bridging research and practice in learning and evaluation, translating scientific findings into practical applications.
Learning Preferences and Evaluation
- Discussion prompts consideration of whether learning design should rely on learners’ preferences.
- Candidates for response highlight categories in which learners may effectively self-judge their learning abilities.
- Encouragement of collaborative discourse among participants to solidify understanding.
Additional Resources
- References to an array of reports on learning and memory available for free.
- Mention of a specific publication titled "Performance Focused Smile Sheet," emphasizing best practices in learning evaluation.### Evaluation in Learning
- Discussions begin with the need to evaluate learning experiences in educational and workplace settings.
- The Capat Level Model, originating from Raymond Kirkpatrick, is critiqued for its limitations in effectively measuring learning outcomes.
- Research indicates that traditional measurement approaches in learning evaluations often yield trivial or misleading results.
Historical Context
- Donald Kirkpatrick highlighted slow progress in training evaluation as early as 1960.
- A 2007 survey revealed only 20% of Learning and Development (L&D) leaders felt they could measure learning effectively.
- Subsequent findings show that frustration in the field has persisted, with over 50% of recent respondents expressing dissatisfaction with current measurement practices.
Learning Evaluation Challenges
- Evaluation is intrinsically difficult, with historical evidence of challenges in achieving meaningful measures.
- Common mistakes in evaluation are emphasized and documented, demonstrating widespread difficulties in addressing learning efectiveness.
Practical Application: Case Study of Hendrick
- Hendrick, a new Chief Learning Officer, implements pilot evaluations focused on leadership training to better manage learning evaluation.
- The new evaluation strategy emphasizes high-level questions about learning impacts rather than superficial measures.
Research Insights
- Meta-analyses of over 150 studies indicate weak correlation (0.09) between smile sheet feedback and actual learning success.
- Leaners' surveys and feedback often fail to provide valuable insights, with most feedback deemed not useful.
Problems with Traditional Measurements
- Likert scales are criticized for ambiguous interpretations and inadequacy in measuring true learner engagement and effectiveness.
- Learners often struggle to assess their learning accurately, leading to potential bias in evaluations.
Recommendations for Improved Evaluation
- Emphasis on tailored, performance-focused questions that yield actionable insights rather than relying on traditional numeric ratings.
- Inclusion of open-ended questions for richer qualitative feedback, allowing for detailed responses on learning experiences.
The New Smile Sheet Approach
- Implementation of a performance-focused smile sheet designed to improve feedback accuracy and utility.
- Questions focused on real-world application and motivation regarding learned skills.
The LT Model (Learning Transfer Model)
- A new framework introduced to enhance learning evaluations, moving beyond the four-level model.
- The LT model categorizes metrics into distinct tiers that differentiate between attendance, learner perceptions, knowledge, competence, and transfer.
Key Goals of New Evaluation Framework
- Aim to understand not just if learners are satisfied but if they are effectively applying knowledge and improving performance.
- Encourage iterative, ongoing evaluation throughout the learning process rather than only at its conclusion.
Conclusion and Future Directions
- Learning evaluation should evolve to reflect detailed, nuanced approaches that prioritize performance and transfer over satisfaction.
- Continued exploration of new technologies and methodologies for gathering and interpreting learning data is necessary for improvement in learning outcomes.
Session Introduction
- Stella Collins leads a session on learning evaluation with a focus on participant engagement.
- Willful Homer presents evidence-based learning initiatives, including the Debunkers Club.
- Participants invited to reflect on their motivations for attending.
Learning Evaluation Insights
- Learning evaluation is frequently overlooked in learning and development.
- Assessing human learning is complex, involving comprehension, motivation, application, and success rates.
- Effective evaluation is necessary for informed decision-making.
Goals of the Session
- Enhance curiosity and thought regarding learning evaluation practices.
- Resources and URLs will be provided for further exploration after the session.
The Importance of Data
- Accurate data collection is essential for sound decision-making across various sectors.
- Learning professionals should aim for data that is valid, relevant, predictive, and cost-effective.
- Agricultural data practices are linked to the data requirements of learning professionals.
Speaker Background
- Willful Homer operates a research and consulting firm focused on learning evaluation.
- His experience includes consulting the Navy SEALs, CDC, Oxfam, and MIT.
- Homer's work bridges research and practice in learning, translating scientific findings into applicable methods.
Learning Preferences and Evaluation
- Participants discuss whether learning design should be based on learner preferences.
- Self-judgment categories in learning abilities are highlighted as potential areas for response.
- Collaborative discussions among participants are encouraged to reinforce understanding.
Additional Resources
- A variety of free reports on learning and memory are available for reference.
- "Performance Focused Smile Sheet" publication emphasizes best practices in learning evaluation.
Evaluation in Learning
- Calls for evaluating learning experiences in educational and workplace environments.
- The Kirkpatrick Model, while influential, is critiqued for inadequacies in measuring learning outcomes.
- Research shows traditional measurement methods in evaluations can yield misleading results.
Historical Context
- Donald Kirkpatrick flagged slow progress in training evaluation in 1960.
- A 2007 survey found only 20% of L&D leaders felt capable of effectively measuring learning.
- Over 50% of recent respondents indicate ongoing frustration regarding current measurement practices.
Learning Evaluation Challenges
- Evaluation poses intrinsic challenges, often producing meaningless measures.
- Documented common mistakes in evaluation illustrate widespread difficulties in assessing learning effectiveness.
Practical Application: Case Study of Hendrick
- Hendrick, as Chief Learning Officer, implements pilot evaluations for leadership training.
- The new evaluation strategy prioritizes high-level impact questions over superficial measures.
Research Insights
- Meta-analyses of 150+ studies reveal a weak correlation (0.09) between smile sheet feedback and actual learning success.
- Learner surveys and feedback are often unhelpful, with most deemed of little value.
Problems with Traditional Measurements
- Likert scales face criticism for ambiguity and inadequacy in capturing true learner engagement and effectiveness.
- Challenges in measuring learning outcomes through traditional means are acknowledged.
Studying That Suits You
Use AI to generate personalized quizzes and flashcards to suit your learning preferences.
Description
Join Stella Collins and Willful Homer as they discuss the often-overlooked aspect of learning evaluation. Homer will share his initiatives such as the Debunkers Club and encourage participants to reflect on their learning motivations. Gain insights into effective evaluation strategies to enhance learning outcomes.
