QHPE 601: Evidence-based Assessment

Assessment

• QHPE 601: Evidence-based Teaching and Learning
• Faculty credited: Dr Ahsan Sethi, Dr Daniel Rainkie
• Learning objectives:
  • Describe assessment types (for/of learning, formative/summative)
  • Match assessment modalities to Miller's pyramid levels
  • Describe and calculate assessment utility index at the institution

Domains of Learning

• The three domains are Knowledge, Psychomotor Skill, and Attitude.

Bloom's Taxonomy of Cognitive Domain

• A hierarchical model of cognitive skills, ranging from remembering to creating.
• Remembering: Recall facts and basic concepts (e.g., define, duplicate, list)
• Understanding: Explain ideas or concepts (e.g., classify, describe, discuss)
• Applying: Use information in new situations (e.g., execute, implement, solve)
• Analyzing: Draw connections among ideas (e.g., differentiate, organize, relate)
• Evaluating: Justify a stand or decision (e.g., appraise, argue, defend)
• Creating: Produce new or original work (e.g., design, assemble, construct)

Dave's Taxonomy of Psychomotor Domain

• A hierarchical model of psychomotor skills.
• Imitation (P1): Observe the different steps of a procedure.
• Manipulation (P2): Assist in the procedure.
• Precision (P3): Perform the different steps of a procedure under supervision.
• Articulation (P4): Perform independently.
• Naturalization: Perform the skill independently.

Bloom's Taxonomy of Affective Domain

• A hierarchical model of affective skills, ranging from receiving to characterising by value concept.
• Receiving: Notice a particular phenomenon (e.g., observe, be conscious, listen).
• Responding: Respond with compliance, then willingly (e.g., engage, consider, accept).
• Valuing: Accept the worth of a thing (e.g., participate, desire).
• Organising and Conceptualising: Adapt behavior to the value system (organise values, determine interrelationships).
• Characterising by Value or value Concept: Adapt behavior to the value system (organising and conceptualising).

Assessment Quotes

• "As long as there are tests, there will be prayer in school." (anecdotal evidence)

What is Assessment?

• Assessment is a systematic measure of student learning by gathering, analyzing, and interpreting data to compare expectations.

Student Assessment

• It's a generic concept with three key areas:
  • Collecting and analyzing information with methods and instruments
  • Determining learning outcomes
  • Decisions, feedback, and review

Assessment as a Measure of Student Learning

• Assessment measures student learning by systematically gathering, analyzing, and interpreting student performance data to compare it with expectations.
• The purpose is to see how well student learning meets expectations.

Why Do We Assess?

• We assess to:
  • Certify learning outcomes attainment
  • Demonstrate progression readiness
  • Grade and rank performance or award prizes
  • Provide feedback on learning
  • Identify learning gaps for intervention
  • Encourage learning and motivation
  • Inform career selection
  • Award qualifications

Assessment at Different Times

• Before teaching: Assessing students' readiness for the course (prerequisites, prior learning).
• During teaching: Assessing how well students are progressing (need for remedial teaching, adjustments to teaching style).
• End of teaching: Assessing mastery of learning outcomes, next stage preparedness, overall competence level.

Assessment Tools

• Various tools are used, such as:
  • MCQ (Multiple Choice Questions)
  • EMQ (Extended Matching Questions)
  • OSCE (Objective Structured Clinical Examination)
  • Viva Voice
  • Logbooks
  • Essays
  • Portfolios

Selecting the Appropriate Assessment Instrument

• Choose an assessment instrument that aligns with the learning objectives and intended outcomes.

Miller's Pyramid

• A pyramid representing the levels of competency - "Knows," "Knows How," and "Does."
• It’s crucial for the learner to be effective in a doctor/professional role

Miller's Pyramid (Cont.)

• It depicts various levels of performance.
• The "does" section is focused on practicing skills and procedures.
• It moves from "Knows" to "Does" and expert proficiency.

"The curriculum instructs teachers what to teach; The exam instructs students what to learn."

• Quotes the principle that exams shape what students learn, by influencing the teaching.

What are the Types of Assessments?

• Diagnostically: Assessments conducted at the beginning of a course or training. (e.g., identifying prior understanding).
• Formatively: Assessments conducted ongoingly during a course, to provide feedback on learning progress. (e.g., providing feedback, identification of areas for improvement).
• Summatively: Assessments conducted at the end of a course to assess learning outcomes. (e.g., examinations, licensing)

Summative Assessments

• Determines performance at a fixed time point typically at the end of a course or program unit.
• Focuses on final product, evaluating proficiency.
• Intended for evaluating proficiency in different learning aspects.

Formative Assessments

• Conducted during a course or program.
• Aims to improve learning experience and performance for students and teachers.
• Provides students with feedback on learning gaps and directions for improvements
• Assessment is an integral part of the learning process.

Formative & Summative Assessments

• Formative assessments are like the chef tasting sauce before serving. Summative assessments are like customers tasting food.

Assessment and Learning

• Assessment reinforces learning (FOR learning).
• Assessment evaluates learning achievements (OF learning).

System of Assessment

• Different tools and approaches are used in assessment, such as OSCEs, MCQs, mini-CEXs etc;

Low & High Stakes Assessment

• Low stakes: Assessment with limited consequences for the learner; used for formative purposes more.
• High stakes: Assessment with far-reaching consequences; used mainly for summative purposes.

Assessment & Stakeholders - Criteria 2011

• Stakeholders (teachers, examinees, patients) have specific responsibilities in the assessment process.

Issues With Assessment

• Potential issues in assessment design and implementation:
  • Curriculum mismatch with assessment content.
  • Question quality.
  • Assessment instrument validity and reliability.
  • Trainer/assessor training
  • Assessment transparency.

Assessment Principles

• Essential principles of effective assessment to gain reliable inferences and evaluate skills.

Principles of Assessment – Assessing 'Knows', 'Knows How' and 'Shows How'

• Focus on specific competencies and skills.
• Assess objective and reliability (e.g., OSCE global score).
• Integrate formative and summative functions.
• Recognize the importance of good qualitative information and narrative.

Principles of Assessment: Assessing 'Does'

• Reliable inferences require a sufficient sample from which one can judge competency.
• Assessors can sometimes be biased.
• Combining formative and summative functions is common.
• Qualitative and non-psychometric (e.g., portfolios) are valuable for making summative decisions.

Utility Index

• A measure of assessment usefulness, combining factors like reliability, validity, educational impact,acceptability, and cost-effectiveness.

Usefulness of Assessment (Utility Index)

• A comprehensive measure of assessment usefulness, encompassing reliability, validity, educational impact, acceptability, cost-effectiveness, and feasibility factors.
• Provides a comprehensive view on the assessment.

This quiz covers key concepts in evidence-based teaching and learning, focusing on assessment types, modalities, and utility indices. It also explores Bloom's Taxonomy and the three domains of learning. Test your understanding and application of these educational theories.