Qualitative Methods for Planning and Evaluation PDF

Qualitative Methods for Planning and Evaluation Learning Objectives With this chapter, readers will be able to: 1. Select an appropriate qualitative design for process and effect evaluations. 2. Describe the basic procedures for analyzing qualitative data. 3. Develop qualitative methods to address...

Qualitative Methods for Planning and Evaluation Learning Objectives With this chapter, readers will be able to: 1. Select an appropriate qualitative design for process and effect evaluations. 2. Describe the basic procedures for analyzing qualitative data. 3. Develop qualitative methods to address evaluation questions. Qualitative Methods in the Planning and Evaluation Cycle • Assessment – Insight into needs, barriers, etc. of target population • Program and effect theory development – Incorporate target audience’s explanatory models – Staff’s theory-in-use during program revision • Process monitoring – Adds different types of information • Effect evaluation – Enhance explanatory power of evaluation design and data – Explain why/how certain outcomes occurred Comparing Qualitative Perspectives Perspective Basic question addressed Planning and evaluation relevance Content analysis Which themes are in the text? Thoughts & perspectives revealed in text and dialogues Critical analysis How has power shaped it? Participants’ view of their ability to be in control of the health problem & solution; staff’s view of their autonomy in improving the program Ethnography What are the norms and values (culture)? Participants’ cultural forces that contribute to the problem and acceptance of the program Grounded theory What are the relationships (theory)? Explanations that participants and staff have for the health problem and possible solutions Phenomenology What does it mean to the person? Participants’ meaning of content and the problem being addressed Qualitative Perspectives and Methods Perspective Typical methods Content analysis Focus groups, surveys with open-ended questions, narrative designs Critical analysis Individual in-depth interviews Ethnography Case study, participant observation, observations Grounded theory Individual in-depth interviews Phenomenology Individual in-depth interviews Comparing Qualitative Methods Method Key benefits Key challenges Case study Understanding context as Complex, overwhelming an influence on the amount of data; program or participant definition of case Observations Identify sequence of causes and effects; may identify new behaviors or events Difficult to obtain reliable data without recording devices; sampling frame difficult to establish Individual in-depth interviews Rich insights into personal thoughts, values, meanings, and attributions Identifying individuals who are willing to be open Comparing Qualitative Methods, Continued Method Key benefits Key challenges Focus groups Inexpensive given amount & type of data collected; get collective views rather than individual views Need training in managing group process; need good data recording method Survey with open-ended questions Very inexpensive Poor handwriting & unclear statements can make data useless Narrative designs Very inexpensive; insights into social & cultural influences on thoughts & actions Need training in data analysis; may not have credibility with stakeholders; difficult to select text most relevant to problem or program Individual In-Depth Interviews • Can be used throughout planning and evaluation cycle – Key informants especially useful in assessment and planning • Focus on meanings and attributions, or insights into poorly understood phenomenon • Interviewer skills are critical Open-Ended Survey Questions • Can be used during assessment, process monitoring, or impact questionnaires • Examples: – Anything else? – What was the best/worst…? – What suggestions do you have? • Challenges include illegible handwriting and overly brief responses Focus Groups • Can be used during assessment and program planning; possibly process monitoring – Not appropriate for outcomes assessment • Group dynamics during the interview may produce new insights • Same data analysis steps as for individual indepth interviews Focus Group Data Collection • Ideally 8 – 12 participants – Balance between homogeneity and heterogeneity – Scheduling can be challenging • Interviewer must elicit responses from quiet participants • Recording must be unobtrusive – Use > 1 method (e.g., audio recording plus handwritten notes) Observation • May be non-participatory or participatory, in a variety of settings • Data collection – May use audio-visual recording – Coding events as they occur with standardized tool – Keeping detailed notes after observation Case Study • Address Qs of how/why something occurred • If used in effect evaluation, program implementation or program itself is the case – Program participants may also be cases in effect evaluations Case Study Data Collection • Defining the case is critical – Consider what makes the case very typical or very unique • Multiple data sources – Primary: interviews, observations, surveys – Secondary: agency documents, program promotional materials, policies • Overwhelming amounts of data  “analysis paralysis” Photovoice • Participants’ photos are analyzed • Interpretation through dialogue and discussion • Useful when there are language difficulties