Qualitative Methods for Planning and Evaluation - PDF

Narrative Designs • Can be used in assessment, process monitoring, and identifying outcomes of policy changes • Text as data – Personal diaries, agency/program records, media reports • Often follow content analysis approach Multiple Methods and Triangulation • Using > 1 method to compensate for l...

Narrative Designs • Can be used in assessment, process monitoring, and identifying outcomes of policy changes • Text as data – Personal diaries, agency/program records, media reports • Often follow content analysis approach Multiple Methods and Triangulation • Using > 1 method to compensate for limitations of each approach – Can use qualitative and/or quantitative methods together – May reveal entirely different results • Triangulation – Confirming, disconfirming, or modifying info gained from one method • Challenges in how to analyze and synthesize data Rigor in Qualitative Methods Dependability – Document analysis steps – Use reliability statistics Confirmability – Document analysis steps – Document researcher’s perceptions Credibility • Triangulation • Use outsiders to aid in interpretation • Refine hypotheses with negative cases Transferability • Applicability to other contexts or respondents Sampling for Qualitative Methods • Two key considerations – Design – Sampling strategy to be used • Purposive samples – Random selection and power analysis not relevant Sampling for Qualitative Methods Method Sampling considerations Case study Choice of case based on being either “usual” or “unusual”; maximum # of cases feasible to conduct Observations Ability to sample behaviors or events without altering their quality; need to obtain category saturation Individual indepth interviews Need to obtain category saturation; choice of individuals based on theoretical sampling Focus groups Participant representativeness within & across groups; max size of each focus group; min # focus groups to capture diversity of views Survey with open-ended questions Linked to sampling strategy for the survey; likelihood of write-in responses Narrative designs Quantity and quality of existing documents available for review; access to existing documents Qualitative Sampling Strategies Sampling strategy Types of cases used Use Convenience Those that are accessible and wiling Saves time and recruitment money Critical cases Exemplar cases; those that are unique in an important way Permits generalization to similar cases Deviant cases Highly unusual cases Reveals factors associated with unique or extreme conditions and may lead to new theory Maximum variation Cases with differing experiences Fosters category saturation with most possible categories Qualitative Sampling Strategies, Continued Sampling strategy Types of cases used Use Random purposeful Cases randomly selected from large sampling pool Adds credibility to the sample and thus some indication of generalizability Typical cases Usual or normal cases Broadly applicable theory or categories, but doesn’t address full breadth of program effects Theory based Cases with theoretical construct Elaborates or refines the theory Overview of the Analytic Process • Determine codable units and identify them in the data – e.g., paragraphs, facial expressions • Understand manifest and implied meanings • Category formation and naming • Define mutually exclusive and exhaustive categories • Present findings to participants to confirm or revise interpretation • Generate working theories based on results Data Analysis Software • Features: – Diagram relationships among categories, count units of analysis, coding text, theory building • Software facilitates analysis work, but does not do the work Analysis Issues to Consider • Tempting to count # of occurences – What is the numerator? Denominator? • Iterative and sometimes messy data analysis • Cost – Travel costs to study sites – Transcription costs (1 hour of interview = 3 hours transcription) • Staff training for data collection and analysis Presenting Findings • Transferability: describe data collection contexts • Dependability and confirmation: document category development • Confirmation: include participants’ words to humanize findings • Diagram relationships among categories • Relate findings to program theory or logic model, if possible