Effective Instruments for Multidisciplinary Research PDF

Effective Instruments for Multidisciplinary Research Detection of Response Biases Jiri Remr Evaluating Questions Standard field pilots Desk-based evaluations Expert panels Respondent debriefing Analysis of existing data Split-ballot experiments Behavioral coding Interviewer...

Effective Instruments for Multidisciplinary Research Detection of Response Biases Jiri Remr Evaluating Questions Standard field pilots Desk-based evaluations Expert panels Respondent debriefing Analysis of existing data Split-ballot experiments Behavioral coding Interviewer rating Cognitive testing 2 Cognitive Interviewing „Process used to study the manner in which target audiences understand, mentally process and respond to the survey questions.“ (Willis 2005) Interview designed to reveal some of the mental processes elicited by questions. Method to examine how respondents perceive and interpret questions and detect possible sources of response bias. Cognitive interviewing is a pretest technique for mapping response variation and error (deviation from the intended meaning). 3 Purpose of Cognitive Interviews To learn:... if respondents understand words and phrases as intended.... about the process of answering the questions.... whether items are acceptable.... about the usefulness of response choices whether response choices are adequate, how respondents use the response choices. 4 Question Response Process Social Social Social Factors Factors Factors Comprehension Retrieval Judgment Response Social Social Social Social Factors Factors Factors Factors 5 Cognitive Interview Methods Think aloud interviews 1) Respondent asked to think aloud as they answer question. 2) Subjects articulate their thoughts as they respond to questions. 3) Asking the participant to verbalize their thinking: “tell me what you’re thinking while answering”. 4) Thorough examination of the entire thought process of creating answers. 5) Require that respondents verbalize their thought processes as they are answering items. Probe interviews 1) Interviewer asks specific questions to elicit how respondent answered question. 2) Scripted and spontaneous probing. 6 Think-aloud Interviews 1. Work better: verbal information (not spatial or nonverbal) problem oriented questions conscious processing of information 2. Less effective: understanding of terms 3. May rely on short term memory recall 7 Verbal Probing Different types of probes: 1) Scripted and unscripted Scripted probes Pros: interview is focused around objectives Cons: might be too rigid Spontaneous probes Pros: allows more flexibility Cons: no coordination of probing across interviewers 8 Verbal Probing Different types of probes: 1) Scripted and unscripted 2) Concurrent and retrospective Concurrent probes (between questions) Pros: Question is very fresh on the mind Cons: Potential bias (switching of tasks can be distracting) Retrospective probes (probe at end of questionnaire) Pros: Avoids bias and task-switching Cons: Long gap between question and probe 9 Verbal Probing Different types of probes: 1) Scripted and unscripted 2) Concurrent and retrospective 3) General and specific 10 Verbal Probing – Other Probes Frame of reference What were you thinking about while answering? Encourage narrative To learn what the short response to the question means Redundancy How is the phrase “give you advice about your diet and exercise” different from the phrase “talk to you about your diet and exercise”? Acceptability 11 When I asked you how often you felt discriminated by doctors because of your race or ethnicity, you answered … Were you offended by this question? Advantages & Disadvantages Think-aloud Verbal Probing Less interviewer bias More control (topics; depth) Less interviewer training Little training of subject Open-ended format (unanticipated answers) Interviewer more free to listen and less potential for bias Need to subject training Greater risk of reactivity Less control Potential for bias More difficult to tell if respondent can answer Need for interviewer training Potential bias in information processing 12