Lecture 8 - Sampling, Surveys, Standardised Interviews PDF

Structured Interviewing and Surveys Department of Government  Today’s Lecture Sampling Surveys: self – completion questionnaires online or by post Standardized interviewing (Face to face or telephone, CATI) Department of Government ...

Structured Interviewing and Surveys Department of Government  Today’s Lecture Sampling Surveys: self – completion questionnaires online or by post Standardized interviewing (Face to face or telephone, CATI) Department of Government  Sampling Selecting participants for your research Generalisability of the findings of the research requires a representative sample Need to decide at the outset: Population Sample Probability or non probability sampling Department of Government  Simple Random Sample Each unit of the population has an equal chance of inclusion in the sample and is independent of each other Who is the population: N Devise sampling frame Decide sample size: n Use random number generator to select respondents Sample = n/N Department of Government  Systematic Sample Variation of the SRS; select units from the sampling frame Assign each member of the population a number and pick each nth member of the population Members of the sample are not independent of each other Department of Government  Stratified Random Sample The population may be stratified according to a specific criterion which would be relevant for the research (age, gender, ethnicity) The stratum size can be proportionate or disproportionate to the population size Use SRS within each of the strata Department of Government  Multi-stage Cluster Sampling Used when the sample is required from a widely diverse population Population is divided into groups known as clusters. 1. Select a random sample of groups 2. Random sample from within the cluster Department of Government  Sample size Factors to consider – Time and cost – Non-response – Heterogeneity of the population The absolute size of a sample is more important that the relative size of a population Increasing sample size does not guarantee an increase in precision Department of Government  Issues in Probability Sampling High quality sampling procedures are vital if a researcher intends to make inferences about a wider population based upon the sample they select. Make claims which can be justified Accurately represent response rate Department of Government  Non – Probability Sampling More strongly associated with qualitative research sample is decided while research is in process, typical cases are more important than actual numbers, suitability to study is more important than representativeness) Snowball sampling Convenience sampling: who is available Quota sampling: I need to get ‘this many’ Department of Government  Stages in a Survey Survey task Skills needed Resources Design Survey Know alternative What resources are research designs and needed? their implementation Survey expert Software Prepare survey Literature Search Questionnaire Expert instrument Gain knowledge of topic Have ability to write questions Form qs into a survey Pilot-test survey Have ability to Administering survey instrument analyse pilot data (paper, postage, Interpret data and computer) revise survey instrument Department of Government  Stages in a Survey Administer Interviews Interviewer training Survey Ask questions in standardised, Training materials efficient manner Telephones Record answers in a Rooms standardised efficient manner Monitor survey administration Mail Postage Understand logistics of mail Paper survey (self addressed envelopes, etc) Graphics consultant Have ability to monitor Incentives for completion Online Software: Qualtrics, Attractive, easy to follow Survey monkey, design Google forms, jotform Department of Government  Stages in a Survey Organize Data Code Data Programmer Enter Data Data Analysis Clean Data consultant Prepare Codebook Analyse Data Select appropriate data Statistician analysis techniques Perform data analysis Report Results Write report Printing Report Prepare slides Department of Government  The Structured Interview Often known as a standardized interview Each interviewee receives the same context of questioning, same interview stimulus. Aim is to aggregate interviewee responses Questions are specific and often pre-coded Department of Government  Sources of Error Poorly worded questions The way in which the question is asked by the interviewee Misunderstanding on the part of the interviewee Memory problems on the part of the interviewee The way the information is recorded by the interviewer Data coding errors or computer entry errors Department of Government  Processing the Information Structured interviews usually rely on closed questions The respondent has a limited number of options available to them Tick the box Reduces interviewer variability Open ended questions can be difficult to interpret and code resulting in coding error People are reluctant to use them They may not give accurate information Department of Government  Respondents will participate if: They are interested in the work They may want to talk to someone They may feel important by being selected They may feel that participating is an act of contribution to the community They may want to help the interviewer to do their job. They may want to be polite They may have nothing else to do with their time Department of Government  Respondents will decline to participate if: They are not interested in the research They do not agree with the research objective The interviewer is unpleasant They do not have time They are unsure of the purpose of the interview They do not have the consent of their spouse, who may be opposed to the research Department of Government  Self Completion Questionnaires Self administered – Post, email, social media Questionnaires need to be clear as there is no interviewer to guide the process. Self completion questionnaires tend to include – Closed questions – Easy to follow designs – No more than two pages of questions (paper); 10-12 mins online Department of Government  Advantages of the Self Completion Questionnaire Online, by post, post-online mix with QR code Cheap to administer (especially online) Quick to administer Absence of interviewer effects Convenient for respondent Department of Government  Disadvantages Cannot prompt the respondent if they are having difficulty with a question No opportunity to probe the answers that are provided Respondents are likely to discard/click out of the survey if they are not interested Limited types of questions are possible Although this is broadening with AI tools for analysis but still limits to effort that respondents will take. Only a small number of questions can be asked Respondents will be able to read the whole questionnaire before answering (if postal option) Department of Government  Lower Response Rates Response rates are important because we need to know that those who respond are not different from those who do not. If a response rate is low, there may be bias in the sample How representative are the results of the survey Department of Government  Steps to Improve Response Rates Write a good cover letter explaining what the research is about Be persuasive about why it matters Postal questionnaires should be accompanied by a stamped, self- addressed envelope (expensive) Follow up individuals who do not respond with reminder post cards, letters, emails Short questionnaires have higher response rates Have clear instructions and an attractive layout Big debate about incentives: draw for an iPad, monetary incentive Department of Government  Design Issues Questions should be clearly presented Instructions should be clear and straight forward Styles can be mixed but should be consistent; all instructions in bold or italics Q and A should appear on the same page Most questions will be closed in a self-completion questionnaire Horizontal or vertical format Likert scales (balanced or unbalanced) If including open questions how will they be analysed Department of Government  Wording Your Questions Is the language simple? Can the question be shortened? Is the question double barrelled? (How often do you visit your grandparents? Everybody has four, which one(s) do you mean?) Is the question leading? (Do you make a habit of drinking?) Is the question negative? (people often mis- interpret questions with the word not in them) Department of Government  Wording Your Questions Does the respondent have the information to answer your question? (do you agree with the Government’s policy on??) Will the word(s) have the same meaning for everyone? (excessive alcohol consumption) Is there a prestige bias in the question? (will the person give what they think is the desirable answer by increasing their salary, alcohol consumption? How will you deal with this) Is the question ambiguous? (poor sentence structure, double negatives) Department of Government  Wording Your Questions Is the frame of reference for the question clear? (How often do you play sport? Weekly, monthly, yearly, which is it) Does the question create artificial opinions? If the person has no opinion, should you include that option. Is the question unnecessarily detailed or objectionable? (age, income, categories are usually better than precise figures) Department of Government  Negative Questions Property taxes should not be introduced. agree disagree Property taxes should be introduced. agree disagree Marijuana use should not be decriminalised. agree disagree Marijuana use should remain illegal agree disagree Department of Government Constructing Questions (4 types) Behaviour: What people do Beliefs: What do people think is true? (does not measure the accuracy of their beliefs) Attitudes: Questions which try to establish what people think is desirable Attributes: Questions designed to obtain information about the respondents’ characteristics (age, gender, ethnicity, marital status, employment status) Department of Government  Types of Questions: Open Respondents can reply how they wish, unusual information may emerge Questions don’t suggest answers so a truer impression of the respondents views will emerge Useful for an examination of new areas or where a person’s subjective experience is under study Increasingly used due to machine learning tools Department of Government  Types of Questions: Closed Easy to process Answers are more comparable Closed question options can clarify the meaning of a question Reduce the variability which emerges in interpreting respondents answers Department of Government  Likert scales Named after Rensis Likert 1.Strongly disagree 2.Disagree 3.Neither agree nor disagree 4.Agree 5.Strongly agree Department of Government Choosing between Closed and Open Questions A closed question to see if a respondent has thought about or is aware of an issue. An open question to get their opinion on the matter. Closed questions address specific aspects of an issue. Open or closed questions to find out respondents reasons for their opinions Closed questions to find out how strong the opinion is. Department of Government  Evaluating Questions Check for redundancy (2 questions asking the same thing) Check for response sets. (agreeing with all questions in a group implies the same opinion: mix question structures) Check likert scales. Department of Government  References Clark et al., (2021) Bryman’s Social Research Methods. Fowler, F.J., (2013) Survey Research Methods; and Improving Survey Questions Keats, D., (1999) Interviewing; a practical guide for students and professionals Department of Government