Week 9 Survey Research I PDF

Survey Research I Angela Zhang  Qualitative and quantitative research Both are trying to describe the world as it exists Both have rich traditions of answering questions/solving problems Both acknowledge individuals have bias and it needs to be accounted for in research Both cr...

Survey Research I Angela Zhang  Qualitative and quantitative research Both are trying to describe the world as it exists Both have rich traditions of answering questions/solving problems Both acknowledge individuals have bias and it needs to be accounted for in research Both create data that are based in qual. and quan. thinking: --all qual data can be coded quant --all quant data are based on qual judgment Both use induction and deduction Both have built-in refutation (reliability and validity checks)  Survey Research An investigation that studies samples of large and small populations taken from larger populations. Theory of survey is that we can find as much as we need to know by studying a sample instead of the entire population. When is survey an appropriate method? Little info about a A lot of info about lot of people __________________________________ a few people Would you rather study, 1 person for 1,000 hours, or 1,000 people for 1 hour? Survey vs. Other Methods Public Polls ---------------Survey Research---------------Experiments (Poli science)-----(Poli science/Social science/JMC)----(Social psych.) (Descriptive)-----------------(Descriptive)--------------------(Predictive) (Correlation)------------------(Correlation)--------------------(Causation) (External validity) ---------(External validity) ---------(Internal validity)  Hourglass logic applied to survey From Populations Draw Samples Survey measures Statistics variables From stats, we Describe parameters of sample Which are inferred to … Populations  Common survey approaches face-to-face (personal interview) telephone (largely killed by the cell phone) mail group administration online Choosing among five types of survey Which ones are self-administered? Any special considerations? When is it best to utilize each? Surveying a geographic area? Reaching underrepresented population? Saving time? On a budget? Long and complex survey? Note: mixed-mode surveys  Web Surveys --web surveys come in all sizes, forms and purposes --cost and ease make them attractive --however, response rates are low, often very low (unless using consumer panel) e.g., Zhang & Muturi, 2021 response rate 25% (n=265 out of n=1037) --*substantial portions of US without access (25-30 percent) --*sampling frames likely do not exist (most non-probability) --recruitment through Email lists Social media/websites Consumer panels/sampling companies Consumer Panels/Sampling Company Panels for specific quota needs for representative sample (nonprobability quota sampling) Not good for smaller geographic areas (e.g., Norman) Except for addressed-based sampling (ABS) (mail surveys) Panel/Sampling companies: e.g., Dynata, Qualtrics, etc. Automated online crowdsourcing panels: e.g., Amazon’s Mechanical Turk, prolific, etc. (reservations about Mturk) See How to use Mturk handout on Canvas See Improving data quality handout for web panel surveys  Strengths of survey research Wide scope of attainable information about large populations If probability sample is used, can be generalized to population *Economically sound (low cost per 1,000) Very accurate if done well realistic, non-obtrusive can be done on your own time, in your home, on the go, etc.  Limitations of survey research *No real depth Possibility of sampling error --no follow-up questions, --rely on volunteers, how do they --limited to what the survey asks. differ from the population? -- extensive data, but not much depth, Difficulty in getting TRUE explanation answers *Time consuming, especially on --ask questions people can’t answer the front end --prestige bias, choosing answers that --develop questionnaires make respondent look good --train interviewers --inability to explain responses, especially attitudes, values --recruitment, follow-ups, data --responses shaped by what collection participants think researchers want  Problems/Prospects for Survey Research Moy and Murphy (2016) --Of 1,150 articles published in our field’s top journals 2008-2014, n=27.7% were survey-based studies Total Survey Error (TSE framework) --frame: biased or incomplete set of people to draw sample (sampling error) --nonresponse: low response raises sampling error, means the survey is likely not generalizable. --measurement: respondent does not interpret the survey as the researcher intended (a reliability error) --specification: a poor statement to measure what is intended (validity issue) --processing: problems with process of data entry, tabulation Improving survey reliability and validity Measurement errors Reducing social desirability bias Issues in questionnaire construction (reliability and validity) Sampling errors Reporting in probability sampling Nonprobability sampling Reducing self-selection bias Increasing response rate  Reporting for Probability and Nonprobability samples (Moy & Murphy, 2016) Probability Nonprobability Disclose researcher/funding Disclose researcher/funding Exact wording of questions Exact wording of questions Full description of population & Sample design (e.g., which type) sampling frame If quota sample, what is the quota used Sample design (e.g., which type) and what population quota is it based on Sample size (plus sampling error) Sample size Type of survey method & dates of data Type of survey method & dates of data collection collection Response rate Response rate (esp. if not using a panel) Results reporting (confidence level Results reporting (confidence level and and confidence intervals/inference to confidence intervals/inference to population)* population with caveat)* Reducing Social Desirability Bias Social Desirability Bias: the tendency to portray oneself as a “good respondent” or someone whose thoughts, attitudes, and behaviors are socially acceptable (Moy & Murpohy, 2016, p. 19) E.g., Sensitive topics, controversial political and social issues, power imbalance (employee research) How to reduce: Survey instruction – include a more permissive frame of reference Change a method (experiment, Q-sort, content analysis, etc.) Issues in Questionnaire Construction Questions to avoid: Incomplete wording Poorly defined terms Multiple questions/double barreled questions Prestige biased questions Biased or loaded-terms (labels) Leading questions Over demanding recall questions Questions that cannot be answered Double-negative terms  To get the truth (& enhance response rates) Gade’s Top 10 list for writing good surveys 10) Make your statements/questions clear 9) Keep statements/questions short 8) Keep statements/questions focused on your research objectives 7) Avoid double-barreled statements/questions 6) Avoid bias or loaded-terms (labels)  Gade’s Top 10 list for writing good surveys II 5) Avoid leading questions (do you agree or disagree?) 4) Avoid “always” and “never” wording 3) Do not ask what can’t be answered (too detailed) 2) Avoid embarrassing questions (they encourage lying or non-response) 1) Direction of statements – word the questions so respondents might agree with some and disagree with others.  Take-home assignment Read through the four survey questionnaires, think about: What are the primary differences between Zhang & Cozma, Perry & Gade AND the National Park and the TCU surveys? For the Zhang & Cozma and Perry & Gade questionnaires: What are the differences and similarities between the two (length, order of questions, instructions, question constructions, type of questions, etc.)? When do we use open-ended vs. close-ended questions? For the National Park and TCU questionnaires: What types of population do you think they are focused on? What are these surveys seeking to find out? Types of measurement used? Based on these types of measurement, what do we know about the intent of the researchers? Critically analyze: survey intro/instructions Can the surveys be self-administered? Any issues with the survey questions? Next Week Read: Salkind, Chap. 6 *Nardi, Chaps. 3-4 *Millar and Dillman, Improving Response to Web and Mixed-mode Surveys Assignment due: Proposal Intro/Outline due