Survey Research Notes PDF

Poll Time What kind of graph is this? A. Histogram B. Bar graph C. Scatterplot D. Line graph  Poll Time What kind of graph is this? A. Histogram B. Bar graph C. Scatterplot D. Line graph  Poll Time What does the Y axis represent on this graph? ...

Poll Time What kind of graph is this? A. Histogram B. Bar graph C. Scatterplot D. Line graph  Poll Time What kind of graph is this? A. Histogram B. Bar graph C. Scatterplot D. Line graph  Poll Time What does the Y axis represent on this graph?  Poll Time What’s wrong with this graph? Week 13: Survey Research  Survey Research What is survey research? Types, advantages and disadvantages Questionnaire design constructing good surveys Sampling issues conducting good surveys What is Survey Research? Survey research uses self-report People are reporting on their own thoughts, feelings, behaviors, etc. Survey research tries to obtain generalizable samples Ideally large and random  Interviews Phone Surveys Questionnaires Types of Survey Research Interviews Structured or unstructured Costly Interviewer bias Social desirability concerns Phone Surveys Structured or unstructured (Used to be) easy to get random samples Cell phones and telemarketing ruined that Cheaper Fewer social desirability concerns Questionnaires Paper or electronic Cheapest Fewest social desirability concerns We are focusing mainly on questionnaires Research Question: Are LGBT attitudes in Canada becoming more positive? In 1997, a poll suggested that 41% of Canadians were in favour of same-sex marriage. How about now? Survey Advantages Can assess non-observable variables, as well as variables that you cannot (ethically) manipulate Demographic information (e.g., sex, age, ethnicity) Attitudes and beliefs Past behavior Current behavior that cannot be observed Motivation and emotion Personality traits Survey Advantages Easy to administer Quick to administer and score Can gather a lot of information Requires few resources “I find it great that in our society, two people of the same sex can get married.” Agree/Disagree Research Question: How does diet affect your mood? Is fruit a mood booster?  How many servings of fruit did you consume last week? _____ How happy did you feel last week? 1 = very unhappy, 7 = very happy Survey Disadvantages Accuracy may be low Participants may lack insight about certain variables May forget previous behavior May respond in a socially desirable manner (i.e., lying) Survey Disadvantages Not manipulating IV, thus cannot demonstrate causation True of all correlational/non-experimental research Poll Time Think of a research question you’re interested in. Would it be well-suited to a survey design, or no?  Survey Research What is survey research? Types, advantages and disadvantages Questionnaire design constructing good surveys What’s Wrong with These Survey Items? When you drink scotch, do you like it on the rocks? Yes/No Should concerned dog owners use tick prevention medication? Yes/No How happy are you with Western as a place to learn and make friends? 1 = very unhappy, 7 = very happy What’s Wrong with These Survey Items? Do you currently, or have you at any time in the past, taken any professional ballet classes, or performed as a ballet dancer in a professional capacity? Yes/No Are you into movies? 1 = not at all, 7= very much Developing Valid Survey Questions Each item should be BRUSO: Brief, Relevant, Unambiguous, Specific, and Objective A Good Survey Item is Brief Avoid long or run-on sentences Avoid unnecessary words Avoid technical terms, acronyms, jargon A Good Survey Item is Relevant Avoid the temptation to include lots of extra items “just in case” Especially personal, “nosy” questions A Good Survey Item is Unambiguous Avoid vague or imprecise terms E.g., “health problems”, “psychological issues” Avoid negative wording “Do you disagree with the idea that parents should not spank their children?” A Good Survey Item is Specific Avoid double-barreled questions – questions that ask two things at once E.g., “Are the Big Macs at McDonald’s delicious and healthy?” A Good Survey Item is Objective Avoid leading questions, emotionally charged words E.g., “Are you in favor of eliminating the wasteful expenses in the budget?” Questions should be neutral E.g., “Are you in favor of reducing the budget?” Reversing the Wording of Some Items Reversing some wording helps prevent yea-saying and nay-saying. When participants agree or disagree with all items because they are not paying attention. What’s Wrong With These Survey Items? How often do you punish your child? Curtailing development and protecting the environment should be a top priority for our city. You wouldn’t say that you are in favor of capping tuition rates, would you? Open-Ended Versus Closed- Ended Items Which genre of music do you enjoy the most? Pop Rap Rock Jazz Death Metal What genre of music do you enjoy the most? _________ Closed-Ended Survey Items Closed-ended questions give a limited number of responses. Must be designed carefully Risk of missing information Easy to score Closed-Ended Categorical Items For categorical questions, simply provide a list of options E.g., What political party are you most affiliated with? Conservative Liberal NDP Green Other _________ Closed-Ended Continuous Items For continuous items, we use rating scales Pick a number on a scale (e.g., 0 to 10) E.g., What is your political orientation? Key researcher decisions: 1. How many points on the scale? 2. What are the anchors - the labels on the ends of the scale? What is your political orientation? 1. Very 2 3 4 5 6 7. Very Liberal Conservative 1. Extremely 2 3 4 5 6 7. Extremely Conservative Progressive 1. Not at all 2. Slightly 3. Somewhat 4. Very 5. Extremely Conservative Conservative Conservative Conservative Conservative 1. Not at all Liberal 2. Slightly Liberal 3. Somewhat Liberal 4. Very Liberal 5. Extremely Liberal 1. Communist 2 3 4. Centrist 5 6 7. Fascist 1. Apathetic/ 2 3 4 5 6 7. Tinfoil hat disengaged wingnut/ moonbat Likert Scale Common type of rating scale used to assess degree of liking or agreement Usually 7 points, but sometimes 5 or 9 Open-Ended Items Allow participants to respond however they want. What is the most important thing in running a business? How often do you drink? If you could be any animal, what would you be and why? Open-Ended Items Provides a lot more information. But: Effortful for the participants Need to decide how to score responses (coding scheme) Scoring the responses takes time Assembling the Survey Order matters because questions can bias responses to later questions! (Item order effects) Place less sensitive/objectionable items before more sensitive items Demographic items are rarely biased and thus go last Assembling the Survey Organized in a coherent, visually pleasing format Font sufficiently large? Not too many items per page (reduce scrolling)? Viewable on different devices?  Survey Research What is survey research? Types, advantages and disadvantages Questionnaire design constructing good surveys Sampling issues conducting good surveys U.S. 1936 Election American magazine, Literary Digest, sampled 2 million people Predicted that Alf Landon would win by a landslide, but Franklin Roosevelt took every state but two What went wrong?  Research Question How happy are 5-year-old Londoners with their kindergarten experiences? Population Entire group we want information about (all potential subjects). Defined by the research question E.g., Children E.g., Children in kindergarten E.g., Children in kindergarten within the London area Samples Subgroup of subjects chosen from the population E.g., Children who we studied E.g., Children in kindergarten who we studied E.g., Children in kindergarten in London who we studied Sampling and Generalization Want to apply results obtained from a sample to the population So, the sample needs to represent the population Simple Random Sampling Everyone in the population has an equal chance of participating Sample should look similar to population E.g., same % of each ethnicity, age, gender, personality traits, etc.. Almost impossible to obtain Limited by location, recruiting methods, resources Stratified Random Sampling Important subgroups are identified E.g., ethnicity, gender, age, income, etc… Obtain a random sample of each subgroup to mirror the population E.g., subsample students from public, Catholic, and private elementary schools in London Nonrandom Sampling Everyone in the population does not have an equal chance of participating May lead to a biased sample – characteristics differ from the population Often due to selection bias – sampling procedures that favor certain characteristics Nonrandom Sampling Convenience sampling – Use participants who are easily available Very common Limits external validity Student Populations 81% of research participants are university students Advantages Easy access to students, free Educate students about research Disadvantages Less variability in age, education, intelligence, wealth Internet populations Recent increase in amount of low paying, online research (e.g., mTurk) Tend to Have a lot of free time Be lower income Be more tech savvy Take research less seriously Voluntary Participation Ethically, participation must be voluntary, but this can affect external validity Volunteer bias: Volunteers are different than non- volunteers More educated Higher social class Higher intelligence Higher need for approval More social More “arousal seeking” Women volunteer more Poll Time A researcher wants to know which kind of billboard is easier to read: black text/white background, or white text/black background. What is the population of interest? Is random sampling needed to test this question? Poll Time A researcher wants to know whether 4th year students are more health-conscious than 1st years. What is the population of interest? Is random sampling needed to test this question? Generalizability Is random sampling always necessary? Need to consider how much participant characteristics are likely to affect results Sometimes, very much E.g., Political polling Sometimes, less so E.g., Vision and reaction time E.g., Mere exposure effect