Conducting A Survey Block 2, 2024 Lecture 1 PDF

Conducting a Survey Block 2, 2024 Week 1: Introduction to Survey Methodology 1  Who are we? Course coordinator Angelo Moretti [email protected] Office: Sjoerd Groenman Gebouw, the Uithof C1.19 Lectures: Angelo Moretti,...

Conducting a Survey Block 2, 2024 Week 1: Introduction to Survey Methodology 1  Who are we? Course coordinator Angelo Moretti [email protected] Office: Sjoerd Groenman Gebouw, the Uithof C1.19 Lectures: Angelo Moretti, Camilla Salvatore WG1: Danielle McColl WG2: Camilla Salvatore Practicals: Daniëlle Remmerwaal 2 2  Goals for Week 1 Week 1. Introduction and specifying your research-question Introduction to the course, its goals and rules, and an introduction to survey methodology. The content of a research plan will be discussed (research question, flow diagram and target population). Students form research groups of 4 during break of the lecture. You will be conducting a survey with this group – Research groups must go together to the WG – Research groups hand in practical assignment per group – Division of groups will appear on Blackboard this week. Research groups: devise an efficient way to work together: → Via Teams >Exchange email & telephone information 3  Course description Week 1: introduction Week 2: Cognitive laboratories, questionnaire quality, special topics and consent Week 3: Designing a sampling frame and non-response Week 4: Best practices in equality, diversity and inclusion in survey research Week 5: coding & screening Week 6: weighting and scaling Week Christmas break 1 Week Christmas break 2 Week 7: New forms of data in survey researc Week 8: Q&A, evaluation Week 9: exam 4 Early exit Send an email to Angelo Moretti ([email protected]) Make sure you are in a group with 4 students (and only one early exit), inform your teammates 5 Book You must read the indicated chapters on the course manual These are part of the course. 6 Goals Conducting a Survey Whole process of survey research You will see the different stages of the process: from research question to developing and organizing the questionnaire to analyzing the results, including the fieldwork. To guide you through the process of conducting a survey, you will work a series of assignments. Group work! The survey will be conducted in groups of about 4 students. Examination: Research plan, practicals (pass), report, exam (individual) 7 More information Course manual! Here you can find all information you need. With handy overviews and deadlines for all assignments. Take a special look at ‘hand in’ dates Several assignments Practicals Research plan Report SPSS/R Exercises in practicals They build up to the actual survey Exam on 21st January 2025 at 1:30pm On Remindo: laptops will be provided. 8 Rules Be on time! Mail in the assignments on time (9.00 am)! In order to pass the course, your final grade AND your exam should be 5.5 or higher. E.g.: if your research plan receives less than a 6, improve it! This will help you to enhance the quality of the final report. 9 Making groups Groups of at least 4 students (no more than 5) You can fill in names on registration list during break via this link: https://forms.gle/Q5uhREFnVg54RvhV6 but see also Practical Information tab for the link I will form groups today. See Brightspace by tomorrow for the full list and final groups. 10 Assessment The final grade is computed as follows: Assignment 1 (research plan): 20% Assignment 2 (final report): 40% Written exam: 40% Practical assignments: Pass/Fail Presentation during the workgroup: Pass/Fail 11 Work group In the workgroup all students must be present Each group will deliver a 10 minutes oral presentation on their progress depending on the weekly topic They will have additional 10 minutes for Q&A and feedback The presentation with the additional documents (depending on the week) will have to be sent to the workgroup instructor according to the guidelines in the table in the Course Overview below The presentation will not be marked (pass/fail). 12 13 With the rise of Big Data, who needs surveys anymore? Google Consumer Surveys, Facebook surveys, DIY systems 14 Big data Three characteristics of big data: » Volume » Velocity » Variability – Administrative data – provided by persons or organizations for regulatory or other government activities – Transaction data – generated as an automatic byproduct of transaction and activities (e.g., credit card data, traffic flow data) – Social media data – created by people with the express purpose of sharing with (some) others – Sensor data – generated by apps on for example smartphones (GPS, accerometers, biomarkers) 15 Limitations of Big Data Couper (2013): Single variable, few covariates Bias through self-selection and self-presentation Volatility or lack of stability Privacy issues Access issues Opportunity for mischief Bigger does not mean better 16 Single Variable, Limited Covariates Surveys are much more than a single variable Limited demographic variables provided or imputed may be wrong – Only about 1/3 of Facebook users and Google Consumer Survey respondents provide demographic information – What is available (derived from cookies) may be wrong, e.g., gender matches about 75% of time – Check your own profile at: https://www.google.com/settings/ads/onweb/ Knowing changing fuel prices is not the same as knowing what people do in response to such changing prices 17 Bias Two sources of bias: – Selection bias – Self-presentation (measurement) bias Selection bias: “haves” versus “have-nots” – Not everyone uses social media! – Need to distinguish between producers and users of users of social media – Not everyone uses loyalty cards or credit cards, or makes purchases online Measurement bias – Impression management is a key element of social media – The average Facebook user has MANY “friends” 18 Technological speed Google.com was registered in 1997 What will Facebook look like 5 or 10 years from now? – Will it even exist? Anyone remember MySpace? Second Life? Twitter (2006) grew 5000% in 5 years – Twitter today is very different from Twitter 5 years ago in fact it is called X! Social media may be good for measuring short term trends, but surveys may be better for longer-run measurement 19 Access issues Social media and transaction data are usually proprietary – Only available to insiders, or at a cost – Exception: Twitter A key strength of surveys is public access to data, permitting replication and reanalysis 20 Bigger does not mean better Example Election Polls USA: – Very large sample (n=10 million) from commercial databases – 2.3 million surveys returned – Correctly predicted last 5 elections But wrong! This was the Literary Digest Poll of 1936 21 So why doing this course? Data-driven society Surveys have an important role to play Probability based surveys cannot be abandoned Important decisions are based on survey data With the increase speed in data collection, there is a clear need for survey professionals! 22 Conducting you own survey Suggestions for topics: Best choice: Your own topic Environment: (how much do people care about the environment, what do they do about it?) Immigration: how do people see immigrants, how do they interact with them? Recession: (do people worry, why so?, and do they change their behavior?) Leisure: To what extend can leisure behavior explain satisfaction with life? Are people who practice “serious leisure” happier than others? 23 Ethics do not collect personal data no invasive topics no special/vulnerable groups such as children, people with mental illness safeguard the data and delete the data once the course is finished! 24 In cooperation with organisation You can do it for an organisation (sports club, restaurant etc.) – Pro’s: results are used; practical experience – Con’s: possible delays; another person to cope with Make sure the organisation can provide you with (e-mail) addresses 25 Assignment 1: Research plan From topic to research question to central constructs Defining the target population and sampling plan Formulate what the independent, dependent, and control variables are and formulate an explicit hypothesis (flow diagram) Mode of data collection Make a planning (questionnaire content, questionnaire design, start fieldwork, reminder, end fieldwork, analyses, writing report) This research plan is graded. 26 Research Process 27 Planning primary data collection 28 Research instrument: survey Questionnaires should be developed and tested carefully before being used! The researcher should know: – What questions to ask (only the relevant ones!) – What form the questions should take (scale, categories, open-ended) – How to word the questions – How to sequence the questions (interesting-personal) Each question should be examined to ensure that it contributes to the research objectives! 29 Introduction to survey methodology Developing and conducting a survey is an iterative process. Use the feedback given by instructors and peers. The following diagram is the backbone of the course. Each week we will discuss a particular part of the diagram. 30 Research Project Research-question - the (sub) research-question(s) must include one or more relationships between constructs/variables Questionnaire: - Contains at least one construct measured by several items - 15 to 30 questions - maximum 10 control (background) variables Rule of thumb: respondents can answer about 3 questions / minute Collecting the data: - web, paper-and-pencil questionnaire; telephone or face-to-face interview - the minimal number of questionnaires you collect is 60 (put more in the field → nonresponse). 40 Research plan: question Research question: What would you like to know/find out? - Descriptive question (what is happening?) - Explanatory question (why?) Set up a meeting of your research group (e.g. on Thursday) and formulate a research question before meeting 41 Research plan: target population Defining the target population: What type of person would you like to study? Define your population as precisely as you can. Is a sampling frame available, e.g. an administration list? How accurate is the list (is every member of the population present on the list, addresses, accuracy, missing info, doubles, ineligibles) 42 Research plan: variables Flow diagram: Which variables do you need to include in your questionnaire in order to be able to answer your research question? Dependent variable(s) Explanatory / independent variables Control (background) variables (demographic, socio- economic, or topic related) Personality Average grade Gender, age 43 Research plan: questions From research question to constructs: How are the constructs going to be measured? - operationalisation of the independent, dependent, and control variables - Which aspects? Which items? - How many items do you need in order to measure the constructs? Investigate existing scales, e.g. collection of scales at www.ipip.ori.org, or at www.yorku.ca/rokada/psyctest/ or search in papers 44 Example: Do dogs have personality? 45  Do dogs have personality? Cross species evidence for the Big 5 Theory Target population Constructs Big five is genetic Dogs (?) dependent independent Big five is cross- Openness (5) About dog cultural Proxy interviews Conscientiousness (5) - Race age, sex Extraversion (5) - Behavior of dog All Dutch dog owners Big five can be Agreeableness (5) About owner: evolutionary explained Neuroticism (5) Age, sex, job, household So why not cross Sample? Frame? situation, species? -Psychological vars - etc. Research question 46 Survey research A method for gathering information about a specified group of people (population) by asking them questions For fact finding and theory testing Questions as well as answers are highly structured Ask yourself: is a survey the best instrument to answer my research question? 47 When is a survey an appropriate tool? It is about populations: what is unit of analysis? ✓ Only when the unit of study is the ‘individual’ How much do members of a population know? ✓ Questions should not be too difficult or complex How much are respondents willing to reveal? ✓ Topics should not be too sensitive It is about non-observable behavior It is only reliable when respondent is aware of behavior A survey is an unreliable tool to discover attitudes and behavior of the distant past 48 Do dogs have personality? Populations? Unit of analysis? ✓ Individuals? ✓ Is there a population frame? How much do members know? ✓ Proxy interviews ✓ Do owners know ‘their’ dogs? Are they willing to reveal true answers? ✓ Wishful thinking ✓ Overgeneralization ✓ Social desirability Observable behaviours? ✓ How to define OCEAN in terms of dog behaviour? Biased answers? ✓ Can we correct for biases using independent variables? 49 GIGO In order to prevent errors, scientists use a methodological framework. In order to reduce the negative consequences of unavoidable errors that enter during the research process, scientists use statistical tools. But remember: You can rarely put right by statistics what you have messed up by design (Thomas Cook, 2006) GIGO = Garbage In Garbage Out 50 Question writing Introduction to question writing (more on the book please always read the indicated chapters!) Examples of different kinds of questions Examples of different kinds of response categories (and related effects) Link between question, the response format and your analyses Scale-analysis, reliability and level of measurement. Consequences for analyses and data entering in SPSS/import to R. 51 Introduction to question writing First question: Is a survey the right tool? While writing your questions constantly keep in mind who your respondent is. While writing your questions constantly keep in mind what your research question is. Developing questions for a survey is as a constant conversation between you and your respondents, keeping in mind the key variables you want to measure. 52 Introduction to question writing Bottom up or Draft writing – Start with writing down all the questions that you can come up with – Find out the logic behind these questions and order them – Cut out what you do not need or cannot measure Top down Start with defining all constructs from theory Operationalize constructs into questions / items We prefer the top down approach: WHY? Respondent’s burden versus 'one go' In top down approach we have the option to look for existing validated measurement instruments 53 Bottom up example Do you trust Do you trust the Dopeople youstrangers? you trust don’t Do you trust government? know? politicians? Would you let a stranger watch your Do you trust luggage on a train? your partner in your Social Trust relation? Would you say you Are you a careful have to be careful person? around other people? 54 Introduction to question writing Top down – Start with defining all constructs from theory – Operationalize constructs into questions / items We prefer the top down approach: WHY? – Respondent’s burden versus 'one go' – In top down approach we have the option to look for existing validated measurement instruments 55 Something to think about (see Czaja and Blair) 1. Does the item measure an aspect of the research question? 2. Does it provide information in conjunction (skip and branch) with other items? 3. Will all respondents understand the question in the same way? 4. Do respondents have the information to answer the question? 5. Will respondents be willing to answer? 6. Can I use this question in my analysis? 56 Introduction into writing questions A question consists of two parts: 1. The statement or question 2. The response categories Think about how you construct your response categories, because they determine how respondents interpret your question! Think about how you construct your response categories, because they determine your analysis strategy! 57 Pitfalls for questions Asking for attitudes is implying that your respondents do have one. – In the past year, would you say that the crime situation in your neighbourhood has become better, stayed the same, or got worse? Double barrelled questions. – Does your company provide training for new employees and retraining for existing staff? Loaded or leading questions. – Are you in favour of higher student grants, as they improve the quality of our educational system? Sensitive questions. – Have you ever taken hard-drugs? 58 Level of measurement Categorical versus continuous variables Nominal Ordinal Interval Ratio 59  Level of Level measurement of measurement Discrete Measurement versus continuous Description Examplesvariables Test Descriptive level Nominal: frequencies, median statistics Statistics Nominal Ordinal Categories gender (m/f) Chi-square Modus Mode colors Interval Ordinal Ratiocategories that can be Education level Chi-square, Modus, Mode, median Median placed in a logical Status (Spearman) order correlation Interval categories in an order, temperature T-test / Modus, Mode, median, Median, where distances (Celsius, ANOVA Mean mean between two values Fahrenheit) Correlation are the same approval rating (all) of government (1-10) Ratio same as interval, Kelvin, number Same Same except there is a of shoes meaningful zero-point 60 Examples of questions: Likert items I can’t live without my smartphone Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree Measurement level? Ordinal / Interval(…) Full labels? Labels chosen more often/anchor ‘Neutral’ category? Is this a viable choice? Don’t know / Is this likely? Alternative to neutral? missing? 61 Examples of questions: completion I am …… aware of the presence of God Never Continually 0 1 2 3 4 5 6 7 8 9 10 This type is called phrase completion it consists of a phrase followed by an 11-point response key. Ratio 62 Examples of questions: more than one answer possible Please tell me which of the following food you consume on a daily basis? 1.Milk products 2.Meat 3.Vegetables 4.Bread 5.Fruit 6.Butter or oil How do you analyse this? How would you bring it into SPSS? Every category is a separate variable (0-1) (v1a, v1b,…, v1f) 63 Examples of questions: forced choice Between the two options, please choose the one that suits you most. 1 2 Comedy Drama ….. Horror Action ….. Comedy Action ….. Horror Comedy ….. Drama Horror ….. Action Drama ….. How do you analyse this? How would you bring it into SPSS? Separate variables (v1a-v1f) 64 Question order Start with interesting question Start with question related to the topic Go from broad to specific Group related questions together End with socio-demographics 65 Introduction text Survey introduction – Inviting, motivating, specific Introduction to specific questions – Difficult questions – Other topic – Sensitive questions – Why do you ask? Instruction text – Take into account… – Think about the last time you… 66 Number of questions Not too many Not too few Grasp every aspect of a construct Prevent wrong measurement from mistakes Scales: at least 3 validated (or 5 unvalidated) per topic The more accurate you need to be, the more questions you need 67 Measurement Error (ME) Deviaton of the answers of respondents from their true values Y=T+e Answer obtained=true score+error ME respondents: lack of motivation,comprehenson problems ME survey instrument: poor wording, poor design, technical flaws 68 Pitfalls for response categories: Response effects The accuracy of surveys depends almost completely on the accuracy of people’s answers Response effects= differences in outcomes that reflect seemingly irrelevant procedural details (such as the order in which the answer categories are presented) These response effects may be due to – problems in understanding the question, – remembering relevant information, – producing an appropriate answer, or – other mental processes 69 Some typical response effects Acquiescence= tendency to agree Social desirability= need to present oneself in a favorably light Primacy and recency effects=more easily selecting the first or last answer (primacy in web surveys) Satisficing= searching for the first option that seems appropriate, rather than evaluating each option individually before arriving at an answer (results in primacy effects) 70 Response effects depend on Central or peripheral routes of question processing – Respondents are more susceptible to response effects when taking a peripheral route Memory-based versus on-line judgments – Respondents are more susceptible to response effects when judging on-line Difficulty of the question – The more difficult the question, the higher the response effect Respondent characteristics – Working memory capacity (e.g. old, low education) Motivation – The lower a respondent’s motivation, the higher the response effect 71 Survey design choices As a researcher you will have to make decisions about the design of your questionnaire – Types of questions – Types of answer categories – Order of questions – Order of response categories – Layout – Etc » Your choices influence respondents’ answers! 72 Visual design Web surveys and paper-and-pencil questionnaires are methods drawing on visual processing Design plays a much larger role in web surveys compared to other methods. More tools are available, e.g. pictures, sound, drop-down boxes, information links, feedback etc. – Researchers often do not know exactly how a question appears on individual respondents’ screens (depends on hardware, software, user preferences) 73 Visual design Web surveys and paper-and-pencil questionnaires are methods drawing on visual processing Design plays a much larger role in web surveys compared to other methods. More tools are available, e.g. pictures, sound, drop-down boxes, information links, feedback etc. – Researchers often do not know exactly how a question appears on individual respondents’ screens (depends on hardware, software, user preferences) 74 Homework Set up a meeting for your research group, and reserve sufficient time to specify: Topic, research question(s) and central constructs Flow diagram dependent, independent, control variables Research hypothesis Target population and sampling frame Mode of data collection Planning Costs Hand in before Tuesday 19th 9:00 am through Brightspace 75