SOC 10 Surveys

• Surveys: SOC 10, 3/10/2025

Overview

• Surveys are used for describing populations
• Surveys are used for constructing surveys
• Surveys are used for administering surveys
• Surveys are used for measuring variables and testing hypotheses
• Surveys should have validity and reliability

Describing Populations

• Survey research is useful for collecting original data about populations too large to observe directly
• Probability sample results can be generalized to the population
• Surveys are used in sociology to measure attitudes, opinions, values, orientations toward social issues, and self-reported behavior like religious attendance
• Surveys facilitate secondary analysis by deidentifying data

Survey Content: Questions and Statements

• Questionnaires use survey items, which are a mix of questions and statements
• Questions and statements are useful for researchers to operationalize concepts for measurement.
• Questions about attitudes toward abortion, for instance, can render empirical measures of attitude
• Statements ask respondents to agree or disagree, sometimes indicating the strength of agreement
• Question items can be open-ended or closed-ended

Open- and Closed-Ended Questions

• Open-ended questions prompt respondents to provide their own answers
  • An example: "What do you feel is the most important issue facing the US today?"
• Open-ended responses must be coded for quantitative analysis
• Open-ended questions capture important issues but are hard to code consistently
• Closed-ended questions ask respondents to select an answer from a provided list
  • An example; "Do you agree with the recent tax increase? Yes or no."
• Closed-ended questions allow response uniformity and are easier to process, but may omit issues important to respondents

Constructing Closed-Ended Questions

• Closed-ended questions should be constructed according to two structural requirements
  • Response categories should be exhaustive
  • Answer categories must be mutually exclusive
• Questions should be precise for clarity.
  • If the question is unclear, it won't measure what it is designed to measure, i.e., it won't be a valid measure
  • Avoid jargon and technical vocabulary

Examples of bad questions

• "Is this person's origin or descent (check one): Mexican, Puerto Rican, Cuban, Central or South American, Other Spanish, or No, none of these" (From the 1970 US Census)
• Double-barreled questions are complex questions with two parts
  • An example: "The United States should abandon its space program and spend the money on domestic programs. Do you agree or disagree?"
  • The word "and" in a question or statement may indicate a double-barreled question
• Respondents should be able to answer questions
  • An example: "At what age did you first talk back to your parents?"
• Respondents must be willing to answer honestly
  • Someone may be afraid of repercussions or sensitive to social desirability
• Questions should be relevant enough for participants to have an opinion.
  • Participants may sometimes claim expertise they don't have, as when they endorse fictional politicians
• Questions should be kept short and succinct.
  • Lengthy, involved questions increase the likelihood of misunderstanding
• Questions should be phrased positively to avoid negative constructions and confusion
  • "The US should not recognize Cuba. Do you agree or disagree?" is an example of a bad question
• Avoid biased items or terms
  • Questions should not encourage respondents to choose one answer over others

Question Wording and Bias

• Respondents to the General Social Survey were asked their opinions on government poverty reduction efforts (Rasinski, 1989)
  • 62.8% felt there was too little assistance for the poor
  • 23.1% felt there was too little welfare
• A survey about Obamacare (Hart Research, 2013) found:
  • 35% had "very negative" feelings toward Obamacare
  • 24% had "very negative" feelings toward the Affordable Care Act

Survey Format

• Survey questionnaires should have spaced questions and an uncluttered layout
• Do not use abbreviations or place more than one question on the same line because it causes confusion
• Respondents indicate answers by checking a box or circling the answer

Contingency Questions

• Not all questions are relevant for each respondent
• Contingency questions are follow-up questions for respondents with particular answers to preceding questions

Matrix Questions

• Matrix questions consist of multiple questions with the same answer categories
  • An example is questions in the Likert scale format
• Matrix questions increase comparability among items
• Matrix questions may encourage respondents to check the same answer (e.g., "agree") for all statements creating a response-set
• Reduce this problem by alternating the emphasis of successive statements with different perspectives

Order Effects

• Item order can influence responses
• An example: If terrorism questions in the US came before an open-ended question about the greatest dangers to the country, the respondent is likely to list terrorism
• Order effects can be anticipated through testing different question orderings with a pilot group i.e., pretesting

Pretesting

• Trying out a survey with a pilot group is known as pre-testing
• Researchers should always pre-test the entire questionnaire before administering it in whole or in part
• Researchers should ask respondents to comment on the questionnaire and incorporate feedback in revisions

Administering Surveys

• Survey research administration can be achieved in these ways:
  • Self-administered mail surveys
  • Face-to-face interviews
  • Telephone interviews
  • Online surveys

Self-Administered Questionnaires

• Self-administered questionnaires are usually sent to respondents by mail
• Questionnaires include a letter explaining the study and a self-addressed, stamped return envelope which in some cases may be sealed/returned without another envelope
• Response rates can be increased via follow-up mailings sent approximately 2-3 times over several weeks
• Low response rates can be a problem because the respondents not returning questionnaires may differ in important ways from those who return them

Self-Administered Questionnaires and Nonresponse Bias

• In cases of low response rate, testing for nonresponse bias should be done
  • Treating respondents after 2-3 reminders as nonrespondents may show evidence of nonresponse bias if significant differences are between early and later responders
• Researchers may incentivize responses by offering money or other compensation
• Delivering or picking up questionnaires in person improves response rates

Interviews

• Typically done face-to-face, but sometimes over the phone
• Survey interviews are structured toward standardization and comparability of responses
• Interviewers receive training to follow standardized protocols
• Protocols include probes which help respondents clarify or elaborate on an answer
  • If a respondent gives an insufficient answer the probe might be something like, "can you tell me more?"
• As with survey questions, probes should be worded neutrally to avoid biasing the response

Face-To-Face Interviews

• Face-to-face interviews commonly yield high response rates, often 80-85%
• The presence of an interviewer decreases the frequency of "I don't know" and "no answer" responses
• Interviewers clarify confusion and observe respondent demeanor, dress, tone of voice, surroundings, etc
• Face-to-face interviews are time and resource intensive, and may raise safety concerns
• The presence of an interviewer may prompt respondents to give socially desirable answers

Telephone Interviews

• Telephone interviews are more cost and time effective than face-to-face interviews
• They used to be a source of class bias, but this is no longer a problem
• Many phone surveys are conducted via Computer Assisted Telephone Interview Systems (CATI)
• Response rates tend to be lower than face-to-face interviews, due partly to telemarketing and phony-survey prevalence
• Survey interviews have higher response rates than mail interviews, but they have declined recently

Online Surveys

• Online surveys have many advantages since they are time and cost effective and have access to large populations as well as voice recognition and virtual "face-to-face" interviews
• Not everyone has internet access, or is equally likely to receive the survey or respond which can lead to nonresponse issues
• Many researchers approach a mixed-methods using online or paper-based questionnaires

Measuring Variables and Specifying Hypotheses

• Survey items allow researchers to operationalize measurement
• Items that measure attitudes, opinions, self-reported behavior, etc., can be examined for relationships between each other
  • Doing responses to variables predicting dependent variable responses

General Social Survey Measurement

• Measure items designed to measure the same construct can combined into an index
• 1996 General Social Survey questions for measuring racial prejudice:
  • If your party nominated an African-American for President, would you vote for him if he were qualified for the job? (yes = 0/no = 1)
  • Do you think there should be laws against marriages between African-Americans and whites? (yes = 0/no = 1)
  • Black-white relations opinions: White people have a right to keep blacks out of their neighborhoods if they want to, and blacks should respect that right. (0 = disagree, 1 = agree)

Racial Prejudice Index

• Analysis to see whether a level of education affects racial prejudice
• Tests can be performed to determine if increased education leads to reduced prejudice

Validity and Reliability

• "Survey research is generally weak on validity and strong on reliability." (Babbie 2021:282)
• Data scores high on reliability if an interviewer presents survey items neutrally and follows protocols for respondents
• Standard format gives superficial access to how respondents interpret questions, making it hard to see if measured attitudes, beliefs, etc, are designed appropriately

Strengths and Limitations

• Surveys are useful for describing large populations' characteristics
• With a proper probability sample, surveys generalize from sample to population
• Because questionnaires are minimally appropriate for the average respondent, peculiarities of any given respondent get missed, and interviewer abilities are limited
• Surveys rely on self-report rather than direct observation of social action
• Survey research is based on unrealistic stimulus-response theory, where researchers assume questionnaires mean the same thing to every respondent and given response although this is impossible to achieve entirely survey questions have been drafted to achieve the ideal as closely as possible.(Babbie 2021:269)