Research Methods: Validity and Surveys

Methods of Research Exam 2 Study Guide

Topic 9: Validity in Research

• Internal validity refers to a study's ability to establish a cause-and-effect relationship between variables.
• External validity is the extent to which a study's results can be generalized to other settings, populations, and times.
• Surveys typically have high external validity but lower internal validity.
• Experiments generally have high internal validity but lower external validity.
• Field research tends to have moderate levels of both external and internal validity.
• Measurement validity is ensured by maintaining consistency, accuracy, and relevance in observed behaviors.

Topic 10A & 10B: Survey Research

• Survey research involves collecting data from a sample of individuals.
• The distribution and return of surveys occur during the data collection phase.
• Survey research can study individuals, households, organizations, communities, social groups, attitudes/opinions, behaviors/practices, demographics, social trends, and policy impact.
• Units of analysis are the subjects being studied, like individuals, groups, or organizations.
• Units of observation refer to the data collected, such as responses.
• Open-ended questions elicit detailed responses, capturing unexpected insights while avoiding response bias and providing context.
• Open-ended questions are time-consuming, difficult to analyze, may have subjective interpretation, and may have irrelevant answers.
• Standardized questions are structured the same way for everyone with fixed response options
• Standardized questions provide consistency, are easy to analyze, reduce bias, improve comparability, and are faster to complete.
• Standardized questions are basic and easy to answer, while open-ended questions take more time to complete.
• Probability sampling ensures the sample accurately represents the population.
• A "parameter" is a number value that describes a characteristic of a population.
• Example of a parameter is the average income of all U.S. households.
• "Target population" is the entire group a researcher aims to study and draw conclusions about.
• Example of target population is all patients diagnosed with type 2 diabetes.
• Three methods of administering survey questionnaires: self-administered, face-to-face, and telephone.
• Self-administered surveys can be online, mail or paper.
• Face-to-face surveys occur in-person.
• Telephone surveys use random digit dialing and pre-existing lists.
• In face-to-face interviews, the interviewer should be verbal, impartial, and professional.
• Interviewing guidelines include maintaining neutrality, staying professional and respectful, following the script, listening carefully, managing the interview flow, and accurately recording responses.
• A "probe" is used to get respondents to provide more details during survey questions.
• Probing techniques involve asking who, what, where, and why.
• Random digit dialing selects a random sample of telephone numbers by generating random phone numbers based on the correct format for a region.
• The "CATI" method involves interviewers using a computer system to conduct telephone interviews, presenting questions on the screen and entering responses in real-time.
• A contingency question is asked only if a respondent's answer to a previous question meets a specific condition or criterion.
• Mail surveys are the most common type of self-administered survey.
• Face-to-face interviews are the most superior method because they yield better data quality, accurate responses, and capture nonverbal cues.
• Strengths of survey research generally are efficiency, cost-effectiveness, standardization, reliability, generalizability, versatility, data analysis, anonymity, and reduced bias.
• Weaknesses of survey research generally are limited depth, response bias, low response rates, sampling issues, interviewer effects, question wording, order effects, and time sensitivity.

Topic 11A & 11B: Sampling

• Sampling involves selecting a subset from a larger population.
• "Sampling" occurs when collecting data phase.
• Sampling is important in social research which allows researchers to gather information from a manageable subset of the population.
• Probability samples are selected so that every member of the population has a known and non-zero chance of being selected.
• Probability samples are random, and every individual in the population has a known chance of being included.
• Non-probability samples are not random, individuals do not have an equal chance of being included, and there is limited ability to generalize findings.
• A "population" is the entire group of individuals, items, or cases a researcher is interested in studying.
• A "sample" is a subset of the population selected for study.
• A "sampling frame" is a list or database of all potential members in a population from which the sample will come from.
• Population is the group to study, sample is a smaller group selected from the population, and sampling frame is the actual list from where the sample is selected.
• A representative sample reflects the characteristics of the broader population from which it is drawn.
• Bias in a sampling frame occurs when the list of potential sample members is not a perfect reflection of the population.
• The extent to which a sample will be representative depends on the sampling method used and how well the sampling frame reflects the population.
• A random or probability sampling method improves the likelihood of obtaining a representative sample.
• Probability sampling ensures that each member of the population has a known and equal chance of being selected for the sample.
• "Sample distribution" is the distribution of a statistic across many possible samples drawn from the same population.
• The standard error measures how much a sample statistic is likely to vary from the true population parameter.
• A lower standard error indicates the sample statistic is likely to be a more accurate reflection of the population parameter, which improves the generalizability of the sample's results.
• A higher standard error suggests more variability between sample statistics, making generalizations less reliable.
• A smaller standard error indicates that the sample mean is close to the true population mean, implying more precise estimates.
• As the sample size increases, the standard error decreases because larger samples tend to provide more accurate estimates of the population parameters.
• Probability samples ensure that each member of the population has a known and non-zero chance of being selected.
• Simple Random Sampling (SRS) gives every member of the population an equal chance of being selected.
• Stratified Random Sampling divides the population into subgroups and takes samples from each subgroup.
• Cluster Sampling divides the population into clusters and chooses a random selection of clusters.
• Systematic Sampling selects every nth individual from a list of the population.
• Simple Random Sampling's strength is complete randomness, while its weakness is inefficiency with large, dispersed populations.
• Stratified Random Sampling ensures representation from all subgroups, but it can be complex and requires subgroup knowledge.
• Cluster Sampling is cost-effective for large populations, but can introduce sampling error if clusters are not homogeneous.
• Systematic Sampling is easy to implement, but can introduce bias if there's a pattern in the list.
• Cluster sampling is used when the population is geographically spread out and it is impractical to sample the entire population.
• Convenience sampling selects individuals who are easiest to reach.
• Judgment Sampling uses the researcher's judgment to select individuals who are representative of the population.
• Quota Sampling is similar to stratified sampling, but the researcher non-randomly selects individuals to meet quotas.
• Snowball Sampling is used when studying hard-to-reach populations, where one participant refers others.

Topic 12: Data Analysis

• "Data analysis" represents step 4 of the wheel of science.
• Univariate analysis examines a single variable at a time.
• Bivariate analysis examines the relationship between two variables.
• Multivariate analysis examines the relationship between three or more variables simultaneously.
• Descriptive analysis summarizes and describes data without inferring relationships.
• Explanatory analysis seeks to explain relationships and causal effects between variables.
• Basic statistics used in univariate analysis include measures of central tendency, measures of dispersion, frequency distribution, skewness & kurtosis
• Statistical tests to test a hypothesis are chi-square test and t-test
• The Chi-Square Test is used when the dependent variable is categorical (nominal or ordinal).
• The T-Test is used when the dependent variable is continuous (interval or ratio).
• Bivariate statistics are considered weak for Explanatory Research as it examines the relationship between two variables at a time, which does not account for other potential influencing factors. It cannot establish causality since other variables might be affecting the relationship.
• Multivariate statistics examines multiple variables at once, allowing researchers to control for confounding variables and better isolate causal effects.
• A standardized regression coefficient shows the strength and direction of a variable's effect on the dependent variable while accounting for differences in measurement scales.
• A small effect is present when ẞ ≈ 0.1
• A moderate effect is present when β≈ 0.3
• A large effect is present when ẞ ≈ 0.5 or higher
• Positive ẞ means independent variable increases, and the dependent variable also increases.
• Negative ẞ means independent variable increases, the dependent variable decreases.

Topic 13: Experiments

• Experiments are well-suited for studies aiming to establish causal relationships between variables.
• The researcher controls of an independent variable (IV) to observe its effect on the Dependent Variable (DV)..
• Participants are randomly assigned to treatment or control groups to reduce bias.
• A control group does not receive the treatment, allowing for comparison with the experimental group.
• Pretesting & Post testing includes measuring outcomes before and after the intervention to track changes to confirm whether the observed effects result from the experimental treatment.
• Hawthorne effects occur when participants change their behavior simply because they are aware they are being observed, not because of the actual experimental treatment.
• Randomization assigns participants randomly to groups to ensure equal distribution of characteristics.
• Matching assigns participants based on similar characteristics to ensure comparability between groups.
• X represents a treatment or intervention (independent variable).
• O represents observation or measurement (dependent variable).
• P represents population or participants in the study.
• R represents randomization (random assignment to groups).
• Internal Validity is the extent to which an experiment accurately measures the causal relationship between the IV and DV.
• External Validity is the extent to which the findings generalize to other settings, populations, or time periods.
• Selection bias, history effects, maturation, testing effects, instrumentation, and experimental mortality (attrition) threatens internal validity in experiments.
• A major disadvantage of experimental methods is that they have limited external validity, ethical concerns, and high cost & time.
• Experiments tend to have low external validity because they are conducted in artificial settings with strict controls.

Topic 14: Field Research

• Field research is a qualitative research method that involves direct observation of people in their natural environments.
• Field research start with observations and then develop theories based on patterns they identify.
• Modes of observation: Complete Observer, Participant Observer, Covert Observer, Overt Observer
• The types of field research observation modalities differ each other by the level of researcher involvement and the structure of data collection.
• Complete observer means the researcher only observes without participating
• Participant observer means the researcher observes while interacting with subjects
• Covert observer means the researcher hides their identity while observing
• Overt observer means the researcher's identity is known to participants
• Complete observer strength is that it is unobtrusive, no influence on behavior and the weakness is limited depth of understanding
• Participant Observer strength is deeper insight and builds trust and the weakness may influence behavior, researcher bias
• Covert Observer strength avoids observer effect and weakness ethical concerns, risk of discovery
• Overt Observer strengths displays ethical transparency and weakness that subjects may alter behaviors (hawthorne effect)
• Observations are recorded in detailed notes, audio/voice recordings and journals or memos
• Focus on expected behaviors, letting personal beliefs influence data collection, forgetting or misinterpreting To observe and record you must be sensitive that observed events and subject may behave differently when they know their being observed
• You should process and analyze the data from field research by coding, categorization, triangulation and narrative analysis
• Field Researchers supplement their observations by interviews, document analysis and surveys
• Strengths of field research is high validity real world behaviors studied in context, can uncover hidden social processes, flexible, generates rich detailed data
• Weaknesses of field research is time consuming, ethical concerns, difficult to replicate results and potential for observer bias

Topic 15: Evaluation Research

• Evaluation research seeks for the definition, purpose and key focus of a study
• Quasi-experimental designs used in evaluation research include nonequivalent control group design, before and after design, interpreted time series design, and multiple times series design.
• A long-time series is valuable in evaluation research as it identifies long-term trends instead of short-term fluctuations. Helps distinguish real effects from random variation and reduces the risk of misleading conclusions.
• The use of multiple times series improves confidence that a given intervention had its intended effects as it compares trends across groups, reduces bias, helps rule out external factors, and strengthens causal inference.
• Using control groups to compare treated vs. untreated cases would increase confidence with quasi- experimental designs.

Topic 16: Critical Consumption of Research

• To be a critical consumer of research one must consider weather the research design is appropriate, is it randomized, observational, cross-sectional etc, is the sample size big enough, was it randomly selected, etc
• When reading social research pay close attention to issues of theory construction and methodology which include clarity of concepts, logical consistency, empirical support, scope & generalizability, causal explanation vs correlation