Research Methods Notes PDF

1- Scientific study of behaviour: Ways of acquiring knowledge - Intuition o Anecdotes, personal experiences, opinions o Understanding decisions based on yours or someone else’s experience, because it feels right emotionally - Authority o Tr...

1- Scientific study of behaviour: Ways of acquiring knowledge - Intuition o Anecdotes, personal experiences, opinions o Understanding decisions based on yours or someone else’s experience, because it feels right emotionally - Authority o Trust in experts or those of power o Trust them since they know more than we do, its easier to trust them then becoming an expert ourselves - Intuition and authority o Do not involve critical thinking o Accepting knowledge soley is susceptible to bias o Common and attractive o Can make incorrect or harmful information - Scientific method o Do not accept authority or intuition without evidence o Scientific skepticism: questioning truth of information and seeking out evidence, used to scrutinize assumptions made and gather data about those assumptions o Empiricism: gaining knowledge based on structured, systematic observations, structural and specific ways - Why use the scientific method: o Objective, systematic way to collect information, evaluate and report, reduces biases. o Poses empirical questions that can be answered by structured observation o Falsifiable: can be shown to be false or can be refuted, fate is not falsifiable, makes it easier to make assertions of reality Goals of psych research - Describe behaviour o What happens, when, for who, how often, how many, what are the characteristics of those who…… o Most common goal - Predicting behaviour  o When will a behaviour occur/not occur o Under what circumstances - Determine causes of behaviour o Related to prediction and relies on: ▪ Covariation of cause and effect When cause is present the behaviour occurs and when its not present it doesn’t occur ▪ Temporal precedence Causal factors occur before the behaviour ▪ No other alternatives Rule out other possible explanations - Explain behaviour o Why does a behaviour occur or change - One question set that doesn’t neatly connect to the other goals o Checking for validity and biases in the data collected o How confident are we in the responses o How good is the evidence o Psychology focuses on intangible phenomenon's, never definitive answers Basic vs applied research - Basic: answer fundamental questions of the nature of behaviour. Goal increasing understanding of behaviour in that area o Niche, specific, purpose is unclear - Applied: addresses practical real world problems aiming to find solutions, ex: comparing models of teaching and learning methods o Practical, answer specific questions 2 – Research starting points How to come up with research design - Assumptions: provide evidence for or against common beliefs - Observations: study what you know, personal intrigues, me search - Practical problems: raise awareness, understand, solve issues - Theories: organize information about a given concept, organize and explain information that exists and help generate new knowledge  - Past research: find gaps and new avenues from past research How to find past research - Library database: specific, narrow search, exact topics, many options to filter – best - Google scholar: less filters, specific for research, cannot limit to peer review, good starting points - mid - Internet wiki: general information and terminology, a lot of effort to evaluate the quality of information, not always empirical – worst Good hypothesis: - Hypothesis: statement about a phenomenon that may or may not be true, requires future evidence to support or refute it, based on theory or past findings - Exploratory hypothesis: insufficient past research or theories to develop clear hypotheses, should at least have a logical explanation of anticipated results 3 – scientific study of behaviour Steps investigating human behaviour: - Ask questions, identify what is known, generate hypothesis, what did you see - Collect data - Analyze data - Interpret results, write a manuscript to share findings - Peer review, publication, dissemination 4 – research proposal Research proposal: - Identify novel research questions/ hypotheses o Reviews and synthesizes existing research to identify gaps in knowledge o Justifies why research question is worth answering or testing - Proposes a study to test the hypothesis o Methods, participant, sampling, procedure, measure, design - Similar to introduction and methods of empirical articles 5 / 6 together – research fundamentals Variables - A variable is any event, situation, behaviour or characteristic that can vary in some way - Variable of interest: any variable in which the researcher is particularly interested - Types of variables o Situational: features of the event, environment participant is exposed to ▪ Ex: color of wall, noise level, dosage o Response – directly elicited by researcher ▪ Participant reactions to event or experience ▪ Ex: amount of time to press a button, change of heart rate due to stimuli o Participant: pre existing characteristics of participant ▪ Ex: gender, ethnicity, age - Operational definitions: specifies operations, techniques that will be used in a particular study to represent that variable. One variable can have multiple. Must include the measurement, predictions and research designs. Basic research designs Nonexperimental: - research design focus on determining the relationship among variables of interest, explore relationships - Aka correlational methods - Variables are measured and observed - Essential question: do the variables of interest change together in a meaningful way? - Observe do not intervene - Linear, curve linearly, unrelated, non linear - Advantages: o Establish trends over large amount of data o Describing behaviour o Can be used to predict future behaviours o Necessary due to ethical issues – observing - Disadvantage o Direction of causal influence – which variable causes the change  o Third variable – a unmeasured variable is responsible for the changes Experimental: - try to determine causal influences among variables of interest - at least one variable is manipulated, one variable is measured or observed - Independent variable – variable that causes in behaviour, manipulated or controlled by researcher o Different levels of IV – different experimental conditions o Ex: treatment or no treatment - Dependent variable – variable that is affected by IV changes, DV depend on the changes, measured by researcher o Ex: therapeutic approaches to treating a mental disorder like depression, different therapies of IV, DV depressive symptoms after therapy The choice depends on the hypothesis, operational definitions, and the conclusions you wish to make. Advantages: - Answer questions about causes of behaviour - High internal validity - More experimental control Disadvantage: - Lack of external validity: can conclusions apply outside the context of the experiment? - Sometimes ethically impossible - impossible to randomly assign participants to various conditions -  How to allow for causal influences? - Temporal precedence: IV comes before DV - IV and DV covary: changes in one accompanied by change in other - Elimination of alternative explanations: remove influence of confounding variables o Experimental control o Random assignment to IV - When ALL THREE CRITERIA ARE MET THERE IS HIGH INTERNAL VALIDITY o Internal validity is the degree to which experiment allows for causal conclusions - Cofounding variable can influence changes in IV and DV - Experimental control - participants all treated the same, only difference is IV manipulation - Individual differences - random assignment of IV, which level they belong to reduce influence of individual differences - Ex: o Experiment: which teaching method is best? o Experimental control - everything about the class is the same other than the IV so the prof is same, content same, modality is same, etc o random assignment - some less and more dedicated students, choosing may be a alternative reason for the conclusions, so we should equally distribute students and don't let them choose 7 /8 together – sampling and measure Population vs sample - Population o Entire group of people of interest, every member o Ex: all 42,000 students at Uottawa - Sample o Smaller group of people who complete study procedures, group from the population o Ex: 200 students from Uottawa - Representative samples o Samples should be similar to population so conclusions are generalizable o More likely with larger samples o Depends on how sample is selected – sample techniques Sampling techniques PROBABILITY Probability: Determine the probability than any member of the population will be selected for the sample - Advantages of probability sample: High representativeness - Disadvantages of probability sample: Resource intensive Difficult to get a list of everyone in the population or to access and identify certain subgroups Cluster: Population divided into groups "clusters", randomly select which cluster to sample Sample techniques NON PROBABILITY  - Advantage of non probability Inexpensive Less time Convenient - Disadvantage of non probability Trade convivence and cost for the possibility of having a less representative sample and less generalizable Measurements - Self reports: individuals explicitly asked to respond on their thoughts, attitudes, beliefs – questionnaire, survey, interview, focus groups o Ex: answers on a scale assessing fear of public speaking o Advantage: ▪ Cheap ▪ Easy to administer o Disadvantage ▪ Rely and assume participants respond accurately and honestly - Behavioural measures: direct observation of behaviour – facial expression coded, counting behaviour occurrences, test performance, reaction time o Ex: number of fearful behaviours observed during public speaking  o Advantage ▪ Behaviours the participant sometimes is unaware of doing o Disadvantage ▪ Costly and time consuming ▪ Sometimes subjectivity within researchers - Physiological measures: recording body responses to stimuli – heart rate, blood pressure, galvanic skin response, EEG, fMRI, blood analysis, saliva analysis o Ex: recording blood pressure and cortisol levels during public speaking event o Advantage: ▪ Not subjective as the others ▪ Observe experiences outside a individual conscious awareness o Disadvantage ▪ Very expensive or special techniques and data analytical assessments. ▪ different physiological responses can mean different things Ex: anxiety and excitement are similar responses How to choose a measure: - Nature of operational definitions and research hypotheses - Cost – monetary and practical - Quality – reliability, validity, reactivity o Reliability – degree to which a measure consistently produces scores that can be reproduced under the same conditions ▪ High reliability – more stable the measure ▪ Internal consistency – how consistent is the measure across items measuring the same concept, relevant to self reports. Cronbach’s alpha – average correlation between items of a measure ▪ Test- retest reliability – how consistent is the measure across time points, relevant to all 3 measures. Test retest correlation - correlation between scores on measure at time 1 and time 2 ▪ Inter rater reliability – how consistent is the measure across different raters, relevant to behavioural observations. Cohen kappa/ interclass correlation – extent of agreement between raters If raters see same ideas then its interrater reliability is high o Validity/ construct validity – degree to which a measure correctly measures variable or behaviour (construct) of interest  High construct validity – measure accurately measures the behaviour/ ▪ construct it is designed to measure ▪ It may not appear to measure the construct on the surface, may not measure all aspects of a variable, does not seem to be related to other well established measures for the same construct o Reactivity – degree to which participants behaviours and responses change as a result of measuring it ▪ Non reactive: a measure that does not greatly alter participants responses or behaviour ▪ Self report – social desirability ▪ Behaviour – knowing your observed causes you to behave different ▪ Physiological – knowing your blood pressure is measured may alter 9/10 together – ethics Research ethics: - Analyzing ethical and legal questions pertaining conducting research, ensure interest and well being of participants is protected, this analysis needs to be done before research starts, government and institutional review board provide guiding framework but ensuring ethical practices is the researchers responsibility. Respect for person - Informed consent: - Fully informed above all aspect of the study that may influence their decision to partake o Including: purpose, procedures, risk, benefits, confidentiality, voluntary nature, rights to refuse or terminate participant no penalty. - Exceptions o Purely observational research o Special populations – assent (verbal approval from child and legal guardian approves) ▪ Ex: children, minors o Research requiring deception ▪ Participants are mislead or not told ▪ When – if researcher believes the participants behaviour or response would change if they knew the real goal ▪ After research participants are debriefed as to the deception and the true research Concern for welfares – risk benefit analysis - Potential benefits of participating are greater than the risks for the participant - Benefits: o Participants ▪ Direct – learning skills, treatments, rewards ▪ Indirect – satisfaction, helping others in the future o Society ▪ Knowledge gained may benefit others ▪ Cost of not doing the research ex: disease studies - Risks o Physical harm – medications o Psychological stress – sensitive topics o Losing privacy or confidentiality ▪ Confidentiality – data kept private and shared only in research team, password protected ▪ Anonymity – identity of participants kept secret, deidentify participants with codes Justice – promoting equity - All individuals have fair access to benefits of research, all same level of risks - Cannot target particular group to study without the research directly benefiting and involving that group o Lacking justice – example Tuscany syphilis study - Inclusion/ exclusion criteria must be justified - Criteria should maximize both scientific validity and potential bnefits to individuals and society WEIRD – western, educated, industrialized, rich, demographic Institutional research ethic boards REB - Scientists, non scientists, legal expert who review proposed studies and deem them ethical or not - Iterative – back and forth with researcher and REB professional ethics: - Researchers have additional responsibilities to avoid scientific misconduct o Data fraud: fabricating data or altering values to influence outcome o Unethical data analysis: misreporting or changing results of statistical test to change outcome - Pubic reform: change to improve transparency and accountability within research process o 1 disclosure of research methods and design o 2 preregistration o 3 open data and material 11/12 together – observational methods Quantitative vs qualitative approaches - Quantitative: o Research approaches that result in numeric data and statistically analyzed o Ex: counting the amount of times a behaviour occurs o More in psychology research o Using numbers and scores to make conclusions o Frequency and intensity - Qualitative o Research approaches that result in not numeric data that are analyzed for meaning, themes, patterns o Ex: test based material - Mixed o Qualitative and quantitative approaches o Used to inform one another Passive: videos or observations Non passive: images Naturalistic observations: - Research method involved the researcher making systematic observations in natural settings, broad range of behaviours are of interest - Can be qualitative – descriptions - Can be quantitative – counting amount of times something occurs - Two methods o Participant observations: researcher joins community they observe, can become slightly bias o Non participant observations: researcher does not actively participate or their presence is hidden, less reactivity, can have ethical issues from concealing information about the research and that they are watched, time consuming, less control - Advantage o High external validity o Rich information o Sometimes only approach able to use  - Disadvantage o Lack control of variables o Many confounding variables o Time consuming o Observer bias o Increased reactivity Structured observation - Research method which researcher observes participants in a controlled setting, typically a lab, behaviours of interest are coded using a coding scheme – a set of rules that defined what behaviours to identify and how, track the frequency of them, the level and degree of behaviour, range of simple to complex. - Behavioural coding: raw observations and assigning meaning to it to analyze it. - Data is usually quantitative derived from coding schemes - Inter rater reliability – eneed to ensure high over 80% agreement between coders of behaviour - Advantages o Good external validity o Some experimental control - Disadvantage o Observer or coder bias o Increase reactivity Case studies - Research method which a researcher makes a detailed observation of behaviour and other factors from a single individual - Data tends to be qualitative - Common for studying rare, low frequency phenomena - Advantages o Rich detailed information o Used when other methods are impossible or unethical - Disadvantages o Low external validity o Observer biases Strategies for proposing a new research study - Combining two different variables together - Studying a variable from a different angle o Group: could be a certain SES, age, gender, ethical group o Condition: certain situation or circumstances - Studying a variable in a different way  o Limitation could be measurement quality issues, incomplete operational definitions 13/14 - self report Divided into questionnaire and interview - Common across these methods is that data are collected by asking participants to describe their own behaviours, attitudes, views - Trusting participants to share their views - Can be experimental or non experimental Questionnaire - Set of questions answered by participants regarding their attitudes or behaviours - Can be administered paper or online - They are convenient way to gather a lot of information - Large samples possible - Many different variables can be measured - Can be open or closed questions o Close – limited choice of responses o Open – any response possible o Quantitative usually close o Qualitative usually open - Issues o Relying on participants honest and meaningful responses o Social desirability biases – respond in a way that they are likely to be viewed favorably – more for sensitive topics o Fraudulent – carelessness o Misunderstanding questions o Using existing or creating a questionnaire ▪ Using good working, choosing appropriate responses ▪ Establishing reliability and validity - Advantages o Affordable and efficient o Large sample size o Anonymity o Flexibility - Disadvantages o Risk of collecting biased or untruthful data Interview research - Interviews are data collections where research asks participants questions orally - Interviewer bias - any influence the interviewer has over the participants responses o Subtle reinforcements of certain responses – smile or frown from the interviewer, judgement can influence response o Social desirability – looking good o Personal characteristics – characteristics of the interviewer can influence responses o Experimenter expectancy effect – probing subtle responses nonverbally - Advantages: o Rich information o Can confirm participant understanding o Can detect careless results - Disadvantages o Time and resource intensive o Interviewer bias o Challenge of consistency  Picking naturalistic over structured Ethical reasons: ex not creating harm, bullying, etc More natural behaviours no observer bias Cheaper and less time consuming Structured over naturalistic Easier to make cause and effect relations Manipulate the variables to study what you really want to study Cofounding variable impacts could be standardized Rare behaviours 15/16- basic experimental design Basics of experimental design: - experimental design aim to determine causal influences o IV: manipulated, causes change in behaviour o DV: measured variable that changes by the IV - Strong experimental design rules out alternative explanations therefore high interal validity o Confounding variable – variable connected to the IV that could explain DV changes - Minimum of one IV Research Design Experimental design divided in two parts – both try to determine causal influences of IV, and reduce confounding variable impacts IV must be operationalized to have at least two levels or conditions - Between subject o Different participants assigned to each level or condition of IV o o This diagram is only when studying 1 IV at 2 level o Starts as a whole group then divided into 2 groups by random assignment to reduce preexisting differences o Each group gets different levels o DV measured in each group and compared, any differences assumed to be because of IV o Main disadvantage of between: ▪ That the two groups have pre-existing differences that result in a difference in the DV instead of the IV influencing the DV o Ways to ensure groups are similar ▪ Random assignment Randomly select participants Reduce risk of systematic differences ▪ Pretests DV measured before and after IV Compare pretest scores across groups and ensure no preexisting differences ▪ Match pairs participants are paired up based on similar characteristics, each person in the pair is assigned to a different group or treatment condition. This design is used to reduce variability between groups and make comparisons more accurate o advantages ▪ allows for causal inference – advantage over non experimental ▪ useful when exposure to multiple levels of IV is not realistic or order effects cannot be eliminated o disadvantages ▪ selection differences could confound results- fixed with random assignment, pretest, match pair ▪ more participants needed  - Within subject o Same group of participants assigned to each level or condition of IV o o One group gets all IV levels, reduces selection bias, identical identification of participants, post test only, compare 2 DV measures o Internal validity of within subject is threatened by issues resulting from repeated measuring of same DV o Order effects - Order of levels or conditions influence DV ▪ Practice effect: preform better cause they practiced the tasks due to a repeated experience ▪ Fatigue effect: preform poorer over time cause they are tired or bored ▪ Contrast effect: responses follow a later condition are impacted by experience in earlier conditions o Order effects reduced by counterbalancing IV levels or conditions ▪ Counterbalancing: Presenting all combinations Still get every level of IV, same groups in two orders Still using within subject design but split group in half and change the order of exposures and compare to see if there is a difference to counter a order effect o Advantages ▪ Allows for causal inference – advantage over non experimental ▪ Fewer participants ▪ Individual differences are not issues ▪ Can detect very small differences o Disadvantages ▪ Order effects – fixed with counterbalance ▪ Participants may guess the hypothesis and change behaviour ▪ Exposure to multiple levels of IV may not be realistic Which experimental design should I use? - Resources o Can you recruit a large sample for between subject o Funds, time, staff - External validity o How would people experience IV in real life? o Could they experience all levels themselves or only one? 17/18 – Quasi - Quasi experimental design: resemble experimental back lack important features that allow for causal influences. Lower internal validity than experimental designs. - Advantages of quasi: o Better internal validity than other NON experimental designs o Sometimes the only way to study a topic - Disadvantages of quasi o Poor internal validity that TRUE experimental designs o A lot of alternative explanations - Why use quasi? o Control and comparison group or random assignment is not possible o Control and comparison group or random assignment is unethical o Can have higher external validity than experimental designs, less control shows more mimics of the real world o Goal is to describe a behaviour - Treats to internal validity o History – historical events affect participants (provides alternative explanations for change between pre and post test) o Maturation – natural changes to participants short term or long term (alternative explanations between pre and post) o Testing – taking pretest influences peoples response to post test (alternative explanations between pre and post) o Regression to mean – extreme scores tend to become less extreme on repeated measures (alternative explanations between pre and post) o Attrition – participants leave before study finishes (creating group differences) o Selection effects – groups are divided for any reason other than random assignment (alternative explanations between pre and post) o Cohort effect -groups are divided by age (alternative explanations between pre and post) - Major categories of quasi: o One group designs: no comparison group, IV has one level or condition. BOTH BETWEEN SUBJECT ▪ One group posttest only design. Group – IV – DV Only one group experiencing one level of a IV then measure DV No comparison group Weak: only descriptive data after the approach, no comparing, not possible to compare knowledge after the IV. ▪ One group pretest and post-test design Group – DV scores – IV- DV Stronger study with the pretest/ baseline Providing more information to researcher if the IV actually affected the DV No causal relationship No manipulation of IV Stronger than only post-test since there is some conclusion that can be made ▪ Major disadvantage: without a comparison or condition it is difficult for the researcher to rule out alternative explanations: Fix this by adding a comparison group o Non equivalent group: comparison group but participants are not randomly assigned to IV level or conditions to create the groups ▪ Non equivalent control group Natural groups There is two or more distinct condition or group, groups are not randomly assigned groups they are grouped based on already existing or natural groups, groups differ on a characteristic (time of class, intelligence, size of class) Two different sections being IV, two level conditions of IV Increase of internal validity: two DV scores that can be compared to determine the significance of the IV.  ▪ Non equivalent control group pretest and post-test Improve internal validity due to the pretest Issue of other inherent unmeasurable variables Cannot rule other variables out, rule out by random assignment of groups 19/20 – complex experiment Basic to complex - Basic experiments can be modified to make it complex o A IV have more than 2 levels or conditions o More than 1 IV each with 2 or more levels or conditions Between with 1 IV and 3 conditions Within with 1 IV and more than 2 levels Factorial research designs – 2 or more IV, each with 2 or more levels, each IV is called a factor - # of levels IV 1 by # levels IV 2 - Ex: 3x2 = 3 levels for IV 1 and 2 levels for IV 2 - Main effects: effect of each individual IV on DV o Amount of IV=amount of main effects o What food do people enjoy more? Do people favor condiments? o - Interaction effects: combined effect of the two IV on the DV (IT DEPENDS) o When the IV doesn't have the same effect on the DV at all levels of the other IV, when IV depends on another level of the other IV o Combination questions? What combination do people prefer o Factorial research can be - Fully between – all between subject IV - Fully within – all within subject IV - Mixed – at least 1 between, and 1 within Must consider - Research question, hypothesis, operational definitions, resources, parsimony (want simple ad useful explanations) Interactions: 21/22 – quantitative data analysis - Research studied collect data from a sample of participants. - Quantitative data are numerical scores or values representing participants responses or behaviours - Researchers analyze data to o Determine trends o Test hypotheses and generalize results - Two types of statistics o Descriptive ▪ Summarize, organize, and simplify ▪ Visual representation or numerical ▪ Measures of central tendency: describe what is typical value that best represents the variable. Ex: mean, median, mode ▪ Measures variability: describe how far apart values are from one another in the data, do the participants in the sample respond similar or different on your measure. Ex: range, standard deviation ▪ Frequencies: describe how many participants in each category of the variable (nominal or ordinal) or how many participants received each score for a variable (interval or ratio)  o Inferential ▪ Test hypotheses ▪ Allow researchers to draw conclusions about their hypotheses ▪ Use sample data to generalize to population ▪ Anything more than describing o Researchers can use a combination of both, depending on study goals - How to choose a statistical analysis o What is the goal of the study ▪ Describing behaviours use descriptive to organise data ▪ Make inferences about differences or relationships Descriptive – examine the data overall Inferential – hypothesized differences or relationships o What scale of measurements was used to measure variables ▪ Nominal Observations classified into named categories with no rank Categories differ in quality not quantity Ex: gender, fav color, political party you voted for ▪ Ordinal Observations classified into ranked categories with non uniform differences between ranked categories Spacing in scale are not equal Ex: low, middle, high, 1st born, 2nd born. 3rd born. ▪ Interval Observations are assigned values on a continuum separated by same sized intervals No real zero, can be negative Ex: Temperature C or F, IQ, Likert Scale ▪ Ratio Observations are assigned values on a continuum separated by same sized intervals Data are numerical scores Zero means an absence of it Ex: # of test questions answered correctly, time, $in bank account, weight, height ▪ Likert scale: technically ordinal because it is not universally spaced, used as a interval for convenience sake o Are you comparing across groups and conditions or examining relationships ▪ Experimental and quasi experiments research typically make comparisons across groups T-tests, ANOVA ▪ Non experimental research look at relationships among variables Chi square, loglinear, correlation, regression o How often did you measure each participant  ▪ Aka do you have a independent measures data or repeated/dependent measures data ▪ Independent measures: each individual participant provides only one score per variable – between subjects ▪ repeated/dependent measures: each individual participant provides more than one score per variable – within subjects - steps of analyzing data o make a data analytic plan – during design o collect data o check and clean data – remove incomplete data o data reduction – scale scores o examine the data – calculate descriptive statistics o test hypotheses – calculate inferential statistics o interpret statistical results and share findings 23/24 – generalize findings - researchers typically collect data from a sample in order to generalize their research to population - representative sample = more generalizable research o sample should be similar to the population to draw generalizable conclusions ▪ generalizable: results can be extended beyond the specific sample or study where they we obtained - but is generalizable enough? How broadly can research results generalize? o Can results generalize beyond immediate sample o Can results generalize beyond specific study design and procedure o How well - Generalize beyond the sample o Do the results generalize beyond who the research participants were? ▪ Reliance on volunteers Selection bias – those who volunteer to participate may be different from those who choose not to participate o Volunteers have more motivation, education, greater need for approval Self selection bias – those who choose to participate may have more personal interest in the topic, influencing their behaviours ▪ Issues of generalizing across gender identities Over representation of cis women in psychology Under representation of trans and non binary identities Lack of research leads to incomplete and incorrect findings ▪ Issues of generalizing across cultures, ethnicities, racial groups Over reliance on western north American white research  Improving gradually in recent years o Western samples have become more diverse o More research happening outside western contexts More diverse more generalizable o Do the results generalize beyond what the research participants did? ▪ Do other researchers groups obtain similar results Experimenter characteristics can influence research o Researcher bias – researcher behaves may influence behaviours of participants o Observer expectancy bias – reinforces to get certain responses Generalizability improves as more researchers yield consistent results ▪ do the findings hold outside a structured lab procedure structured, controlled lab experiments are often designed to measure (controlling) behaviours that also occur in real life o higher internal validity of experiments might mean lower external validity o do the same results occur in a less controlled setting field experiments on a similar topic can be conducted to confirm o meta analytical findings show most lab and field experiments yield similar results but some found opposite conclusions if structured and natural get the same results they are trustworthy ▪ replication conducting another study with the same hypothesis, variables, methods and different sample to increase confidence in results direct – study is reconducted using exact same measures and procedure to ensure the first wasn’t a fluke conceptual replication – study is reconducted with same hypothesis and variables but different measures or procedures either type could be completed in different type of sample to see if results are generalizable beyond original population what if a research fails to replicate o if one doesn’t work mother others will look at a set not just one replication ▪ meta analysis can statistically analyze how well researchers generalize across multiple studies using a meta analysis  meta analysis – statistical technique that analyze the average effect size a large set of existing research studies on a given topic o effect size: the magnitude of difference between groups or the strength of relationships between variables ▪ effect size = r stronger closer to 1 neg or pos o pools effect size across all studies selected to give average effect size o can evaluate overall generalizability of research results from a large body of research o compile results in a average to determine its presence asses large body of information regardless of method some over some under looked look for unpublished data Important Diagrams: