Doc1.docx PDF

## 8.2 Overarching Concepts: Sampling, Credibility, Generalizability and Bias **The Concepts** Four overarching concepts are used to describe a research study and make a judgment about its quality. The concepts are applicable to all research methods, but they can be approached very differently by...

## 8.2 Overarching Concepts: Sampling, Credibility, Generalizability and Bias **The Concepts** Four overarching concepts are used to describe a research study and make a judgment about its quality. The concepts are applicable to all research methods, but they can be approached very differently by qualitative and quantitative researchers. Sometimes different sets of terms may be used in each approach to refer to the same concept. For example, the idea of "credibility" may be referred to as "internal validity" in experimental research and "trustworthiness" in qualitative studies. For this reason, it is important to clearly understand both the meaning of the four overarching concepts and the way they are manifested in experimental, correlational and qualitative research. * **Sampling** * A sample is the group of individuals taking part in the research study. * Sampling is the process of recruiting these individuals for participation. * **Credibility** * Credibility is the extent to which results of the study can be trusted to reflect the reality. A study is credible when we have reasons to believe that its findings are true. * **Bias** * Bias is the flipside of credibility. It characterizes various distortions introduced to the findings by the researcher, the research procedure, mistakes in the process of measurement, unnatural behaviour of participants, and so on. * **Generalizability** * Generalizability is the extent to which the results of the study can be applied beyond the sample and the settings used in the study itself. The way qualitative and quantitative researchers approach generalizability is distinctly different. **Table 8.2 Overarching Concepts** | Overarching Concepts | Experimental Studies | Quantitative Research (Correlational Studies) | Qualitative Research | |---|---|---|---| | **Sampling** | - Random Sampling - Stratified Sampling - Self-Selected Sampling - Opportunity Sampling | Same as Experimental Studies | - Quota Sampling - Purposive Sampling - Theoretical Sampling - Snowball Sampling - Convenience Sampling | | **Generalizability** | - External Validity - Ecological Validity - Population Validity - Construct Validity | - Population Validity- Construct Validity | - Sample-to-Population Generalization - Case-to-Case Generalization - Theoretical Generalization | | **Credibility** | - Internal Validity | Credibility | - Credibility / Trustworthiness | | **Bias** | - Threats to Internal Validity: -- Selection -- History -- Maturation -- Testing Effect -- Instrumentation -- Regression to the Mean -- Experimental Mortality -- Experimenter Bias -- Demand Characteristics | - On the level of measurement of variables, depends on the method of measurement - On the level of interpretation of findings: -- Curvilinear relationships -- The third variable problem -- Spurious correlations | - Participant Bias: -- Acquiescence -- Social Desirability -- Dominant Respondent -- Sensitivity - Researcher Bias: -- Confirmation Bias -- Leading Question Bias -- Question Order Bias -- Sampling Bias -- Biased Reporting | ## 8.3 The Experiment **Constructs and Variables** * Quantitative research operates with variables. A variable is any characteristic that is objectively registered and quantified. * In relation to this, it is important to make the distinction between constructs and operationalizations. * A construct is any theoretically defined variable, for example, violence, attraction, memory, anxiety. * Constructs need to be operationalized. * Operationalizing a construct means expressing it in terms of observable behaviour. **Variables in the Experiment** * The simplest experiment includes one independent variable (IV) and one dependent variable (DV), while the other potentially important variables are controlled. * The independent variable is the variable that is manipulated by the experimenter. * The dependent variable changes as a result of this manipulation. * We need to ensure that it is the change in the IV that causes the change in the DV, and for this reason, we need to control the potential influences of other variables. These other variables that can interffere in the relationship between the IV and the DV are called confounding variables. **Table 8.3** * **Manipulate the IV:** Ensure that any differences between the two groups can be attributed to the IV, (and nothing else), researchers ensure that potential confounding variables are either eliminated or kept constant in both groups. * **Control Confounding Variables:** **One confounding variable that may affect the results is participants' familiarity with the words.** Researchers eliminate it by ensuring that all participants are perfectly familiar with all words: they exclude participants whose native language is not English and they use words commonly used in everyday language. **Another variable that may affect the results is background noise in the room.** One cannot eliminate it entirely, but researchers make sure that both groups are tested in the same room and at the same time of the day. If noise is the same in both groups, it will affect them equally, which will not compromise the comparison. * **Measure the DV:** After hearing the list of words, participants are required to recall it in the right order, and the researcher measures how many words they recalled correctly. * **Analyze the Results:** Compare the DV in the two groups. If the number of words correctly recalled in the long-word group is significantly smaller than the number of words correctly recalled in the short-word group, the experimental hypothesis will be supported. **Figure 8.1 A Simple Experiment** * **Independent variable (IV)** * **Dependent variable (DV)** * **Confounding variables** **Sampling in Quantitative Research** * The target population is the group of people to which the findings of the study are expected to be generalized. * The sample is the group of people that take part in the experiment. It is a sub-set of the target population. * We need to be sure that results of quantitative research can be generalized from the sample to the target population. For this to be possible, the sample must be representative of the target population. * A sample is said to be representative if it reflects all of the essential characteristics of the target population. **Table 8.4 Sampling Techniques in Quantitative Research** | Sampling Technique | Explanation | Advantages | Disadvantages | |---|---|---|---| | **Random Sampling ** | Create a list of all members of the target population and randomly select a sub-set. This way every member of the target population has an equal chance of becoming part of the sample. | If the sample size is sufficient, unexpected characteristics are fairly represented in the sample. Allows researchers to control representativeness of some key characteristics without relying on chance. | It is practically impossible to carry out truly random sampling, for example, the target population might be geographically dispersed. Requires more knowledge about characteristics of the target population; harder to implement. | | **Stratified Sampling** | First decide on the list of essential characteristics of the population that the sample has to reflect (such as age, occupation, language and so on). Then study the distribution of these characteristics in the target population (for example, age distribution). Then recruit participants randomly, but in a way that keeps the same proportions in the sample as has been observed in the target population. | Useful when the researcher is certain about which characteristics are essential and when the sample sizes are not large. | Generalization from opportunity samples is very limited because of the sampling bias. For example, psychology students are typically familiar with experimental procedures and sometimes can even guess the aim of the study, which does not apply to the wider population. | | **Convenience Sampling** | Recruiting participants that are easily available, for example, undergraduate psychology students who participate in psychological research to earn credit as part of the course. | Useful when financial resources are limited. In some studies, there may be reasons to believe that people are not that different. For example, some basic properties of memory or sense perception may be common for most people and are independent of culture, education or other characteristics. Also useful when generalization of findings is not the primary purpose of the study (for example, in a pilot research study). | Representativeness and generalization are limited. A typical volunteer is more motivated than the average participant from a bigger population, and volunteers could also pursue monetary incentives for their participation. | | **Self-Selected Sampling** | Recruiting volunteers, for example, through newspaper advertisements. Anyone who wants to participate is included in the sample. | A quick and easy method to recruit participants while at the same time having wide coverage (for example, many different people read newspapers). | | **Figure 8.2 Random and Stratified Sampling:** Images of people being randomly allocated within a population, and people being allocated within a population while maintaining the same proportions as the target population. | **Experimental Designs** * There are three types of experimental design, depending on how the independent variable is manipulated: * Independent measures * Matched pairs * Repeated measures. **Independent Measures Design** * In the independent measures design, the independent variable is manipulated by randomly allocating participants into different groups. * The rationale behind random group allocation is that all potential confounding variables cancel each other out. **Figure 8.3 Independent Measures Design** * A diagram of people being randomly allocated to treatment and control groups and then compared to the original data. **Matched Pairs Design** * Matched pairs design is similar to independent measures, but instead of completely random allocation, researchers use matching to form the groups. * To ensure that the groups are equivalent in terms of age, while all other characteristics are kept random. **Figure 8.4 Matched Pairs Design** * A diagram of people being ranked on their specific attribute and then randomly allocated into groups based on that attribute. **Repeated Measures Design** * The same group of participants is exposed to two (or more) conditions, and the conditions are compared. * This way participants are compared to themselves, so these designs are also called "within-subject" designs. * To ensure that order effects are controlled, the researcher needs to use counterbalancing. **Figure 8.5 Repeated Measures Design with Counterbalancing** * A diagram of two groups of people being allocated to two different conditions, and then comparing the results of the two groups. **Table 8.5 Advantages and Disadvantages of Experimental Designs** | Design | Advantages | Disadvantages | How to Overcome The Disadvantages | |---|---|---|---| | **Independent Measures:** | - Can have multiple groups. - Participants only take part in one condition, so there are no order effects. - More difficult for participants to figure out the true aim of the study. | - Participant variability: since these are different people, it is likely that participants in the groups will not be completely equivalent at the start of the study. - More difficult to implement because matching variables need to be measured first. | When allocation into groups is random and groups are large enough, it is likely that pre-existing individual differences will cancel each other out and the groups on average will be equivalent. Keeping the experiment simple: matching is easier to implement when there is one matching variable and two groups. | | **Matched Pairs:** | - Useful when the researcher is particularly careful about certain confounding variables and wants to keep them constant in all groups. | - Participant variability is not a problem because participants are compared to themselves. | - It is more difficult to implement because matching variables need to be measured first. - Theory-driven: the researcher needs to know what variables are particularly likely to be confounding. | | **Repeated Measures:** | - Useful when the sample size is not large and there is a chance that random allocation will end up producing groups that are not equivalent. - Participant variability is not a problem because participants are compared to themselves. This also means that sample sizes can be smaller. | - Participants take park in more than one condition, which increases the chances that they will figure out the true aim of the study. | - Counterbalancing, however, is difficult when there are many conditions (for example, counterbalancing of three conditions requires six groups of participants: ABC, ACB, BAC, BCA, CAB, CBA). | **Credibility and Generalizability in the Experiment: Types of Validity** * The quality of experiments is characterized by their construct, internal and external validity. * Internal validity relates to credibility of the experiment, while external and construct validity characterize generalizability of results. * **Construct Validity:** Is a characteristic of the quality of operationalizations. Operationalizations express constructs in terms of observable behaviour. * Moving from operationalization to a construct is always a bit of a leap. * The construct validity of an experiment is high if this leap is justified and if the operationalization provides sufficient coverage of the construct. * In this sense, construct validity relates to the overarching concept of generalizability- it characterizes generalizability of findings to the theory. * **External Validity:** Is a characteristic of generalizability of findings to other people and other situations. * **Population Validity:** Refers to the extent to which findings can be generalized from the sample to the target population. It depends on how representative the sample is. * **Ecological Validity:** Refers to the extent to which findings can be generalized from the experiment to other settings or situations. It depends on how artificial the experimental procedure is. * **Internal Validity:** Is a characteristic of the methodological quality of an experiment. * In terms of the overarching concepts, it relates to the credibility of the research study. * Internal validity is high when confounding variables have been controlled and we are quite certain that it was the change in the IV (not something else) that caused the change in the DV. * In other words, internal validity links directly to bias- the less bias, the higher the internal validity of the experiment. * Usually there is an inverse relationship between internal validity and ecological validity. **Bias In The Experiment: Threats to Internal Validity** * Bias in experimental research comes in the form of confounding variables that can reduce internal validity. **Table 8.6 Threats to Internal Validity** | Threats To Internal Validity | Explanation | How It Can Be Counteracted | |---|---|---| | **Selection:** | For some reason groups are not entirely equivalent at the start of the experiment, and the way in which they differ affects the relationship between the IV and the DV (like comparing apples and oranges). | Random allocation into groups; sufficiently large group sizes. | | **History:** | Outside events that happen to participants in the course of the experiment. It is especially a problem in lengthy experiments where the DV is measured sometime after the onset of the study. | Standardize experimental procedures as much as possible in all groups to avoid history effects created during the experiment itself. | | **Maturation:** | The natural changes that participants go through in the course of the experiment, such as fatigue or growth. (If the procedure is extended in time) | Having a control group. If we can assume that the rates of maturation are the same in both groups, the comparison will not be affected. | | **Testing Effect:** | The first measurement of the DV may affect the second (and subsequent) measurements. Sometimes in independent measures designs the DV is measured twice. For example, if you asses the effectiveness of a training session to reduce anxiety, you could use an anxiety questionnaire both before and after the training. In repeated measures designs testing effect is a special case of order effects. | In independent measures designs there must be a control group, the same test and retest, but no experimental manipulation. In repeated measures designs, counterbalancing must be used. | | **Instrumentation:** | Occurs when the instrument measuring the DV changes slightly between measurements, compromising standardization of the measurement process. In psychology the "instrument of measurement" is often a human observer. For example, if observations are happening throughout the day, observers may grow tired and miss more important details by the end of the day. | Standardize measurement conditions as much as possible across all comparison groups and all observers. | | **Regression to the Mean:** | This becomes a threat when the initial score on the DV is extreme (either very low or very high). For example, if you are assessing the effectiveness of a training session to reduce anxiety and your participants are a group of people whose initial anxiety score is very high, these participants will naturally become less anxious even if no session is conducted with them. | A control group with the same starting score on the DV, but no experimental manipulation. | | **Experimental Mortality:** | Occurs when some participants drop out of the experiment. It only becomes a problem when the rate of dropping out is not the same in every experimental condition. | Whenever possible, design experimental conditions in such a way that participants do not feel discomfort causing them to withdraw from participation. | | **Demand Characteristics:** | Occurs when participants understand the true aim of the experiment and alter their behaviour (intentionally or unintentionally) because of that. Demand characteristics are a bigger problem in repeated measures designs because participants take part in more than one condition. | Deception to conceal the true aim of the study (however, ethical considerations arise). Post-experimental questionnaires to find out to what extent participants were able to guess the true aim study. | | **Experimenter Bias:** | Occurs when the researcher unintentionally influences participants' behaviour and the results of the study. | Using the double-blind design where neither the participants nor the experimenter knows who has been assigned to what condition. | **Table 8.7 Special Types of Experiments** | Type of Experiment | Independent Variable | Settings | Can We Infer Causation? | |---|---|---|---| | **True Laboratory Experiment** | Manipulated by the researcher | Laboratory | Yes | | **True Field Experiment** | Manipulated by the researcher | Real-life | Yes (but there may be confounding variables) | | **Natural Experiment** | Manipulated by the nature | Real-life | Strictly speaking no | | **Quasi-Experiment** | Not manipulated; pre-existing difference | Laboratory or Real-Life | No | ## 8.4 Correlational Studies **What is a Correlation?** * In correlational studies no variable is manipulated by the researcher, therefore cause-effect inferences cannot be made. * Two or more variables are measured and the relationship between them is mathematically quantified. * A correlation is a measure of linear relationship between two variables. * A correlation coefficient can vary from -1 to +1. * A negative correlation means that there is an inverse relationship between two variables: the higher A, the lower B. * A positive correlation means a direct relationship: the higher A, the higher B. * A correlation close to zero means that there is no relationship between the two variables. **Figure 8.6 Correlations** * Various scatterplots representing both positive and negative correlations, as well as no correlation. **Effect Size and Statistical Significance** * Just like with any other inferential statistical test, a correlation may be characterized by two parameters: *effect size* and *statistical significance*. **Table 8.9 Interpretation** | Correlation Coefficient (Effect Size (r)) | Interpretation | |---|---| | Less than 0.10 | Negligible | | 0.10-0.29 | Small | | 0.30-0.49 | Medium | | 0.50 and larger | Large | * The effect size is the absolute value of the correlation coefficient (a number from 0 to 1). * It shows how large the correlation is. **Table 8.10 The Probability** | The Probability That The Result Is Due to Random Chance | Notation | Interpretation | |---|---|---| | More than 5% | p ≥ .05 | Result is non-significant | | Less than 5% | p < .05 | Result is statistically significant (reliably different from zero) | | Less than 1% | p < .01 | Result is very significant | | Less than 0.1% | p <.001 | Result is highly significant | * Statistical significance shows the likelihood that a correlation of this size has been obtained by chance. * If this likelihood is less than 5%, the correlation is accepted as statistically significant. * There are conventional cut-off points used in interpreting statistical significance. * When interpreting correlations, one needs to take into account both the effect size and the level of statistical significance. * If a correlation is statistically significant, it does not mean that it is large, because in large samples even small correlations can be significant (reliably different from zero). **Table 8.11 Bias on the Level of Interpretation of Findings** | Source of Bias In Interpretation of Findings | Explanation | How Bias Can Be Counteracted | |---|---|---| | **Curvilinear Relationships Between Variables:** | In calculating the correlation between two variables, we assume that the relationship between them is linear. Mathematically the formula of a correlation coefficient is a formula of a straight line. However, curvilinear relationships cannot be captured in a standard correlation coefficient. | If suspected, curvilinear relationships should be investigated graphically. | | **The Third Variable Problem:** | There is always a possibility that a third variable exists that correlates both with A and B and explains the correlation between them. If you only measure A and B, you will observe a correlation between them, but it does not mean that they are related directly. | Consider potential "third variables" in advance and include them in the research study to explicitly investigate the links between A, B and these "third variables." | | **Spurious Correlations:** | Spurious correlations are correlations obtained by chance. They become an issue if the research study includes multiple variables and computes multiple correlations between them. If you measure 100 correlations, there is a chance that a small number of them will be significant, even if in reality the variables are not related. | Results of multiple correlations should be interpreted with caution. Effect sizes need to be considered together with the level of statistical significance. | **Figure 8.7 Curvilinear Relationship** * A scatterplot showing a curved relationship between two variables, demonstrating the difficulty of trying to measure a relationship with a straight line. **Credibility and Bias in Correlational Research** * Bias in correlational research can occur on the level of variable measurement and interpretation of findings. * The less bias there is in a correlational research study, the more credible it is. * Credibility in correlational research is the same idea as internal validity in experimental research; however, the term "internal validity" is not used in correlational studies. **Bias on the Level of Variable Measurement** * Depending on the method used to measure the variables, bias may be inherent in the measurement procedure. * This bias is not specific to correlational research and will occur in any other research study using the same variable measured in the same way. **Bias on the Level of Interpretation of Findings** * Depending on the method used to measure the variables, bias may be inherent in the measurement procedure. * This bias is not specific to correlational research and will occur in any other research study using the same variable measured in the same way. **Sampling and Generalizability in Correlational Studies** * Sampling strategies in correlational and experimental research are the same: random, stratified, opportunity, self-selected. * Generalizability depends on how representative the sample is of the target population. * In its turn, representativeness of the sample depends on the sampling strategy. * Random and stratified samples tend to be more representative than opportunity and self-selected samples. * This aspect of generalizability is similar to the idea of population validity in experimental research. * Another aspect of generalizability is construct validity- the extent to which the way variables are measured (operationalized) captures the theoretical nature of the construct. * This is also similar to experimental research (see Table 8.2 near the start of this unit).