Lec 2: Research Design

A researcher aims to study the effect of a new teaching method (IV) on student test scores (DV). To claim that the changes in test scores are primarily due to the new method, what aspect of experimental validity should the researcher prioritize?

A physical therapist designs a new assessment tool for measuring balance in elderly individuals. What type of validity is most important to establish if the therapist wants to ensure the tool accurately reflects the theoretical concept of balance?

Which of the following scenarios best illustrates the application of the 'known groups method' in establishing construct validity?

A researcher develops a new survey to measure 'emotional intelligence'. While the survey demonstrates high reliability, scores on the survey do not correlate with real-world measures of social skills or job performance. What type of validity is most likely lacking in this scenario?

A language test includes sections on phonology, morphology, syntax, and usage. If the test effectively covers all these key components of language, what type of validity is primarily being addressed?

A researcher aims to study the impact of a new teaching method on student performance. To ensure internal validity, which of the following strategies would be MOST effective in controlling for subject attrition?

In a study examining the effectiveness of a new drug, researchers provide some participants in the control group with extra counseling sessions to compensate for not receiving the drug. This situation primarily threatens internal validity due to:

A school conducts a study where teachers are informed which students are expected to perform well and which are not, regardless of the students' actual abilities. What effect is MOST likely to influence the results of this study, potentially compromising internal validity?

Researchers are conducting a study on a weight loss program. Participants in the program group start showing improved results, but the researchers suspect that the attention they are receiving from being in the study is a contributing factor. This situation is an example of which threat to internal validity?

A researcher uses a depression scale that primarily assesses physical symptoms of depression (e.g., sleep disturbances, appetite changes) but neglects emotional symptoms (e.g., sadness, loss of interest). This situation presents a threat to internal validity related to:

In hypothesis testing, what does rejecting the null hypothesis imply when the p-value is less than or equal to the alpha level?

What is the primary consequence of having a statistical test with very low power?

Which of the following factors does NOT affect the statistical power of a study?

A researcher aims to determine the minimum important difference between two groups before conducting a study. What type of effect size is the researcher trying to define?

In the context of effect sizes, what does Cohen's d primarily indicate?

What is the key distinction between 'statistical significance' and 'practical significance'?

What do eta squared ($\eta^2$) and omega squared ($\omega^2$) measure?

When a p-value is between 0.06 and 0.10, how do researchers typically describe the statistical finding?

What does the Standard Error of the Mean (SEM) primarily estimate?

A researcher aims to determine if a new teaching method improves student test scores. What is the primary purpose of stating a null hypothesis ($H_0$) in this context?

In hypothesis testing, what does the alternative hypothesis ($H_1$) represent?

Why is it important to consider the sample size when interpreting the probability (p-value) in statistical hypothesis testing?

A researcher conducts 5 independent statistical tests on the same dataset, each with a significance level of p = 0.05. What is the main concern regarding Type I error in this scenario?

What is the purpose of applying a Bonferroni correction when conducting multiple statistical tests?

If a study reports a 95% confidence interval (CI) for a sample mean, how should this CI be interpreted?

In the context of statistical testing, what does 'degrees of freedom' typically refer to?

What is the relationship between the Standard Error of the Mean (SEM) and the width of a confidence interval (CI)?

In hypothesis testing, after computing a test statistic, what is the next step to determine statistical significance?

How do floor and ceiling effects primarily impact the validity of a measure?

A researcher aims to assess the effectiveness of a new rehabilitation technique. What is the MOST important consideration regarding the measure used?

A sensitive measure is required for detecting subtle changes in a patient's condition. Which two characteristics are MOST important for ensuring a measure's sensitivity?

To what extent can the findings of a study be generalized to a larger population if the sample is not representative?

What is the key distinction between conscious and unconscious sampling bias?

A researcher intentionally limits their study population to high-functioning individuals with a specific condition. What type of sampling bias is this, and how does it affect the generalizability of the study?

What is the primary advantage of probability sampling over non-probability sampling?

Which of the following is NOT a characteristic of probability sampling?

When is non-probability sampling MOST appropriate to use?

A researcher uses a random number generator to select participants from a list. What probability sampling technique is being used?

A researcher is studying the effectiveness of a new teaching method. They obtain an alphabetized list of students and select every tenth student to participate. What technique is being used, and what is a potential concern?

A researcher wants to ensure their sample includes the same proportion of different ethnic groups as the population. Which sampling technique should they use?

In what scenario would disproportional sampling be MOST appropriate?

A researcher wants to study elementary school children across a large state, but cannot obtain a list of all students. Instead, they randomly select several school districts and then randomly select students within those districts. Which sampling technique are they using?

A researcher posts flyers in a community center and recruits participants who respond to the advertisement. Which non-probability sampling technique is being used, and what is a potential source of bias?

A researcher is conducting a study on a new memory-enhancing drug. Participants in the control group, feeling disadvantaged, begin using various mnemonic techniques to improve their memory performance. This situation primarily threatens internal validity due to:

In a study examining the effectiveness of a new reading intervention program, some students in the control group receive additional tutoring in reading from a different source. This situation presents a threat to internal validity primarily related to:

Researchers are evaluating a new therapy for depression. They notice that participants who were initially the most depressed at the start of the study show the most improvement by the end, regardless of whether they received the therapy or a placebo. This phenomenon is MOST likely an example of:

A researcher is studying the impact of a new exercise program on weight loss. However, many participants drop out of the study before it is completed, and the researcher notices that those who remain in the exercise group are generally more motivated and health-conscious than those who dropped out. This situation primarily threatens internal validity due to:

A teacher implementing a new reading program is unintentionally more enthusiastic and encouraging with students in the experimental group compared to the control group. This difference in behavior primarily introduces bias related to:

A researcher is developing a new test for social anxiety. To establish convergent validity, with which existing measure should the new test show a strong positive correlation?

A university admissions officer uses high school GPA to predict college performance. This is an example of what kind of validity?

A new depression scale is developed, but the test scores are highly influenced by participants' current mood on the day of testing. This primarily threatens:

What type of validity is threatened when a test designed to measure mathematical ability inadvertently measures reading comprehension skills as well?

Researchers suspect that their own expectations about participant performance are subtly influencing how they score the tests. This is an example of:

When evaluating a new assessment for functional independence in seniors, which outcome would provide evidence supporting construct validity?

In factor analysis, what does the clustering of highly correlated variables or items associated with a construct indicate?

A panel of experts is asked to review a new test designed to measure anxiety in teenagers. They evaluate whether the test items adequately cover all aspects of anxiety, such as cognitive, behavioral, and emotional symptoms. What type of validity is being assessed?

A researcher develops a questionnaire to assess job satisfaction. To establish face validity, what approach would be MOST appropriate?

A new self-report measure of social anxiety is developed. To establish concurrent validity, researchers administer the new measure and a well-established, validated measure of social anxiety to the same group of participants. What statistical analysis would be MOST appropriate to assess concurrent validity in this scenario?

A researcher is developing a new diagnostic test for a rare disease. They want to compare the results of their new test with the 'gold standard' diagnostic procedure. Which type of validity is the researcher MOST concerned with establishing?

What is a key limitation of relying solely on face validity when evaluating a psychological measure?

A researcher creates a survey about job satisfaction, but many questions inadvertently ask about stress levels instead. Although employees generally find the survey easy to understand, what is the PRIMARY concern regarding its validity?

A researcher increases the alpha level from 0.01 to 0.05 in a study. How does this change MOST likely affect statistical power and the risk of Type I error?

In a study comparing two treatments, the effect size (Cohen's d) is found to be 0.3. How should this effect size be interpreted?

A new study with low statistical power fails to find a significant difference between a new treatment and a control. Given the low power, what is the MOST appropriate conclusion?

Which of the following scenarios BEST exemplifies 'practical significance' in the context of a weight loss intervention?

A researcher is planning a study and wants to determine the minimum sample size needed to detect a clinically meaningful effect. What type of effect size is MOST relevant for this purpose?

In what situation might a researcher choose to report a statistical trend (p between 0.06 and 0.10), and what caution should accompany this decision?

A researcher conducts a study and calculates eta-squared ($\eta^2$) to be 0.04. How should this value be interpreted in terms of the variance explained by the independent variable?

How does increasing the sample size affect the likelihood of achieving statistical significance, assuming the effect size remains constant?

A researcher calculates a 95% confidence interval for the mean blood pressure of a population. What does this interval estimate?

A researcher is planning a study and wants to increase its statistical power. Which of the following strategies would be MOST effective?

Why is the null hypothesis tested against the alternative hypothesis?

Researchers conduct a study with three groups and decide to perform multiple t-tests to compare all possible pairs of groups. What is the primary concern with this approach?

A study finds a statistically significant difference between two groups (p < 0.05), but the effect size is very small (Cohen's d = 0.1). What is the MOST appropriate interpretation of this result?

A researcher wants to determine if there's a significant difference in exam scores between students who use a new study technique and those who use the traditional method. They set their alpha level at 0.05. What does this alpha level represent?

What does a smaller Standard Error of the Mean (SEM) indicate?

In hypothesis testing, what is the purpose of comparing the calculated test statistic to a critical value?

A researcher wants to reduce the risk of a Type I error when conducting multiple statistical tests. Which of the following methods is MOST appropriate?

Researchers are investigating the effectiveness of a new drug. The null hypothesis states that the drug has no effect. If they reject the null hypothesis, what conclusion can they draw?

A researcher discovers that participants in the control group of a drug trial are seeking additional treatments outside of the study to manage their symptoms. What threat to internal validity does this BEST represent?

In a study comparing a new therapy to a standard treatment, therapists unintentionally spend more time and express greater enthusiasm with patients receiving the new therapy. This situation primarily threatens internal validity due to:

A researcher is conducting a longitudinal study on the effects of exercise on cognitive function in older adults. Over the course of the 5-year study, some participants move away, some experience health issues unrelated to the study, and others simply lose interest and drop out. If the participants who drop out are systematically different from those who remain, how might this affect the study?

To mitigate the potential impact of participants guessing the research hypothesis and altering their behavior, what strategy could a researcher implement?

Researchers are evaluating the effectiveness of a new educational program. They find that students in the control group, aware that they are not receiving the new program, try harder and seek additional resources to improve their performance. What is this effect known as?

A researcher wants to determine if a new test accurately predicts future academic success. Which type of validity evidence should they prioritize?

A researcher conducts a study in a highly controlled laboratory setting to maximize internal validity. What is the MOST likely trade-off they will face concerning external validity?

Which of the following scenarios best illustrates the application of convergent validity?

A researcher is developing a new test to measure a specific construct. To ensure discriminant validity, the new test should demonstrate which of the following?

A physical therapy clinic is trialing a new assessment tool for lower back pain. The researchers want to assess how well the tool's results correspond to a well-established questionnaire designed to measure functional disability due to lower back pain. What type of validity assessment is MOST appropriate in this case?

In a study examining the effectiveness of a new teaching method, researchers discover that the pre-test administered to participants inadvertently influenced their responses on the post-test, regardless of the teaching method used. Which type of validity is MOST threatened by this 'reactive testing' effect?

A clinician is assessing hypernasality in a patient with a cleft palate before and after surgery. If the clinician's judgment of hypernasality is influenced by their knowledge of whether the patient has undergone surgery, which type of bias is MOST likely affecting the reliability and validity of the assessment?

A researcher is developing a new standardized test to assess mathematical ability in high school students. To ensure the test scores can be meaningfully compared across different schools and districts, what type of measurement framework is MOST appropriate?

In a study where participants complete the same cognitive test twice, their performance on the second administration is affected by their experience with the first test. This situation primarily threatens which aspect of measurement?

A team of researchers is assessing the inter-rater reliability of a new observational tool designed to evaluate social interactions among children. Which of the following scenarios would BEST demonstrate high inter-rater reliability?

A rehabilitation center is evaluating a new functional assessment tool. Initial testing reveals that a large percentage of patients score nearly the maximum possible score. What type of validity is most likely affected by this observation?

A researcher is testing a new anxiety intervention. Post-intervention, almost all participants report minimal to no anxiety symptoms based on the measurement scale used. What measurement issue could be affecting the researcher’s ability to accurately assess the interventions effectiveness?

Which of the following strategies is most effective in ensuring a sample adequately represents key subgroups within a population?

A researcher wants to study the experiences of physical therapists working in rural settings. Due to the difficulty in accessing a comprehensive list of all such therapists, the researcher initially contacts a few known therapists and asks them to refer other colleagues, which would be an example of:

A researcher is conducting a study on the effectiveness of a new treatment for a rare disorder. Given the limited number of individuals with the condition, what sampling method is MOST appropriate to ensure sufficient representation of this population in the study?

In a study examining the impact of a community-based exercise program, researchers recruit participants from several local community centers. What type of sampling is being employed?

A polling company wants to predict the outcome of an election but only surveys individuals who own smartphones. This introduces which type of bias?

A school administrator distributes questionnaires to all teachers in the district regarding their job satisfaction. What type of sampling method is being used if participation is voluntary?

A researcher aims to survey undergraduate students at a large university. To ensure proportional representation across all academic years (freshman, sophomore, junior, senior), which sampling method should be used?

A public health researcher wants to assess the prevalence of smoking among adults in a city. Due to resource constraints, they randomly select a few neighborhoods and survey all households within those neighborhoods. Which sampling technique is being used?

To evaluate the impact of a new rehabilitation program, a therapist recruits every fifth patient from an alphabetical list. What sampling approach is this?

A researcher is studying the effectiveness of a new therapy technique for improving motor skills in children with cerebral palsy. They recruit participants by posting flyers in local pediatric clinics and therapy centers. This recruitment strategy is MOST likely to result in:

A researcher manually selects participants for study. This selection involved strategic limiting of the population. This introduces which type of bias?

A researcher is conducting a study on stress levels of nurses in a hospital setting. They obtain a list of all nurses and randomly select a starting point, then choose every tenth name on the list to participate. What type of sampling method is being used?

A researcher is investigating the effectiveness of a new educational program across different school districts. Due to logistical constraints, they randomly select several school districts and then include all the schools within those selected districts in their study. What sampling method is being employed?

How does factor analysis contribute to construct validity of a measurement instrument?

In the context of construct validity, a measure of functional independence in seniors should exhibit which of the following relationships?

When evaluating content validity, what is the primary role of expert opinion and literature review?

A new test is designed to measure a specific psychological construct. To provide evidence of concurrent validity, what should researchers do?

A researcher develops a new diagnostic test for a common disease. To establish criterion-related validity, what step is most critical?

What is a key limitation of face validity when used as the sole measure of a test's validity?

A researcher correlates scores from a new measure of anxiety with scores from an established depression inventory. A high correlation is found. What type of validity is being primarily assessed, and what does this suggest?

A content validity assessment reveals that a test intended to measure critical thinking skills heavily emphasizes mathematical reasoning while neglecting other important aspects, such as logical argumentation and analytical skills. What is the primary concern?

A researcher is using a statistical test with a set alpha level. What is the direct consequence of decreasing the alpha level from 0.05 to 0.01?

In a study comparing two groups, the p-value obtained is 0.03, and the alpha level was set at 0.05. What decision should be made regarding the null hypothesis?

Which of the following is NOT a factor that affects the statistical power of a study?

A researcher calculates Cohen's d and obtains a value of 0.65. How would this effect size typically be interpreted?

Which of the following provides a measure of the proportion of variance in the dependent variable that is explained by the independent variable?

A study finds a statistically significant difference between a new treatment and a control group, but the actual improvement in patients' quality of life is minimal. What does this primarily indicate?

In the context of statistical hypothesis testing, what is the potential consequence of increasing statistical power?

A researcher observes a p-value of 0.07 in their study. How might they typically describe this finding in a research report?

A researcher conducts a study and calculates a confidence interval (CI) for the population mean. A wider CI indicates:

A researcher states the null hypothesis as: $H_0: \mu_1 = \mu_2$. What would be the appropriate non-directional alternative hypothesis?

A researcher is planning a study and wants to increase its statistical power. Which of the following actions would directly contribute to achieving this goal?

In statistical hypothesis testing, what is the purpose of setting an alpha level (significance level)?

A researcher runs multiple statistical tests on the same dataset. What is the primary reason for applying a Bonferroni correction?

If a study's results are described as 'statistically significant', what does this primarily indicate?

Which of the following best describes the Standard Error of the Mean (SEM)?

A researcher observes a large difference between two group means, but the statistical test is not significant. Which factor could MOST likely be contributing to this non-significant result?

Which of the following is MOST directly related to the calculation of a confidence interval?

A researcher aims to replicate a study but obtains a non-significant result despite using the same methods and a similar sample size as the original study, which reported a significant finding. What is the MOST likely explanation for this discrepancy?

Extraneous Variable

Independent Variable

Dependent Variable

Diffusion or Imitation of Treatment

Hawthorne Effect

Internal Validity

External Validity

Measurement Validity

Construct Validity

Known Groups Method

Selection of Participants

Norm-referenced

Criterion-referenced

Measurement Reliability

Decision Rule (p ≤ alpha)

Type I Error

Type II Error

Statistical Power

Practical Significance

Effect Size

Cohen’s d

Statistical Trend

Standard Error of Measurement (SEM)

Standard Error of the Means (SEM)

Confidence Intervals (CI)

Inferential Statistics

Null Hypothesis (Ho)

Alternative Hypothesis (H1)

Alpha Level (Significance Level)

Correction for Multiple Tests

Within-Group Differences (Variance)

Test Statistic

Floor and Ceiling Effects

Responsiveness to Change

Sampling Bias

Conscious Bias

Unconscious Bias

Probability Sampling

Non-probability Sampling

Simple Random Sampling

Systematic Sampling

Stratified Random Sampling

Disproportional sampling

Cluster Sampling

Consecutive Sampling

Convenience sampling

Snowball Sampling

Inclusion/Exclusion Criteria

Subject Attrition (Mortality)

Compensatory Equalization

Experimenter Expectancies

Placebo Effect

Instrument Responsiveness

Construct Dimensions

Factor Analysis

Content validity

Face Validity

Criterion-Related Validity

Concurrent Validity

Establishing Content Validity

Predictive Validity

Convergent Validity

Discriminant Validity

Threats to Measurement Validity & Reliability

Sensitivity

Sample

Target Population

Accessible Population

Representative Sample

Stratified Sampling

Cohen's d

Clinical Significance

Population Parameters: σ & σ²

Significant Difference

Hypothesis Testing Steps

Null Hypothesis (H₀)

Alternative Hypothesis (H₁)

Alpha Level (α)

Bonferroni Correction

Between Group Differences

Compute Test Statistic

Experimental Arrangements

Maturation

Testing Effects

Statistical Regression to the Mean

Compensatory Equalization of Treatment

Rosenthal Effect

Threats to Measurement

Blinded Measurement

A priori effect size

Calculated effect size

Personal Significance

P-value

Null Hypothesis

Alternative Hypothesis

Multiple comparisons correction

Determinants of Significance

Calculated Value

Critical Value

Within-Group Differences