Psychology Experiment Insights

Podcast

Play an AI-generated podcast conversation about this lesson

Questions and Answers

What was a major issue faced by researchers attempting to replicate Bargh's priming studies?

The studies were based on rigorous methodologies that were never questioned.

They often found a stronger effect than originally reported.

Replication efforts were universally accepted and published.

Many were unable to achieve the same results, leading to unpublishable findings. (correct)

What was the response of Bargh to critics of his work?

He issued an apology for his unreplicable studies.

He welcomed constructive criticism and revised his methods.

He dismissed critics and labeled them as incompetent. (correct)

He publicly praised the replicability efforts made by others.

What significant index emerged in response to concerns about replication in psychological research?

Effectiveness Index

Replicability Index (correct)

Methodological Rigor Index

Research Integrity Index

What unethical action did the Principal Investigator (PI) take in Kate's story?

He manipulated the experimental setup to misrepresent results. Signup and view all the answers

Which of the following studies was mentioned as an example of popular research that has failed to replicate?

Power Posing Signup and view all the answers

What is the main approach used by Data Detectives to identify fraudulent data?

Examine the paper like a crime scene for evidence Signup and view all the answers

How can descriptive statistics reveal issues in published research, according to the examples provided?

By analyzing standard deviations Signup and view all the answers

What specific tool in Excel helps track data changes during analysis?

CalcChain Signup and view all the answers

What was the outcome of the investigation into Francesca Gino's work?

Four of her papers were retracted due to manipulation Signup and view all the answers

What is the stance of the Tri-Agency Statement of Principles on Digital Data Management regarding public grants?

Data collected with public funds should be publicly available Signup and view all the answers

What is a primary responsibility of research agencies in managing data?

To recognize data as an important research output Signup and view all the answers

What resulted from Francesca Gino's lawsuit against Harvard?

The case was dismissed Signup and view all the answers

Why is it essential to treat data as a research output, as emphasized by research agencies?

To satisfy funding requirements Signup and view all the answers

What psychological phenomenon was suggested by Bem's experiment concerning participants' memory?

Participants had an enhanced memory for words studied after the test. Signup and view all the answers

What is a key principle associated with Karl Popper's philosophy of science?

A theory is scientific if it can be disproven. Signup and view all the answers

Which of the following is true regarding reproducibility in scientific research?

Only a small percentage of studies in some psychology journals can be reproduced. Signup and view all the answers

What does the concept of confirmation bias refer to in scientific research?

The tendency to favor information that confirms existing beliefs. Signup and view all the answers

What percentage of psychological studies was found to be successfully repeatable in the Many Labs 2 project?

50% Signup and view all the answers

What is a characteristic of pseudo-science according to the discussion in the notes?

It seeks to prove hypotheses through confirmation. Signup and view all the answers

What issue was highlighted by the evaluation of 250,000 p-values in published psychology papers?

A significant portion had inconsistencies in reported values. Signup and view all the answers

Which of the following is NOT a criterion of the scientific method mentioned?

Derivable Signup and view all the answers

What is the primary difference between confirmatory and exploratory research?

Confirmatory research commits to a specific hypothesis. Signup and view all the answers

Which of the following best describes the practice of p-hacking?

Excluding irrelevant data to shape desired outcomes. Signup and view all the answers

In the context of optional stopping, how can researchers appropriately manage peeking at their data?

They should decide how many times to check before analyzing the data. Signup and view all the answers

What encourages researchers to engage in questionable research practices (QRPs)?

The necessity of producing statistically significant results. Signup and view all the answers

How does the example of sampling 100 people with 40 excluded compare to multiple studies with smaller samples?

Excluding data points can lead to a misleading perception of results. Signup and view all the answers

Why is it problematic that researchers often do not present the entire set of data?

It creates a culture of selective reporting which distorts reality. Signup and view all the answers

What analogy is used to describe p-hacking in research?

Dealing a magic poker hand. Signup and view all the answers

What misconception do many people have about p-hacking?

It is acceptable if done occasionally. Signup and view all the answers

What was a significant outcome of the article 'False-Positive Psychology'?

It highlighted QRPs as a prevalent issue akin to serious crimes. Signup and view all the answers

Study Notes

Daryl Bem Experiment

Participants had a better memory for words they would study after the test.
Implied participants had psychic powers.

Confirmation Bias

Journal of Personality + Social Psychology had a policy where articles would not be published if they disconfirmed previous publications.
Card flipping experiment:
- Flipping a 'D' card is a confirmation choice.
- Flipping a '7' card is a falsification choice, which can provide evidence to refute the hypothesis.
Karl Popper:
- A theory not falsifiable is not a scientific theory because science disconfirms theories, while pseudo-science confirms them.

Scientific Methods

Testable, refutable, and falsifiable.

Reproducibility

Molecular Brain journal:
- Out of 41 manuscripts, 21 were withdrawn (didn't want to provide raw data), and 19 couldn't be reproduced.
- Only 3% could be reproduced.
"The Prevalence of Statistical Reporting Errors in Psychology (1985-2013)" article:
- 250,000 p-values were evaluated using R statcheck.
- Half of all published papers had inconsistencies between p-value and the test statistic.

Replicability

Many Labs 2:
- Addressed criticisms of Many Labs 1.
- Sample sizes were 60x larger, repeated in labs worldwide.
- Only half of psychological studies repeated successfully.
- "Published and true are not synonyms."

Good Science

What was true before the replication crisis is not true after it.
Our knowledge changes our universe.

John A. Bargh - Priming Research

Paper on priming elderly people with words to walk slower cited over 3800 times.
Many researchers had trouble replicating.
- Pashler (2008) failed to find the effect and posted it on the FileDrawer website.
- Doyen failed to replicate but got the paper published in PLoS ONE.
- Bargh responded by attacking people for calling him out, claiming they are "incompetent and ill-informed," and "have nothing in their heads."
Led to the creation of the Replicability Index: Percentage of findings in a journal that replicate.

Examples of Unreplicated Studies

Power Posing - Amy Cuddy
Pen in mouth (facial feedback hypothesis)
Ego depletion

Kate's Story

PI conducted a biased experiment with undergrad students.
PI asked undergrads to lie about equipment placement to support his hypothesis.
Undergrads sought advice from Kate, who advised them to go to the department chair.
Kate was fired from the lab, had to finish her Ph.D. elsewhere, and her career was delayed 10 years.
Whistleblower faces the most consequences for reporting QRPs.

Darren Brown, the Garden of Forking Paths - Khadisha

Darren Brown used "the system" to create a reality for Khadisha through p-hacking.
Academics use the same system to create a reality for readers of their research.
Researchers present one path (e.g., horse betting) but don't show the numerous other paths.
Readers lack knowledge of the broader paths when reading academic research.

Exploratory vs. Confirmatory Research

Confirmatory research:
- Outcome is a conclusion about something that might be true.
- Committed to a specific hypothesis or research question.
Exploratory research:
- Outcome is a hypothesis, not a conclusion.
- No set hypothesis beforehand.
- Needs to be followed up by a confirmatory analysis.
In brief:
- Confirmatory = Scientific conclusions
- Exploratory = Hypothesis (needs further testing)

Joseph Simmons, Leif Nelson, Uri Simonsohn: DataColada

"A researcher may run a regression with/without outliers, with/without a covariate, with one and then another DV, and only report the significant analyses in the paper."
"Undisclosed Flexibility" - "When I'm 64" by the Beatles.

Result of DataColada Research

Everyone thought p-hacking was wrong, but they thought it was wrong like jaywalking.
"False-Positive Psychology" revealed it's wrong like robbing a bank.

Saving Science from the Scientist - Dorothy Bishop

P-hacking is comparable to dealing a magic poker hand, researchers deal many hands and pick the exciting ones (similar to Khadisha).
P-hacking is often due to ignorance.
- Researchers don't understand the need to present all data.
- It has become a culture to conduct research in this way.
- Strong incentive: Journals don't publish without low p-values.

Optional Stopping

A way to conduct optional stopping appropriately.
Pre-determine how many times to "peak" at data.
- Peaking 3 times = dividing significance (p = .05) / 3 = p = .0017, which is much harder to achieve.

How Common are QRPs?

9% of researchers admitted to falsifying data (making stuff up).

Mathematically Identical Situations, but with a Better Look

Researchers sample 100 people in one study and find no effect.
- Removing 40 people (data points) and finding an effect is unethical.
Researchers sample 100 people across 5 studies.
- 3 samples = significant effects (n = 60).
- 2 samples = no effect (n = 40).
- Reporting only the 3 significant findings is the same as manipulating the first example.

Data Detectives

Treat a paper like a crime scene.
Look for evidence of manipulation, fabrication, or fraud in data.
Can often find fraud by looking at sample statistics alone.
Example using descriptive statistics:
- Simonsohn (2013) found issues in Sanna, Chang, Miceli, & Lundberg (2013) based solely on standard deviations.
Example using raw data:
- Francesca Gino - Harvard researcher.
- Someone moved participants between conditions to ensure significance.
- "Under the hood" in excel can track changes made to a data file using a calcChain.
- This can be applied to specific statistics.
  - DataColada showed 6 rows were out of sequence, suggesting manual manipulation of variables with large effects on the DV.
Results: Manipulated results were significant, original ones were not.

Francesca Gino

4 retracted published papers showed evidence of data manipulation.
Harvard commissioned an investigation, resulting in a 1300-page report.
Gino sued Harvard for $25 million and the DataColada team.
The case against DataColada was dismissed.
- "Scientists cannot effectively sue other scientists for exposing fraud/errors in their work."

Tri-Agency Statement of Principles on Digital Data Management

Grants come from taxpayer money.
Tri-agency believes research data collected with public funds should be in the public domain for reuse.
You don't own data collected with public funds.
This policy is mandatory for everyone receiving grants, even if they're not specifically from the Tri-agency.

Responsibilities of Research Agencies

"Recognizing data as an important research output and fostering excellence in data management."
Data itself is considered a research output, not just published papers.

Studying That Suits You

Use AI to generate personalized quizzes and flashcards to suit your learning preferences.

Description

Explore the intriguing findings from Daryl Bem's experiments, including the phenomenon of retroactive memory improvement. Dive into topics like confirmation bias, the importance of falsifiability in scientific theories, and challenges faced by psychology in reproducibility. This quiz will test your understanding of these critical concepts in psychology research.