Research Credibility and Empiricism Overview

Credibility Crisis in Research

• Publish-or-perish phenomenon pressures researchers to publish in high-impact journals, potentially sacrificing credibility for publication.
• Biased research designs prioritize expected outcomes over disproving hypotheses.
• Peer review systems face challenges in adequate review time and training, impacting rigor.
• P-hacking compromises data integrity by manipulating data-collection to achieve desired outcomes.

Solutions to Research Issues

• Preregistration badges on research pieces
• Open material for accessible research resources (e.g., surveys, data)
• Open data accessibility for comparable past research.

Foundations of Empiricism

• Plato's rationalism emphasizes reasoning based on internal knowledge.
• Aristotle's empiricism prioritizes knowledge derived from sensory experience.
• Modern empiricism includes systematic empiricism and skepticism.

Non-empirical Ways of Thinking

• Trusting experts: experts are susceptible to mistakes, biases, and misinformation.
• Anecdotal evidence: personal experiences are limited, lack comparison groups, and contain potential confounds.

Importance of Skepticism

• Being unsure about things is valuable, even when data is available.

Theory-Data Cycle

• Theory: general statements about variables.
• Hypothesis/Prediction: specific guesses derived from theory.
• Data Collection: observations and measures.

Assessing Research Claims

• Variables definitions (IVs, DVs, Constants)
• IVs (independent variables) get manipulated; DVs (dependent variables) get measured.
• Random assignment of participants to conditions
• Quasi-experiments establishes cause-and-effect relationship
• Confounds (confounding variables) that affect both independent and dependent variables must be controlled.

Types and Claims

• Frequency claims describing variables alone
• Association claims between two or more variables
• Causal claims one variable causing changes in other variable

Assessing Research Claims

• Frequency: construct, external validity
• Association: construct, external, statistical validity
• Causal: construct, external, statistical, internal validity

Research on Rats

• Tryon's inbred rats experiment.
• Construct validity of experiments
• Frequency claims
• Association claims
• Causal claims

Validity Types

• Construct validity
• Content validity
• Criterion validity
• Face validity
• Empirical assessment

Survey Research

• Describing characteristics of populations, comparing groups/populations, time trends, relationships between variables
• Understanding population characteristics
• Testing psychological theories and models

Developing a Good Survey

• Converting research goals to specific survey topics
• Defining survey items

Item Formatting Choices

• Open-ended items
• Close-ended items
• Multiple choice items (single/multiple response)
• Likert/Likert-type scales
• Ranking scales

Visual Analog Scales

• Rating on a continuous scale without numerical values.

Leading Items & Wording Issues

• Leading/loaded items nudge respondents toward certain answers.
• Double-barreled items ask about multiple topics or concerns using one question.
• Careful consideration of language, terms, and sentence structure is important to avoid confusion, bias and misunderstanding.

Order of Survey Questions

• Order effects.
• Leading items (they influence the subsequent answers)
• Loaded items (force certain responses, usually untrue ones)
• Double-barreled items (ask about more than one topic).
• Items using double negatives (e.g., "not unwilling").
• Carefully worded items with unambiguous punctuation.
• Items in logical order.

Issues in Survey Responses

• Yea-saying (always agreeing)
• Nay-saying (always disagreeing)
• Fence-sitting (always neutral).

Socially Desirable Responding

• Participants respond in ways they believe are socially appropriate to look good.

Solutions for Socially Desirable Responding

• Anonymity is crucial for honesty in surveys (because people are less likely to provide dishonest answers)
• Bogus pipeline technique creates perceived consequences to encourage honesty about answers.

Types of Observational Research

• Natural Observation.
• Participant Observation.
• Structured Observation.

Observational Research Considerations

• Focusing on certain subjects.
• Choosing an area/location to observe.
• Determining how behaviors are recorded (e.g., systematic, continuous).

Correlation and Causation

• Correlation does not equal causation.
• Temporal precedence (cause must come before effect)
• Internal validity (ruling out other explanations)
• Third-variable problem (confounding variable)
• Directionality problem

Partial and Multiple Correlations

• These methods help determine relationships between variables and factors.
• Multiple regression analysis
• Controlling extraneous/confounding variables

Internal Validity

• Control group
• Random assignment
• Proper control over conditions
• Consistency in procedures or treatment

Experimental Designs

• Single-factor, between-groups
• Single-factor, within-groups
• Factorial designs (including two, three, and mixed designs)

Threats to Internal Validity

• Design confounds
• Selection effects
• Order effects
• Experimenter expectancy effects

Instrumentation Threats

• Measuring instruments may change over time, decreasing reliability/validity.
• Training coders/observers to ensure consistent use of instruments.
• Finding alternate instrument forms in pre- and post-test assessments to maintain consistency.
• Ensure sufficient detail in measurement descriptions.

Null Results

• Results indicate no direct relationship between treatments and outcomes.
• Low variability between conditions
• High variability within conditions
• Reverse confounds: probability that the observed results are associated with a lurking variable instead of an actual treatment effect.

Validity in Research

• Convergent validity
• Discriminant validity (how well a measure distinguishes between different constructs)

Ethical Concerns in Research

• Informed consent
• Right to withdraw
• Deception
• Debriefing

Types of Claims (in Research)

• Frequency claims
• Association claims
• Causal claims

Factors for choosing design types (for research)

• Degree of complexity in researcher's question
• Amount of control desired by researcher
• Practicality and feasibility
• External validity