Psychology Online Research Methods

Publishing Psychology Studies

• Computer scientists began archiving their work online in the 1970s.
• Physicists followed suit in the 1990s, using a journal called "arXiv" to host their research.
• Psychology publications had early online journals such as "Psycoloquy" (APA, 1990-2002) and "Journal of Vision" (ARVO, 2001 to date).

Types of Data Collection

• Online surveys are used to collect data on various topics.
• Examples of online survey platforms: Qualtrics, LimeSurvey, SurveyMonkey
• Perception and cognition experiments can benefit from online data collection with stimulus presentation and response timing.
• Examples of platforms used for this purpose: PsychoPy, Gorilla.

Advantages of Using the Internet for Research

• Researchers can access historical records, digital transactions, and various quantitative and qualitative data.
• This makes recruitment and data collection much easier.
• It allows for accessing large, diverse samples at a lower cost.
• Researchers can focus on specific groups, including those from different countries and cultures.
• Online research potentially reduces experimenter bias as the standardisation removes the influence of the experimenter.
• Data collection becomes faster with global participation enabling 24-hour data collection.
• The internet's familiarity and engagement for younger participants increases motivation.

Ethical Benefits

• Anonymity provided by online research encourages increased self-disclosure.
• Online participation can be less effortful and costly for participants.
• Decreased social pressure is a positive aspect of online research.

Ethical Disadvantages

• The Facebook "emotional manipulation" study highlights potential harm resulting from direct participation.
• Online participants may become distressed.
• Breaches of confidentiality are a concern for online research.

Disadvantages of Using the Internet

• Sample biases and lack of generalizability present challenges.
• Drop-out rates are a concern.
• Researchers have no control over the environment in which data is collected.
• Participants might invest less energy in online research compared to in-person interaction.
• Data security risks can lead to leaks.
• Technology failures are potential obstacles.
• Multiple submissions by the same participant can skew data.
• Researchers need to enhance their technological expertise.

Threats to Data Quality and Solutions

• Pilot testing and pre-testing are crucial to improve data quality.
• Data should be collected from a trustworthy source.
• Good participant management, such as recording IP addresses, helps avoid multiple submission.
• Establish objective exclusion criteria (timing, attention checks) to ensure data quality.

Survey Methods in Research Practice

• The first step in using survey methods is to find measures that answer your research question.
• It's important to assess the quality of the measure for your purpose.
• Researchers may need to adapt an existing measure or create a new one.
• Choosing the appropriate survey type and platform for creation is crucial.
• Distributing the survey and collecting responses is essential.
• Determining the validity of survey responses is vital.
• Handling invalid responses within the dataset is crucial.

Where to Find Measures

• Electronic databases such as the APA PsycTEST website are excellent resources for finding measures.

Determining the Quality of the Measure

• Consider whether the measure is appropriate for your research question, target demographic, and method of delivery.
• Assess the measure's psychometric properties (validity and reliability).
• Consider practical aspects like accessibility and cost of the measure.

Adapting or Creating Measures

• When adapting an existing measure, ensure the items are clear and understandable.
• Maintain consistency in response scales across measures.
• Group frequency and attitude scales together to minimize confusion.

Benefits of Using or Adapting Existing Measures

• Using an existing measure saves time and resources.
• It allows for easier comparison with previous studies.
• Reviewers are more likely to support research using existing measures.

Creating a New Measure

• Conduct thorough research on existing measures, theories, and resources.
• Incorporate focus group feedback.

Steps in Creating a New Measure

• Drafting Survey Items:* Ensure the items are reliable and valid.
• Reliability: Consider whether items are interpreted consistently across all respondents, on different days, and as intended.
• Validity: Determine whether items capture the necessary information accurately and relevantly.
• Piloting and Editing Items:* Collect feedback and edit items to improve quality.
• Obtain feedback from colleagues or supervisors.
• Pilot the draft survey with a small group.
• Edit items based on feedback.

Deciding on the Survey Type and Platform

• Mail Surveys: Access to target population, inexpensive, convenient.

• Disadvantages: Poor response rates, not suitable for all populations, no control over administration, manual data entry.

• Interviews (F2F or Phone): Suitable for all populations, captures non-verbal cues, keeps the interviewee focused, minimizes missing data.

• Disadvantages: High cost, time-consuming, can limit sample size, anonymity compromised.

• Online Surveys: Convenient, time and cost-effective, automatic data collection, access to large samples, global reach.

• Disadvantages: May not be accessible to all populations, increased chance of survey fraud. Incentives can encourage falsification.

Creating a New Online Survey

• The first step is determining the sensitivity level of the data to be collected.

• Data sensitivity can be classified into three categories:

  • Highly Sensitive: Data containing information about health, criminality, genetics, race and religion, and finances.
  • Sensitive: Data containing information about personal information not related to factors above, cultural heritage, location, and ecological data about endangered species.
  • General: Anonymized or publicly available data in aggregated form.

• Use a Data Sensitivity Calculator to determine the sensitivity level of the dataset.

• Choose the appropriate research platform based on the data sensitivity level.

Suitable Platforms

• Highly Sensitive Data: Suitable for: collecting and safely storing survey data involving sensitive topics.
• Sensitive Data: Suitable for: collecting survey data, storing data, and archiving datasets.
• General Data: Suitable for: data storage, archiving data, and accessing datasets.

Distributing Surveys and Collecting Responses

• Use various methods to distribute surveys:
  • Collaboration with organizations (schools, hospitals, workplaces)
  • Personal or professional emails
  • Research participant pools (e.g., MQ SONA, Qualtrics)
  • Social media
  • "Word of mouth"

Determining Survey Response Validity

• Inattentive or careless respondents can negatively impact data quality.
• This decreases statistical power, obscures meaningful results, and can lead to spurious results.
• Other forms of invalidity include:
  • Faking good
  • Faking bad
  • Social desirability.

Detecting Invalid Responding

• Include items designed to detect inattentive responding.
• Conduct multivariate outlier analysis to assess statistically unlikely response patterns.
• Analyze survey response time.
• Utilize long string analyses to measure repeated answer selection.
• Include self-reported diligence measures.

Handling Invalid Responses

• Delete cases with inappropriate responses.
• Create a cut-off score for attention items.
• Conduct sensitivity analyses.
• Replace invalid responses with plausible values.