Research Methodology Quiz

Research Orientations

• Ways of knowing include experience, authority, tradition, and intuition.
• Many of these rely on cognitive logic processes, which can be flawed.
• Epistemology asks how we know what we know.
• Necessary and sufficient conditions for knowledge must be determined.

Social Science Epistemology

• Knowledge in social science relies on reliably observed phenomena.
• Epistemology focuses on what counts as knowledge in the social sciences.

Social Scientific Inquiry

• Different research questions require different methods.
• Scientific/Quantitative methods include surveys and experiments.
• Interpretive/Qualitative methods include interviews, focus groups, and ethnography/participant observation.

Social Scientific Theory

• Quantitative/Deductive theory aims to create testable and falsifiable hypotheses.
• Theoretical constructs in quantitative theory must be observable.
• Interpretive/Inductive theory seeks plausible interpretations of social data.

Hypotheses Testing

• Hypotheses propose relationships between variables.
• Hypotheses commonly take the form of H1 (X is related to Y) or H2 (Group A will differ from Group B on X).
• Examples include communication apprehension affecting public speaking grades or social media use affecting mediated activity time.

Quantitative Methodology

• A quantitative methodology aims to produce unbiased and replicable knowledge.
• Methodologies produce quantifiable results, ideally generalizable.

Elements of Quantitative Research

• Understanding what is being studied (conceptualization).
• How the study will be conducted (operationalization).
• Who is participating (sampling).
• Clarity of data collection methods (replicability).
• Confidence in findings.

Variables

• Variables are entities that take on different values.
• Variables are the building blocks of research design.

Conceptualization

• Refines and specifies abstract concepts.
• Conceptualizations are working agreements, not dictionary definitions.

Language - The Problem

• Key concepts in communication studies (e.g., love, leadership, satisfaction, control, culture) can be problematic due to language creating confusions and limitations in communication.
• Conceptualization, indicators, and dimensions of communication apprehension include nervousness, butterflies in stomach, worry, speechlessness, fear, etc.

Quantitative Approaches: Confidence in Findings

• All studies are likely flawed.
• The "coffee problem" highlights how preconceived notions bias the interpretation of research findings.
• Overall effects of an action (e.g., coffee consumption) may be complex, preventing clear conclusions from individual studies.
• The Central Limit Theorem describes how sample statistics approximate population parameters.

Normal Distribution

• Data clusters around the mean.
• To gain confidence in quantitative findings:
  • Carefully conceptualize the variables being studied.
  • Precisely operationalize the variables.
  • Employ appropriate sampling techniques.
  • Ensure data collection methods allow for replication.

Variables (continued)

• Variables are entities that take on diverse values.
• Variables underpin research design.

Categorical Variables

• Mutually exclusive categories; e.g., hair color, gender

Ordinal Variables

• Ordered categories (e.g., paper grades); meaningful rank but not precise differences

Interval Variables

• Ranked, meaningful differences, but no true zero point (e.g., temperature in Celsius)

Ratio Variables

• Ranked, meaningful differences, and a meaningful zero point (e.g., height, weight).

Measure variables...

• at the highest possible level whenever possible.

Conceptualization of Variables

• Conceptualization involves defining abstract concepts in a study-specific manner.
• Conceptualizations are working agreements, not dictionary definitions.
• Careful analysis of study variables is essential, especially for abstract concepts.
• Indicators are elements that demonstrate the presence or absence of a concept.
• Variables can be multi-dimensional (e.g., communication apprehension).

Connecting Variables to Questions

• Research questions propose relationships between variables, general in nature.
• Hypotheses propose specific relationships between variables, usually in a directional form.

Hypotheses

• Hypotheses can be correlational (e.g., a relationship between variables) or comparative (e.g., differences between groups).

Week 3: Measurements

• Operationalization: The development of specific research procedures resulting in empirical observations representing those concepts in the real world.
• Types of Operationalization in Social Sciences: Direct Observation, Survey/Self-Report Data.

Developing Measures

• Drawbacks: May not capture the richness of experiences; structure may not match respondent experiences.
• Indicators: Identifying indicators of your variable.
• Crafting Items: Creating questions or items based on indicators.
• Composite Measures: Using multiple items to create a comprehensive measure.

Example Scales

• Likert Scale: Used for measuring attitudes or feelings.
• Semantic Differential: Measures attitudes using pairs of adjectives.

Choosing a Scale

• Measure data at the most sophisticated level possible.
• Avoid using categorical/ordinal scales for ratio variables.
• Write more items than needed to ensure reliability.
• Use precise and clear language.
• Keep items short.
• Avoid jargon.
• Use language respondents understand.
• Provide specific frames (e.g., "When thinking of...").
• Avoid absolutes like "always" or "never" in items.

Language

• Language: Use language that respondents understand.
• Specific Frames: Provide specific contexts for questions.
• Avoid Absolutes: Avoid words like "always" or "never."

Respondent Considerations

• Ensure respondents are willing and able to answer.
• Be aware of social desirability bias.
• Ensure respondents have the knowledge or experience to answer.
• Consider if respondents can remember the information.

Sampling

• Sample: A subset of elements or units selected from a population.
• Population: The entire group you wish to generalize the results to.
• Types of Population:
  • All the people in the world.
  • Population of Canada.
  • Married couples.
  • Teenagers and screen time.
  • Organizations, messages, words, locations.
• Why Not Use the Whole Population?: It becomes difficult to create an accurate census.
• Subset of the Population: Generally, impossible to study everyone, so we study a small sample.
• Goal: To get a sample that is representative of the population we wish to study.

Probability Sampling (Gold Standard)

• Random Sampling: Each population member has an equal chance of being selected.
• Simple Random Sampling:

Nonprobability Sampling

• Why Use Nonprobability Sampling?: Probability sampling is theoretically best but practically impossible.
• Convenience Sampling: Reliance on available participants, introducing biases like nonrepresentative sample and self-selection.
• Purposive Sample: Also called judgmental sample, specific people chosen based on the research question.
• Snowball Sample: Data from a few known members of a target population, asking them to provide other members.
• Quota Sampling: Participants selected nonrandomly based on their known proportion in the population.
• Are Nonprobability Samples OK?: Convenience sampling can contribute to valid inferences if representative and repeated (replication is key).

Pragmatic Approaches...

• Specify the population → Consider bias → Measure demographics.

Week 4: Experimental Design

• Experimental Design: Introducing some action/manipulation/treatment by the experimenter and observing the consequences.
• Control: Essential for establishing causality.

Criteria for Causality

• X is related to Y.
• X temporally precedes Y.
• X and Y are not related through some third variable.
• Relationship between X and Y is not spurious.

True Experiments

• Control Group: Helps control participant-related threats to internal validity (e.g., Hawthorne Effect, Placebo Effect, Maturation, Experimental Mortality).
• Experimental Group: Also called an experimental condition or treatment condition.
• Random Assignment: Ensures experimental and control groups are equivalent, eliminating selection bias.

Experimental Designs

• Double-Blind Experiment
• Manipulation Checks
• Pretest-Posttest Control Group Design
• Posttest Only Control Group Design
• Solomon Four-Group Design
• Quasi-Experimental Design
• Time Series Design
• Nonequivalent Control Group Design
• Multiple Time Series Design
• One-Shot Case Study
• One-Group Pretest-Posttest
• Static Group Comparison

Reliability and Validity

• Reliability: Consistency of a measure (same result with repeated application).
• Techniques:
  • Test-Retest: Consistency over time.
  • Internal Consistency:
    • Split-Half Reliability: Correlation between two subsets of items.
    • Item-Total Reliability: Correlation between individual items and the total score
• Validity: Accuracy of a measure (reflects intended concept)
• Types:
  • Content Validity:
    • Face Validity: Measure appears to represent the concept.
    • Expert Panel Validity: Experts evaluate the measure.
  • Criterion Validity:
    • Predictive Validity: Measure predicts future behavior.
    • Concurrent Validity: Measure correlates with similar measures.
  • Construct Validity:
    • Convergent Validity: Measure correlates with related variables.
    • Divergent Validity: Measure does not correlate with different variables.

Threats to Validity

• Internal Validity Threats: Placebo Effect, Hawthorne Effect, Maturation, Mortality, Researcher Threats.
• External Validity Threats: Ecological Validity.

Week 5: Limitations of Quantitative Research

• All Studies Are Wrong: How wrong? What are the limitations of the study? How much confidence should we have in the results?
• Robust Findings: Findings that can be consistently replicated.
• Quantitative Concerns: Ecological Validity
• Threats to External Validity: Testing Interaction, Selection Interaction, History Interaction.

Week 7: Research Ethics and REBs

• Technology: Makes large datasets easily searchable, scrapable, and analyzable.
• Ethical Considerations: Is it ethical to use this data?
• Informed Consent: Understanding Algorithms, Reading Terms of Service, Public Data, Participant Perspective
• Risk Minimization Strategies: Obtain informed consent, Delete names and identifiable information.

2. Association of Internet Researchers Guidelines

• Types of Research:
  • Big Data from Scrapes/API
  • Social Listening
  • Social Analytics
  • Building Dashboards
  • Algorithm Design
  • User Experience
  • Semantic/Sentiment Analysis
  • Digital Rhetoric
• Venue/Platform Considerations
• Legal Considerations
• Ethical Considerations
• Data Handling: (Data Management)
  • Management, storage, and representation of data.
  • Securing sensitive data (e.g., information on self-harm or criminal activity).
  • Anonymization -Ethical considerations of anonymizing data.
• Data Minimization: Collect and store only the necessary data necessary to answer research questions.
• Legal Considerations: Compliance, adherence to Terms of Service, Avoiding legal risks, Maintaining privacy
• Data Utility: Can the data answer relevant questions? Understanding the sample (data, users, etc.). Proprietary nature of social media and internet data limits generalizability for independent researchers.

3. Whose Ethics?

• Ethics Legislation: Historical Context; Tuskegee Syphilis Studies, The Stanford Prison Study, Milgram’s Obedience Studies

Week 8: Qualitative Approaches

• Theoretical Process: What are the theoretical processes?
• Observations: What observations support these concepts?
• Evolution of Concepts: Evolution of Concept
• Number of Concepts: How many concepts are there?
• Systematic Relationships: Are those concepts systematically related?
• Conceptual Density; Development: Are the concepts well-developed?

Variation

• Range of Conditions: Were the concepts observed under a range of different conditions?
• Theoretical Accounting: Is the variation accounted for by the theoretical concepts?

Substantive

• New Information: Does the study deliver new information?
• Guidelines for Action: Does it produce guidelines for action?
• Theory-Building: Are the results theory-building?
• Contribution: What has society or the discipline gained by having this researcher do this project?

Qualitative Coding

• Definition: Shorthand devices to label, separate, compile, and organize data.
• Interpretation: Breaking data down into component parts and naming those parts.
• Deductive Coding (from a developed codebook):
  • Development: from theory or extant research.
  • Process: often indicative of a more quantitative process.
• Inductive Coding (emergent from the data):
  • Source: Emergent coding from the data.
  • Development: Theory develops from the codes.
  • Process: often indicative of a more qualitative process.

Iterative Process

• Analysis: Takes place after some data has been collected, influencing further data collection

Stages of Coding

• Open Coding
• Axial Coding
• Selective Coding
• Code Books
• Explanations

Qualitative Approaches

• Qualitative Ontology: Multiple Realities.
• Purpose of Qualitative Research: Explain and Describe, Focus: Crafting plausible or credible explanations of social processes and human behavior

Common Methods

• Grounded Theory
• In-Depth Interviews
• Focus Groups
• Ethnography
• Autoethnography
• Content/Discourse Analysis

Qualitative Terms

• Saturation: Point at which no new information is being discovered.
• Member Validation: Participants validate the findings.
• Sensitizing Concepts: Initial ideas that guide the research.
• Iterative: Repeated cycles of data collection and analysis.
• Rich/Thick Descriptions: Detailed understanding, providing detailed and nuanced descriptions of the research context and findings.

Week 9: Interviews

• Definition: An interview develops a view of something between (inter) people (Brenner, 1985).
• Purposes of the Qualitative Interview

Interview Structures

• Structured Interview: (Fixed format)
• Semi-Structured Interview: (Specific topics but flexible)
• Unstructured Interview: (Conversational)

Interview Questions

• Interview Schedule: Variety of questions
• Chronological Order
• Broad to Specific (Funnel)
• Specific to Broad (Inverted Funnel)
• Types of Questions (Introducing, Direct, Indirect, Follow Up).

Focus Groups

• Definition

• Purpose (collaboration; deeper understanding)

• Focus Group Size (7-12 people)

• Focus Group Makeup (homogenous; members not knowing each other)

• Advantages

• Captures real-life data

• Flexible

• High face validity: Non-researchers can understand the results

Week 10: Ethnography

• Ethnography: Long-term immersion in a field site to write about cultures.
• Ethnography Components (ethno: culture + graph: something written).
• Researcher Roles (Participant? Observer? Positionality)
• Characteristics:
  • Time-Consuming
  • Ethical questions emerge
• Good Field Note Habits (Journaling, Allocating Time)
• Field Types & Notes (full and scratch notes, allocates time).

Access (in Ethnography)

• Preparation: How to access the site of the study and the community
• Questions to Consider: What is the goal? Are they a member? What resources are needed?
• Types of Access: Overt, Covert
• Overt: direct permission from group leaders/members.
• Covert: Don't tell participants about the research.

Week 11: Limitations of Qualitative Research

• Qualitative is not Quantitative
• Qualitative is Empirical
• Qualitative is NOT Generalizable (thick descriptions)

Qualitative Criteria

• Support for Conclusions: Does the process adequately support conclusions?
• Qualitative Reporting
• Theoretical Consistency
• Major Categories, Events, Incidents, Actions, Relationships
• Credibility Considerations
• Triangulation

Qualitative Research Methodology Summary

• Comparison across two or more forms of evidence.
• Can be with multiple sources, multiple methods, and/or multiple researchers.
• Negative Case Analysis (new data disconfirming previous explanations.)
• Member Validation (checking perceptions of findings; participants)
• Time Consuming
  • Field Time
  • Data Preparation (coding/etc).
• Balance
• Participant and Researcher Expertise
• Writing
• Trust
• Avoiding Exploitation: Important to avoid community exploitation.
• Audience-Dependent: The goal of qualitative research is ultimately audience-dependent.

Week 12: Transcending Quantitative and Qualitative Research

• Methods as Tools (question-driven)
• Analogy: "If all you have is a hammer, everything looks like a nail."
• Common Goals: Deeper knowledge of human behavior and social processes

Quantitative vs Qualitative Research

• Quantitative: Focus: Behavior, Approach: Deductive, Data: Numbers, Setting: Artificial
• Qualitative: Focus: Meaning, Approach: Inductive, Data: Words, Setting: Natural

Multi-Strategy Research

• Purpose: Use one type of research to corroborate findings from another type.
• Example: Follow up on survey data with qualitative interviews, focus groups, create possible survey items.
• Complementarity: Combining aspects of an investigation using different strategies (e.g., using survey methods to assess an experience and interviews to get more in-depth reactions).

Rich Understanding

• Rich Understanding: Understanding symbolic and cultural engagement in depth.