Research Design and Methodology in IT

Research Design and Methodology

• Research design and methodology are explained within the domains of IT
• Research questions often involve "how many," "how frequently," "what percentage," "what proportion," and "to what extent"

Typical Research Question

• Outcomes are typically sets of numerical values statistically analyzed to draw inferences and comparisons.

Example

• In computer science, researchers might test the effectiveness of a new algorithm.
• Empirical Observation: Researchers could observe algorithm performance in controlled or real-world environments.
• Objective Measurement: Algorithm efficiency, accuracy, and speed are measured against existing algorithms.
• Independent of Researcher's Mind: Results of tests are valid regardless of researchers' preconceptions.
• Scientific Theories and Concepts: Established concepts and metrics guide data interpretation.
• Replicability: Other researchers should be able to replicate the experiment and achieve similar results.
• Research involves quantifiable, observable, and verifiable data.

Methodologies

• Techniques for understanding social reality, including organizational functions.
• Understanding Social Phenomena: Researchers interpret social aspects using documents like public policies or meeting minutes.
• Data Collection through Interviews: Gather data from people familiar with the phenomenon.
• Analysis Shaped by Experiences: Participant accounts heavily influence analysis and conclusions in constructivist research.

Typical Interpretive Research Question

• Interpretivists use qualitative thinking to discover, generate, explore, identify, or describe events.
• Qualitative questions commonly begin with "how," "why," "in which ways," or "to what extent."

Example (Interpretivism)

• In computer science, interpretivism might be applied to studying how people interact with technology, or how organizational culture impacts new IT system adoption.
• Qualitative Data Collection: In-depth interviews or focus groups gather rich, qualitative data about user experiences.
• Contextual Factors: Social and cultural contexts, like backgrounds or job roles, are considered during the study.
• Data Interpretation: Researchers aim to understand how users perceive and interpret the software’s usability.

Pragmatism

• Pragmatism in research focuses on understanding and solving specific problems.
• Researchers using pragmatism are not restricted to any one methodology; they choose the methods most effective for gaining understanding.
• Methods from both quantitative (surveys, experiments) and qualitative (interviews, observations) realms may be used.

Typical Pragmatic Research Question

• Example questions:
  • Increasing literacy in adult learners
  • Impact of online distance learning (ODL) on student satisfaction
  • Enhancing satisfaction among software developers in organizations
  • Effective incentives for increasing software developer productivity

Example (Pragmatism)

• Improve user experience on a social media platform.
• Identifying the Problem: A decline in user engagement is a problem.
• Quantitative Methods: An initial quantitative survey gathers data on how frequently users interact with platform features.
• Qualitative Methods: Qualitative interviews or focus groups follow identifying user preferences.
• Iterative Development: A prototype feature emerges from initial findings.
• Outcome Focused: The goal is a tangible improvement in user engagement.
• Adaptive Methods: Methods are adjusted throughout the process according to practical results.

Realism

• Focuses on the true nature of the research problem.
• Realists believe that positivist and interpretivist approaches may not fully grasp the research problem.
• Pragmatic combinations of methods often aren't sufficient in achieving deep understanding.
• Realists use constructivist case studies to gain deep insight.
• Aims to uncover truth by studying properties and processes as they exist in reality.

Sample Critical Research (Realism) Questions

• Questions concerning issues with Facebook content ownership.
• Marginalization of unconnected users due to the internet power.
• Benefits and drawbacks surrounding data disclosure.
• Reasons for the failure of the One Laptop per Child project.
• Exploiting vulnerabilities and rights to privacy through learning analytics.
• Impact of massive open online courses (MOOCs) and online education in exposing learners to more educational failure.

Example (Realism)

• Studying the real-world performance of a new AI algorithm.
• Objective Measurement: AI algorithm performance is measured in various realistic scenarios.
• In-depth Case Studies: Case studies investigate how the AI interacts with other systems.
• Real-World Applications: Actual customer satisfaction and workflows in real-world business environments are examined.

Methodology

• Methodology refers to the nature and structure of the research process, encompassing how specific methods, principles, and techniques are applied in a study.
• Research Process Steps: Identifying, selecting, collecting, processing, analyzing data, and drawing conclusions in a particular field of study.
• Consideration in Method Selection: When choosing a methodology, researchers must consider the nature of the research problem.

Research Design

• A structured plan that details steps for generating knowledge, defining the research problem, formulating hypotheses, and developing an analytical framework.
• Involves selecting appropriate methods for data collection, presentation, and analysis.
• Inclusion of Methodological Approaches: Encompasses action plans for qualitative, quantitative, or mixed-methods approaches.

Foundational Elements of Research Design and Methodology

• Research design can be illustrated visually.
• Five elements guide a researcher: Philosophical Assumptions, Worldviews, Research Design Strategy, Research Methods , and Methodology

Conceptualization of Elements of Research Design

• Positivism, Interpretivism, Pragmatism, Realism are influences on Research Design, impacting Design Strategy and Methods used for Data Collection. Influences from global, organizational, technological, process and people factors make up Context. Analysis and Interpretation lead to Collection.

IT Methodology Thinking Framework

• Philosophical assumptions (Positivist, Interpretivist, Pragmatic, Realist) guide mixed, quantitative, and qualitative research methodologies, influencing research design (e.g., descriptive, correlational, experimental).
• Research Methods (Data Collection, Data Analysis, Measurements, Interpretation, Validation, and Utilization) are detailed.

The Goal

• The goal of the research proposal is to ensure suitability of the research design for obtaining valid, meaningful answers to research questions.
• Sound research design ensures effective evidence-gathering that unambiguously addresses the research problem.

Steps for Research Design

• Identifying the research problem, and the design type (positivism, interpretivism, pragmatism, or realism).
• Selecting the Research Philosophy to tailor to the selected methodology.
• Clarifying Research Methods (e.g., quantitative survey, case study, action research).
• Evaluating the Research Design.

Types of Research Design

• Various types: Descriptive, Experimental, Comparative, Causal-Comparative, Correlations designs.
• Further details can be read in McCombes & Bhandari (2021-2022), a guide with examples.

Descriptive Research Design

• Purpose: Answers basic questions ("who," "what," "when," "where," and "how") regarding a phenomenon.
• Gathering data on a phenomenon's characteristics and status.
• Limitations: Cannot definitively explain causes.
• Preliminary step for quantitative research.

Experimental Design

• Control of variables affecting outcomes.
• Determining cause-and-effect relationships using statistical methods.
• Ideal for examining likelihood of independent variables impacting dependent variables.
• Can occur within/outside of laboratory settings.

Examples of Experimental Studies

• Examples of areas appropriate for this type of research: Software development impact of AI tools, impact of malware on individuals, artificial intelligence applications in retail settings.

Experimental Design Advantages

• Setting limits for experiment, controlling independent variables to understand 'what causes the occurrence of an event', determining the dependent variable based on independent variables, distinguishing effects, and determining deductive analysis and predictive analysis.

Comparative Study Design

• Comparing objects, concepts, or parameters to understand relationships.
• Simple application: comparing two solutions.
• Examples: comparing Agile and Spiral Software Development, ranking antivirus software, analyzing the effectiveness of learning methods.

Causal-Comparative/Quasi-Experimental Design

• Aims to identify cause-and-effect relationships among variables.
• Unlike true experiments, quasi-experiments do not rely on random assignment to groups.
• Examples: banking customers with prior cybercrimes in cybersecurity research, etc.

Continue

• Application with malware: Studying malware impact using honeypot systems.
• Use in Software Engineering and Cybersecurity: Observing behavioral differences in usability studies.

Correlational Research Design

• Identifying and understanding the extent of relationships between variables using statistical data.
• Focus on trends, patterns, and relationships in data, and interpreting interconnections.
• Variables studied in their natural state as researchers don't manipulate or control them.

Examples of correlational design

• Examining the relationship between insider threats and cybersecurity breaches.
• Examining the association between educational levels and ease of information system use.
• Analyzing the correlation between available data and artificial intelligence performance.

Research Methodology

• Systematic approach for finding explanations for a phenomenon.
• Deals with research strategy and approach.
• Research Methods: Specific approaches, procedures, or techniques used for data identification, selection, processing, and analysis to draw conclusions.
• Impact on Validity and Reliability: Broader methodology and specific methods affect the validity and reliability of research studies.

Three Main Methodological Issues

• Data collection/generation, analysis, and conclusion drawing

Three Main Methodologies

• Quantitative, Qualitative, and Mixed Methods

Quantitative Methodology

• Emphasizes objective measurements and numerical data.
• Ensures: repeatability, transparency, and credibility.
• Structured interviews focus on measurable questions.

Continue

• Hypothesis testing: Designing and testing hypotheses using mathematical, computational, and statistical means to draw quantifiable inferences.
• Statistical Analysis: Often involves statistical analysis of numerical data and interpreting data.
• Focus on Measurable Quantity.
• Examples: questions like "how many?", "how frequently?", and "what proportion?"
• Data Collection: Types of Quantitative Data (discrete/continuous), Examples (counting security breaches, measuring physical attributes), Quantitative Data Collection Techniques (experiments, probability sampling, etc.).

Data Analysis

• Extracting information, drawing inferences, and presenting data in a clear, understandable format.
• The stage is pivotal to ensure meaningful results and build a foundation for informed decision-making.
• Common analysis techniques.

Data analysis techniques for Quantitative data

• Trend Analysis: examining and interpreting changes over time.
• Total Unduplicated Reach and Frequency (TURF): analyzing market reach and combinations.
• SWOT Analysis: evaluating strengths, weaknesses, opportunities, and threats.
• Gap Analysis: comparing expected and actual results to identify areas of improvement.
• Verification and Validation: evaluating research data, findings to ensure correctness.
• Software tools support data interpretation and conclusion drawing relevant to the research question (e.g. heatmaps.)

Qualitative Approach

• Focus on non-numerical data.
• Understanding individual perceptions.
• Narrative data use (rather than numerical).
• Inductive reasoning approach (explanations are drawn from data).
• Interpretive nature to explore human perceptions.
• Descriptive/Interpretive methodology to understand economic events.
• Analyzing and understanding phenomena from the perspectives of key informants.

Qualitative Data Collection Methods

• Various methods: documents, observation, participant observation, interviews, focus groups, and visual or photo essays.
• Semi-structured interviews allow respondents to express their views freely and provide detailed responses.
• Data collection methods support gain of deep insights into how individuals perceive their social environment.

Qualitative Data Analysis

• Data analysis is complex, involving the derivation and construction of interpretations using techniques like:
• Stages of Coding: Open coding, axial coding, and selective coding.
• Data is non-numerical, relying on perceptions, interpretations, and lived experiences of key informants.
• Approach to research acknowledges its subjective nature and that it may not always represent single, definitive reality.

Mixed-Methods Approach

• Combining quantitative and qualitative research methods for investigating problems where each method may be insufficient in isolation.
• Data Triangulation: applying both approaches for enriched analysis
• Complementary Methods: quantitative methods for measurable information against qualitative's perception-based information.

Benefits of Mixed Methods

• Expansion: building upon prior research
• Enrichment: combined measurable and unmeasurable aspects
• Complementarity: combining illustration and explanation of phenomena.
• Repositioning ideas: revealing contradictions and leading to new perspectives.
• Methodological development: enhancing one methodology based on the outcome of another.

Application in IT Studies

• Applicable to multidisciplinary tasks, like requirements engineering or service quality in institutional settings.
• Examples include: studies on how agile software development impacts user experience in software products.

Quantitative Component

• Data Collection: surveys among software developers and project managers.
• Data Analysis: statical methods analyzing survey results (e.g., correlations).

Qualitative Component

• Data Collection: in-depth interviews with software users and UX designers.
• Data Analysis: thematic analysis of interview transcripts.

Mixed-Method Integration

• Triangulation: comparing and contrasting findings for a comprehensive understanding of the impact.
• Complementarity: quantitative for general trends and efficacy, qualitative providing in-depth user experiences.

Research Output and Application of Findings

• Multidimensional analysis yielding insightful findings about Agile practices, human-centric experiences, and potential best practices.
• Information will enhance user experience, and balancing technical efficiency.

Concluding Note

• Researchers encouraged to tailor research methodology to their specific research problem and questions.
• Selecting a research philosophy and methodology, then detailing data collection and analysis processes.

This quiz explores the principles of research design and methodology specifically within the context of IT. It covers various research questions and how numerical data can be analyzed to draw conclusions about algorithm effectiveness and performance. Test your understanding of empirical observation and scientific methods used in IT research.