Research Design in IT

Research Design and Methodology

• Research design and methodology are explained within the domains of IT.

Typical Research Question

• Research questions often ask about frequency, proportions, percentages, and extents.
• Outcomes are often numeric values suitable for statistical analysis and comparisons.
• An example in computer science might involve testing algorithm effectiveness.

Empirical Observation and Objective Measurement

• Researchers might observe algorithm performance in controlled or real-world applications.
• Data collection would focus on efficiency, accuracy, and speed, comparing it to existing algorithms.
• Results are considered valid regardless of researcher biases.
• Scientific concepts and metrics are used to interpret data and express algorithm performance.
• Replicability is key; other researchers should be able to repeat the experiment and achieve similar results.

Methodologies and Understanding Social Phenomena

• Research incorporates various methodologies to understand social reality, including organizational function.
• Researchers interpret social aspects by examining documents (policies, minutes, rules).
• Interviews are used to gather data from individuals closely involved with the phenomenon.
• Analysis is influenced by participants' lived experiences (constructivist research).

Typical Interpretive Research Question

• Interpretive research often uses qualitative approaches.
• Questions generally start with "how," "why," "in which ways," or "to what extent."
• This type of research aims to discover, generate, explore, identify, or describe events, in an exploratory manner.

Example: Study on User Experience (UX)

• Interpretivism can be applied to study user interaction with technology and the effect of organizational culture on the adoption of IT systems.
• Qualitative data collection (in-depth interviews, focus groups) is used to understand user experiences with a new software interface.
• Contextual factors, such as user backgrounds and job roles, are considered.
• Data interpretation focuses on how individual users perceive and construct the meaning of software usability.

Pragmatism

• Pragmatism is problem-oriented, focused on understanding and solving specific problems.
• Researchers using pragmatism aren't restricted to a single methodology.
• They choose the methods best suited to gain understanding, drawing from quantitative (surveys, experiments) and/or qualitative (interviews, observations) realms.

Typical Pragmatic Research Questions

• Pragmatic research examples include increasing adult literacy, evaluating Online Distance Learning (ODL) effectiveness, boosting software developer satisfaction, and identifying effective incentives for developer productivity.
• The focus is on practical, applied research (what works) and solving the problem.

Example: Improving User Experience (UX) on a Social Media Platform

• Identifying the problem, such as declining user engagement.
• Using quantitative methods (e.g., surveys) initially to gather data on user engagement with platform features.
• Using qualitative methods (e.g., interviews, focus groups) to delve deeper into reasons for user preferences or dislike of certain features.
• Developing a prototype feature based on the findings.
• Using quantitative measures to test effectiveness.
• Using qualitative feedback for further refinements.
• Focus on tangible improvement in user engagement rather than confirming a specific theory.

Realism

• Realism in research aims to understand the true nature of the research problem, by exploring the reality of the phenomenon being studied.
• Realists often find the combination of quantitative and qualitative data insufficient for a deep understanding.
• Case studies are frequently used to delve deeper into specific problems.
• Realists aim to uncover the truth about the world, studying properties and processes as they exist.

Sample Critical Research Questions (Realism)

• Research questions, such as why Facebook owns content, impact of the internet on those unconnected, benefits of data disclosure, reasons behind the One Laptop Per Child program failure, impact of learning analytics on student vulnerabilities and right to privacy, and the true cost of MOOCs, are considered relevant examples under a realist approach.

Example: Al Algorithm Performance

• In computer science, a realist might study the actual performance of a new AI algorithm in real-world settings.
• Objective measurement is crucial, going beyond ideal laboratory conditions.
• In-depth case studies can explore how the AI algorithm interacts with other systems.
• Real-world applications can analyze the algorithm's impact on customer satisfaction and business workflows.

Methodology (Definition)

• Methodology encompasses the nature and structuring of a research process.
• It focuses on how methods, techniques, and principles are applied within a given study.
• Researchers select methods and approaches best suited to addressing the specific research problem.

Research Process Steps & Method Selection

• Identifying, selecting, collecting, processing, and analyzing data, to draw conclusions within a specific area.
• Considering whether a quantitative, qualitative, or mixed-methods approach is best for data collection and analysis, considering the research problem.

Research Design

• This structured plan lays out the necessary steps for generating knowledge, including defining the problem, forming hypotheses, and outlining analysis approaches.

Inclusion of Methodological Approaches

• The structure or design of a research study encompasses plans incorporating qualitative, quantitative, or mixed-method approaches.

Foundational Elements of Research Design and Methodology

• Research design is visualized in a generic diagram, incorporating key elements like philosophical assumptions, worldviews, research design, and methodologies.
• The elements guide researcher choices in IT research.

Conceptualization of Elements of Research Design

• This concept involves illustrating the key elements that influence research design in a conceptual diagram.

IT Methodology Thinking Framework

• Key concepts (philosophical assumption, methodology, research design, research collection methods) are interconnected within this framework that guides IT research thinking.

The Goal of Research Design

• Well-designed research should produce valid, objective/subjective, and meaningful answers to research questions.
• Sound research design addresses the research problem effectively and unambiguously, and the quality of the design and its effectiveness are evaluated critically.

Steps for Research Design

• Identifying the research problem clearly.
• If positivism guides the study, determining the type of positivism (descriptive, correlational, causal comparative, experimental or quasi-experimental).
• Identifying the research philosophy (positivist, interpretivist, pragmatic, realist) to guide the methodology (quantitative, qualitative, or mixed-methods)—and clarifying data collection methods (surveys, interviews, case studies, action research, etc.).
• Specifying appropriate methods for data collection, representation, and analysis (Step 6).

Types of Research Design

• This section primarily covers descriptive and experimental research designs commonly applied within IT research. Other examples (comparative, causal-comparative, correlational) are also introduced.
  • Descriptive Research Design aims to describe the nature and characteristics of specific phenomena (who, what, when, where, how).
  • Experimental Research Design allows for control over variables in order to determine (cause-and-effect) relationships.

Information Gathering and Limitations in Research

• Descriptive research gathers data on the phenomenon(s), focusing on specific variables or circumstances, but can't explain causes.
• This type of research often forms a foundation for more in-depth research endeavors.

Examples in IT Research

• Examples within cybersecurity (e.g., studying organizational attitudes toward cybercrimes, examining the insider factor in cybersecurity breaches) and computer science (analyzing how processors impact computer performance) are included.

Experimental Design

• Experimental design allows manipulation and control of variables.
• This design, often used to study (cause and effect), uses statistical methods to determine the relationships between variables.
• This design allows full control of independent variables, enables determination of cause-and-effect, and facilitates predictive analytics.
• Real-world experimentation is emphasized as critical in practical contexts. Applications in areas like AI tool effects, malware attack impacts, and customer support roles within AI are mentioned.

This quiz covers research design and methodology specific to the field of Information Technology. It includes topics such as formulating research questions, empirical observation, and objective measurement. Participants will explore how to assess algorithm performance and understand the importance of replicability in research outcomes.