Research Design and Methodology PDF

Research Design and Methodology In this chapter, research design and methodology are explained within the domains of IT Typical Research Question How many? How frequently? What percentage? What proportion? To what extent? The outcome is usually a set of numeric values that...

Research Design and Methodology In this chapter, research design and methodology are explained within the domains of IT Typical Research Question How many? How frequently? What percentage? What proportion? To what extent? The outcome is usually a set of numeric values that can be statistically analyzed to draw inferences and comparisons. Example In computer science, might be study where researchers are testing the effectiveness of a new algorithm. For example: Empirical Observation: Researchers could empirically observe the performance of the algorithm in a controlled environment or in real-world applications. Objective Measurement: They would collect data on its efficiency, accuracy, and speed compared to existing algorithms. Independent of Researcher's Mind: The results of these tests would be considered valid irrespective of the researchers' preconceived notions or expectations about the algorithm's performance. Scientific Theories and Concepts: They would use established scientific concepts and metrics to interpret the data and to express the nature of the algorithm's performance. Replicability: Other researchers should be able to replicate the experiment under the same conditions and achieve the same results, which is a key principle. So, in a computer science context, involves conducting research in a manner that is quantifiable, observable, and verifiable by others.  Methodologies: incorporates a range of methodologies, tools and techniques aimed at understanding social reality, including how organizations function Understanding Social Phenomena: Researchers interpret the social aspects of phenomena by reviewing documents like public policies, meeting minutes, and institutional rules Data Collection through Interviews: Interviews are conducted to gather data from individuals who are closely acquainted with the phenomenon being studied. Analysis Shaped by Experiences: The analysis and conclusions in constructivist research are heavily influenced by the participants’ accounts of their lived experiences. Typical Interpretive Research Question The interpretivist usually applies qualitative thinking The requirement is to: o ‘discover’, ‘generate’, ‘explore’, ‘identify’ or ‘describe’ events in an exploratory manner. Qualitative research questions generally start with: o ‘how’, ‘why’, ‘in which ways’, ‘to what extent’ Example In computer science, interpretivism might be applied in the study of how people interact with technology or how organizational culture affects the adoption of new IT systems. Here's an example: Study on User Experience (UX): A researcher might want to understand how users perceive and interact with a new software interface. Instead of just measuring click rates or error rates (as in positivist research), an interpretivist study would delve into users' subjective experiences. Qualitative Data Collection: The researcher conducts in-depth interviews or focus groups with users to gather rich, qualitative data about their experiences, thoughts, and feelings regarding the software. Contextual Factors: The study would consider the social and cultural context of the users, such as how their backgrounds or job roles influence their interaction with the software. Data Interpretation: Instead of seeking objective truths, the researcher interprets the data to provide insights into how different users uniquely perceive and construct the meaning of the software's usability. Pragmatism Problem-Oriented Approach: Pragmatism in research is centered around understanding and solving specific problems. Researchers using pragmatism are not restricted to any one methodology; they choose whatever methods are necessary to gain understanding. Researchers may use several methods either from the quantitative realm (like surveys and experiments) or qualitative realm (like interviews and observations). Typical Pragmatic Research Question 1. What can be done to increase literacy of adult learners? 2. Does ODL (Online Distance Learning) increase student satisfaction and completion rates? 3. How to increase software developers satisfactions in an organization? 4. What incentives are effective for encouraging software developers to be more productive ? It focuses on practical applied research: ‘what works’. It focuses on solving the problem. Example In computer science, an example of pragmatism could be a study that aims to improve user experience on a social media platform: Identifying the Problem: The researcher begins by identifying a problem, such as a decline in user engagement on the platform. Quantitative Methods: Initially, a quantitative survey might be conducted to gather data on how frequently users engage with different features of the platform. Qualitative Methods: Subsequently, qualitative interviews or focus groups might be used to delve deeper into the reasons behind users' preferences or aversions to certain features. Iterative Development: Based on these findings, the researcher might develop a prototype of a new feature and then use quantitative measures to test its effectiveness, followed by qualitative feedback for refinement. Outcome Focused: The ultimate goal of the research is to create a tangible improvement in user engagement, rather than to confirm a specific theoretical perspective. Adaptive Methods: Throughout the process, the researcher remains open to using new methods or altering the research design based on what is most practical for solving the problem at hand. Realism Realism in research focuses on understanding the true nature of what is real with respect to the research problem. Realists find the data from positivist (quantitative) and interpretivist (qualitative) approaches to be insufficient for a deep understanding of research problems. Realists believe that even the pragmatic combination of methodologies does not go far enough in depth to understand the research problem fully. Realists often select constructivist case study methods to gain a deep insight into a problem. Realists aim to uncover the truth about the world by studying the properties and processes of things as they exist in reality. Sample Critical Research (Realism) Questions Why does Facebook own all the content that we supply? Does the power of the net further marginalize the non- connected? Who benefits from data disclosure? Why did the One Laptop Per Child fail? Does learning analytics exploit student vulnerabilities and right to privacy? Are MOOCs (Massive open online course) really free? Who owns and for what use are learning analytics? Does Online education only expose learners to more educational failure? Realism is regarded as metaphysical (aiming to understand the relationship between the mind and the physical universe) Example In the context of computer science, a realist might be interested in studying the actual performance and effects of a new artificial intelligence (AI) algorithm in practical, realworld settings. Here’s how realism could be applied: Study on AI Algorithm Performance: A computer science researcher may want to understand the true performance of an AI algorithm in various conditions. Objective Measurement: The realist would objectively measure the performance of the algorithm, not just under ideal laboratory conditions but also in real-world applications where variables are less controlled. In-depth Case Studies: The researcher might conduct in-depth case studies on how the AI algorithm interacts with other systems and affects them, aiming to reveal a comprehensive understanding of the AI’s functionality and its implications. Real-World Applications: For instance, studying how an AI system for automated customer service affects actual customer satisfaction and company workflows in a real business environment. Methodology Definition of Methodology: Methodology refers to the nature and structure of the research process, encompassing how specific methods, principles, and techniques are applied in a study. Research Process Steps: It involves identifying, selecting, collecting, processing, analyzing data, and drawing conclusions in a particular field of study. Consideration in Method Selection: When choosing a methodology, researchers must consider the nature of the research problem to decide whether a quantitative, qualitative, or mixed-methods approach is most suitable for data collection and analysis. Research Design Research design provides a structured plan detailing the steps necessary to generate knowledge, including defining the research problem, hypotheses, and analytical framework. It involves selecting appropriate methods for data collection, presentation, and analysis. Inclusion of Methodological Approaches: The design encompasses action plans for qualitative, quantitative, or mixed-methods approaches. Foundational Elements of Research Design and Methodology Research design can be represented visually, as exemplified by the generic diagram shown in Figure IT Research Design The next figure shows five elements of research design that can guide a researcher in the fields IT. Conceptualization of Elements of Research Design IT Methodology Thinking Framework The goal The goal is to present a proposal that ensures that the research design is suitable to obtain valid, objective/subjective and meaningful answers to the research questions since research is about creating meaning from the relative chaos of ideas and data. Only sound research design can ensure that the evidence-gathering will address the research problem effectively, in an unambiguous way. Reviewers and examiners take great care in considering the quality of research design and its effectiveness in addressing the research problem. The following steps can be used to clarify the methodology and frame the research design and methods: Steps for research design Step 1: Since you have identified the research problem clearly. Step 2: If the research philosophy is positivist, then decide what type of positivism applies, for example descriptive, correlational, causal comparative, experimental or quasi-experimental. Step 3: Selecting a research philosophy will help to clarify the research methodology: If you chose a positivist research philosophy, then the methodology will be quantitative or mixed methods. If you chose an interpretivist research philosophy, then the methodology will be qualitative. If you chose a pragmatist research philosophy or a realist philosophy, then the methodology will be qualitative or mixed methods.  Step 4: The next step is to clarify the detail of the research methods, deciding whether to adopt a quantitative survey, a design and creation method, a qualitative survey, a case study or action research. Step 5: With respect to data collection methods, descriptive research may adopt a quantitative survey design, while a constructivist case study design could include interviews, focus groups, documentary review and observation (or some of these). For further guidance on questionnaires, interviews and focus groups, read Williamson (2018). Step 6: The final step in research design will include deciding on the methods for evaluating the design (Oates, 2006) prior to submission to critical readers/assessors. At this juncture, the researcher should appreciate that the research problem determines the nature of the research design, not vice versa. The understanding and knowledge of the problem is central to the choice of the approaches to data collection and data analysis. Types of Research Design The section primarily discusses research designs commonly used in IT research. 1. Descriptive Research Design 2. Experimental Design 3. Comparative Study Design 4. Causal-Comparative/Quasi-Experimental Design 5. Correlational Research Design For more details read : McCombes, S., & Bhandari, P. (2021–2022). Research design: A step-by- step guide with examples. https://www.scribbr.com/methodology/research- design/ Descriptive Research Design Purpose of Descriptive Research Design: This design is used to answer basic questions about who, what, when, where, and how, helping to describe the nature and characteristics of a specific phenomenon. Information Gathering: It enables researchers to collect data on the status of the phenomenon, focusing on specific variables or circumstances. Limitation in Explaining Causes: While descriptive research provides detailed information, it does not offer insights into why a phenomenon occurs. Foundation for Further Research: This type of research is often a preliminary step that informs further quantitative research by identifying important variables for hypothesis testing. Examples in Cybersecurity: In cybersecurity, descriptive research might involve studying organizational attitudes towards global cybercrimes or examining the insider factor in cybersecurity breaches. Example in Computer Science: An application in computer science could be analyzing how different processors affect computer performance. Experimental Design Experimental design allows researchers to control variables that might affect the outcome of an experiment. Establishing Cause-and-Effect: It employs statistical methods to determine the cause-and-effect relationships between variables in a study. Suitability for Studying Relationships: This design is ideal for examining the likelihood that independent variables consistently affect dependent variables, thereby assessing the strength of the relationship. Not Limited to Lab Settings: While often associated with laboratory work, a true experiment can occur outside a lab setting. It's characterized by the manipulation and control of independent variables to observe their effects. Real-world Experimentation: In practical applications, experiments must involve controlled, randomized, and manipulated measurements to ensure accuracy and reliability. Examples of Experimental Studies: Some examples of experimental study may include the following: 1. The effect of AI tools on software development. 2. The effect of malware attacks on individuals. 3. The effect of artificial intelligence on supporting customers in a retailing shop. 4. It can be used to substantiate algorithm theories, network theories, memory performance assumptions, AI theories, cyber risk theories, user experience theories, etc. Experimental design has the following advantages: It helps the researcher to set the limits of the experiment by scoping the boundaries of the study using independent variables. The researcher is in full control of the independent variables, which allows the researcher to address the question, ‘what causes an event to occur?’ It helps the researcher to determine the dependent variables based on the independent variables. This design allows the researcher to determine cause-and-effect relationships between variable factors and to differentiate dummy effects from actual effects. It enables the researcher to determine what has happened (deductive analysis) or predict what may happen (predictive analytics). Comparative Study Design This design is used to understand the relationship between two or more objects, ideas, or parameters by observing and comparing them analytically. A simple application could involve comparing two solutions to determine which is better. Example in Software Engineering: Comparing between Agile and Spiral Software Development models. Example in Computer Science: For instance, comparing two algorithms solving the same problem to see which has superior performance based on certain attributes. Cybersecurity Context: In cybersecurity, this might involve studying and ranking different types of antivirus software based on predefined performance criteria. Practical Example: An example of a comparative study could be investigating the effects of accelerated computational skills training versus conventional learning methods for college students.  Causal-Comparative/Quasi-Experimental Design This design aims to identify the cause-and-effect relationships among variables. Difference from True Experiments: Unlike true experiments, quasi experiments do not use random assignment to experimental and control groups; participants are assigned based on non-random criteria. Example in Cybersecurity: For instance, in a cybersecurity study, banking customers with prior cybercrime experiences might be assigned to an experimental group, while those without such experiences are placed in a control group. Continue … Application Example with Malware: For example, the impact of malware on systems could be studied using a 'honeypot' system instead of a live production environment. Use in SE and CY Behavior Studies: In Software Engineering (SE) and cybersecurity (CY), quasi-experimental design can be used to observe behavioral differences in usability studies, particularly when examining psychological factors among different user groups. Correlational Research Design This design focuses on identifying and understanding the extent of the relationship between two or more variable factors using statistical data. Focus on Trends and Relationships: The primary interest is in observing trends, patterns, and relationships in the data, as well as interpreting the interconnections among these variables. In correlational studies, variables are not manipulated or controlled by the researcher; instead, they are identified and studied in their natural state. Examples of correlational design include the following: 1. The relationship between insider threats and cybersecurity breaches 2. The relationship between the level of education and users’ ease of use of information systems 3. The relationship between the amount of data available and artificial intelligence performance In all these cases, statistical analysis techniques are used to calculate the relationship between the identified variables, and often a correlation coefficient governs the association between the variables, where the value lies between −1 and +1 and where the coefficient of +1 depicts a positive relationship between the variables and −1 indicates a negative relationship between the variables. Research Methodology Research Methodology: According to Mkansi & Acheampong (2012) and Sileyew (2019, Chapter 3), research methodology is a systematic approach to finding explanations for a phenomenon. It deals with the overall strategy and approach of the research. Research Methods: These are specific approaches, procedures, or techniques used for identifying, selecting, processing, and analyzing data to draw conclusions about a research problem. Impact on Validity and Reliability: Both the broader methodology and the specific methods chosen affect the validity and reliability of a research study. Three Main Methodological Issues: Methodology encompasses: 1. how data will be collected or generated, 2. how it will be analyzed, 3. and how conclusions are drawn. Three Main Methodologies: The primary research methodologies are: 1. quantitative, 2. qualitative, 3. and mixed methods. There is a wide range of specific research methods within these methodologies. Quantitative Methodology The quantitative approach emphasizes objective measurements and numerical analysis of data collected through documentation, surveys, and questionnaires. Attributes of Quantitative Research: This approach ensures repeatability, transparency, and credibility in studies. Structured Interviews: When interviews are used in quantitative research, they focus on measurable questions rather than exploring reasons or motives. Continue … Hypothesis Testing: It involves designing and testing hypotheses using mathematical, computational, and statistical means, suitable for drawing quantifiable inferences through deductive reasoning. Statistical Analysis: Quantitative research often involves statistical analysis and the interpretation of numeric data, including intervals, ratios, percentages, and the use of graphs or diagrams for results explanation. Focus on Measuring Quantity: This paradigm is concerned with measuring quantities and is framed with questions like "how many?", "how frequently?", "what proportion?", or "to what extent?" Data Collection Types of Quantitative Data: In quantitative studies, data collected must be either discrete or continuous. Discrete data consists of finite numbers and constant values, while continuous data includes values with fractions or decimals. Examples of Data Types: For example, counting the number of security breaches in financial institutions would yield discrete data (whole numbers), whereas measuring physical attributes like height, weight, age, or distance results in continuous data. Quantitative Data Collection Techniques: Techniques for gathering quantitative data include experiments, probability sampling, methodical observations, longitudinal studies, surveys, polls, questionnaires, interviews, and reusing data from statistical reports. Data Analysis The primary goal is to extract evidential information, draw inferences, and present outcomes in a clear, understandable manner. Importance of Data Analysis: This stage is crucial for ensuring that the research results are meaningful and can serve as a foundation for informed decision-making. The following are popular quantitative data analysis techniques  quantitative data analysis techniques Trend Analysis: This is a statistical technique that examines quantitative data over time to identify patterns and changes in events. It helps understand how changes in independent variables affect the dependent variable. Total Unduplicated Reach and Frequency Analysis (TURF): TURF analyzes the potential market reach of various product and service combinations. It provides insights into potential market reach and suggestions for enhancing product performance or consumer appeal. SWOT Analysis: This method can be quantitative or qualitative and involves assigning numerical values to strengths, weaknesses, opportunities, and threats. It's used for comparison and prediction of comparative advantages for events, products, or services. Continue … Gap Analysis: This involves using a matrix to present quantitative data that measures the variance between expected and actual results. It's suitable for assessing performance gaps or service quality and suggests ways to bridge these gaps. Verification and Validation: Part of the research process includes verifying and validating the analysis. Verification involves reviewing the data analysis against the initial design concepts and theories to see how they contribute to answering the research question. Validation might require external evaluation to confirm the correctness of the research process, findings, and analysis. Software Tools for Data Interpretation: Various software tools assist researchers in interpreting data and making meaningful conclusions relevant to their research questions. Such as heat map Qualitative Approach Focus on Non-Numerical Data: Qualitative research primarily deals with unstructured or semi-structured data that is not numerical in nature. Understanding Individual Perceptions: It aims to understand the experiences of a specific population based on the individual perceptions or words of its members. Narrative Data Use: This approach utilizes narrative data rather than numeric data. Inductive Reasoning Approach: Qualitative research employs inductive reasoning, where explanations are derived from the data, often based on documentary, observational, or narrative evidence. Exploratory Nature: It is exploratory in nature, seeking to provide insights into how people or groups understand various aspects of the social world, including the economy. Descriptive and Interpretive: The research is descriptive or interpretive, focusing on events that can be observed but not quantified. Interpretation of Phenomena: Researchers in this field strive to interpret or understand the phenomena as perceived or experienced by people, particularly key informants. Data Collection Diverse Data Collection Methods: Qualitative data can be gathered through various methods including documents, observation, participant observation, interviews, focus groups, and visual or photo essays. Use of Semi-Structured Interviews: A common approach in qualitative research is to use semi-structured interviews with open-ended questions, allowing respondents to express their views freely. Encouraging In-Depth Responses: Respondents in these interviews are encouraged to provide detailed answers in their own words. Gaining Deep Insights: The results from qualitative data collection methods provide a profound understanding of how individuals perceive their social environment. Visual Aid for Methods: next Figure illustrates the possible data collection methods applicable in a qualitative research paradigm. Qualitative Data Collection Methods Data Analysis Data analysis in qualitative research is complex, involving the derivation and construction of interpretations using various techniques. Techniques include coding qualitative data into categories and subcategories, identifying patterns, and understanding interconnections. Stages of Coding: The analysis typically starts with open coding (assigning initial codes), followed by axial coding (creating connections based on initial codes), and then selective coding (identifying a main category and relating other categories to it). The data in qualitative research is non-numerical, relying on perceptions, interpretations, and lived experiences of key informants. This research approach is subjective and acknowledges that it does not represent a single, definitive reality. Mixed-Methods Approach Mixed-Methods Approach: This approach combines quantitative and qualitative methods in a single study, used when neither method alone is sufficient to investigate a problem. triangulation of Data: Mixed-methods research enables triangulation, enriching study results by applying both quantitative and qualitative methods. Complementary Data Collection and Analysis: Quantitative methods provide close-ended, measurable information, typically analyzed mathematically, while qualitative methods offer open-ended, perception-based information. Benefits of Mixed Methods: Expansion: It allows researchers to build upon previous research using new mixed methods. Enrichment: By combining measurable and unmeasurable aspects, mixed methods enrich the understanding and reliability of the research findings. Complementarity: The approach provides both illustration (quantitative) and explanation (qualitative) of phenomena. Repositioning Ideas: Mixed methods can uncover contradictions between quantitative and qualitative results, leading to new perspectives. Methodological Development: The approach uses the outcome of one methodology to enhance the development of the other. Application in IT Studies: Mixed-methods are suitable for multidisciplinary of IT studies in for tasks like: 1. requirements engineering or 2. understanding service quality in institutional settings. Example of a Mixed-Methods Approach in Software Engineering Research: Studying User Experience (UX) in Agile Software Development Research Objective: To evaluate how Agile software development practices impact the user experience (UX) of software products.  Quantitative Component: Data Collection: Conduct a survey among software developers and project managers involved in Agile projects. The survey includes closed-ended questions rating their satisfaction with Agile methodologies, efficiency in meeting deadlines, and perceived quality of the final product. Data Analysis: Use statistical methods to analyze the survey results, focusing on correlations between Agile practices and project success metrics, such as time to market, bug frequency, and client satisfaction scores. Qualitative Component: Data Collection: Perform in-depth interviews with a select group of software users and UX designers who have interacted with products developed through Agile methodologies. Focus on open-ended  questions to explore their subjective experiences, perceptions of usability, and satisfaction with the software features and design. Data Analysis: Employ thematic analysis on the interview transcripts to identify common themes, perceptions, and experiences related to UX in Agile-developed software. Mixed-Methods Integration: Triangulation: Compare and contrast the findings from both quantitative and qualitative components to provide a comprehensive understanding of the impact of Agile practices on UX. Complementarity: Use quantitative data to illustrate the general trends and efficacy of Agile methodologies, while qualitative data offers deeper insights into the user experiences and practical implications of these trends.  Research Output: The mixed-methods approach will result in a multidimensional analysis of Agile practices in software engineering, highlighting both the measurable outcomes and the human-centric experiences of UX in Agile environments. Application of Findings: The study could inform best practices in Agile software development, particularly in areas related to enhancing user experience and balancing technical efficiency with user-centric design principles. Concluding Note: Researchers are encouraged to develop their own research philosophy, design, and methodology tailored to their specific research problem and question. This involves selecting a research philosophy and methodology, then detailing data collection and analysis processes.