Sampling Methods PDF

SAMPLING METHODS Sampling methods are techniques used to select a subset of individuals or items from a larger population for the purpose of data collection and analysis. Different sampling methods are employed based on the research objectives, population ch...

SAMPLING METHODS Sampling methods are techniques used to select a subset of individuals or items from a larger population for the purpose of data collection and analysis. Different sampling methods are employed based on the research objectives, population characteristics, and available resources. Here are some common sampling methods: 1. Simple Random Sampling (SRS): In SRS, every individual or item in the population has an equal chance of being selected for the sample. This method is straightforward and easy to implement, ensuring that each member of the population has an equal probability of inclusion. Example: Suppose we want to conduct a survey to estimate the average income of households in a city. We assign each household a unique number and use a random number generator to select a sample of households to survey. 2. Stratified Random Sampling: Stratified sampling involves dividing the population into homogeneous subgroups called strata based on certain characteristics (e.g., age, gender, income level) and then selecting a random sample from each stratum. This method ensures representation from each subgroup, allowing for more precise estimates for each stratum. Example: A market researcher wants to study consumer preferences for a new product. They first divide the population into age groups (e.g., 18-25, 26-35, 36-45) and then randomly select a sample from each age group. 3. Systematic Sampling: Systematic sampling involves selecting every kth individual from the population after a random start. The sampling interval (k) is calculated as the population size divided by the desired sample size. Systematic sampling is efficient and easy to implement, but it may introduce bias if there is a periodic pattern in the population. Example: A university wants to survey students about their satisfaction with campus facilities. Instead of surveying all students, they select every 10th student from the student directory after selecting a random starting point. 4. Cluster Sampling: Cluster sampling involves dividing the population into clusters or groups, randomly selecting some clusters, and then sampling all individuals within those selected clusters. This method is practical for large and geographically dispersed populations, as it reduces costs and logistical challenges. However, it may lead to increased variability if clusters are not homogeneous. Example: A health researcher wants to assess the prevalence of a disease in a region. They first divide the region into districts, randomly select some districts, and then test all individuals for the disease within the selected districts. 5. Convenience Sampling: Convenience sampling involves selecting individuals or items that are readily available and accessible to the researcher. This method is convenient and time- efficient but may introduce bias because it does not ensure representativeness of the population. Example: A researcher conducting a study on smartphone usage in a shopping mall approaches shoppers and surveys those who are willing to participate. 6. Snowball Sampling: Snowball sampling involves initially selecting a few individuals who meet the criteria for the study and then asking them to refer other potential participants. This method is useful for studying hard-to-reach or hidden populations, but it may result in biased samples if referral chains are not representative of the population. Example: A researcher studying rare medical conditions starts by identifying a few patients with the condition and then asks them to refer other patients they know who have the same condition. 7. Purposive Sampling: Purposive sampling involves selecting individuals or items based on specific criteria or characteristics relevant to the research objectives. This method is suitable for targeted or specialized populations but may lack representativeness and generalizability. Example: If the researcher would want to focus on all diabetic persons, then he should get individuals diagnosed with the rare medical condition by medical professionals. 8. Quota Sampling: Quota sampling involves selecting individuals based on pre-defined quotas or proportions for certain characteristics (e.g., age, gender, occupation) to ensure representation from different segments of the population. This method is flexible and allows for control over sample composition but may not fully represent the population if quotas are not set appropriately. Example: The interviewer approaches a 30-year-old individual in a shopping mall. If the quota for the 26-40 age group has not been met, the individual is invited to participate in the survey. However, if the quota for the 26-40 age group has already been filled, the interviewer may politely decline participation from the 30-year-old individual and continue approaching other potential participants until the quota is met These sampling methods offer various advantages and limitations, and the choice of method depends on factors such as the research objectives, population characteristics, budget, and time constraints. Researchers should carefully consider the strengths and weaknesses of each method to ensure the validity and reliability of their study findings. METHODS OF DATA COLLECTION Data collection is a critical step in the research process, involving the gathering of information or data for analysis and interpretation. There are various methods of data collection, each with its advantages, limitations, and suitability for different research objectives. Let's explore some common methods of data collection: 1. SURVEY. Surveys involve gathering information from individuals through the administration of structured questionnaires or interviews. Surveys are cost-effective, allow for standardized data collection, and can reach a large number of participants quickly. Limitation of using this method is that surveys may suffer from response bias, low response rates, and difficulty in capturing nuanced or qualitative information. 2. INTERVIEWS: Interviews involve direct interaction between a researcher and a participant to gather information through structured, semi-structured, or unstructured questioning. Interviews allow for in-depth exploration of topics, clarification of responses, and the collection of qualitative data. Limitation of using this method is that interviews can be time- consuming, resource-intensive, and may be influenced by interviewer bias or social desirability bias. Structured, semi-structured, and unstructured interviews are different approaches to conducting qualitative research interviews, each characterized by varying degrees of flexibility and organization in the questioning process. Here's an explanation of each: A. Structured Interview: In a structured interview, the researcher follows a predetermined set of questions, often in a standardized format, with little to no deviation from the script. Questions are typically closed-ended, meaning respondents choose from a list of predefined response options or answer with a simple "yes" or "no." The primary aim of structured interviews is to collect specific, quantifiable data that can be easily analyzed and compared across respondents. Structured interviews are commonly used in quantitative research or when the research objectives require consistency and uniformity in data collection. Example: A structured interview might involve asking all participants a series of fixed- choice questions about their purchasing habits, such as "How often do you shop online?" with response options ranging from "never" to "daily." B. Semi-Structured Interview: In a semi-structured interview, the researcher follows a flexible interview guide containing a list of predetermined topics or themes to be explored, but allows for open-ended questioning and probing. While the interviewer has a framework to follow, they have the freedom to adapt the conversation based on the participant's responses and delve deeper into specific areas of interest. Questions in semi-structured interviews are often a mix of open-ended and closed- ended, allowing for both qualitative insights and quantitative data. Semi-structured interviews are widely used in qualitative research to gather rich, detailed narratives and perspectives from participants. Example: A semi-structured interview with customers of a particular brand might begin with broad questions like "Can you tell me about your experiences with this brand?" and then explore specific aspects in more depth based on the participant's responses. C. Unstructured Interview: In an unstructured interview, there is no predetermined set of questions or topics, and the conversation flows freely based on the participant's interests, experiences, and perspectives. The interviewer acts more as a facilitator, guiding the discussion without imposing a rigid structure or agenda. Questions in unstructured interviews are entirely open-ended, allowing participants to express themselves in their own words and explore topics in any direction they choose. Unstructured interviews are often used in exploratory research or when the researcher seeks to uncover new insights, patterns, or themes without predefined expectations. Example: An unstructured interview might involve inviting participants to share their thoughts, feelings, and experiences related to a particular phenomenon or issue, with the interviewer simply prompting with phrases like "Tell me more" or "Can you elaborate?" 3. OBSERVATIONS: Observational methods involve systematically observing and recording behaviors, events, or phenomena in natural or controlled settings. Observations provide firsthand data, allow for the study of behavior in real-life contexts, and can capture non-verbal cues. Limitation of using this method is that observations may be subject to observer bias, lack of generalizability, and ethical concerns regarding privacy and consent. Types of Observations Participant Observations: In participant observations, the researcher actively participates in the setting or context being observed, immersing themselves in the environment and interacting with participants. The observer may adopt a dual role as both a researcher and a participant, enabling them to gain insider perspectives and deeper insights into the social dynamics and cultural norms of the setting. Participant observations are commonly used in ethnographic research, where researchers aim to understand the lived experiences and behaviors of individuals within a particular culture or community. Non-Participant Observations: In non-participant observations, the researcher remains detached from the setting or context being observed, acting solely as an observer and refraining from direct participation or interaction with participants. The observer maintains an external perspective, focusing on documenting behaviors, interactions, and events as they unfold without influencing the natural dynamics of the setting. Non-participant observations are often employed in studies where objectivity and neutrality are essential, such as observational studies in healthcare or social science research. 4. EXPERIMENTS: Experiments involve manipulating one or more variables to observe the effects on another variable under controlled conditions. Experiments allow for causal inference, precise control over variables, and the establishment of cause-and-effect relationships. Limitation of using this method is that experiments may lack ecological validity, may be impractical or unethical for certain research questions, and require careful design to minimize confounding variables. 5. Secondary Data Analysis (Documents): Secondary data analysis involves using existing datasets or sources of information collected for other purposes. Secondary data analysis is cost-effective, time-saving, and can provide access to large datasets or historical data. Limitation of using this method is that secondary data may be incomplete, outdated, or not tailored to the specific research question, and researchers may have limited control over data quality or variables. 6. Focus Group Discussion (FGD) A Focus Group Discussion (FGD) is a qualitative research method that involves bringing together a small group of participants (usually 6-12 individuals) to engage in facilitated discussions on a particular topic of interest. FGDs are used to explore shared beliefs, attitudes, experiences, and perceptions among participants and to generate in-depth qualitative data through group interaction and discussion. FGDs are typically conducted in a comfortable and neutral environment, with a skilled facilitator guiding the discussion using a predetermined set of open-ended questions or topics. Participants are encouraged to express their views freely, respond to each other's comments, and explore different viewpoints. FGDs are commonly used in market research, program evaluation, needs assessment, and policy development to gather diverse perspectives, identify common themes, and generate rich qualitative data. 7. Key Informant Interview (KII) A Key Informant Interview (KII) is a qualitative research method that involves conducting one-on-one interviews with individuals who possess specialized knowledge, expertise, or firsthand experience relevant to the research topic. KIIs are used to gather in-depth information, insights, and expert opinions on specific issues or areas of interest from individuals who are considered key informants or stakeholders in the field. KIIs are conducted through semi-structured interviews, where the interviewer follows a flexible interview guide containing a list of key topics or questions. The interviewer engages the informant in a conversational manner, allowing for probing, clarification, and exploration of relevant themes and perspectives. KIIs are commonly used in qualitative research studies, needs assessments, policy analysis, and program evaluation to obtain expert insights, validate findings from other data sources, and gain a deeper understanding of complex issues from knowledgeable informants. ORGANIZING DATA Organizing data is a crucial step in the research process, as it helps researchers manage, structure, and prepare their data for analysis. Here are several ways to organize data effectively: 1. Create a Data Management Plan: Before collecting data, develop a clear plan outlining how the data will be organized, stored, and documented throughout the research process. Define naming conventions, file structures, and data documentation standards to ensure consistency and reproducibility. 2. Use Data Tables or Spreadsheets: Organize data into tables or spreadsheets using software like Microsoft Excel, Google Sheets, or specialized statistical software. Each row represents a separate observation or case, while each column represents a variable or attribute. 3. Standardize Variable Names: Use clear, descriptive variable names that accurately represent the data they contain. Standardize variable names across datasets to facilitate data merging and analysis. 4. Arrange Data Hierarchically: If the data have a hierarchical structure (e.g., nested data), organize them accordingly to reflect the relationships between different levels. Use indentation, grouping, or subheadings to visually represent hierarchical relationships. 5. Group Related Variables: Group related variables or variables with similar characteristics together within the dataset. This helps organize the data logically and facilitates data analysis by grouping variables that are conceptually related. 6. Add Descriptive Metadata: Include descriptive metadata or annotations alongside the data to provide context, clarify variable definitions, and document any transformations or data cleaning procedures. This helps ensure transparency and reproducibility in data analysis. 7. Use Data Coding Schemes: If applicable, develop coding schemes or categorization systems to code qualitative or categorical data. Assign numerical or alphanumeric codes to represent different categories, themes, or responses consistently across the dataset. 8. Maintain Data Versioning: Keep track of different versions of the dataset, especially if modifications or updates are made during the research process. Use version control systems or file naming conventions to distinguish between different versions and document changes. 9. Implement Data Security Measures: Protect sensitive or confidential data by implementing appropriate security measures, such as encryption, password protection, and restricted access controls. Follow ethical guidelines and regulations for data handling and storage. 10. Document Data Sources and Collection Methods: Clearly document the sources of data and the methods used for data collection, including sampling procedures, measurement instruments, and data collection protocols. This helps ensure transparency and reproducibility in data analysis and interpretation. ANALYSIS OF DATA Analyzing data is a critical step in the research process, allowing researchers to derive meaningful insights, test hypotheses, and draw conclusions from their data. Here are several common methods and techniques for analyzing data: 1. Descriptive Statistics: Descriptive statistics involve summarizing and describing the main features of a dataset, such as measures of central tendency (e.g., mean, median, mode), measures of variability (e.g., range, variance, standard deviation), and frequency distributions. Descriptive statistics provide an overview of the data and help researchers understand its basic characteristics. 2. Inferential Statistics: Inferential statistics are used to make inferences or predictions about a population based on a sample of data. Common inferential statistical techniques include hypothesis testing, confidence intervals, and regression analysis. Inferential statistics help researchers draw conclusions and make generalizations beyond the observed data. 3. Qualitative Data Analysis: Qualitative data analysis involves interpreting textual, visual, or non-numeric data to identify patterns, themes, and meanings. Qualitative data analysis techniques include content analysis, thematic analysis, and grounded theory. Qualitative data analysis provides rich, detailed insights into complex phenomena and allows researchers to explore the perspectives and experiences of participants. 4. Quantitative Data Analysis: Quantitative data analysis involves analyzing numerical or categorical data using statistical methods. Quantitative data analysis techniques include correlation analysis, regression analysis, analysis of variance (ANOVA), and factor analysis. Quantitative data analysis enables researchers to test hypotheses, identify relationships between variables, and make statistical inferences. 5. Data Visualization: Data visualization techniques involve representing data visually using graphs, charts, and other graphical representations. Common data visualization techniques include bar charts, line graphs, scatter plots, histograms, and heatmaps. Data visualization helps researchers communicate findings effectively, identify trends and patterns, and gain insights from the data. 6. Cluster Analysis: Cluster analysis is a multivariate statistical technique used to group similar observations or cases into clusters based on their characteristics or attributes. Cluster analysis helps identify natural groupings within the data and can be used for segmentation, classification, or pattern recognition. 7. Factor Analysis: Factor analysis is a statistical technique used to identify underlying factors or dimensions that explain the correlations among a set of observed variables. Factor analysis helps reduce the complexity of data by identifying common patterns and relationships among variables. 8. Time Series Analysis: Time series analysis involves analyzing data collected over time to identify trends, seasonal patterns, and other temporal dependencies. Time series analysis techniques include trend analysis, seasonal decomposition, and forecasting methods such as moving averages and exponential smoothing. 9. Text Mining: Text mining techniques involve extracting information and insights from textual data, such as documents, social media posts, or customer reviews. Text mining techniques include sentiment analysis, topic modeling, and natural language processing (NLP). Text mining allows researchers to analyze unstructured text data and uncover valuable insights hidden within large volumes of text. 10. Machine Learning: Machine learning techniques involve using algorithms and computational methods to analyze data, identify patterns, and make predictions or decisions. Common machine learning techniques include classification, regression, clustering, and anomaly detection. Machine learning can be applied to both quantitative and qualitative data to automate analysis tasks and uncover complex relationships within the data.  PRESENTATION OF DATA The presentation of data refers to the manner in which data is visually or verbally communicated to an audience. It involves transforming raw data into meaningful information through various methods such as tables, charts, graphs, infographics, narratives, or interactive displays. The goal of data presentation is to effectively convey insights, trends, patterns, and conclusions derived from the data in a clear, concise, and engaging manner. Here are several ways to present data: 1. TABLES Tables are structured arrangements of data organized into rows and columns, providing a clear and systematic representation of numerical or categorical information. They are commonly used for presenting detailed data sets, allowing viewers to compare values and identify patterns easily. 2. CHARTS AND GRAPHS Charts and graphs visually represent data using various graphical elements such as lines, bars, and points. They are effective for illustrating trends, comparisons, and distributions within data sets, making complex information more accessible and understandable. a. Bar Charts: Compare values across categories or groups. b. Line Charts: Display trends over time or across continuous variables. c. Pie Charts: Show the composition of a whole in terms of proportions or percentages. d. Scatter Plots: Explore relationships between two continuous variables. e. Histograms: Illustrate the distribution of a continuous variable.  f. Box Plots: Present the distribution, variability, and outliers of a dataset. 3. INFOGRAPHICS Infographics combine text, visuals, and graphics to present data in a visually engaging and concise manner. They condense complex information into easy-to-understand formats, enabling viewers to quickly grasp key insights and trends. 4. DASHBOARDS - Dashboards are interactive displays that provide a comprehensive overview of key performance indicators (KPIs), metrics, and trends. They allow users to customize their view of the data, drill down into specific details, and monitor real-time changes, facilitating data- driven decision-making. 5. MAPS Maps visualize spatial data and geographic patterns using cartographic techniques. They are useful for displaying regional variations, geographic trends, and spatial relationships within data sets, providing context and insights into location-based phenomena. 6. NARRATIVE REPORTS Narrative reports provide a written analysis and interpretation of data, offering context, insights, and recommendations for decision-making. They typically follow a structured format, guiding readers through the data analysis process and highlighting key findings and implications. 7. DATA VISUALIZATIONS Data visualizations are custom-designed graphical representations of data created using specialized software tools or programming languages. They allow for advanced analysis and visualization techniques, enabling users to explore data in depth and uncover hidden patterns or insights. 8. INTERACTIVE PRESENTATIONS Interactive presentations engage audiences through dynamic and multimedia-rich content, encouraging active participation and exploration of data. They often include clickable elements, animations, and embedded media to enhance understanding and engagement. 9. STORYTELLING Storytelling uses narrative techniques to convey data-driven insights in a compelling and memorable way. It frames data within a narrative context, making it more relatable and impactful for audiences, and prompting action or decision-making based on the insights shared. 10. DATA SUMMARIES AND HIGHLIGHTS Data summaries and highlights distill key findings and insights from complex data sets into concise and easily digestible formats. They provide an overview of the most important information, allowing viewers to quickly grasp the main points and implications of the data analysis.  OVERVIEW OF DATA, DATA SCIENCE ANALYTICS, AND TOOLS DARIOS B. ALADO, DIT DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 HISTORY OF DATA SCIENCE The history of data science and analytics is rich and spans several centuries, involving contributions from various fields such as statistics, mathematics, computer science, and domain-specific knowledge. Here’s a chronological overview highlighting key milestones: Early Foundations (1600s - 1800s) 17th Century: The development of probability theory by mathematicians such as Blaise Pascal and Pierre de Fermat laid the groundwork for statistical analysis. 18th Century: Thomas Bayes formulated Bayes' Theorem, providing a mathematical approach to probability inference. 19th Century: The emergence of statistics as a distinct discipline, with contributions from figures like Carl Friedrich Gauss (normal distribution) and Florence Nightingale (statistical graphics). DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 HISTORY OF DATA SCIENCE 20th Century: The Rise of Statistics and Computing Early 1900s: Karl Pearson and Ronald A. Fisher advanced the field of statistics with the introduction of correlation coefficients, hypothesis testing, and analysis of variance (ANOVA). 1930s: The development of the first mechanical computers by pioneers like Alan Turing and John Atanasoff marked the beginning of the computing era. 1950s: The advent of electronic computers enabled more complex data analysis and the birth of computer science as a field. The term "artificial intelligence" was coined, and the first neural networks were conceptualized. 1960s: The introduction of the term "data processing" as businesses began to use computers for managing data. Edgar F. Codd proposed the relational database model, revolutionizing data storage and retrieval. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 HISTORY OF DATA SCIENCE Late 20th Century: The Digital Age 1970s: The development of structured query language (SQL) facilitated efficient data management and querying. Statistical software like SAS and SPSS became widely used in academia and industry. 1980s: The rise of personal computers made data analysis tools more accessible. The concept of data warehousing emerged, enabling organizations to consolidate and analyze large datasets. 1990s: The explosion of the internet led to an unprecedented increase in data generation. The term "business intelligence" (BI) became popular, emphasizing data-driven decision-making. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 HISTORY OF DATA SCIENCE 21st Century: The Era of Big Data and Data Science 2000s: The advent of big data technologies such as Hadoop and NoSQL databases enabled the processing of vast amounts of unstructured data. The term "data science" gained prominence, reflecting the interdisciplinary nature of modern data analysis. 2010s: The proliferation of machine learning and artificial intelligence applications in various industries. Data science became a recognized profession, with dedicated academic programs and roles such as data scientists and data engineers. The rise of cloud computing facilitated scalable data storage and processing. 2020s: The integration of data science with other emerging technologies such as IoT, blockchain, and quantum computing. Emphasis on ethical considerations, data privacy, and interpretability of AI models. The COVID-19 pandemic highlighted the importance of data science in public health and policy-making. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 KEY MILESTONES AND CONTRIBUTION Development of Machine Learning Algorithms: Algorithms such as decision trees, support vector machines, and neural networks have become fundamental tools in data science. Advances in Statistical Methods: Techniques like bootstrapping, Bayesian inference, and time series analysis have enhanced the ability to draw meaningful insights from data. Growth of Open-Source Tools: The development of open- source languages and libraries such as Python, R, TensorFlow, and Scikit-learn has democratized data science, making powerful tools accessible to a wider audience. Data Visualization: Innovations in data visualization, through tools like Tableau, D3.js, and Matplotlib, have improved the ability to communicate complex data insights effectively. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 KEY MILESTONES AND CONTRIBUTION Development of Machine Learning Algorithms: Algorithms such as decision trees, support vector machines, and neural networks have become fundamental tools in data science. Advances in Statistical Methods: Techniques like bootstrapping, Bayesian inference, and time series analysis have enhanced the ability to draw meaningful insights from data. Growth of Open-Source Tools: The development of open- source languages and libraries such as Python, R, TensorFlow, and Scikit-learn has democratized data science, making powerful tools accessible to a wider audience. Data Visualization: Innovations in data visualization, through tools like Tableau, D3.js, and Matplotlib, have improved the ability to communicate complex data insights effectively. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 KEY MILESTONES AND CONTRIBUTION Current Trends Automation and AutoML: The use of automated machine learning (AutoML) tools to streamline model development and deployment. Explainable AI: Growing focus on making AI models transparent and interpretable. Ethics and Data Privacy: Increasing emphasis on ethical considerations, fairness, and data privacy in data science practices. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 WHAT IS DATA SCIENCE? Data science is an interdisciplinary field that focuses on extracting knowledge and insights from structured and unstructured data through scientific methods, processes, algorithms, and systems. It combines principles and techniques from mathematics, statistics, computer science, and domain-specific knowledge to analyze and interpret complex data. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 DATA SCIENCE LIFE CYCLE The Data Science Life Cycle is a systematic approach to solving data-driven problems and extracting actionable insights from data. It encompasses several stages, each involving specific tasks and methodologies. 1. Problem Definition: Objective: Understand the business problem or research question to be addressed. Tasks: Collaborate with stakeholders to define clear goals and objectives, identify key metrics for success, and establish a project plan. 2. Data Collection: Objective: Gather data from various sources that are relevant to the problem. Tasks: Collect data from databases, APIs, web scraping, sensors, surveys, or third-party sources. Ensure data is in a usable format. College of Computing Science, Information and Communication Technology 7/4/2024 DATA SCIENCE LIFE CYCLE 3. Data Cleaning and Preprocessing: Objective: Prepare the data for analysis by addressing quality issues. Tasks: Handle missing values, remove duplicates, correct errors, standardize formats, and normalize data. Perform exploratory data analysis (EDA) to understand data distributions and relationships. 4. Data Exploration and Analysis: Objective: Gain insights and identify patterns in the data. Tasks: Use statistical methods and visualization tools to explore data, summarize key characteristics, and identify trends or anomalies. Develop hypotheses based on initial findings. 5. Feature Engineering: Objective: Create relevant features that improve model performance. Tasks: Transform raw data into meaningful features, perform dimensionality reduction, create interaction terms, and normalize or scale features. 7/4/2024 DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology DATA SCIENCE LIFE CYCLE 6. Model Building: Objective: Develop predictive or descriptive models based on the data. Tasks: Select appropriate algorithms, train models using training data, tune hyperparameters, and use techniques such as cross- validation to assess model performance. 7. Model Evaluation: Objective: Assess the accuracy and robustness of the model. Tasks: Evaluate model performance using metrics such as accuracy, precision, recall, F1-score, ROC-AUC, or others depending on the problem type (classification, regression, etc.). Validate the model on unseen test data. 8. Model Deployment: Objective: Implement the model in a production environment for real-time or batch processing. Tasks: Integrate the model into business processes or applications, set up APIs or batch processing pipelines, and ensure scalability and reliability. 7/4/2024 DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology DATA SCIENCE LIFE CYCLE 9. Model Monitoring and Maintenance: Objective: Ensure the model continues to perform well over time. Tasks: Monitor model performance in production, track key metrics, detect and address issues such as data drift or model degradation, and retrain or update the model as needed. 10. Communication and Reporting: Objective: Share insights and results with stakeholders. Tasks: Create reports, dashboards, and visualizations to present findings. Communicate the implications of the results and provide recommendations for decision-making. 11. Iteration and Improvement: Objective: Continuously improve the model and the overall process. Tasks: Use feedback from stakeholders and performance monitoring to refine the model, explore new features or data sources, and enhance the data science pipeline. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 TYPES OF DATA ANALYTICS DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 TYPES OF DATA ANALYTICS 1. Descriptive Analytics Objective: Understand past and current data to identify trends and patterns. Methods and Tools: Data Aggregation: Summarizing data to extract meaningful information. Data Visualization: Using charts, graphs, dashboards, and reports to represent data visually. Basic Statistical Analysis: Calculating averages, percentages, and other summary statistics. Examples: Generating sales reports to show monthly revenue trends. Visualizing website traffic data to understand user behavior over time. Summarizing customer feedback to identify common themes and sentiments. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 TYPES OF DATA ANALYTICS 2. Diagnostic Analytics Objective: Determine the reasons behind past outcomes or events. Methods and Tools: Root Cause Analysis: Identifying the underlying causes of specific outcomes. Drill-Down Analysis: Breaking down data into finer details to explore specific aspects. Correlation Analysis: Assessing relationships between different variables. Examples: Analyzing a sudden drop in sales to identify contributing factors. Investigating customer churn rates to determine the reasons for customer loss. Examining production delays to find the root causes in a manufacturing process. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 TYPES OF DATA ANALYTICS 3. Predictive Analytics Objective: Forecast future outcomes based on historical data. Methods and Tools: Statistical Modeling: Using techniques such as regression analysis to predict future values. Machine Learning: Applying algorithms like decision trees, random forests, and neural networks to make predictions. Time Series Analysis: Forecasting future values based on historical trends and patterns. Examples: Predicting future sales based on past trends and seasonal patterns. Forecasting stock prices or market trends using historical financial data. Anticipating equipment failures in manufacturing using sensor data and historical maintenance records. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 TYPES OF DATA ANALYTICS 4. Prescriptive Analytics Objective: Recommend actions to achieve desired outcomes. Methods and Tools: Optimization Techniques: Using linear programming, integer programming, and other methods to find the best course of action. Simulation: Modeling different scenarios to assess the impact of various decisions. Decision Analysis: Evaluating different decision options and their potential outcomes. Examples: Recommending inventory levels to minimize costs while meeting demand. Suggesting marketing strategies to maximize customer engagement and conversion rates. Optimizing delivery routes to reduce transportation costs and improve efficiency. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 WHAT IS DATA? Data refers to raw, unprocessed facts and figures collected from various sources. It can take various forms, such as numbers, text, images, videos, and sounds. Data is the fundamental building block for information and knowledge generation when it is processed, analyzed, and interpreted. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 TYPES OF DATA Types of Data: 1.Structured Data: 1. Definition: Data that is organized in a predefined format or structure, often in rows and columns, making it easily searchable and analyzable. 2. Examples: Databases, spreadsheets, CSV files. 3. Sources: Relational databases (MySQL, PostgreSQL), spreadsheets (Excel). 2.Unstructured Data: 1. Definition: Data that does not have a predefined format or structure, making it more complex to process and analyze. 2. Examples: Text documents, social media posts, images, videos, emails. 3. Sources: Social media platforms, multimedia files, emails. 3.Semi-Structured Data: 1. Definition: Data that does not fit into a rigid structure like structured data but contains tags or markers to separate elements, making it somewhat easier to organize and analyze. 2. Examples: XML files, JSON files, HTML documents. 3. Sources: Web pages, APIs. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 CHARACTERISTICS OF DATA DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 LEVEL OF MEASUREMENT OF DATA Nominal Data Definition: Nominal data is a type of categorical data where the categories do not have any inherent order or ranking. It is used for labeling variables without any quantitative value. Characteristics: Categories: Represents different categories or groups. No Order: No logical order or ranking among the categories. Qualitative: Describes qualities or characteristics. Examples: Gender: Male, Female, Non-binary. Marital Status: Single, Married, Divorced, Widowed. Types of Cuisine: Italian, Chinese, Mexican, Indian. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 LEVEL OF MEASUREMENT OF DATA Ordinal Data Definition: Ordinal data is a type of categorical data where the categories have a meaningful order or ranking, but the intervals between the categories are not necessarily equal or known. Characteristics: Order: Categories have a logical order or ranking. Unequal Intervals: The difference between categories is not uniform or known. Qualitative or Quantitative: Can describe both qualitative attributes and ranked quantitative measures. Examples: Education Level: High School, Bachelor's, Master's, Doctorate. Customer Satisfaction: Very Unsatisfied, Unsatisfied, Neutral, Satisfied, Very Satisfied. Class Ranks: Freshman, Sophomore, Junior, Senior. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 LEVEL OF MEASUREMENT OF DATA Interval Data Definition: Interval data is a type of quantitative data where the difference between values is meaningful, but there is no true zero point. Characteristics: Equal Intervals: The difference between values is consistent and measurable. No True Zero: Zero is arbitrary and does not indicate the absence of the attribute. Quantitative: Represents numerical values with equal intervals. Examples: Temperature (Celsius or Fahrenheit): The difference between 20°C and 30°C is the same as between 30°C and 40°C, but 0°C does not mean 'no temperature.' Calendar Dates: The difference between years is consistent, but there is no 'zero year.' DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 LEVEL OF MEASUREMENT OF DATA Ratio Data Definition: Ratio data is a type of quantitative data that has all the properties of interval data, but with a meaningful zero point that indicates the absence of the measured attribute. Characteristics: Equal Intervals: The difference between values is consistent and measurable. True Zero: Zero indicates the absence of the attribute being measured. Quantitative: Represents numerical values with equal intervals and a true zero. Examples: Height: Measured in centimeters or inches, with 0 indicating no height. Weight: Measured in kilograms or pounds, with 0 indicating no weight. Income: Measured in currency units, with 0 indicating no income. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 SOURCES OF DATA 1. Primary Data Sources Primary data is data collected directly from first-hand sources for a specific research purpose or analysis. It is original and unique to the study at hand. Surveys and Questionnaires: Collect data directly from individuals through structured questions. Example: Customer satisfaction surveys, market research questionnaires. Interviews: Gather in-depth information through one-on-one or group conversations. Example: Employee feedback interviews, qualitative research interviews. Observations: Collect data by observing behaviors, events, or conditions. Example: Observing consumer behavior in a retail store, recording traffic patterns. Experiments: Conduct controlled tests or experiments to gather data on specific variables. Example: A/B testing in marketing, clinical trials in healthcare. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 SOURCES OF DATA 2. Secondary Data Sources Secondary data is data that has already been collected and published by others for a different purpose. It is readily available and can be used for further analysis. Government Reports and Publications: Official documents and statistics provided by government agencies. Example: Census data, economic reports, public health records. Academic Journals and Research Papers: Published studies and research findings from academic institutions. Example: Articles from scientific journals, conference proceedings. Books and Reference Materials: Information compiled in books, encyclopedias, and other reference sources. Example: Textbooks, industry handbooks. Commercial Data: Data collected and sold by commercial entities. Example: Market research reports, syndicated data services. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 SOURCES OF DATA 3. Digital and Online Sources With the rise of the internet and digital technologies, a vast amount of data is generated and available online. Websites and Online Databases: Information available on the internet through various websites and online repositories. Example: Company websites, online encyclopedias, databases like PubMed and Google Scholar. Social Media Platforms: Data generated by users on social media platforms. Example: Tweets, Facebook posts, Instagram photos. E-Commerce Platforms: Data from online transactions and user interactions. Example: Purchase history, product reviews, browsing behavior. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 SOURCES OF DATA 4. Machine-Generated Data Data generated automatically by machines and sensors, often in large volumes and at high velocity. IoT Devices and Sensors: Data from interconnected devices and sensors in various environments. Example: Smart home devices, industrial sensors, environmental monitoring systems. Log Files: Records of events and transactions automatically logged by software applications and systems. Example: Server logs, application logs, security logs. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 SOURCES OF DATA 5. Internal Organizational Data Data generated and stored within an organization, often used for operational and strategic purposes. Customer Databases: Information about customers collected through interactions and transactions. Example: Customer profiles, purchase history, customer support tickets. Financial Records: Data related to financial transactions and performance. Example: Sales records, expense reports, profit and loss statements. Human Resources Data: Information about employees and workforce management. Example: Employee records, payroll data, performance evaluations. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 SOURCES OF DATA 6. Open Data Sources Data that is freely available for anyone to use, often provided by governments, organizations, or communities. Open Government Data: Publicly accessible data released by government entities. Example: Open data portals, public datasets on health, education, and transportation. Community-Contributed Data: Data shared by communities or collaborative projects. Example: Wikipedia, open-source software repositories, community science projects. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 DATA ANALYTICS TOOL DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 DATA ANALYTICS TOOL Data analytics tools are software applications and platforms designed to process, analyze, visualize, and interpret data. These tools enable organizations and individuals to derive meaningful insights from large and complex datasets. Here are some popular data analytics tools categorized based on their functionalities: DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 DATA ANALYTICS TOOL Data Collection and Integration Tools 1.Apache Kafka: A distributed streaming platform for collecting and processing real-time data streams. 2.Apache NiFi: An open-source data integration tool that allows for the automation of data flow between systems. 3.Talend: Provides data integration and data quality services through a unified platform for ETL (Extract, Transform, Load) processes. Data Storage and Management Tools 1.Apache Hadoop: An open-source framework for distributed storage and processing of large datasets across clusters of computers. 2.Apache Spark: A unified analytics engine for large-scale data processing, offering both batch processing and real-time data streaming capabilities. 3.Amazon S3 (Simple Storage Service): A scalable object storage service offered by Amazon Web Services (AWS) for storing and retrieving data. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 DATA ANALYTICS TOOL Data Analysis and Exploration Tools 1.Tableau: A powerful data visualization tool that allows users to create interactive and shareable dashboards and reports. 2.Power BI: Microsoft's business analytics service for creating interactive visualizations and business intelligence reports. 3.QlikView / Qlik Sense: Business intelligence and data visualization tools for exploring and analyzing data from multiple sources. Statistical Analysis and Modeling Tools 1.R: A programming language and software environment for statistical computing and graphics, widely used for data analysis and machine learning. 2.Python (with libraries like NumPy, Pandas, SciPy, scikit-learn): A versatile programming language with extensive libraries for data manipulation, analysis, and machine learning. 3.IBM SPSS Statistics: Statistical software used for data analysis, including descriptive statistics, regression analysis, and predictive modeling. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 DATA ANALYTICS TOOL Machine Learning and AI Tools 1.TensorFlow / Keras: Open-source frameworks for deep learning and machine learning applications, developed by Google. 2.PyTorch: A machine learning library for Python, developed by Facebook's AI Research lab, known for its flexibility and ease of use. 3.Azure Machine Learning: Microsoft's cloud-based service for building, training, and deploying machine learning models. Big Data Processing and Querying Tools 1.Apache Hive: A data warehouse infrastructure built on top of Hadoop for querying and managing large datasets stored in Hadoop's HDFS. 2.Apache Drill: A schema-free SQL query engine for big data exploration, supporting a wide range of data sources and formats. 3.Google BigQuery: A serverless, highly scalable, and cost-effective cloud data warehouse for running SQL queries on large datasets. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 DATA ANALYTICS TOOL Data Governance and Security Tools 1.Collibra: A data governance platform that provides tools for data cataloging, data lineage, and data stewardship. 2.IBM InfoSphere Information Governance Catalog: Offers capabilities for metadata management, data lineage, and governance policies enforcement. 3.Varonis Data Security Platform: Provides tools for data access governance, data security, and threat detection across on-premises and cloud environments. Data Visualization and Reporting Tools 1.D3.js: A JavaScript library for producing dynamic, interactive data visualizations in web browsers. 2.Plotly: A graphing library for creating interactive plots and charts in Python, R, and JavaScript. 3.Microsoft Excel: Widely used spreadsheet software that includes features for data analysis, visualization, and reporting. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 DATA ANALYTICS TOOL Business Intelligence (BI) Platforms 1.Sisense: BI software that allows users to prepare, analyze, and visualize complex datasets using AI-driven analytics. 2.Looker: A data exploration and business intelligence platform that offers data modeling, exploration, and real-time analytics capabilities. 3.Yellowfin BI: Provides tools for data visualization, dashboards, and reporting, with embedded analytics and collaboration features. Workflow and Automation Tools 1.Alteryx: A platform for data blending and advanced data analytics, offering workflow automation and predictive analytics capabilities. 2.Apache Airflow: A platform to programmatically author, schedule, and monitor workflows, with support for task dependencies and data pipelines. 3.Knime: An open-source platform for creating data science workflows, including data integration, preprocessing, analysis, and visualization. DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024 CHAPTER ACTIVITIES Activity 1 Instructions: 1. Briefly explain the concept of data types in data science: nominal, ordinal, interval, and ratio. 2. Provide examples for each type to ensure understanding. Activity 2 1. Distribute a list of examples (see examples below) or display them on a screen. 2. Classify each example into one of the four data types. Example list: 1. Temperature readings in Celsius 2. Types of animals in a zoo 3. Educational attainment (e.g., high school diploma, bachelor's degree) 4. Customer satisfaction ratings (e.g., very satisfied, satisfied, neutral, unsatisfied, very unsatisfied) 5. Heights of students in a class 6. Gender (e.g., male, female, non-binary) 7. Years of experience in a job 8. Scores on a 1-10 happiness scale Activity 3 1. After classifying the examples, discuss as a group why each example belongs to its respective data type. 2. Explore the implications of each type for data analysis and decision-making. 3. Ask participants to brainstorm real-world scenarios where understanding data types is crucial (e.g., healthcare, marketing, finance). DARIOS B. ALADO, DIT | Data Science Analytics | Isabela State University - College of Computing Science, Information and Communication Technology 7/4/2024