Data Science Research Methodology

Data Science Research Approaches

• Question-driven approach: formulate hypothesis, design experiment, collect data, analyze data, accept or reject hypothesis
• Data-driven approach: explore data, formulate research question, structure and annotate data, develop and apply learning techniques, evaluate on data, answer research question

Steps in Data Science Research

• Formulate a clear research question and task definition
• Collect data
• Explore and preprocess data
• Develop and apply learning techniques
• Evaluate model performance
• Answer research question

Challenges of Working with Data

• Noisy data
• Large data
• Small data
• Incomplete data
• Different sampling rates
• Different formats
• Wrongly chosen or irrelevant variables
• Large or unknown number of classes
• Class imbalance
• Heterogeneous data or features
• New domain

Defining a Task in Data Science

• Clearly define the research question
• Specify whether the task is supervised or unsupervised
• Determine the type of task: classification, regression, ranking, etc.
• Identify the data and its characteristics
• Identify the labels or targets and their characteristics

Simple and Multiple Linear Regression

• Simple linear regression: Y = βX + b, where Y is the dependent variable, X is the independent variable, β is the slope, and b is the intercept
• Multiple linear regression: y = β⋅X + b, requires normalization or standardization of values before training

Logistic Regression

• A discriminative model that learns to distinguish between two classes
• Learns from a training set to minimize the loss function L(y ̂,y)
• Uses gradient descent to optimize the parameters w and b
• Uses the sigmoid function to transform output to a probability: σ(z) = 1/(1+e^(-z) )

Data Types and Exploration

• Structured data: a database with rows, columns, and a relational key
• Semi-structured data: data with additional information, such as references to images
• Unstructured data: unorganized data, such as images, text, or time series data
• Use median instead of mean when data is skewed or has outliers
• Boxplots are more useful for outlier detection than histograms

Learn about the question-driven and data-driven approaches in data science research, including the steps involved in each methodology. Understand how to set up and execute an experiment within data science.