Introduction to Data Science

Introduction to Data Science

• Data science is utilized in autonomous cars for real-time decision-making, such as accelerating, braking, or turning
• Self-driving cars could potentially prevent around 2 million deaths annually caused by car accidents per a study
• Data science addresses issues in the airline industry like flight delays, demand prediction, route planning, and equipment selection
• Effective use of data science can reduce problems for both airlines and passengers.
• Airlines use data science for better route planning, delay prediction and promotional offers
• Logistics companies such as FedEx use data science models to optimize routes, cut costs, and determine the best delivery times and transport modes.
• Data science is used for better decision making, predictive analysis, and pattern discovery

Data Science Process Overview

• The process involves asking the right question and thoroughly exploring the data
• Modeling includes choosing the right algorithm, training the model, and refining it for accuracy
• The final step involves visualizing the results in an understandable format and communicating them effectively
• Initial phases of using data included automation of selling, manufacturing, and ERP and CRM systems.

Business Intelligence vs. Data Science

• Business intelligence (BI) primarily uses structured data from sources like ERP and CRM systems
• BI methods are mainly analytical, reporting historical data through reports and dashboards
• BI typically requires visualization skills and less focus on in-depth statistics
• BI focuses on historical data reporting; for example, sales reports from the past year.
• Data science uses structured and unstructured data, including web blogs and customer comments
• Data science seeks to deeply understand the reasons behind behaviors, going beyond simple reporting with statistical analysis
• Data science requires strong statistical skills in addition to visualization, for tasks like correlation and regression analysis
• Data science uses historical data and other information to predict future outcomes, going beyond historical reporting

Prerequisites for Data Science

• Essential traits for a data scientist include curiosity, common sense, and communication skills
• Machine learning is a core component, requiring expertise in algorithms and model training
• Strong statistical knowledge is fundamental for data analysis and interpretation.
• Programming skills, particularly in Python or R, are necessary for executing data science projects
• Understanding databases and data handling is essential

Tools and Skills in Data Science

• Common programming languages are Python and R.
• Python is favored for its ease of learning and extensive libraries.
• Essential skills include programming, statistics, and knowledge of data analysis tools
• SAS is used, but is proprietary; Python and R are open source.
• Jupyter notebooks and RStudio are used as interactive development environments
• ETL (Extract, Transform, Load) is required and SQL querying for data extraction is useful
• Hadoop is important for handling large, unstructured data
• Spark is an engine for data analysis in distributed mode and is often used with Hadoop
• Data visualization includes tools such as Tableau and Cognos.
• Machine learning tools include Python, Spark Mlib, Apache Mahout, and Microsoft Azure ML Studio.

Daily Activities of a Data Scientist

• A data scientist addresses business problems by asking questions to define the problem
• Data scientists gather raw data from various sources
• The data scientist processes the collected data, analyzes it, and converts it into a usable format.
• The processed data is fed into analytics systems, like machine learning algorithms or statistical models, to generate insights
• The data scientist organizes the results and presents them to stakeholders in a clear, understandable way.

Machine Learning Algorithms

• Regression models predict continuous numerical values (e.g., temperature, stock prices)
• Clustering is an unsupervised learning technique used to group unlabeled data for analysis (e.g., categorizing cricketers based on performance)
• Decision trees classify data in a logical, understandable manner, useful for classification problems
• Support Vector Machines (SVM) are used for classification purposes.
• Naive Bayes is a statistical, probability-based classification method.

Data Science Project Lifecycle

• The concept study involves understanding the business problem, goals, budget, and available data
• Data preparation involves gathering, cleaning, and transforming raw data into a usable format
• Data integration is part of data prep and transforms data, resolves conflicts, and removes redundancies in order to proceed with an organized data set
• Data cleaning involves handling missing, null, or incorrect values
• Missing values can be addressed by removing rows (if few) or filling gaps with mean or median values
• Data splitting divides data into training (80%) and testing (20%) sets to assess model accuracy
• Exploratory data analysis involves understanding data types, cleaning data, and identifying max/min values
• Visualization techniques, such as histograms and scatter plots, are used for quick identification of data patterns
• During model planning, decisions are made on which models to use
• Statistical models may be used and also maching learning models depending on complexity
• Models are trained using training data and validated with test data through multiple iterations
• Common tools are R, RStudio, and Python offering integrated environments and libraries for data analysis
• Matlab and SAS are also useful for statistical analysis and data science tasks
• Model building may include creating simple models such as a linear regression model

Linear Regression Details

• Linear regression models the relationship between independent and dependent variables.
• Linear regression calculates y = mx + c
• After the training process is complete it will produce a new value of m & c that is then used for predicting new values that will come, for example predicting a price
• A straight line (y = mx + c) is determined to best fit the data The training process determines the values of 'm' and 'c' based on the given data
• The trained model, with determined 'm' and 'c' values, is used to predict values for new data
• The model is validated using test data
• If validation is good the model is deployed and if not it is retrained

Model Building and Implementation

• Python, with libraries like pandas and NumPy, can be used to build and implement models.
• Implementation details will be covered in a separate tutorial.

Communicating Results

• Presenting results to stakeholders is an essential step for data scientists.
• This involves creating presentations or dashboards to explain findings.
• Recommendations should be provided to address the problem.

Operationalization

• Operationalization is the process of putting accepted data science presentations into practice.
• It helps improve or solve the problem defined in the initial step.

Data Science Life Cycle Summary

• The lifecycle includes:
  • Concept study
  • Data preparation
  • Model planning
  • Model building
  • Result communication
  • Operationalization

Demand for Data Scientists

• There is high demand and low supply of data scientists.
• Industries with high demand include:
  • Gaming
  • Healthcare
  • Finance
  • Marketing
  • Technology

Summary of Key Topics

• This session covered the need for and definition of data science.
• Required skills, programming languages, and tools were discussed.
• Tools like Python and R were compared.
• The differences between business intelligence and data science were outlined.
• The data science project lifecycle was detailed with an example.
• The global demand for data scientists was highlighted.