Data Validation and Standardization Quiz

Introduction to Data Science

• Dr. Amal Fouad is a Senior Data Scientist
• Introduction to Data Science presentation

Agenda

• Why study Data Science?
• How Does Data Science Impact Organizations?
• Application and Competitive Advantage
• Importance of Data Science
• What Will Be Discussed

DS vs AI vs ML vs DL?

• Artificial Intelligence: Any technique that allows computers to mimic human behavior
• Machine Learning: A subset of AI techniques that use statistical methods to enable machines to improve with experience
• Deep Learning: A subset of Machine Learning that makes the computation of multi-layer neural networks feasible

DS vs AI vs ML vs DL? (Timeline)

• Early Artificial Intelligence (1950s)
• Machine Learning Flourishes (1990s-2000s)
• Deep Learning Breakthroughs (2010s)

Why Data Science?

• Information Created Worldwide: Expected to continuously accelerate, data is growing exponentially.
• 2005 - 0.1 ZB of data
• 2010 - 2 ZB of data (9%)
• 2015 - 12 ZB of data (9%)
• 2020 - 47 ZB of data (16%)
• 2025 - 163 ZB of data (36%)
• Data accumulating at 28% annual growth rate
• Data Analysts in the workforce growing at 5.7% annual growth rate
• Data Analyst shortage
• Data is a huge field with lots of opportunities to study

Components of Data Science

• Artificial Intelligence (AI): This is a broad field encompassing many techniques to mimic human behaviour
• Machine Learning (ML): Methods used in AI to enable machines to learn from data
• Deep Learning (DL): A specific technique of ML that involves networks of multiple layers
• Maths & Statistics: Statistical models used for data analysis
• Visualization: Using charts,graphs to visually present the data for better understanding
• EDA (Exploratory Data Analysis): Process of discovering patterns in the data

Data Definition

• Data is a collection of details, figures, symbols, descriptions
• Raw data can be in various forms like letters, numbers, images, characters
• Computer data is represented (in most cases) by a combination of 0's and 1's
• Information is data to which meaning has been attached

Databases

• Oracle
• MySQL
• Microsoft SQL Server
• MongoDB
• PostgreSQL

Data Definition (Databases aspect)

• Databases are logical structures that rely on set theory, relationships between tables, and unique primary and foreign keys.
• This keeps data integrity between tables
• Data Science methods, processes and insight concerning structured or unstructured data

Data Science Definition

• Data science is an interdisciplinary field that employs scientific methods, processes, algorithms, and systems to obtain knowledge and insights from structured or unstructured data.
• Data Science is applied in many areas

Data Science Definition (Unified concepts)

• Data science is a concept to unify statistics, data analysis, machine learning and their related methods.
• Used for understanding and analyzing actual phenomena with the help of data
• Drawn from mathematics, statistics, computer science, and information science

Data Science Definition (Roles & skills)

• Analyst - Business administration. Exploratory Data Analysis.
• Data Scientist - Advanced algorithms. Machine learning. Data preparation. Data governance. Data product engineering. SQL.
• Data Engineer - Builds data pipelines.
• Process Owner - Project management. Management of stakeholder expectations. Maintaining a vision. Facilitating processes

Data Scientist Starter Pack - Libraries

• NumPy: A Python library for large, multi-dimensional arrays and matrices in order to use high-level mathematical numerical functions.
• Scikit-learn: A free software library for classification, regression and clustering algorithms.
• Pandas: A Python library for data manipulation and analysis of data structures.

Data Scientist Starter Pack - Tools

• Jupyter: An open-source web application to create and share documents containing live code, equations, diagrams and narrative text.
• Colab: similar to Jupyter Notebook but provides sharing and collaboration features.

Data Scientist Applications

• E-commerce: Identifying customers, recommending products, analyzing reviews
• Manufacturing: Predicting problems, monitoring systems, automating manufacturing units, maintenance scheduling, anomaly detection
• Healthcare: Medical image analysis, drug discovery, bioinformatics, virtual assistants
• Transport: Self-driving cars, enhanced driving experience, car monitoring system, enhancing passenger safety
• Banking: Fraud detection. Credit risk modeling. Customer lifetime value
• Finance: Customer segmentation, strategic decision making, algorithmic trading, risk analysis

Data Science - Importance

• Data science helps brands to deeply understand their customers
• Data science provides powerful and engaging ways for brands to communicate their story
• The field of Big Data is a continuously evolving field

Data Science - Importance (Specific areas)

• Data Science findings and results are applicable to sectors like travel, healthcare and education.
• Data science is accessible to most sectors

Data Science Roadmap

• Includes different areas of expertise, math, probability, statistics, programming, machine learning, deep learning, NLP, database, visualization tools, and deployment.
• Includes project on various topics including credit card fraud detection, movie recommendation, data science at Netflix, data science at Flipkart

How YouTube uses AI

• Automatically removes objectionable content
• Recommends videos based on user choice
• Organizes content

How Facebook uses AI

• DeepText
• Machine Translation
• Automatic Image Recognition
• Chatbots
• Deepfakes
• Targeted ads

How Google uses AI

• DeepText
• Gmail (smart replies, spam detection)
• Google Search (suggestions based on past searches)
• Google maps (ETA based on location, day of the week, trip time)
• Google Translate
• Speech Recognition

Companies using data science

• Facebook
• Amazon
• Google
• LinkedIn
• Netflix
• YouTube
• Microsoft

Roles in Data Science

• Data Scientist: Proves of disproves hypothesis. Gathering and wrangling data, develop and communicate

• Data Engineer: Builds Data Driven Platforms, Operationalize algorithms, and handle data integration.

• Data Analyst: Storytelling, build dashboards and other data visualizations. Providing insights through visuals

• Process Owner Project management to maintain a vision for data that facilitates process

Learning Data Science with Python (Libraries)

• NumPy: numerical computing library
• Scikit-learn: machine learning library
• Pandas: data manipulation and analysis library

Learning Data Science with Python (Tools)

• Matplotlib: plotting library
• TensorFlow: open-source library for dataflow programming. Used for machine learning, applications, and neural networks.
• Keras: Neural network library that runs on TensorFlow
• Jupyter: interactive computing environment
• Colab: similar to Jupyter for collaborating and sharing

Data Science - Applications (List)

• Consumer Identification
• Product Recommendation
• Review Analysis
• Potential Problem Prediction
• Automated Systems in Manufacturing
• Maintenance Scheduling
• Anomaly Detection
• Fraud Detection
• Credit Risk Modeling
• Customer Lifetime Value
• Medical Image Analysis
• Drug Discovery
• Bioinformatics
• Virtual Assistants
• Self Driving Cars
• Enhanced Driving Experience
• Car Monitoring System
• Improving Passenger Safety
• Customer Segmentation
• Strategic Decision Making
• Algorithmic Trading
• Risk Analysis

Data Science - Additional key definitions

• Data is a collection of details and facts.
• Information is derived from processed data.
• Analytics are methods to get insight from data
• Data Science is an interdisciplinary field
• AI is mimicking human behavior
• Machine Learning enables machines to learn
• Deep learning uses multiple layer neural networks

Data Cleaning

• Incomplete data
• Noisy data
• Inconsistent data

Handling Missing Values

• Replace with default values
• Calculate mean/mode, and fill in missing numerical/categorical fields respectively
• Generate random values from existing data distribution (as needed)

Handling Outliers

• Data Values that exceed expected values.
• Potentially due to erroneous data processing errors
• Remove or adjust the values
• Check data for consistency

Handling Noisy Data

• Errors in Data collection, entry, transmission or technology limitations.
• Data validation and correction process is used for fixing errors

Handling Inconsistent Data

• Data values that differ from expected values
• Use data dependencies to detect and rectify inconsistencies

Data Science Process

• Understanding the problem
• Prepare data samples
• Develop a model
• Apply model to dataset
• Deploy, maintain models

Prior Knowledge

• Understand the objective of the problem being investigated
• Understand subject matter
• Understand the data context

Data Preparation

• Data exploration (to understand the data characteristics)
• Handling data quality (validating or fixing erroneous data)
• Handling missing data (managing or replacing missing values)
• Data type conversion (converting between different data types, if necessary)
• Handling data anomalies (handling outliers or highly correlated attributes)
• Feature Selection (Selecting relevant/important features from large datasets)
• Sampling (Representing a sample from a dataset)

Data Modeling

• Creating a model that provides representation of data and relationships within context
• Making predictions based on a trained model

Data Application

• Assimilation of model results into the business process
• Deployment and maintenance of the model

Data Visualization

• Visualizing data helps to identify trends & correlations in data
• Univariate visualization (analyzing one attribute at a time)
• Multivariate Visualization (analyzing multiple attributes together)
• Charts can display trends and relationships.
• Graphs can help in interpreting trends and patterns from the data effectively