Big Data Fundamentals

• Big data refers to nontraditional techniques and tech for collecting, aggregating, processing, and gaining insights from massive datasets
• The big data process involves data acquisition, preprocessing, mining, prediction, and visualization
• Skilled analysts are needed to effectively leverage big data, basic big data analytics include: data preparation, model planning, and model building

Analytics for Data Science

• Big data's components consist of structured, semi-structured, and unstructured data
• Structured data has a well-defined structure in columns and rows for easy management and accessibility
• Semi-structured data does not adhere to formal structures but uses tags to distinguish semantic elements
• Unstructured data lacks a defined structure or standard hierarchy
• Big data has five characteristics known as the 5 V's: volume, velocity, variety, veracity, and value

The 5 V's of Big Data

• Volume is a huge amount of information and the size of data is and indicates whether data should be classified as "big"
• Internet traffic in 2016 measured 6.2 exabytes/month; expected to reach 40,000 exabytes by 2020
• Velocity refers to the speed of data generation, from machines, networks, social media, and mobile phones
• More than 3.5 billion searches occur on Google daily, and Facebook users are rising by about 22% yearly
• Variety talks about the nature of data that can be structured, semi-structured, and unstructured and data comes from inside and outside enterprises
• Veracity signifies data's vulnerability to inconsistencies and uncertainty because data are collected from various sources in huge amounts and monitoring data quality is challenging
• Value means data processed to extract value/knowledge, and insights derived are important

Examples of Data Analytics

• Fraud detection reports identify fraudulent transactions and unauthorized account access
• Live tracking reports are used track cars, customer requests, payment processing, and find needs and revenue
• Sales forecast and plan analysis are made to assess customers' sales, profits, needs, and evaluate future targets
• Live data are handled to provide stock market data and other real-time reports
• Alerts are generated based on events like data center alerts, using examples of big data analytics notifications
• Google Analytics reports get user visit counts, user location, and client computer specs

Data analytics life cycle

• The Data analytics life cycle is required for problems with big data and has data science applications
• Consists of six basic phases
• The process is iterative
• Data discovery involves analyzing the problem, finding data sources, and forming initial hypotheses
• Data preparation includes cleaning, sampling, combining, and aggregating data, often requiring extensive time and labor
• System preparing
• Plan construction
• Results communication
• Operationalization
• An analytical sandbox is created for data preparation, aggregating relevant data
• A copy of financial data from the computational sandbox is accessed over dealing with the production version of the organization’s main database, as this will be closely managed and required for financial reporting
• The ETL (Extract, Load, Transform) process is a type of data integration
• Analytical sandbox size should be at least 5–10 times the original datasets
• ETL involves extraction, transformation, and loading

ETL

• Extraction involves being aware of ODBC/JDBC drivers, understanding data structures, and handling resources, CDC (data capture modification) extracts data that has changed over time
• Transformation cleans data for further use, installation, and data integration
• Loading loads data into target structure for further processing
• Data scientists wanting raw data in warehouses lead to big ETL
• Application Programming Interfaces (APIs) are popular for accessing data
• Hadoop allows data scientists to explore data complexities and store data without needing to grasp all specifics
• Alpine Miner develops analytical workflows with a GUI for data manipulation and analytics
• OpenRefine is a standalone open-source tool for cleaning, transforming, and wrangling data
• Trifacta Wrangler enables analysts to get data from various sources to prepare for analytical or visualization tools

Model Planning

• The team selects an analytical model and appropriate variables
• Key consideration is assessing the structure of datasets and ensuring the analytical technique meets objectives
• Data exploration aims to know relationships among variables and variable reduction addresses significant correlations
• Model selection chooses an analytical technique based on project goals and revisits analytic challenges
• R manipulates data easily, runs on multiple platforms, has many packages
• Tableau Public links any data source, generates visualizations, allows file sharing
• SAS accesses/handles data, has customer intelligence products/marketing analytics, can predict, monitor, and refine behaviors