Big Data Analytics

• Big Data Analytics is a field focused on extracting meaningful insights from large and complex datasets.
• Cost reduction, faster/better decision-making, and new product/service development can be achieved through Big Data Analytics.

Learning outcomes

• Data Analytics
• Big Data
• Big Data Characteristics
• Types of Analytics

Big Data Analytics Textbook & Reference Books

• "Big Data Analytics: A Hands-On Approach" by Arshdeep Bahga
• "Big Data Fundamentals Concepts, Drivers & Techniques" by Thomas Erl
• "Hadoop: The definitive guide" by Tom White from O'Reilly Media, Inc., 2012
• "Learning Spark: lightning-fast big data analysis" by Karau, Holden, Konwinski, Wendell, and Zaharia from O'Reilly Media, Inc., 2015
• "MapReduce design patterns: building effective algorithms and analytics for Hadoop and other systems" by Miner and Shook from O'Reilly Media, Inc., 2012.

Big Data

• Big Data is defined by collections of datasets whose volume, velocity, or variety are very large.
• Big Data Volume, velocity or variety is so large that it becomes difficult to store, manage, process, and analyze with traditional databases.
• Exponential growth in structured and unstructured data occurs due to information technology, industrial applications, healthcare, and the Internet of Things.
• Approximately 3.5 quintillion bytes of data are created every day.

Characteristics of Big Data

• Volume: Big data is a form of data with a scale that is too large for a single machine, requiring special tools/frameworks for storage, processing, and analysis.
• Social media platforms process billions of messages daily.
• Industrial and energy systems can generate terabytes of sensor data each day.
• Cab aggregation apps might process millions of transactions daily.
• Velocity: Velocity of data refers to the speed at which data is generated.
• High velocity results in high volume after a short span of time.
• Some applications require real-time analysis with strict deadlines like trading platforms.
• Variety: Variety refers to the different forms of data such as structured, semi-structured, and unstructured, including text, image, audio, video, and sensor data.
• Value: Value of data refers to the usefulness of data for the intended purpose
• Veracity: Veracity refers to the accuracy of the data that it contains
• Data needs to be cleaned of noise to extract value, and faulty data must be filtered out.
• Data-driven applications require meaningful and accurate data to reap the benefits of big data.

Data Analytics

• Analytics is a broad term encompassing the processes, technologies, frameworks, and algorithms used to extract meaningful insights.
• Data analysis is achieved through filtering, processing, categorizing, condensing, and contextualizing data.
• Data analytics is the process of exploring and analyzing large data sets to identify hidden patterns, unseen trends, and valuable correlations/insights.

Types of Analytics

• Descriptive Analytics: Focuses on what has happened by looking at data.
• Diagnostic Analytics: Focuses on why something happened.
• Predictive Analytics: Focuses on what will happen in the future.
• Prescriptive Analytics: Focuses on what the solution is.

Computational Tasks

• The National Research Council has characterized computational tasks for massive data analysis, known as the "seven giants".
• The seven computational tasks:
• Basic Statistics
• Generalized N-Body Problems
• Linear Algebraic Computations
• Graph-Theoretic Computations
• Optimization
• Integration

Domain specific examples of Big Data

• Homes
• Cities
• Environment
• Energy Systems
• Retail
• Logistics
• Industry
• Agriculture
• Internet of Things
• Healthcare

Industry specific analytics

• Web:
• Web Analytics
• Performance Monitoring
• Ad Targeting & Analytics
• Content Recommendation
• Financial:
• Credit Risk Modeling
• Fraud Detection
• Healthcare:
• Epidemiological Surveillance
• Patient Similarity-based Decision Intelligence Application
• Adverse Drug Events Prediction
• Detecting Claim Anomalies
• Real-time health monitoring
• Internet of Things:
• Intrusion Detection
• Smart Parking
• Smart Roads
• Structural Health Monitoring
• Smart Irrigation
• Environment:
• Weather Monitoring
• Air Pollution Monitoring
• Noise Pollution Monitoring
• Forest Fire Detection
• River Floods Detection
• Water Quality Monitoring
• Logistics and Transportation
• Real-time Fleet Tracking
• Shipment Monitoring
• Remote Vehicle Diagnostics
• Route Generation and Scheduling
• Hyper-local Delivery
• Cab/Taxi Aggregators
• Industry:
• Machine Diagnosis and Prognosis
• Risk Analysis of Industrial Operations
• Production Planning and Control
• Retail:
• Inventory Management
• Customer Recommendations
• Store Layout Optimization
• Forecasting Demand

Analytics flow

• Data Collection.
• Data Preparation.
• Analysis Types.
• Analysis Modes.
• Visualizations.

Big Data Stack

• A Big Data stack refers to the collection of software and technologies used to handle and process large volumes of data.
• Big data stacks are designed to manage the challenges associated with storing, processing, and analyzing massive datasets.
• Big data analytics is the process of extracting insights from large, complex datasets.
• Big data stack is the collection of technologies and tools used to manage, process, and analyze that data.
• The stack is the infrastructure and analytics are the activity performed on it.

Elements of the Big Data Stack

• Raw Data Sources consist of Logs, Transactional Data, Social Media, Databases, Sensor Data, Clickstream Data, Surveillance Data, Healthcare Data and Network Data
• Logs are records of events within an application or server environment for performance monitoring.
• Transactional data is generated by eCommerce, Banking, and Financial applications.
• Social Media data is generated by social media platforms, while databases hold structured data in relational formats.
• Sensor Data is generated by Internet of Things (IoT) systems.
• Clickstream Data is generated by web applications to analyze browsing patterns.
• Healthcare Data is generated by Electronic Health Record (EHR) and other healthcare apps
• Network Data is generated by network devices such as routers.
• Data Access Connectors consist of Publish-Subscribe Messaging, Source-Sink Connectors, Database Connectors, Messaging Queues and Custom Connectors
• These connectors consist of tools and frameworks for collecting and ingesting data from various sources into big data storage and analytics frameworks
• Data Storage consists of Non-relational (NoSQL) databases and the Hadoop Distributed File System (HDFS).
• Hadoop is an open-source software framework designed for distributed storage and processing of large datasets, enabling efficient analysis by splitting workloads.
• Batch Analytics include Hadoop-MapReduce, Pig, Oozie, Spark, Solr and Machine Learning.
• It allows analysis of data in batches.
• Real-time Analytics tools are Apache Storm and Spark Streaming.
• Apache Storm is a framework for distributed and fault-tolerant real-time computation.
• Storm handles data from publish-subscribe messaging frameworks (Kafka or Kinesis), messaging queues (RabbitMQ or ZeroMQ), and custom connectors.
• Spark Streaming is a component of Spark for analyzing streaming data like sensor data, clickstreams, and web server logs.
• The streaming data is ingested and analyzed in micro-batches leading to scalable and high throughput stream processing.
• Interactive Querying systems.
• Spark SQL and Hive can be used to query structured and semi-structured data using SQL-like queries, while supporting Apache Hadoop.
• Amazon Redshift to handle queries on datasets of sizes up to a petabyte or more parallelizing the SQL queries.
• Google BigQuery querying datasets using SQL-like queries.
• Serving Databases, Web & Visualization Frameworks consist of:
• Databases include MySQL, Amazon DynamoDB, Cassandra and MongoDB
• Visualization frameworks would be Lightning, Pygal and Seaborn
• Analytics results are stored in these serving databases for subsequent presentation and visualization tasks.

Analytics mapping

• In any big data application, the next step is to map the analytics flow to specific tools and frameworks in the big data stack, based on the chosen analytics flow.

Analytics flow case study

• Weather Data Analysis uses
• Data Collection (collection processes)
• The Data Preparation stage
• The analysis stage
• The Batch
• The interactive tools
• Real-time Analytics stages.
• Batch
• Interactive
• Data analytics