IoT and Big Data Fundamentals

IoT and Big Data Overview

• IoT growth is influencing the world.
• The Internet has expanded to include mobile devices, cars, TVs, and refrigerators.
• Expanded connectivity helps to understand relationships between different types of data.
• Big Data is critical for processing the large amount of data generated by IoT.
• Big Data technologies process data from Internet-connected devices.
• Early adopters gain business advantages from IoT and Big Data

IoT Big Data Characteristics

• Understanding IoT characteristics and challenges is key for designing a big data system.

Number of IoT Devices and Cost Reduction

• As costs decrease, adoption increases leading to a larger scale of usage.
• Expect billions of connected devices in the future.
• IoT products will be tested on a large scale.

Multiple IoT Devices & Data Types

• IoT uses devices from various manufacturers.
• Data from different device manufacturer models can differ.
• Data can be structured, semi-structured, and unstructured.
• Uses data types like XML, JSON, plain text, audio, video, and sensory data.

Veracity of Data

• IoT devices can send incorrect data or fail, causing data issues.
• Veracity is the quality, consistency, and trustworthiness of data.
• Veracity impacts the accuracy of analytics.

Update Frequency

• Devices vary in data frequency.
• Remote sensors have low frequency.
• Sophisticated devices such as cars have high frequency.
• Advanced tools are needed to analyze reported data.

Historical Data

• Big Data insights are derived from current and historical IoT data.
• Historical data enables smart monitoring and control.
• Building this history can take time.

Context Data

• Context adds value to IoT data.
• Context can be static, or dynamic.
• Location is static, weather is dynamic.
• Establishing relationships can be challenging.

Privacy Issues

• Data privacy is a risk when storing data from devices.
• Data can be shared with external systems.
• IoT systems should ensure user control through security policies.

IoT Big Data Architecture

• Well considered design solves Big Data challenges from IoT systems.
• Reference architectures from big players come from research and implementation.
• A review of Microsoft's logical Big Data architecture include options from Azure, AWS, and open source.

Cloud Gateway

• Cloud Gateway captures and stores messages.
• The cloud gateway acts as message buffer, implements queuing, ensures delivery, and scales processing.
• Cloud gateway options consist of Azure Event Hubs, Azure IoT Hub, AWS Device Gateway, Amazon Kinesis, and Apache Kafka.
• Amazon Kinesis Data Streams collects and processes large data streams in real time.
• Apache Kafka is an open-source system that streams process real-time data and integration at scale.
• Kinesis is easier to deploy.
• Kafka is better for DevOps teams that can configure its deployment.

Stream Processing

• Solution filters, aggregates, prepares data for the next step, and passes it to subscribers.
• Azure Stream Analytics and Amazon Kinesis provide managed stream processing to run SQL queries on data.
• Apache Storm™™ and Apache Spark™™ in an HDInsight cluster provide open-source options.
• A speed layer examines data streams and recognizes time patterns, detects anomalies, and triggers special alerts.

Cold Storage

• Data for non-real time processing is stored in an economical distributed file store known as a data lake.
• Azure Data Lake Store and Amazon S3 are options for implementing cold storage

Batch Analytics

• Batch jobs filter, aggregate, and prepare data for additional analysis of data sets.
• Steps generally reading source files, processing, and writing the output to new files.
• Options include running U-SQL jobs in Azure Data Lake Analytics, using AWS Glue with Amazon Athena, or open-source Apache Hive, Apache Pig , or custom Map/Reduce jobs in an HDInsight Apache Hadoop cluster or Amazon Elastic MapReduce.

Analytical Data Store

• Big Data solutions prepare data for analysis and allow queries with analytical tools.
• Azure SQL Data Warehouse and Amazon Redshift provide cloud-based data warehousing.

Analysis & Reporting

• Key objective of Big Data is to deliver data insights via analysis and reporting.
• A data modeling layer is essential for this.
• Azure Analysis Service gives a multidimensional Online Analytical Processing (OLAP) cube or tabular data models.
• Tools, for example Microsoft Power BI and Amazon Quicksight visualize the results.
• Data exploration may have data scientists or data analysts, with analytical notebooks such as Jupyter™.

Orchestration

• Orchestration automates repetition for fixed data workflow, routing data, loading data into a data store, and generating dashboards

Big Data Tools and Technologies

• NoSQL databases like Cassandra, MongoDB handle unstructured IoT data.
• Apache Kafka streams data.
• Tableau assists in interpreting large datasets through charts and dashboards.

Big Data Platforms for IoT

• Big data platforms for IoT collect, store, process, and analyze vast amounts of data from connected devices.

AWS IoT Analytics

• A fully managed service performs analytics on IoT data.
• Integrates with AWS such as S3, Lambda, and Sage Maker.
• Best for scalable cloud-based IoT solutions.
• Strengths are real time analytics, integration with the AWS ecosystem, and serverless architecture.

Google Cloud IoT Core

• Google Cloud IoT Core integrates with Big Query and Dataflow.
• The features are secure device connection, real-time and batch data processing, and machine learning integration.
• Best for real-time analytics and AI-powered insights.
• Its strengths are strong machine learning and integration with Google's cloud ecosystem.

Microsoft Azure IoT Hub & Azure Synapse Analytics

• The features include scalable device management, stream processing, and integration with Power BI.
• Best for enterprises using Microsoft products.
• Strengths include end to end IoT solutions with advanced analytics and security.

IBM Watson IoT Platform

• The features include device management, real-time insights, and AI-powered analytics.
• Best for industries like manufacturing, automotive, and healthcare.
• Strengths include AI/ML integration with Watson services.

Cloudera Data Platform (CDP) for IoT

• The features are real-time streaming analytics with Apache Kafka, Spark, and NiFi.
• Best for handling large-scale, distributed IoT data.
• Its strength is an open-source foundation with enterprise-grade features.

Thing Speak

• Collects, Visualizes, and Analyzes Live IoT Data streams
• Quick IoT projects that are academic and research.
• Simple Integration with MATLAB for analysis.

GE Predix

• GE Predix is part of GE Digital.
• Focuses on predictive maintenance and digital twins.
• Works with Industrial and manufacturing sector.
• Strong Analytics for operations and performance management.

SAP Leonardo IoT

• Integrates IoT Data with business processes.
• Strong with business process and industry solutions
• Enterprises that use SAP products.

Apache Hadoop

• HDFS stores large data sets.
• MapReduce processes data across distributed systems.
• Use to Analyze sensor data to detect long-term patterns.

Apache Spark

• Faster, processing in-memory.
• Works with real-time and data streams.
• Use case would include real-time anomaly detection

IoT Big Data Applications

• UPS captures 200 data points from 80,000 vehicles to track optimize fleet usage.

Barcelona

• Has smart parking meters with real-time updates and supports mobile payments.

John Deere Field Connect

• Monitors air and soil temperature, wind speed, humidity, solar radiation, rainfall, and leaf wetness to assist farmers.

Disney

• Uses RFID-based wearable bands to manage guest activities in the parks.

Alex and Ani

• Tracks customer activity in their stores, sends users offers as they enter the store.

King’s Hawaiian

• Connected machines that monitor factory performance to reduce downtime and lower maintenance costs.

BC Hydro

• Track hourly electricity usage and trends to curb theft; and alert company of outages.

Challenges of Big Data in IoT

• Encryption is used to ensure Data Security and Privacy.
• Cloud scaling is for Scalability.

Real-Time Processing

• Real time processing involves data coming from Kafka, edges, streams.

Data Integration

• Data integration involves middleware and APIs

Conclusion

• Data from IoT solutions is different and poses unique challenges
• Popular platforms from Azure and AWS are available.
• IoT and Big Data have significant business benefits.