Streaming Data Processing Systems

Streaming Data

• There are two main types of stream data processing systems: one based on existing relational database principles and another more suitable for streaming data.
• Flink is a second-generation stream processing system that utilizes automated scaling.
• Lambda architecture supports both stream and batch functionalities, providing accurate and up-to-date, near-real-time results.
• Kappa architecture does not have a batch layer, and all data are processed exclusively as streams.
• Stream analytics is important for sensing what is happening in the radio frequency electromagnetic spectrum with relatively low-cost sensors.
• Spectrum Streamer is a streaming architecture and an open-source library used to ingest time, frequency, and energy values of data in real-time.
• Real-time notifications are detected transmissions that can be used to update spectrum occupancy databases or directly notify other devices that a transmission is happening in a certain channel.
• Statistical reporting references a report delivered by computational statistics of the detected events, primarily targeting human users for report generation.
• Custom applications can be developed to subscribe to the output of the spectrum streamer for report generation.

Organizations

• Standards development organizations (SDOs) work to formulate health and safety standards.
• Industrial Internet Consortium (IIC) is a nonprofit organization created for the promotion of open standards and interoperability for technologies used in industrial and machine-to-machine (M2M) environments.

Processing Data

• Batch processing is a process that accumulates new data into disjunctive groups and processes them later.
• Stream processing is a process that handles each new piece of data at the time the data is being created, either individually or inside of a rolling window.
• Five main operations performed on streaming data: compression, dimensionality reduction, summarization, learning, and visualization.
• Compression reduces the occupied space for storing and/or the required bandwidth for transmitting data.
• Dimensionality reduction reduces the number of random variables (or dimensions) under consideration.
• Summarization transforms the data to provide a representation that best describes the time series under consideration.
• Learning and mining involve automating model building from recorded data and using models to extract insights and understand large datasets.
• Visualization transforms the data into a visual representation targeted at a human consumer.

Information Consumer

• An information consumer wants to join the data revolution and build a new application that draws on multiple disparate sources of data.

IoT Standards

• HyperCat is a specification for representing and exposing IoT platform catalogues over Web technologies.
• DCAT is a W3C recommendation aimed at facilitating interoperability between data catalogues.

Interoperability

• Three main aspects of interoperability: discovery, access control, and usage.
• Discovery involves identifying which aggregators contain information of interest and finding specific data sets.
• Access control involves facilities that allow information providers to retain control over access.
• Data access involves reducing technical barriers faced by both providers and consumers of information.

Types of Data

• Sensor data is the initial value required for metadata to be gathered.
• Metadata is a set of data that describes and presents information about other data.
• Dataset is a key-value pair of data that is relational to the initial query of data from a sensor.

IoT Evolution

• First Wave: Hardware related breakthrough that aimed to connect everyday objects to networks.
• Second Wave: Development of new sensors and materials as well as communication technologies and protocols.
• Third Wave: Data management processing and analytics as well as security, privacy, and trust.

IoT Ecosystem

• IoT ecosystem is a set of stakeholders that participate in the deployment of IoT technology and the relationship between them.
• Stakeholders include information providers, platform providers, application developers, and end users.

IoT Components

• Actuator: a device that converts an electrical signal into a corresponding physical quantity such as movement, force, or sound.
• Edge processing: a process of data aggregation, manipulation, bandwidth reduction, and other logic directly on an IoT sensor or device.
• Access network: part of the network concerned with moving data to and from a sensor or edge.
• Customer edge device: receives data from and sends commands to IoT devices, provides limited local storage, processing, and networking functions.
• Mobile edge computer: reduces transmission cost and provides fast interactive response in the computation of off-loading resources.

IoT Architecture

• Industrial Internet Reference Architecture (IIRA) is a standards-based architectural template and methodology that enables Industrial Internet of Things.

Security and Privacy

• BSI Kitemark is a recognizable symbol that indicates levels of trust, compliance, security, etc.
• California Consumer Privacy Act (CCPA) is a state statue intended to enhance privacy rights and consumer protection for residents of California in the United States.
• Vulnerabilities Equities Process (VEP) is a process used by the U.S. federal government to determine how it should treat zero-day computer security and vulnerabilities.

Blockchain

• Distributed Ledger Technology (DLT) is a decentralized database managed by multiple participants across multiple nodes.
• Permissionless DLTs allow any actor to become a node.
• Federated DLT is a permissioned DLT shared across more than one organization.
• Features of blockchains: decentralization, immutability, auditability, fault tolerance, and resilience.
• Benefits of blockchain: trust, resilience, adaptability, fault tolerance, security, and complete decentralization.

Streaming Data

• Two main types of stream data processing systems:
  • First type based on existing relational database principles
  • Second type suitable for streaming data, enables creation of custom operators and generation of new data streams
• Flink: second-generation stream processing system that utilizes automated scaling
• Lambda architecture: supports both stream and batch functionalities, provides accurate and up-to-date results
• Kappa architecture: batch layer is not present, and all data are processed exclusively as streams

Stream Analytics

• Important for sensing what is happening in the radio frequency electromagnetic spectrum, with relatively low-cost sensors
• Spectrum Streamer: streaming architecture and open-source library used to ingest time, frequency, and energy values of data in real-time
• Real-time notifications: detected transmissions that can be used to update spectrum occupancy databases or directly notify other devices
• Statistical reporting: report delivered by computational statistics of the detected events, targeting human users for report generation
• Custom applications: can be developed to subscribe to the output of the spectrum streamer for report generation

Organizations

• Standards development organizations (SDOs): work to formulate health and safety standards
• Industrial Internet Consortium (IIC): nonprofit organization created for the promotion of open standards and interoperability for technologies used in industrial and machine-to-machine (M2M) environments

IoT Concepts

• Journeys: represented as geospatial features (e.g., point, line, or polygon) with arbitrary associated properties described in metadata
• Geographical features: represented as geospatial features (e.g., point, line, or polygon) with arbitrary associated properties described in metadata
• Situations: includes an aggregation of sensor-based data and other data, as well as interpretation, relevant to a time and specified locations

Processing Data

• Batch processing: accumulates new data into disjunctive groups and processes them later, defined by criteria such as batch size or age
• Stream processing: handles each new piece of data at the time the data is being created, either individually or inside of a rolling window
• Five main operations performed on streaming data:
  • Compression: reduces occupied space for storing and/or the required bandwidth for transmitting
  • Dimensionality reduction: reduces the number of random variables (or dimensions) under consideration
  • Summarization: transforms the data to provide a representation that best describes the time series under consideration
  • Learning and mining: automated model building from recorded data and using models to extract insights and understand large datasets
  • Visualization: transforms the data into a visual representation targeted at a human consumer

IoT Ecosystem

• IoT Ecosystem: a set of stakeholders that participate in the deployment of IoT technology and the relationship between them
• Stakeholders in IoT Ecosystems:
  • Information Providers: owners of sensor deployments that may share information and data for the general good
  • Platform Providers: provide storage and computing infrastructure as well as analytics services
  • Application Developers: produce applications that process the available data within a specific context to produce actionable insight for end users
  • End Users: use the information and applications that are made available by other stakeholders

IoT Concepts

• Actuator: device that converts an electrical signal into a corresponding physical quantity such as movement, force, or sound
• Edge processing: process of data aggregation, manipulation, bandwidth reduction, and other logic directly on an IoT sensor or device
• Access network: part of network which is concerned with moving data to and from a sensor or edge
• Customer edge device: receives data from and sends commands to IoT devices & provides limited local storage, processing, and networking functions
• Mobile edge computer: reduces transmission cost and provides fast interactive response in the computation of off-loading resources
• Centralized IoT platform: powerful central storage and processing capability for IoT use cases delivering important capabilities
• IoT Runtime Environment: provides additional functionality beyond the container management level that enables remote management of information flows between multiple IoT applications and external endpoints
• Device management: encompasses the areas of initial device configuration and on-going management
• Secure Runtime Environment: component represents a comprehensive collection of tools that reduce the attack space through measures such as identity access management (IAM), secure boot, device attestation, TPMs, and trusted execution environments (TEEs)

IoT Tools and Technologies

• NodeJS: JavaScript for servers
• Node-RED: flow-based development tool meant for IoT, built on NodeJS
• Protocol adapter: protocol specific module enabling the transformation of incoming sensor data or outgoing actuator commands to a common format
• Information broker: on-board storage module that allows storage of recently received data from sensors or from the cloud or other modules
• Rules Engine: module to route the incoming data from other modules based on predefined rules
• Use case specific: module that is produced by a developer to perform a specialist function
• Management and Security: module that allows registration of the IoT device to the system and manages modules and configuration, handles security features

Healthcare and Energy

• Electronic health record (EHR): the electronic, systematized collection of patient and population health information stored in a digital format that can be shared across different healthcare settings
• Laboratory information system (LIS): software-based solution with features that support a modern laboratory's operations
• NoSQL: provides a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations
• Advanced metering infrastructure (AMI): integrated system of smart meters, communications networks, and data management systems that enable two-way communication between utilities and customers
• SM: smart meter
• IED: intelligent energy device
• PMU: phasor measurement unit
• VNF: virtual network function
• vMCM: virtual machine cloud machine function, offers digital twin services of the on-field physical assets
• vBCP: virtual blockchain processing function, gateway to blockchain infrastructures, allowing interaction of vBCP service-compliant devices with the blockchain
• vAAA: VNF delivering AAA, features two distinct service modes, namely blockchain-based and OAUTH2.0 service
• vMME: virtual mobility management entity responsible for registering devices to the network and advertising their presence on it
• vSON: virtual self-organizing networks enabling multihop and multiprotocol connectivity of IoT devices without direct access to 5g networking
• vTSD: virtual terminal service discovery responsible for effectively handling IoT devices services discovery

Streaming Data

• Two main types of stream data processing systems:
  • First type based on existing relational database principles
  • Second type suitable for streaming data, enables creation of custom operators and generation of new data streams
• Flink: second-generation stream processing system that utilizes automated scaling
• Lambda architecture: supports both stream and batch functionalities, provides accurate and up-to-date results
• Kappa architecture: batch layer is not present, and all data are processed exclusively as streams

Stream Analytics

• Important for sensing what is happening in the radio frequency electromagnetic spectrum, with relatively low-cost sensors
• Spectrum Streamer: streaming architecture and open-source library used to ingest time, frequency, and energy values of data in real-time
• Real-time notifications: detected transmissions that can be used to update spectrum occupancy databases or directly notify other devices
• Statistical reporting: report delivered by computational statistics of the detected events, targeting human users for report generation
• Custom applications: can be developed to subscribe to the output of the spectrum streamer for report generation

Organizations

• Standards development organizations (SDOs): work to formulate health and safety standards
• Industrial Internet Consortium (IIC): nonprofit organization created for the promotion of open standards and interoperability for technologies used in industrial and machine-to-machine (M2M) environments

IoT Concepts

• Journeys: represented as geospatial features (e.g., point, line, or polygon) with arbitrary associated properties described in metadata
• Geographical features: represented as geospatial features (e.g., point, line, or polygon) with arbitrary associated properties described in metadata
• Situations: includes an aggregation of sensor-based data and other data, as well as interpretation, relevant to a time and specified locations

Processing Data

• Batch processing: accumulates new data into disjunctive groups and processes them later, defined by criteria such as batch size or age
• Stream processing: handles each new piece of data at the time the data is being created, either individually or inside of a rolling window
• Five main operations performed on streaming data:
  • Compression: reduces occupied space for storing and/or the required bandwidth for transmitting
  • Dimensionality reduction: reduces the number of random variables (or dimensions) under consideration
  • Summarization: transforms the data to provide a representation that best describes the time series under consideration
  • Learning and mining: automated model building from recorded data and using models to extract insights and understand large datasets
  • Visualization: transforms the data into a visual representation targeted at a human consumer

IoT Ecosystem

• IoT Ecosystem: a set of stakeholders that participate in the deployment of IoT technology and the relationship between them
• Stakeholders in IoT Ecosystems:
  • Information Providers: owners of sensor deployments that may share information and data for the general good
  • Platform Providers: provide storage and computing infrastructure as well as analytics services
  • Application Developers: produce applications that process the available data within a specific context to produce actionable insight for end users
  • End Users: use the information and applications that are made available by other stakeholders

IoT Concepts

• Actuator: device that converts an electrical signal into a corresponding physical quantity such as movement, force, or sound
• Edge processing: process of data aggregation, manipulation, bandwidth reduction, and other logic directly on an IoT sensor or device
• Access network: part of network which is concerned with moving data to and from a sensor or edge
• Customer edge device: receives data from and sends commands to IoT devices & provides limited local storage, processing, and networking functions
• Mobile edge computer: reduces transmission cost and provides fast interactive response in the computation of off-loading resources
• Centralized IoT platform: powerful central storage and processing capability for IoT use cases delivering important capabilities
• IoT Runtime Environment: provides additional functionality beyond the container management level that enables remote management of information flows between multiple IoT applications and external endpoints
• Device management: encompasses the areas of initial device configuration and on-going management
• Secure Runtime Environment: component represents a comprehensive collection of tools that reduce the attack space through measures such as identity access management (IAM), secure boot, device attestation, TPMs, and trusted execution environments (TEEs)

IoT Tools and Technologies

• NodeJS: JavaScript for servers
• Node-RED: flow-based development tool meant for IoT, built on NodeJS
• Protocol adapter: protocol specific module enabling the transformation of incoming sensor data or outgoing actuator commands to a common format
• Information broker: on-board storage module that allows storage of recently received data from sensors or from the cloud or other modules
• Rules Engine: module to route the incoming data from other modules based on predefined rules
• Use case specific: module that is produced by a developer to perform a specialist function
• Management and Security: module that allows registration of the IoT device to the system and manages modules and configuration, handles security features

Healthcare and Energy

• Electronic health record (EHR): the electronic, systematized collection of patient and population health information stored in a digital format that can be shared across different healthcare settings
• Laboratory information system (LIS): software-based solution with features that support a modern laboratory's operations
• NoSQL: provides a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations
• Advanced metering infrastructure (AMI): integrated system of smart meters, communications networks, and data management systems that enable two-way communication between utilities and customers
• SM: smart meter
• IED: intelligent energy device
• PMU: phasor measurement unit
• VNF: virtual network function
• vMCM: virtual machine cloud machine function, offers digital twin services of the on-field physical assets
• vBCP: virtual blockchain processing function, gateway to blockchain infrastructures, allowing interaction of vBCP service-compliant devices with the blockchain
• vAAA: VNF delivering AAA, features two distinct service modes, namely blockchain-based and OAUTH2.0 service
• vMME: virtual mobility management entity responsible for registering devices to the network and advertising their presence on it
• vSON: virtual self-organizing networks enabling multihop and multiprotocol connectivity of IoT devices without direct access to 5g networking
• vTSD: virtual terminal service discovery responsible for effectively handling IoT devices services discovery

Streaming Data

• There are two main types of stream data processing systems: one based on existing relational database principles and another more suitable for streaming data.
• Flink is a second-generation stream processing system that utilizes automated scaling.
• Lambda architecture supports both stream and batch functionalities, providing accurate and up-to-date, near-real-time results.
• Kappa architecture does not have a batch layer, and all data are processed exclusively as streams.
• Stream analytics is important for sensing what is happening in the radio frequency electromagnetic spectrum with relatively low-cost sensors.
• Spectrum Streamer is a streaming architecture and an open-source library used to ingest time, frequency, and energy values of data in real-time.
• Real-time notifications are detected transmissions that can be used to update spectrum occupancy databases or directly notify other devices that a transmission is happening in a certain channel.
• Statistical reporting references a report delivered by computational statistics of the detected events, primarily targeting human users for report generation.
• Custom applications can be developed to subscribe to the output of the spectrum streamer for report generation.

Organizations

• Standards development organizations (SDOs) work to formulate health and safety standards.
• Industrial Internet Consortium (IIC) is a nonprofit organization created for the promotion of open standards and interoperability for technologies used in industrial and machine-to-machine (M2M) environments.

Processing Data

• Batch processing is a process that accumulates new data into disjunctive groups and processes them later.
• Stream processing is a process that handles each new piece of data at the time the data is being created, either individually or inside of a rolling window.
• Five main operations performed on streaming data: compression, dimensionality reduction, summarization, learning, and visualization.
• Compression reduces the occupied space for storing and/or the required bandwidth for transmitting data.
• Dimensionality reduction reduces the number of random variables (or dimensions) under consideration.
• Summarization transforms the data to provide a representation that best describes the time series under consideration.
• Learning and mining involve automating model building from recorded data and using models to extract insights and understand large datasets.
• Visualization transforms the data into a visual representation targeted at a human consumer.

Information Consumer

• An information consumer wants to join the data revolution and build a new application that draws on multiple disparate sources of data.

IoT Standards

• HyperCat is a specification for representing and exposing IoT platform catalogues over Web technologies.
• DCAT is a W3C recommendation aimed at facilitating interoperability between data catalogues.

Interoperability

• Three main aspects of interoperability: discovery, access control, and usage.
• Discovery involves identifying which aggregators contain information of interest and finding specific data sets.
• Access control involves facilities that allow information providers to retain control over access.
• Data access involves reducing technical barriers faced by both providers and consumers of information.

Types of Data

• Sensor data is the initial value required for metadata to be gathered.
• Metadata is a set of data that describes and presents information about other data.
• Dataset is a key-value pair of data that is relational to the initial query of data from a sensor.

IoT Evolution

• First Wave: Hardware related breakthrough that aimed to connect everyday objects to networks.
• Second Wave: Development of new sensors and materials as well as communication technologies and protocols.
• Third Wave: Data management processing and analytics as well as security, privacy, and trust.

IoT Ecosystem

• IoT ecosystem is a set of stakeholders that participate in the deployment of IoT technology and the relationship between them.
• Stakeholders include information providers, platform providers, application developers, and end users.

IoT Components

• Actuator: a device that converts an electrical signal into a corresponding physical quantity such as movement, force, or sound.
• Edge processing: a process of data aggregation, manipulation, bandwidth reduction, and other logic directly on an IoT sensor or device.
• Access network: part of the network concerned with moving data to and from a sensor or edge.
• Customer edge device: receives data from and sends commands to IoT devices, provides limited local storage, processing, and networking functions.
• Mobile edge computer: reduces transmission cost and provides fast interactive response in the computation of off-loading resources.

IoT Architecture

• Industrial Internet Reference Architecture (IIRA) is a standards-based architectural template and methodology that enables Industrial Internet of Things.

Security and Privacy

• BSI Kitemark is a recognizable symbol that indicates levels of trust, compliance, security, etc.
• California Consumer Privacy Act (CCPA) is a state statue intended to enhance privacy rights and consumer protection for residents of California in the United States.
• Vulnerabilities Equities Process (VEP) is a process used by the U.S. federal government to determine how it should treat zero-day computer security and vulnerabilities.

Blockchain

• Distributed Ledger Technology (DLT) is a decentralized database managed by multiple participants across multiple nodes.
• Permissionless DLTs allow any actor to become a node.
• Federated DLT is a permissioned DLT shared across more than one organization.
• Features of blockchains: decentralization, immutability, auditability, fault tolerance, and resilience.
• Benefits of blockchain: trust, resilience, adaptability, fault tolerance, security, and complete decentralization.

Streaming Data

• There are two main types of stream data processing systems: one based on existing relational database principles and another more suitable for streaming data.
• Flink is a second-generation stream processing system that utilizes automated scaling.
• Lambda architecture supports both stream and batch functionalities, providing accurate and up-to-date, near-real-time results.
• Kappa architecture does not have a batch layer, and all data are processed exclusively as streams.
• Stream analytics is important for sensing what is happening in the radio frequency electromagnetic spectrum with relatively low-cost sensors.
• Spectrum Streamer is a streaming architecture and an open-source library used to ingest time, frequency, and energy values of data in real-time.
• Real-time notifications are detected transmissions that can be used to update spectrum occupancy databases or directly notify other devices that a transmission is happening in a certain channel.
• Statistical reporting references a report delivered by computational statistics of the detected events, primarily targeting human users for report generation.
• Custom applications can be developed to subscribe to the output of the spectrum streamer for report generation.

Organizations

• Standards development organizations (SDOs) work to formulate health and safety standards.
• Industrial Internet Consortium (IIC) is a nonprofit organization created for the promotion of open standards and interoperability for technologies used in industrial and machine-to-machine (M2M) environments.

Processing Data

• Batch processing is a process that accumulates new data into disjunctive groups and processes them later.
• Stream processing is a process that handles each new piece of data at the time the data is being created, either individually or inside of a rolling window.
• Five main operations performed on streaming data: compression, dimensionality reduction, summarization, learning, and visualization.
• Compression reduces the occupied space for storing and/or the required bandwidth for transmitting data.
• Dimensionality reduction reduces the number of random variables (or dimensions) under consideration.
• Summarization transforms the data to provide a representation that best describes the time series under consideration.
• Learning and mining involve automating model building from recorded data and using models to extract insights and understand large datasets.
• Visualization transforms the data into a visual representation targeted at a human consumer.

Information Consumer

• An information consumer wants to join the data revolution and build a new application that draws on multiple disparate sources of data.

IoT Standards

• HyperCat is a specification for representing and exposing IoT platform catalogues over Web technologies.
• DCAT is a W3C recommendation aimed at facilitating interoperability between data catalogues.

Interoperability

• Three main aspects of interoperability: discovery, access control, and usage.
• Discovery involves identifying which aggregators contain information of interest and finding specific data sets.
• Access control involves facilities that allow information providers to retain control over access.
• Data access involves reducing technical barriers faced by both providers and consumers of information.

Types of Data

• Sensor data is the initial value required for metadata to be gathered.
• Metadata is a set of data that describes and presents information about other data.
• Dataset is a key-value pair of data that is relational to the initial query of data from a sensor.

IoT Evolution

• First Wave: Hardware related breakthrough that aimed to connect everyday objects to networks.
• Second Wave: Development of new sensors and materials as well as communication technologies and protocols.
• Third Wave: Data management processing and analytics as well as security, privacy, and trust.

IoT Ecosystem

• IoT ecosystem is a set of stakeholders that participate in the deployment of IoT technology and the relationship between them.
• Stakeholders include information providers, platform providers, application developers, and end users.

IoT Components

• Actuator: a device that converts an electrical signal into a corresponding physical quantity such as movement, force, or sound.
• Edge processing: a process of data aggregation, manipulation, bandwidth reduction, and other logic directly on an IoT sensor or device.
• Access network: part of the network concerned with moving data to and from a sensor or edge.
• Customer edge device: receives data from and sends commands to IoT devices, provides limited local storage, processing, and networking functions.
• Mobile edge computer: reduces transmission cost and provides fast interactive response in the computation of off-loading resources.

IoT Architecture

• Industrial Internet Reference Architecture (IIRA) is a standards-based architectural template and methodology that enables Industrial Internet of Things.

Security and Privacy

• BSI Kitemark is a recognizable symbol that indicates levels of trust, compliance, security, etc.
• California Consumer Privacy Act (CCPA) is a state statue intended to enhance privacy rights and consumer protection for residents of California in the United States.
• Vulnerabilities Equities Process (VEP) is a process used by the U.S. federal government to determine how it should treat zero-day computer security and vulnerabilities.

Blockchain

• Distributed Ledger Technology (DLT) is a decentralized database managed by multiple participants across multiple nodes.
• Permissionless DLTs allow any actor to become a node.
• Federated DLT is a permissioned DLT shared across more than one organization.
• Features of blockchains: decentralization, immutability, auditability, fault tolerance, and resilience.
• Benefits of blockchain: trust, resilience, adaptability, fault tolerance, security, and complete decentralization.

Learn about the two main types of stream data processing systems, including Flink and Lambda architecture, which support both stream and batch functionalities.