Internet-of-Things (IoT) Systems Architectures, Algorithms PDF

Dimitrios Serpanos Marilyn Wolf Internet-of-Things (IoT) Systems Architectures, Algorithms, Methodologies Dimitrios Serpanos Marilyn Wolf Electrical & Computer Engineering School of ECE University of Patras Georgia Institute of Technology Patras, Greece Atlanta, GA, USA ISBN 978-3-319-69714-7 ISBN 978-3-319-69715-4 (eBook) https://doi.org/10.1007/978-3-319-69715-4 Library of Congress Control Number: 2017956194 © Springer International Publishing AG 2018 Preface The Internet of Things is the evolutionary step of the Internet that creates a world- wide infrastructure interconnecting machines and humans. As the Internet became public in the early 1990s, the first wave of its exploitation and deployment was mainly focused on the impact to everyday services and applications that changed the known models for financial transactions, shopping, news feeding and informa- tion sharing. It was a revolution that digitized a wide range of services as we knew them, from banking and retail shopping to face-to-face communication and govern- ment services. The first two decades of the Internet revolution focused strongly on consumer services and businesses, but human-centric. New business models appeared for banking, for online shopping, video communication, etc. for consum- ers. Business to business models and the cloud have impacted businesses signifi- cantly, wiping out large sectors of industry that did not adjust to the fast pace of the revolution. The impact on the economies has been tremendous. Now, more than two decades later, we witness and experience a new way of life because of the Internet’s reach to our homes and work environments. The advances of communication technology that enables the deployment and success of the Internet at home and work had an additional effect: the development of sophisticated interconnections among machines in the operational environment; we contrast the operational technology (OT) environment, which controls physical machines, to the information technology (IT) environment where humans are using computers for work. The already automated industrial environment received well the emerging technologies, adopted the suitable ones and created a, private mostly, network infrastructure that enables highly productive industrial processes. It has only been a natural step to evolve the Internet itself to include these processes. Additionally, the control models of the industrial environment, taking advantage of the smart devices –i.e. devices that include processing, memory and networking resources- that are deployed in various environments, have been extended and used in a wide variety of application domains. Conventional application domains like transportation, aeronautics, energy production and distribution, manufacturing and health adopt similar control models, exploiting smart sensors, actuators and devices that enable control automation for sophisticated applications. Critical infrastructure of countries is run using these technologies today. This emerging Internet-of-Things (IoT) is the natural evolutionary step of the Internet revolution that started about three decades ago. Importantly, IoT is building a worldwide infrastructure that will influence all facets of our life, from agriculture to mining, from health services to manufacturing and transportation. Clearly, it will provide the infrastructure over which the new emerging AI revolution will be based. This book addresses the fundamental IoT technologies, architectures, applica- tion domains and directions. Development of a complete IoT system and service includes several components. The hardware base includes embedded processors, memories of different types, sensors, actuators, cloud servers, intermediate process- ing systems, network systems and gateways. The software base includes operating systems, data bases and control applications for several application domains, to the very least. The combination of hardware and software components for control applications constitutes the base for the evolution of cyber-physical systems. VLSI capabilities play a huge role in the design of IoT systems. Event-driven, distributed operation shapes the design of architectures and applications. Specialized network protocols enable efficient communication in this environment, including appropri- ate machine-to-machine (M2M) communication models. These technologies are emerging with constraints and restrictions for the IoT environment that are different from the typical IT environment, because of the requirements for safety, real-time responses, low power operation, etc. Security, privacy, and safety require particular attention and special techniques. IoT is a fast-changing field. This book provides a snapshot of its current state. We continue to work in this area and hope to create updates to this book as the field progresses. Atlanta, GA, USA Marilyn Wolf Patras, Greece Dimitrios Serpanos Contents 1 The IoT Landscape 1 1.1 What Is IoT? 1 1.2 Applications 2 1.3 Architectures 4 1.4 Wireless Networks 4 1.5 Devices 4 1.6 Security and Privacy 5 1.7 Event-Driven Systems 5 1.8 This Book 6 Reference 6 2 IoT System Architectures 7 2.1 Introduction 7 2.2 Protocols Concepts 7 2.3 IoT-Oriented Protocols 10 2.4 Databases 12 2.5 Time Bases 13 2.6 Security 13 References 14 3 IoT Devices 17 3.1 The IoT Device Design Space 17 3.2 Cost of Ownership and Power Consumption 18 3.3 Cost per Transistor and Chip Size 19 3.4 Duty Cycle and Power Consumption 20 3.5 Platform Design 22 3.6 Summary 22 References 22 4 Event-Driven System Analysis 25 4.1 Introduction 25 4.2 Previous Work 26 4.3 Motivating Example 27 4.4 IoT Network Model 27 4.4.1 Events 27 4.4.2 Networks 28 4.4.3 Devices and Hubs 28 4.4.4 Single-Hub Networks 29 4.4.5 Multi-hub Networks 29 4.4.6 Network Models and Physical Networks 30 4.5 IoT Event Analysis 30 4.5.1 Event Populations 30 4.5.2 Stochastic Event Populations 32 4.5.3 Environmental Interaction Modeling 34 4.5.4 Event Transport and Migration 34 References 36 5 Industrial Internet of Things 37 5.1 Introduction 37 5.2 Industrie 4.0 39 5.3 Industrial Internet of Things (IIoT) 41 5.4 IIoT Architecture 42 5.5 Basic Technologies 49 5.6 Applications and Challenges 50 References 52 6 Security and Safety 55 6.1 Introduction 55 6.2 Systems Security 60 6.3 Network Security 62 6.4 Generic Application Security 64 6.5 Application Process Security and Safety 65 6.6 Reliable-and-Secure-by-Design IoT Applications 66 6.7 Run-Time Monitoring 67 6.8 The ARMET Approach 68 6.9 Privacy and Dependability 72 References 73 7 Security Testing IoT Systems 77 7.1 Introduction 77 7.2 Fuzz Testing for Security 78 7.2.1 White-Box Fuzzing 80 7.2.2 Black-Box Fuzzing 80 7.3 Fuzzing Industrial Control Network Systems 82 7.4 Fuzzing Modbus 82 7.4.1 The Modbus Protocol 82 7.4.2 Modbus/TCP Fuzzer 85 References 86 Index 91 Chapter 1 The IoT Landscape 1.1 What Is IoT? The Internet of Things (IoT) has become a common news item and marketing trend. Beyond the hype, IoT has emerged as an important technology with applications in many fields. IoT has roots in several earlier technologies: pervasive information systems, sensor networks, and embedded computing. The term IoT system more accurately describes the use of this technology than does Internet of Things. Most IoT devices are connected together to form purpose-specific systems; they are less frequently used as general-access devices on a worldwide network. IoT moves beyond pervasive computing and information systems, which con- centrated on data. Smart refrigerators are one example of pervasive computing devices. Several products included built-in PCs and allowed users to enter informa- tion about the contents of their refrigerator for menu planning. Conceptual devices would automatically scan the refrigerator contents to take care of data entry. The use cases envisioned for these refrigerators are not so far removed from menu planning applications for stand-alone personal computers. Sensor network research spanned a range of configurations. Many of these were designed for data collection at very low data rates. The collected data would then be sent to servers for processing. Traditional sensor network research did not empha- size in-network processing. Embedded computing concentrated on either stand-alone devices or tightly cou- pled networks such as those used in vehicles. Consumer electronics and cyber- physical systems were two major application domains for embedded computing; both emphasized engineered systems with well-defined goals. Given the wide range of advocates for IoT technology, no single, clear definition of the term has emerged. We can identify several possibilities: Internet-enabled physical devices, although many devices don’t use the Internet Protocol 2 1 The IoT Landscape Soft real-time sensor networks Dynamic and evolving networks of embedded computing devices This book is primarily interested in IoT systems. We use this term to capture two characteristics. First, the system is designed for one or a set of applications, rather than being an agglomeration of Internet-enabled devices. Second, the IoT system takes into account the dynamics of physical systems. An IoT system may consist primarily of sensors; in some cases it may include a significant number of actuators. In both cases, the goal is to process signals and time-series data. Interest in the Internet of Things has been spurred by the availability of micro- electromechanical (MEMS) sensors. Integrated accelerometers, gyroscopes, chemi- cal sensors, and other forms of sensor are now widely available. The low cost and power consumption of these sensors enables new applications well beyond those of traditional laboratory or industrial measurement equipment. These sensor applica- tions push IoT systems toward signal processing. IoT is also enabled by the very low cost of VLSI digital and analog electronics. As we will see, IoT nodes do not rely on state-of-the-art VLSI manufacturing pro- cesses. In fact, they are inexpensive because they are able to make use of older manufacturing lines; the lower device counts available in these older technologies are more than sufficient for many IoT systems. IoT systems must consume very little power. Power consumption is a key factor in total cost of ownership for IoT systems. Achieving the necessary power levels requires careful attention to hardware design, software design, and application algorithms. Security and safety are key design and operational requirements for IoT systems. As we have argued elsewhere, safety and security are no longer separable problems. The merger of computational and physical systems requires us to merge the previ- ously separate tasks of safe physical system design and secure computer system design. 1.2 Applications IoT systems are useful in a broad range of applications: Industrial systems use sensors to monitor both the industrial processes them- selves – the quality of the product – and the state of the equipment. An increasing number of electric motors, for example, include sensors that collect data used to predict impending motor failures. Smart buildings use sensors to identify the locations of people as well as the state of the building. That data can be used to control heating/ventilation/air condi- tioning systems and lighting systems to reduce operating costs. Smart buildings and structures also use sensors to monitor structural health. Smart cities use sensors to monitor pedestrian and vehicular traffic and may inte- grate data from smart buildings. 1.2 Applications 3 Vehicles use networked sensors to monitor the state of the vehicle and provide improved dynamics, reduced fuel consumption, and lower emissions. Medical systems connect a wide range of patient monitoring sensors that may be located at the home, in emergency vehicles, the doctor’s office, or the hospital. Use cases help us understand the requirements on an IoT system. Sensor network The system may act strictly as a data gathering system for a set of sensors. Alert system Data from sensors may be gathered and analyzed. Alerts are gener- ated when particular criteria are met. Analysis system Data from sensors is gathered and analyzed, but in this case, the analysis is ongoing. Reports on analytic results may be generated periodically – hourly, daily, etc. – or may be continuously updated. Reactive system Analysis of sensor data may cause actuators to be triggered. We reserve the term reactive for systems that don’t implement typical control laws. Control system Sensor data is fed to control algorithms that generate outputs for actuators. We can identify a class of nonfunctional requirements that apply to many IoT systems. Nonfunctional requirements on the system impose nonfunctional require- ments on the components. Event latency Latency from capture of an event to its destination may not be important for batch-oriented applications but becomes important for online analysis. Event throughput The rate at which events can be captured, transported, and pro- cessed depends on the throughput of the nodes, network bandwidth, and cloud throughput. Event loss rate and buffer capacity In the absence of strict upper bounds on event production rates, the environment may produce more events in an interval than the system can produce. Event loss rate captures the desired capability, while buffer capacity is a more pragmatic requirement that can be directly tied to component capabilities. Service latency and throughput Ultimately, events will be processed by services. We can also specify the latency and throughput for services. Reliability and availability Since IoT systems are distributed, reliability is more likely to be specified over parts of the network rather than reliability of the complete system. Availability is commonly used to describe distributed systems. Service lifetime IoT systems are often expected to have longer lifetimes than we expect for PC systems. The lifetime of the system or a subset of the system may be considerably longer than that of a component, particularly if the system uses redun- dant sensors and other components. 4 1 The IoT Landscape 1.3 Architectures A key aspect of IoT is event-driven or aperiodic sampling. Traditional digital signal processing and control assume periodic samples resulting in time-series data. However, time series consume too much power at the nodes and too much band- width on the network. Not all applications are amenable to aperiodic data acquisition. Constraints on power and bandwidth also encourage distributed computing over sensor events. Relatively small processors can perform useful processing on many data streams. Recognizing interesting events using edge processing reduces the amount of network bandwidth consumed; it also reduces power consumption since wireless communication requires large amounts of power. Cloud computing- (centralized servers) or fog computing (servers closer to the edge) can be used to perform further processing on those extracted events. 1.4 Wireless Networks Wireless networks are integral to IoT systems. Wireless network connections sim- plify installation and operation of wireless networks. However, wireless networks introduce some important problems and restric- tions. Radio communication requires more power than does wired communication. Some of the wireless networks used in today’s IoT devices were designed for other purposes, such as telephony and multimedia. As a result, they are not optimized for event-driven communication and consume significant amounts of power in the com- munications protocol. One of the ironies of IoT is that many edge devices and their wireless networks don’t operate on the Internet Protocol (IP). IP introduces significant overhead with an extra level of packetization and associated processing. Many IoT devices avoid IP and rely on upstream nodes to provide them with an Internet presence. IoT networks are typically run by noncomputer experts. IoT wireless networks must be easy to deploy and relatively self-managing. 1.5 Devices The characteristics of event-driven systems allow IoT nodes to be relatively simple. The realities of low-power operation also push nodes toward relatively low levels of integration. VLSI technology and Moore’s law are key factors in the rise of IoT systems because they allow nodes to be manufactured extremely cheaply. Very small chips can provide enough computation, memory, and networking for useful IoT node 1.7 Event-Driven Systems 5 functions. In contrast to traditional microprocessor and consumer electronic appli- cations, where chip areas range around or even higher, chips of several square mil- limeters are large enough for many IoT node devices. 1.6 Security and Privacy Security has finally been recognized as an essential requirement for all types of computer systems, including IoT systems. However, many IoT systems are much less secure than typical Windows/Mac/Linux systems. IoT security problems stem from a range of causes: inadequate security features in hardware, poorly designed software with a range of vulnerabilities, default passwords, and other security design errors. Insecure IoT nodes create problems for the security of the entire IoT system. Because nodes typically have lifetimes of several years, the large installed base of insecure devices will create security problems for some time to come. Insecure IoT systems also cause security problems for the rest of the Internet. IoT devices are plentiful; insecure IoT nodes are ideally suited to denial-of-service attacks. The Dyn attack [Sch16] is one example of an IoT-based attack on traditional Internet infrastructure. Privacy is related to security but requires specific measures at the application, network, and device levels. Not only must user data be protected from outright theft, but the network needs to be designed so that less-private data cannot easily be used to infer more private data. 1.7 Event-Driven Systems We believe that the event is a fundamental data type in IoT systems and that event- driven systems are an important structuring technique for IoT. Many of the building block technologies used for IoT today show some holdover from traditional, transaction-oriented systems. Event processing pushes us to treat time as a first- class concept and to consider the relationship between events in event sequences. We use the term event more broadly then do simulation engineers. We consider events as time-value sets. Event-driven system simulation is widely used for model- ing a wide range of engineering systems. In that context, an event is generally used to mean a change in the state of a variable. Given the decentralized nature of IoT systems, we are willing to consider stuttering – the repetition of an event value – as part of the event model. We also use events to model sampled data and time-series data. We believe that all these uses of the term event can be unified to create rich system structures. 6 1 The IoT Landscape 1.8 This Book The rest of this book describes a range of topics in IoT systems in more detail: Chapter 2 studies IoT system architectures, including wireless networks. Chapter 3 considers VLSI IoT devices. It describes the relationship between cost of ownership, power consumption, and duty cycle. Chapter 4 introduces analysis methods for event-driven IoT systems. These anal- ysis methods allow us to study the memory requirements implied by event com- munication and processing. Chapter 5 describes the Industrial Internet of Things and applications of IoT systems in smart energy systems. Chapter 6 studies security and safety issues in IoT systems. Computer and cyber- physical system security is closely tied to safety in sensor and closed-loop con- trol systems. Chapter 7 describes fuzz testing, a technique for testing the security of IoT sys- tems. Bugs and crashes can provide exploits for attackers; fuzz testing is designed to help identify such problems. Reference [Sch16] Schneier, B. (2016, October 22). DDoS attacks against Dyn. Schneier on Security. https://www.schneier.com/blog/archives/2016/10/ddos_attacks_ag.html Chapter 2 IoT System Architectures 2.1 Introduction In this chapter, we study architectures for IoT systems. We will study typical com- ponents used for networks, databases, etc. Figure 2.1 shows the organization of an IoT system: The plant or environment is the physical system with which the IoT system inter- acts. We will use these two terms interchangeably. A set of devices form the leaves of the network. A node may include sensors and/ or actuators, processors, and memory. Each node has a network interface. A node may or may not run the Internet Protocol. Hubs provide first-level connectivity between the nodes and the rest of the net- work. Hubs are typically run IP. Fog processors perform operations on local sets of nodes and hubs. Keeping some servers nearer the nodes reduces latency. However, fog devices may not have as much compute power as cloud servers. Fog devices also introduce sys- tem management issues. Cloud servers provide computational services for the IoT system. Databases store data and computational results. The cloud may provide a variety of services that mediate between nodes and users. 2.2 Protocols Concepts Several protocols are used for data services in IoT systems. Communication protocols may not provide sufficient abstraction for many appli- cations. IoT systems need multi-hop, end-to-end communication. They also may exhibit complex relationships between data sources and sinks. Higher-level proto- cols can provide services that model more closely the needs of IoT systems. Given 8 2 IoT System Architectures Fig. 2.1 Organization of an IoT system Fig. 2.2 The publish/subscribe model the heterogeneous and long-lived nature of most IoT systems, standards are often used rather than custom protocols. Several different protocols have been proposed and, to varying degrees, used for IoT systems [Duf13]. The user space has not yet converged on a single standard for IoT communication services. Given the prevalence of event-oriented models in IoT systems, a protocol should support event-style communication. The HTTP protocol uses a request/response design pattern. A client issues a request for a hypertext object; the server then replies with the object in response. A publish/subscribe protocol [Twi11] requires less coupling between the client and server as illustrated in Fig. 2.2. The server, known as a publisher, classifies mes- sages into categories. Clients subscribe to the categories of interest to them. Publish/ subscribe systems are typically mediated by brokers which receive published 2.2 Protocols Concepts 9 essages from publishers and send them to subscribers. Messages may be orga- m nized by topic; all message of a given topic are distributed by the brokers to the subscribers for that topic. The broker knows the identities of subscribers but the publisher does not. Brokers may interact with each other using a bridge protocol. A bridge allows indirect publication of messages, with a message going from the pub- lisher to a first broker, then to a second broker, and finally to subscribers who are not connected to the first broker. Data Distribution Service (DDS) (http://portals.omg.org/dds/) [Obj16] is a pub- lish/subscribe software architecture; several implementations of DDS are in use. A DDS domain maintains a logical global data space; the data is managed over a set of local stores. Publishers and subscribers are dynamically discovered across the network. Publishers can specify a number of quality of service parameters that are enforced by the brokers. Real-Time Publish/Subscribe Protocol (RTPS) [Obj14] is a so-called wire proto- col that defines a protocol for communication with DDS and other publish/sub- scribe systems. RTPS provides QoS properties, fault tolerance, and type safety. Esposito et al. [Esp09] developed an architecture for time-sensitive publish/sub- scribe systems that would be scalable to Internet-sized systems. They identified three major design goals: predictable latency, guaranteed delivery in the presence of multiple faults, and continued performance under scaling. They identified several types of fault models for publish/subscribe systems: network anomalies (loss, order- ing, corruption, delay, congestion, partitioning), link crash, node crash, and churn of nodes unexpectedly joining and leaving the system. Their architecture has three abstraction layers: the network layer consists of domains composed of nodes; the nodes layer consists of clusters, with each cluster’s members belonging to the same stub domain; and a coordinators layer. The coordination layer routes messages using a tree-based topology built on top of a distributed hash table. The coordinator is p-redundant to provide fault-tolerant coordination. To provide fault-tolerant over- lays, they formulate a model for path diversity that can be computed with limited knowledge of the network connections. Kang et al. [Kan12] used a semantics-aware communication mechanism to reduce overhead and improve reliability. They use state-space estimators at both the publisher and subscriber to maintain continuity of sensor values in the presence of network variations. Their state estimator is of the form xk + 1 = Fk + 1xk. The designer sets a model precision bound δ for each sensor. The bound is used to manage band- width requirements. Their system also dynamically adjusts the model precision bound. Choi et al. [Choi16] combined DDS with the OpenFlow software-defined net- working protocol to ensure that DDS can implement the QoS parameters. They added two QoS parameters that could not be easily deduced from the standard DDS parameters: MINIMUM_SEPARATION and an E2E_LATENCY specified by subscribers. 10 2 IoT System Architectures 2.3 IoT-Oriented Protocols We can divide protocols into two major categories: those that are tied to a specific physical layer and those that are not. Generally speaking, protocols that rely on a specific physical layer do not use the Internet Protocol, while protocols that are physical layer agnostic do use IP. Zigbee [Zig14, Far08] is a mesh network designed for low-power operation. A variety of derivative application standards specialize the protocol for applications such as smart homes and utilities. Zigbee is based on the IEEE 802.15.4 PHY and MAC standards. 802.15.4 operates in three bands: 868 MHz, 915 MHz, and 2.4 GHz. It delivers bit rates from 20 to 250 kbps, depending on the frequency band. The Zigbee NWK layer sits on top of the 802.15.4 MAC layer and provides data and management services. The APL layer includes three sections: the application sup- port sublayer, the Zigbee Device Objects layer, and the application framework. Zigbee provides two types of network security models: a centralized security network can be started only by a Zigbee coordinator/trust center; distributed secu- rity networks do not have a central trust center. Nodes can join either type of net- work and adapt to the type of network they have joined. Networks are formed by either coordinators or routers after scanning to select an available channel. Coordinators form centralized security networks, while routers form distributed security networks. Network steering is the name for the process by which a node joins a network. After identifying an open network, the node associates with that network and receives a network key. Clusters define interfaces for features and domains. Bluetooth Low Energy (BLE) (https://www.bluetooth.com/what-is-bluetooth- technology/how-it-works/low-energy) [Hay13] is a part of the Bluetooth standard designed for low-power operation such as devices powered from coin cell batteries. A BLE device can work as a transmitter, receiver, or both. Figure 2.3 illustrates the Bluetooth Classic protocol stack. The link layer provides an advertising service; devices can scan to identify nodes and networks. Devices can act as gateways to the Internet based on network address translation. The BLE protocol is stateful. BLE includes a number of optimizations to reduce power consumption. LoRa (http://lora-alliance.org) [LoR15] is designed for wide-area IoT applica- tions with a base station covering hundreds of square kilometers. It is designed to support a network topology with gateways for end devices, with gateways organized into their own star network. Data rates range from 0.3 to 50 kbps. MQTT (http://www.mqtt.org) [IBM12, Oas14] is an IoT-oriented protocol with publish/subscribe semantics. The protocol is designed for low overhead and is agnostic to the data payload. MQTT provides three levels of quality of service: at most once provides best-effort service, at least once assures delivery but may incur duplicates, and exactly once ensures the message is delivered without duplication. 2.3 IoT-Oriented Protocols 11 Fig. 2.3 The Bluetooth stack MQTT is based on a publish/subscribe model. A message is given a retention attribute when it is published; messages with QoS designations of at least once or exactly once should set the retention flag. A new subscriber to the topic will receive the last publication on that topic. When setting up a connection, a client can provide a will to the server to specify a message to be published if the client is unexpectedly disconnected. Messages are classified using topic strings similar to hierarchical file names. The set of topics is organized into a topic tree. Topic names follow the names of the nodes in the topic tree path, with node names separated by “/”. Subscribers can use wildcards in the topic string: ‘+’ denotes a wildcard match at one level of the topic tree; “#” denotes a match at any number of levels of the topic tree. XMPP (http://xmpp.org) is a protocol for streaming XML. It provides security, authentication, and information about network availability, and rosters of clients. XMPP-IoT (http://xmpp-iot.org) is a dialect of XMPP designed for IoT applications. REST [Vaq14, Rod15] is widely used for Web services and has received some use as an IoT service model. REST is a design pattern for stateless HTTP transfers. It exposes directory-structured form resource indicators. REST can be used to trans- fer XML or JSON data. Clients access resources using GET, PUT, POST, and DELETE methods. CoAP (http://coap.technology) [IET14] is a REST-based Web transfer protocol designed for IoT devices. It can be used with several types of data payloads, includ- ing XML and JSON. Google Cloud Pub/Sub [Goo17A, Goo17B] can be used to provide publish/sub- scribe service to IoT and other systems. Topics and subscriptions are exposed as 12 2 IoT System Architectures REST collections. The system is divided into a data plane for messages and a con- trol plane for allocation to servers known as routers; data plane servers are known as forwarders. The routers balance consistency and uniformity of data using a con- sistent hashing algorithm. A message life cycle includes several steps. When a pub- lisher sends a message, it is written to storage. The subscribers receive the message, and the publisher receives an acknowledgment. Subscribers acknowledge the mes- sage to Google Cloud Pub/Sub. The message is deleted from storage once at least one subscriber for each subscription has acknowledged the message. The system monitors itself to detect and mitigate service problems. Amazon Web Services (AWS) IoT [Bar15] is a managed cloud service for IoT devices, which are termed things. A thing shadow is a cloud model of a thing. A rule engine transforms messages based on rules and routes the results to AWS services. The message broker is based on MQTT. A Thing Registry assigns unique identity to things. Microsoft Azure (https://azure.microsoft.com/en-us/services/iot-hub/) provides IoT-oriented services. Its Service Fabric is a middleware communication system that supports microservices running on a cluster. A microservice may be either stateless or stateful. It also provides a container model for applications; a container provides an isolated environment but relies on the operating system, in contrast to a virtual machine which runs underneath the operating system. It provides databases using both structured and unstructured approaches. It also provides APIs for artifi- cial intelligence services. 2.4 Databases Databases are used for both short-term and long-term storage. Applications may rely on databases to retrieve data over a time window for analysis. Some use cases may require archival storage of values. Unstructured databases, known as noSQL, are used in many IoT systems. A noSQL database does not have a schema. Simple noSQL databases represent data as key-value pairs, but other representations are possible. The lack of a schema allows quick deployment but may cause maintenance problems. The Amazon Simple Storage Service (Amazon S3) (https://aws.amazon.com/ s3/) is an object store with a Web service interface. Data can be pushed to other, lower-cost storage services for long-term, infrequent use. Notifications can be issued when objects operated upon. Google Cloud Storage (https://cloud.google.com/storage) is an object store for unstructured data. It provides three different service models at different latency/ latency/price points. Cloud SQL can be used to perform database operations. Streaming transfers are supported using HTTP chunked transfer encoding. Time-series data possesses structure that may require special handling to provide proper database performance. Time series are sometimes stored as blobs in rela- tional databases to allow specialized algorithms. 2.6 Security 13 Dynamic time warping (DTW) [Rat04, Rak12] is widely used to search over time-series data. DTW was originally used to compare waveforms for speech pro- cessing. Correlation provides a direct comparison of two waveforms. By warping one waveform, non-exact matches can be found. Dynamic programming can be used to find the minimum warp match between two-time series; a limit on maxi- mum warping is typically applied to avoid obviously bad matches. Very efficient algorithms have been developed to provide high-speed search. Among other tech- niques, these algorithms abandon a warp computation early when partial results exceed a given bound. Fast DTW algorithms have been used to search very large databases. 2.5 Time Bases Many IoT systems require a notion of global time. Several algorithms, starting with Lamport’s algorithm [Lam78], have been developed for the synchronization of clocks in a distributed system. The Network Time Protocol (RFC1305) is used on the Internet for distributed time synchronization. 2.6 Security Security is a system property; the system can be only as secure as its weakest com- ponent. Security features are provided by components at several layers in the IoT stack: devices, physical networks, and middleware. A unified view of IoT system security architectures has not yet emerged. Some, but not all processors for low-power operation, provide security features such as encryption accelerators and root of trust. The National Security Agency has developed families of lightweight block ciphers [Sch13]: SIMON targets hardware implementations, and SPECK is intended for software implementations. Gulcan et al. [Gul14] developed a low-power implementation of SIMON. Several networks provide security features. Bluetooth Low Energy provides a Simple Secure Pairing protocol to protect against passive eavesdropping. It also provides address randomization. As discussed above, Zigbee provides two network security models: centralized and distributed. LoRa provides unique network keys, unique application keys, and device-specific keys. MQTT does not specifically require encryption, but it can be used with several different security standards. MQTT and the NIST Framework for Improving Critical Infrastructure Cybersecurity [Oas14B] describe the relationship between MQTT and the NIST Cybersecurity Framework. We will study IoT system security in more detail in Chap. 6. 14 2 IoT System Architectures References [Bar15] Barr, J. (2015, October 8). AWS IoT: Cloud services for connected devices. AWS Blog. https://aws.amazon.com/blogs/aws/aws-iot-cloud-services-for-connected-devices/ [Choi16] Choi, H.-Y., King, A. L., & Lee, I. (2016). Making DDS really real-time with OpenFlow. 2016 international conference on embedded software (EMSOFT) (pp. 1–10). Pittsburgh, PA. [Duf13] Duffy, P. (2013, April 30) Beyond MQTT: A Cisco view on IoT protocols. Cisco Blogs. https://blogs.cisco.com/digital/beyond-mqtt-a-cisco-view-on-iot-protocols [Esp09] Esposito, C., Cotroneo, D., & Gokhale, A.. 2009. Reliable publish/subscribe middle- ware for time-sensitive internet-scale applications. Proceedings of the third ACM international conference on distributed event-based systems (DEBS’09). ACM, New York, Article 16, 12 pages. [Far08] Farahani, S. (2008). Zigbee wireless networks and transceivers. Amsterdam: Newnes. [Goo17A] Google. (2017, April 19). What is Google Cloud Pub/Sub? https://cloud.google.com/ pubsub/docs/overview [Goo17B] Google. (2017, April 3). Google Cloud Pub/Sub: A Google-scale messaging service. https://cloud.google.com/pubsub/architecture [Gul14] Gulcan, E., Aysu, A., & Schaumont, P. (2015). A flexible and compact hardware archi- tecture for the SIMON block cipher. In T. Eisenbarth & E. Öztürk (Eds.), Lightweight cryptog- raphy for security and privacy. LightSec 2014, Lecture Notes in Computer Science (Vol. 8898, pp. 34–50). Cham: Springer. [Hay13] Heydon, R. (2013). Bluetooth low energy: The developer’s handbook. Prentice Hall: Upper Saddle River, NJ. [IBM12] IBM International Technical Support Organization (2012, September). Building smarter planet solutions with MQTT and IBM WebSphere MQ telemetry, Redbooks. [IET14] Internet Engineering Task Force (2014, June). The constrained application protocol (CoAP), RFC 7252, Shelby, Z., Hartke, K., & Bormann, C. [Kan12] Kang, W., Kapitanova, K., & Son, S. H. (2012). RDDS: A real-time data distribu- tion service for cyber-physical systems. IEEE Transactions on Industrial Informatics, 8(2), 393–405. [Lam78] Lamport, L. (1978). Time, clocks, and the ordering of events in a distributed system. Communications of the ACM, 21(7), 558–565. [LoR15] LoRa Alliance (2015, November). LoRaWAN: What is it? A technical overview of LoRa and LoRaWAN. [Oas14] Oasis. (2014, 29). MQTT version 3.1.1. Oasis standard. [Oas14B] Oasis (2014, May 28). MQTT and the NISTG cybersecurity framework version 1.0. Committee note 01. [Obj14] Object Management Group. (2014). The real-time publish-subscribe protocol (RTPS) DDS interoperability wire protocol specification, Version 2.2. [Obj16] Object Management Group. (2016). What is DDS? http://portals.omg.org/dds/what-is- dds-3/, accessed May 4, 2017. [Sch13] Schneier, B. SIMON and SPECK: New NSA encryption algorithms. Schneier on Security. https://www.schneier.com/blog/archives/2013/07/simon_and_speck.html, retrieved May 8, 2017. [Vaq14] Vaqqas, M. (2014, September 23) RESTful web services: A tutorial. Dr. Dobb’s. http:// www.drdobbs.com/web-development/restful-web-services-a-tutorial/240169069 [Rak12] Rakthanmanon, T., Campana, B., Mueen, A., Batista, G., Westover, B., Zhu, Q., Zakaria, J., & Keogh, E.. 2012. Searching and mining trillions of time series subsequences under dynamic time warping. Proceedings of the 18th ACM SIGKDD international conference on knowledge discovery and data mining (KDD’12) (pp. 262–270). ACM, New York. [Rat04] Ratanamahatana, C. A., & Keogh, E. (2004, August 22–25). Everything you know about dynamic time warping is wrong. Third workshop on mining temporal and sequential data, in References 15 conjunction with the tenth ACM SIGKDD international conference on knowledge discovery and data mining (KDD-2004). Seattle, WA. [Rod15] Rodriguez, Alex. (2008, November 6). RESTful web services: The basics. IBM devel- operWorks, updated February 9, 2015. https://www.ibm.com/developerworks/library/ws-rest- ful/index.html [Twi11] Twin Oaks Computing, Inc. (2011). What can DDS do for you? [Zig14] Zigbee Alliance (2014, December 2). ZigBee 3.0: The open, global standard for the Internet of Things. http://www.zigbee.org/zigbee-for-developers/zigbee/ Chapter 3 IoT Devices 3.1 The IoT Device Design Space The design space for IoT devices is very different from that for mobile or cloud processors. Both mobile and cloud systems require very large chips. IoT devices should operate at extremely low power levels but often not operate continuously. They must integrate processors, memory and storage, communication, and sensors. They will also be sold in quantities that dwarf even those of mobile processors, which in turn require a very low price. Purchase price is, however, only one compo- nent of the IoT device cost model. Total cost of ownership will drive many IoT markets – these devices will be installed for use over a lifetime of several years. Installation cost is an important element in the decision to purchase and install these devices. We will see that cost of ownership is directly tied to power consumption. The sensors and MEMS communities have long been interested in IoT as an application for integrated sensors and actuators. Many commentators have called for a trillion sensor world. This goal is in fact very realistic given current industry capabilities. According to Semi.org [Die16], worldwide manufacturing capacity for 200 mm wafers is expected to be 5.4 million wafers per month in 2018. If all this capacity is used for IoT, it translates to 678 billion chips per month of size 1 mm2 or 68 billion per month of 10 mm2 chips. That capacity puts the industry within range of producing a trillion sensors per year. We could reach the trillion sensors per year mark simply by reallocating existing capacity. Even if production does not com- pletely reach the trillion sensor mark, the industry can clearly manufacture huge volumes of sensors. 18 3 IoT Devices 3.2 Cost of Ownership and Power Consumption Lifetime cost of ownership is a key metric for IoT devices [Wol16]. The cost of an IoT silicon includes several components: sensing and actuation, computation, net- working, as well as packaging. The completed IoT device includes power supply and packaging. However, installation cost is a significant factor in the cost of owner- ship. The cost of installing a cable drop in an existing building in the USA is, in the authors’ experience, around $150. That cost overwhelms the cost of hardware. Eliminating all wiring – both power and networking – substantially reduces instal- lation cost. The cost of replacing batteries is significant. Our colleague Rajesh Gupta reported that the computer science building at University of California, San Diego, requires a full-time employee to replace batteries on electronic door locks (Rajesh Gupta, personal communication, February 2014). The ability to power devices entirely by energy harvesting would eliminate that cost but imposes con- straints on the devices. The high cost and effort of wired power have encouraged the development of energy-scavenging (also known as energy-harvesting) technologies. A range of physical mechanisms can be used to convert energy for use by the environment. Since most scavenging sources provide varying amounts of power, the harvested energy is stored for later use. Electric power may be stored in a battery, a capacitor, or a supercapacitor. On-chip power management circuitry stores harvested energy and then regulates the power as it is used by the rest of the chip. Paradiso and Starner [Par05] identified several widely different sources of energy, including radio frequency, ambient light, thermoelectricity, and heel strikes. They pointed out that indoor lighting provides much lower ambient light levels than are available from the sun. Sudevalayam and Kulkarni [Sud11] surveyed energy- harvesting technologies for sensor nodes. They identified a range of technologies with different sources, conversion efficiencies, and harvest yield. They reported, for example, that light converged by solar cells typically provided 15 mW/cm2, wind by anemometer provided 1200mWh/day, and provided footfalls 5W. Romani et al. [Rom17] survey power conversion and management architectures for ambient-powered IoT devices. Their reference architecture for a no-battery power management system includes several components. A transducer extracts power from an external power source with efficiency η. Several sources of internal power consumption further limit the overall system efficiency: power control cir- cuitry consumes intrinsic power Pint; the storage element leaks power at a rate Pleak; monitor circuits consume Pvmon. A bootstrap circuit may be used to initialize the system from discharge. They note that a key challenge of the power management controller is to match the effective load impedance to the power source’s internal impedance. 3.3 Cost per Transistor and Chip Size 19 3.3 Cost per Transistor and Chip Size Commentators have noted that established technology nodes offer cost-effective manufacturing for many products [Whi15]. One article [Hru12] quotes an NVIDIA presentation claiming that cost per transistor for 20 and 14 nm nodes was barely lower than that of the previous node and that 20 nm is “essentially worthless.” Maly [Mal94] developed an early cost model for the cost of silicon as a function of manu- facturing node. His model computed cost per transistor as a function of design den- sity, minimum feature size, wafer area, and wafer cost. An even simpler cost model is based on the total cost of a manufactured wafer, including the cost of the wafer itself and all processing. As shown in Fig. 3.1, that cost will decrease slightly as the manufacturing pro- cess matures. However, the cost of a manufactured wafer grows significantly at advanced nodes [Wol17]. Double patterning became required for lithography at 20 nm. This technique uses two masks for each feature, roughly speaking one per edge; the size of a fabricated feature can be smaller than the size of a feature on either of the masks. Double patterning requires two masks per step rather than one; since mask costs are a large part of the cost of the manufactured wafer, double pat- terning (and the more recent use of triple patterning) substantially increases manu- factured wafer cost. Increasing the number of masks adds costs beyond those of the masks themselves: more time must be spent with the wafers in expensive equip- ment; wafers spend longer in the manufacturing plant. We can write a formula for the cost per transistor based on the manufactured wafer cost Cm and the number of working transistors per wafer ntr: Cm Ctr =. ntr In the standard Moore’s Law scenario, we expect the number of transistors per wafer to double from one generation to the next. If the cost of processing the wafer increases by less than that factor, cost per transistor goes down; if not, Fig. 3.1 Cost of processed wafers over time and technology node 20 3 IoT Devices cost-per-transistor increases. Put another way, if Cm(B)/Cm(A) > rtr for technology nodes A and B, where rtr is the factor increase in working transistors per wafer from technology A to B, then cost per transistor increases. The transition from 28 to 20 nm was an inflection point at which the cost per manufactured wafer grew enough to offset density gains. Given that these costs continued at smaller nodes, the cost- per-transistor reached a local minimum at 28 nm. It is likely that 28 nm will prove to be the global minimum of cost-per-transistor; more advanced lithography meth- ods have their own costs. Another major component of chip cost is silicon area. The traditional emphasis in VLSI has been on large chips to maximize functionality. However, silicon tech- nology has advanced to the point where we can provide interesting functionality on very small amounts of silicon, thereby providing low-cost chips. This is true even for technology nodes such as 28 nm, which are large relative to the nodes used for latest-generation chips but still very dense relative to historical standards. The transistor counts of early microprocessors provide context for the circuitry required to provide useful functionality. The IBM PC’s CPU was an Intel 8088 run- ning at 4.77 MHz [Wik16A]; the 8088 contained approximately 4000 transistors [Wik16B, Wik16C]. Packaging is another significant component of the cost of integrated circuits. A wide range of system-in-package technologies have been developed that provide several improvements over traditional single-die packaging: the ability to combine chips from several manufacturing processes, each with its own native devices; reduced inter-die parasitic values; and lower cost. The DARPA SHIELD chip [Ral16] provides an example of a very low-cost, highly integrated IoT chip. SHIELD is designed to be attached hardware modules – chips, boards, etc. – to provide a secure, traceable identifier for that module. Since each module in a system would require its own identifier, SHIELD targets a very low manufacturing cost of one cent. The system includes several die combined in a leadless package: a CMOS module, an RF pickup coil, and a thin-film temperature sensor. The CMOS module is 100 μm × 100 μm in a 14 nm FinFET technology; it combines a digital CPU and communication, onetime programmable memory, a physically unclonable function (PUF), an analog-digital converter, and power con- version and management circuitry. The device is powered by near-field RF energy; the RF coil is used for both power delivery and communication. 3.4 Duty Cycle and Power Consumption The duty cycle model is widely used to analyze IoT devices. As shown in Fig. 3.2, the model assumes periodic activation of the device. The duty cycle is the percent- age of time for which the device is on: 3.4 Duty Cycle and Power Consumption 21 Fig. 3.2 The IoT device duty cycle O D= ´100%. T Lower duty cycles mean lower energy consumption. We can change the duty cycle through a combination of changes to the operating time O and the period T. Reducing the operating time may reduce the device’s functionality; increasing its period lowers its data rate. Let the on-state power consumption of the device be Pon. If we assume zero leak- age, then the power consumption under duty cycle operation is O Pideal = Pon. T If the device has a leakage power of Poff, then its average power consumption over the duty cycle is O æ Oö Pleak = Pon + ç 1 - ÷ Poff. T è Tø We can also solve for fractional duty cycle as a function of on-state and off-state power and total power consumption: O Pleak - Poff =. T Pon - Poff This model carries several implications for the design of IoT devices: the device must be good at idling at low power; it should provide low energy and time to shut down and to turn back on. Communication power is a large fraction of the total power consumption of many IoT devices. Many IoT devices transmit small amounts of data during the on portion of their duty cycle. In this scenario, the overhead associated with setting up a communication is a significant part of the total communication power; many com- munication systems are designed for connection-oriented service that allows setup costs to be amortized over a longer communication. Dementyev et al. [Dem13] measured the power consumption of several wireless protocols. They used their data to determine the optimal period T for each protocol: 14.3 s for Zigbee and 10.0 s for Bluetooth Low Energy (BLE). 22 3 IoT Devices 3.5 Platform Design Unlike mobile devices, most IoT devices do not operate continuously. Nonetheless, they need to retain state from activation for a range of purposes: communication status, DSP filtering, etc. SRAM requires power to retain state and thereby length- ens the allowable duty cycle. Flash memory must be written in blocks. Emerging technologies offer the promise of bit-level persistent-state devices that can be used within the processor, not just as memory. Soerken et al. [Soe17] developed a programmable logic-in-memory (PLiM) using resistive RAM (RRAM) devices. An RRAM device has persistent state – it can be written and retains its state after the power supply is removed – making it well suited to the duty cycle characteristics of IoT devices. They designed their processor to take advantage of the majority-logic characteristics of RRAMs. They developed a compiler to translate Boolean functions into instruction streams for their processor. 3.6 Summary IoT systems open up a new horizon for VLSI design. IoT systems require ultra-low power systems that combine disparate elements – computation, communication, and sensing – at very low price points. IoT systems emphasize small, capable chips in contrast to the large chips that have driven the industry for many years. We are at the early stages in the development of this new category of chip. References [Dem13] Dementyev, A., Hodges, S., Taylor, S., & Smith, J. (2013). Power consumption analysis of Bluetooth Low Energy, ZigBee and ANT sensor nodes in a cyclic sleep scenario. Wireless Symposium (IWS), 2013 IEEE International, Beijing, 2013, pp. 1–4. [Die16] Dieseldorf, C. G. (2016). Foundries Take Over 200mm Capacity Fab by 2018. www. semi.org, January 25, 2016. [Hru12] Hruska, J. (2012). Nvidia deeply unhappy with TSMC, claims 20 nm essentially worthless. extremetech.com, http://www.extremetech.com/computing/123529-nvidia-deeply- unhappy-with-tsmc-claims-22nm-essentially-worthless, March 23, 2012. [Mal94] Maly, W. (1994). Cost of Silicon Viewed from VLSI Design Perspective. Design Automation, 1994. 31st Conference on, San Diego, CA, USA, 1994, pp. 135–142. [Par05] J. A. Paradiso and T. Starner, Energy scavenging for mobile and wireless electronics IEEE Pervasive Computing, vol. 4, no. 1, pp. 18–27, 2005. [Ral16] Ralston, P., Fry, D., Suko, S., Winters, B., King, M., & Kober, R. (2016). Defeating counterfeiters with microscopic dielets embedded in electronic components. Computer, 49(8), 18–26. References 23 [Rom17] Romani, A., Tartagni, M., & Sangiorgi, E. (2017). Doing a lot with a little: Micropower conversion and management for ambient-powered electronics. Computer, 50(6), 41–49. [Soe17] Soeken, M., Gaillardon, P. E., Shirinzadeh, S., Drechsler, R., & Micheli, G. D. (2017). A PLiM computer for the internet of things. Computer, 50(6), 35–40. [Sud11] Sudevalayam, S., & Kulkarni, P. (2011, Third Quarter). Energy harvesting sensor nodes: Survey and implications. IEEE Communications Surveys & Tutorials, 13(3), 443–461. [Whi15] White, M. (2015). IoT, Cost-per-Transistor Extend Lifetimes of Established Technology Nodes. Electronic Design, May 15, 2015, http://electronicdesign.com/eda/ iot-cost-transistor-extend-lifetimes-established-technology-nodes [Wik16A] Wikipedia. (2016). IBM Personal Computer. https://en.wikipedia.org/wiki/IBM_ Personal_Computer. Accessed October 19, 2016. [Wik16B] Wikipedia. (2016). Intel 8088. https://en.wikipedia.org/wiki/Intel_8088. Accessed October 19, 2016. [Wik16C] Wikipedia. (2016). Transistor count. https://en.wikipedia.org/wiki/Transistor_count, Accessed October 19, 2016. [Wol16] Wolf, M. (2016). Ultralow power and the new era of not-so-VLSI. IEEE Design & Test, 33(4), 109–113. [Wol17] Wolf, M. (2017). The physics of computing. Cambridge MA: Elsevier. Chapter 4 Event-Driven System Analysis 4.1 Introduction This chapter describes modeling and analysis methods for Internet of Things (IoT) system design. IoT systems require new types of analysis because events do not necessarily result in immediate actions or maintain their order relative to other events. Traditional methods such as the distributed control-oriented methods of Thiele and Ernst consider possibly infinite streams of events or samples, but the lifetime of an event/sample in the system is relatively short. In contrast, IoT systems must deal with event lifetimes at multiple time scales: some events may schedule activity only seconds in the future, while other events may schedule activity days, weeks, or months ahead. IoT also do not maintain temporal order of causality – one event may cause an event in the near future, while another event may cause an event in the far future. We need new analytical methods for multiple time scales and complex cau- sality relationships. The primary goal of our analysis is the understanding of the required character- istics of the IoT platform. We propose a model of the IoT system as a network with devices as leaf nodes and hubs as non-leaf nodes. Hubs perform routing functions but for our purposes their key role is to control the timing of event activity through the use of timewheels. While we assume that events carry key-value pairs, we are not concerned here with the semantics of events. Instead, we analyze the lifetimes of event populations. Event populations depend in part on the activity of the envi- ronment in which the IoT operates. To accommodate a wide range of realistic sce- narios, we develop models based on both deterministic and stochastic event timing. 26 4 Event-Driven System Analysis 4.2 Previous Work Several lines of work have established event-based models for real-time networked systems. One of the goals of these projects has been to unify the analysis of network- oriented events and the computation on the network nodes that transform one stream of events into another. Chakraborty et al. [Cha03] developed a real-time calculus that models events and resources. They model an event stream R[s, t) over the prescribed time interval as a pair of arrival curves: αl(∆) for the lower bound on the number of events in the interval ∆ and αu(∆) for the upper bound of events in the interval. They show how to model event streams with jitter. They use β functions to model the service pro- vided by computational and communication components. They show how to ana- lyze single streams, multiple interacting streams, and platforms with multiple computing and communication resources. Maxiaguine et al. [Max04] used work- load curves to characterize the computational workload of real-time systems. They showed how to use their methods to analyze both a rate-monotonic system and streaming architectures. Henia et al. [Hen05] give definitions and formulas for events and event streams. Many of their results apply to our model; we summarize some of their applicable results here. Event time applies to both generation and release time. An event time includes a nominal time and jitter: T,J A periodic event stream has parameters period and jitter: P, J The upper event function ηu(Δt) gives the maximum number of events in the interval Δt. Similarly, the lower event function ηl(Δt) gives the minimum number of events in the interval Δt. The upper and lower event functions for a periodic event stream with jitter are Dt + J æ Dt - J ö h Pu + J = , h Pl + J = max ç 0, P è P ÷ø They give formulas for the jitter of the output of components that combine event streams using AND and OR combination methods. 4.4 IoT Network Model 27 4.3 Motivating Example IoT systems are built for a variety of applications: industrial control, environmental monitoring, logistics, etc. We will use examples in this paper derived from our experiments with IoT systems for long-term care [Wol15]. This application pro- vides us with use cases typical of smart homes (turning on and off lights, energy management, etc.) as well as use cases associated with health care (scheduling med- ications, checking on the condition of residents, etc.). Our example IoT system operates in a home with several residents, as a rotating set of staffers, and visitors. A variety of sensors monitor activity in the home: cameras, utility sensors, smart objects, etc. The IoT system is designed to track the activity of residents and staffers and to alert staffers of situations that may deserve their attention. The system architecture consists of several elements: A set of sensors A local hub that monitors the sensors as I/O devices A cloud-based node for some analytical functions A key feature of the local hub is its internal timewheel (Coelho, D., 2014, August 2, private communication). Timewheels are used in event-driven simulation to man- age simulator event activity; in this case, we use the timewheel to manage events in the real world as mediated by the I/O devices. Events are timestamped with a time at which they should occur, which may be later than the time at which the event was generated. The timewheel is a time-sorted queue; when the clock time equals the timestamp of the event at the head of the timewheel, that event is dequeued and processed. 4.4 IoT Network Model Our IoT network model is oriented toward the analysis of event behavior in the system. Because events have long lives, memory in the form of timewheel queues plays a critical role in the model. 4.4.1 Events The model of computation is based on long-lived events. An event is generated at a node and stored until it is ready to be completed, at which time it is consumed by a node. An event is a 5-tuple: key, value, dest, gen _ time, release _ time 28 4 Event-Driven System Analysis The semantics of the event is given by the key-value pair. The destination of the event is the device that should process the given key-value pair. The modeling meth- ods described in this paper are not concerned with the semantics of key-value pairs. We also need to know the temporal behavior of an event, which is given by two values. The generation time ϑ is the time at which the event was created. The gen- eration time is useful in our analysis; an implementation may or may not keep track of this value. The release time ρ of an event allows the IoT system to perform delayed actions – one event in the environment may not cause an immediate response but rather one that happens some time later. We refer to the difference between generation and release time as lifetime of an event is λ = ρ − ϑ. Events may be generated periodically or aperiodically. Activation or release times may be peri- odic or aperiodic. 4.4.2 Networks A network consists of nodes and links. We will discuss nodes in more detail below. We model communication links are unidirectional. Most physical hubs are full duplex, but we model links as unidirectional to advance our analysis. 4.4.3 Devices and Hubs A node may be one of two types: device or hub. A device appears only as leaves in the network. A device has at most one input and at most one output link. Logically, a device receives or generates events. Physically, event generation can be caused either by physical events or by internal node activity; physical event receipt may cause the physical node to initiate a physi- cal event or change its internal state. Hubs are non-leaf nodes in the network. A hub may have more than one input or output link. Hubs may include computing and storage. However, for analytical pur- poses, the key role of a hub is to sequence events. A hub maintains a timewheel, a time-ordered queue of events, and a clock. (The timewheel is a notion borrowed from an event-driven simulation, but we use it in this case to manage events in cyber-physical systems.) Events arriving from the hub’s devices are entered into the timewheel in order of activation time. The head event of the queue is removed from the timewheel when its activation time equals the clock time. The hub then sends the event to its destination. Hubs must keep track of real time in order to dispatch events. Devices may or may not need to keep track of real time, depending on their functions. Given that events are dispatched by the hubs, they can rely on the hub’s notion of real time to initiate events. When scheduling events, they may be able to set the release time for the event relative to the current time, which avoids the need for the device to directly keep track of real time. Our model only assumes that hubs are required to know real 4.4 IoT Network Model 29 time, which simplifies analysis. Allowing devices to avoid maintaining real-time clocks may have some advantage in implementation as well. 4.4.4 Single-Hub Networks A single-hub network consists of one hub mode and one or more device nodes and their associated links. The hub manages the exchange of events between its device nodes. In a single-hub network, input traffic arrives at the hub from its device nodes, while output traffic is generated by the timewheel and goes to the devices. As a simple example, consider scheduling medications for residents of the home. If a resident receives medicines twice per day, once in the morning and again in the evening, the device responsible for scheduling the medicines must generate an event for each administration. The event for the next medicine administration is probably generated when the current medicine administration is released, giving an event lifetime of 12 h. The morning routine of the residents presents a more complex set of events and more scheduling choices. Each resident’s routine will generate a series of events (getting up, toileting, eating breakfast, etc.); depending on the activity, all the events in the routine may be scheduled at once, or some may be scheduled on the comple- tion of other events. If all residents get up at once, they create both congestion in the house and congestion in the hubs and their timewheels – the maximum number of events in the system will be a function of the number of residents as well as the complexity of their routines. By staggering the timing of their activities, we can both reduce physical congestion as well as reduce the number of events that the hubs must deal with at any given time. 4.4.5 Multi-hub Networks A more general network may contain more than one type of hub. One link or a pair of links is used to connect the hubs. For the moment, we consider only tree- structured networks. In our example system, the in-house system consists of a hub and a set of devices. The cloud analytics system also uses a hub and timewheel to manage the times at which events should be processed. For modeling purposes, the analytics engine itself is a device. We model event routing as hub-to-hub transfers in which an event is removed from one timewheel and placed on another. When an event is transmitted to another hub, we may use additional queue operators to remove the event before it reaches the head of the queue. We will discuss the effects of event routing in more detail in the next section. 30 4 Event-Driven System Analysis 4.4.6 Network Models and Physical Networks The mapping between model nodes/links and nodes/links in the physical network need not be one-to-one. A single physical device may house several logical nodes. A single network physical link may be used to transport several logical links. We can rely on results from parallel computing [Dua02] for techniques for rout- ing events in multi-hub networks. Physical networks may use separate memories for queues and buffers on network links. The network model helps us to understand the behavior of more complex physi- cal links. We can first separately analyze half-duplex traffic on links and then use that analysis to understand the characteristics of full-duplex links. 4.5 IoT Event Analysis The theories for event-based analysis of distributed control networks described in Sect. 4.2 were designed for transducer networks in which events maintain their time order. In contrast, our events may be generated in one order but released in another order. The reordering effects of release times and the timewheel substantially change the analysis of IoT networks as compared to distributed control. We start with analy- sis of event populations using simple models of event generation. We then go on to identify stochastic models that are useful for the analysis of IoT event systems. 4.5.1 Event Populations Because events in IoT systems are long-lived, we must consider the lifetimes of event populations. Because events may be released long after they are generated, the system may need to accommodate a large number of events even if no events are currently being generated. The event population is the number of events that are still alive, given by the dif- ference between the number of generated and released events. We can evaluate event population over the entire network or over a set of components in the network. When events are generated and released with jitter, we can write formulas for the upper and lower population; here we concentrate on a jitter-free form of the analysis to emphasize basic principles. A general form for the population count is to enumerate all events from the sys- tem start time: t P ( t ) = ò éëJ ( t ) - r ( t ) ùû dt. 0 4.5 IoT Event Analysis 31 This formulation is cumbersome since it requires the entire system history. However, without some knowledge of the event lifetimes, we can do no better. And without a bound on event lifetime, the number of events in the system can increase without limit. A practical consequence of this observation is that useful IoT systems must put an upper bound on the lifetimes of events. If we have a maximum lifetime L on the lifetime of an event, we can write the event population as t P ( t ) = P ( t - L ) + ò éëJ ( t ) - r ( t ) ùû dt L We can also write a version of this equation taking into account event jitter. We need to know the event population at time t − L because some events may have been generated that have not yet expired. If no events are generated in an interval L then we can guarantee that the event population at the end of that interval is zero. Event population determines buffer requirements for components. The maxi- mum population determines the memory requirements of the timewheel queue. Maximum populations on links help to determine the queue sizes on those links. As a simple example, consider a single-hub network. The event stream controls medication dispensing, with medications being dispensed every 12 h. Scheduling medications may be done separately from dosing them, but let us assume for the moment that each medication dispensation also causes the next dispensing event to be scheduled. If we let T = 12 h and assume that one person is in the system, then γu(T) = γl(T) = 1, αu(T) = αl(T) = 1, and ηu(T) = ηl(T) = 2. The maximum lifetime is L = 12 h. The event population at t = 12 − ϵ, just before the first set of prescription dispensed is released and the second set generated is 12 - P (12 -  ) = ò éëJ ( t ) - r ( t ) ùû dt = 2 0 We can easily generalize this formula to the case of n people. The maximum population in an interval [t1, t2] is Pmax ( t1 ,t2 ) = max P ( t ) t [ t1 ,t2 ] Event generation in many IoT systems is not strictly periodic – some events or event streams may be activated aperiodically. In this case, the event population depends on the use case. We can evaluate event populations when event characteristics are stochastic. For example, consider an event stream that is generated periodically at a rate of one per second. The activation times (measured relative to the generation time) are given by a uniform distribution over the interval [1, 10]. The maximum population is 32 4 Event-Driven System Analysis Fig. 4.1 A multi-hub network t2 Pmax ( t1 ,t2 ) = max ò éëJ ( t ) - r ( t ) ùû dt t [ t1 ,t2 ] t1 We will develop in Sect. 4.5.3 techniques to characterize event populations under several different models of event generation. For the moment, consider the case of the morning routine for n people. Let us assume for concreteness that a stream of four events is generated, each 1 min apart, with events released 5 min after generation. If everyone gets up at once, then the event population in the first 4 mins is P(4) = 4n and Pmax = 4n. If we stagger the schedules of the residents so that each gets up 5 mins apart, then the maximum population reduces to 4. In the case of a multi-hub system, different nodes may have different event popu- lations and different maximum event populations. In the smart home, we perform some operations locally and some in the cloud. We model this with the network shown in Fig. 4.1: two hubs are in the home, one for input devices and one for out- put devices (a choice made here for modeling clarity); one hub is in the cloud. The cloud hub is connected to a single device that performs analysis algorithms. The analysis algorithms consume events, process them, and then possibly generate out- put events. (One example of such analysis is tracking.) 4.5.2 Stochastic Event Populations A wide variety of assumptions and stochastic models are possible for events. In this section, we use some basic models to derive important design metrics. Although no one to our knowledge has gathered large traces of IoT activity, we can use models of traffic from related domains to help us understand IoT design. We can gain some intuition by considering the simpler case of the Poisson distri- bution. A common model for telephone traffic is that call arrivals and departures are 4.5 IoT Event Analysis 33 each modeled as Poisson processes. In our case, we use the Poisson distribution to model event generation at a rate λ. If successive events have non-overlapping life- times, then their maximum population in that interval is 1; if their lifetimes overlap, then the maximum population is 2, which must be accommodated by buffering. The probability that two events have overlapping lifetimes L is P [t < L ] = 1 - e- l L This simple formulation suggests that λL, the product of event generation rate and lifetime, is a useful metric for judging maximum event populations. The Erlang-B distribution provides a more accurate model for event populations. In the case of IoT events, the event dwell time corresponds to call duration; the queues correspond to telephone lines. (The Erlang-C distribution models call wait- ing with queues. In our case, consider the queue as a set of servers consisting of memory locations. One memory location/server is required for each event to wait for its release time.) The offered traffic is in units of erlangs: E = lL In our case, λ is the event arrival rate and L is the event lifetime. The probability of blocking (i.e., dropping an event due to a full queue) is E m / m! Pb = B ( E,m ) = å m i =0 E i / i! where m is the size of the queue. The offered event traffic in erlangs is a useful rule-of-thumb metric for IoT sys- tem traffic – both the frequency of events and their dwell times must be considered to understand their effect on timewheel size. We can use Pb to design the timewheel capacities of the hubs, either using the maximum traffic as a guide or evaluating the traffic at different points in time using the population functions. Given the systemwide offered traffic, we can find Pb for the entire network. However, in a multi-hub system, we must determine how to partition the timewheel memory between the hubs. We can model the traffic hub by hub: n n åE = ål L i =1 i i =1 i i From this, we can determine the Pbs. However, this approach does not minimize total system memory. If we assume that all the hubs share the same values for arrival rate and event lifetime, then E < ∑1 ≤ i ≤ nEi. We describe in Sect. 4.5.4 how to transfer events between hubs to balance queue sizes. 34 4 Event-Driven System Analysis 4.5.3 Environmental Interaction Modeling We can identify three methods for modeling the interaction of the IoT system with its environment, each with its own degree of accuracy and detail. The simplest model treats both the device and the user as timed finite-state machines. Given a path through the user machine that defines a given use case, we can form the product of the device machine and the user machine path. The result- ing FSM, along with a timing regimen that is specified by the use case, tells us when events are generated by the device. That trace can be used to build the event popula- tion trace. A more sophisticated model treats the user as a Markov decision process (MDP) with fixed timing. A Markov decision process is a stochastic model used for optimi- zation. An MDP is defined by a set of states and possible actions out of each state. Each action is assigned a reward R. Transitions out of the action to the next state are assigned probabilities. We can use any of several different algorithms (dynamic programming, linear algebra) to find the path that maximizes the reward. In this scenario, we solve for the optimal reward path and form its product with the device model, using a fixed time model. Figure 4.2 shows an example of a simple device model and user model. The device model combines the actions of all the component devices related to the routine into a single state machine for simplicity. The actions in the user model MDP correspond to states in the device model. A yet more complex model uses a continuous time Markov decision process (CTMDP). The most common mathematical form of this model is as an MDP with the timing of state transitions modeled as a Poisson process [Buc11]. Standard MDP approaches can be used to solve for the optimal path with timing given by the Poisson process. 4.5.4 Event Transport and Migration An event does not necessarily have to be stored on the hub that owns either the event’s source or destination. In a multi-hub system, we can station events at nonlo- cal hubs to avoid overflowing a hub’s queue capacity or improve its battery life. If an event is queued nonlocally, we must factor transmission time into its release to ensure that it reaches its destination device at the proper time. For simplicity, we consider the case of no energy cost for transporting events across the network. Let Pe be the power consumption of storing one event in mem- ory for a unit time. Given a population of events Π, the energy required to store all events in the population until their release times is Epop = å éë r ( e ) - J ( e ) ùû Pe eP 4.5 IoT Event Analysis 35 Fig. 4.2 Models of the morning routine for devices and residents We have a set of H hubs each with available battery energy Eh(i). We want to find an allocation of events to hubs such that "iH : Epop ( i ) £ Eh ( i ) This is a classic bin-packing problem, although we want to solve it as a distributed problem without a centralized list of events. In practice, transmission energy reduces the set of plausible event allocations. We propose a heuristic algorithm for event migration: Find a partial ordering of the hubs such that no two adjacent hubs are in the same set and all hubs are covered. 36 4 Event-Driven System Analysis Hubs proceed in order so that no two adjacent hubs off-load simultaneously. Each hub off-loads enough events to meet its battery requirements. Events are moved to adjacent hubs with the greatest available battery capacity. Acknowledgment Thanks to the team at Alya Networks for useful discussions on key-value- based IoT networks. References [Buc11] Peter Buchholz (2011). Continuous time Markov decision processes: Theory, applica- tions, and computational algorithms. TU Dortmund Informatik IV lecture notes. [Cha03] Chakraborty, S., K¨unzli, S., & Thiele, L. (2003). A general framework for analysing system properties in platform-based embedded system designs. Proceedings sixth design, auto- mation and test in Europe (DATE) (pp. 190–195). Munich, Germany. [Dua02] Duato, J., Yalamanchili, S., & Ni, L. (2002). Interconnection networks. San Francisco: Morgan Kaufman. [Hen05] Henia, R., Hamann, A., Jersak, M., Racu, R., Richter, K., & Ernst, R. (2005). System level performance analysis: The SymTA/S approach. IEE Proceedings: Computer Design Techniques, 152(2), 148–166. [Max04] Maxiaguine, A., Kunzli, S., & Thiele, L. (2004). Workload characterization model for tasks with variable execution demand. Proceedings of the conference on seventh design, auto- mation and test in Europe (DATE) (pp. 1040–1045). Paris, France. [Wol15] Wolf, M., van der Schaar, M., Kim, H., & Xu, J. (2015). Caring analytics for adults with special needs. IEEE Design & Test, 32(5), 35–44. Chapter 5 Industrial Internet of Things 5.1 Introduction The Internet of Things (IoT) has already brought a revolution to our understanding of applications in a wide range of human activity. This trend is expected to increase in the near future, as the potential economic impact of IoT is expected to be between 900 billion USD and 2.3 trillion USD on a yearly basis up to 2025 [Man13]. IoT applications are spreading to various sectors including smart energy, manufactur- ing, agriculture, health, security and safety, smart cities, smart buildings, and smart environment. All these application areas repeat the same basic model: a large num- ber of smart devices, interconnected over wired or wireless media, interacting and coordinating to achieve a goal. In the industrial environment, the effort for smart factories [Zue10], the Industrie 4.0 strategy [Ind14], the Industrial Internet [GE17], and the European initiative for the Factories of the Future [FoF] have initiated the adoption of IoT in industry with the goals of increasing flexibility and productivity, while reducing production cost. The developing concept is the Industrial IoT (IIoT). The Industrial Internet of Things is part of the general IoT evolution. However, it faces challenges that are unique and differentiate it from the other systems and ser- vices of IoT due to the need to integrate programmable logic controllers (PLC) and supervisory control and data acquisition systems (SCADA). PLC and SCADA sys- tems, together with the related industrial networks that interconnect them, constitute the infrastructure of operational technology (OT), which has traditionally evolved independently from the typical IT technology, because it addresses the needs of systems in the field – industrial floor, energy production facilities, energy distribu- tion networks, etc. – with strong requirements such as continuous operation, safety, real-time operation, etc. The capabilities offered by the emerging IIoT technology pose challenges for the integration of these OT systems with the traditional enter- prise IT systems at many levels, from enterprise management to cyber security. For example, enterprise resource planning systems (ERP) need to be expanded to 38 5 Industrial Internet of Things include manufacturing operations, which are managed currently by manufacturing execution systems (MES) that have grown independently and present significant interoperability challenges to their integration. Clearly, an integrated system that manages the complete enterprise/factory hierarchy, from business processes to sen- sors, provides significant flexibility and presents new opportunities to enterprises. Industrial technology is not part only of manufacturing or factories. The maturity of the technology and its cyber-physical control capabilities has spread its use out- side traditional factory environments, and now they constitute a significant part of the critical infrastructure at many fronts. Energy production and distribution infra- structure includes OT systems, which are the indispensable infrastructure on which modern smart grids are built. Actually, the energy sector is a high priority in the evolution of IIoT, not only because there is increasing need to consumers for energy, especially in light of the population growth, but also because energy management is a critical factor in the industrial sector and the desired low-cost production of goods and services. In addition to the energy sector, industrial systems are widespread in many other sectors of critical infrastructure, such as water management and transportation. The interoperability challenges to the convergence of IT and OT are only a part of the challenges in the emerging IIoT. Appropriate architectures need to be devel- oped to build and manage effective IIoT systems, technologies for the design and management of cyber-physical systems, sensors and networks need to be devel- oped, and, importantly, safety and security need to be addressed in a unified way in the context of IIoT. Safety and security are significant challenges, because, tradi- tionally, security has been a concern in the IT sector, while safety has been the major concern in the OT sector. Bringing the two together has brought the realization that safety cannot be achieved without security, while, at the same time, security needs to include technologies that combine dependability and meet strong requirements for real time and low power in many application domains. Although the security issues of industrial control systems have attracted attention in the last decade and standards have been evolving at a much faster pace than in the past, e.g., the ISO/ IEC 27000 and the ISA/IEC 62443 families of standards [IEC16, ISA16], there are still significant challenges at the technology, architecture, and management fronts to obtain solutions for the unified IIoT. In this chapter, we present the concepts and evolution of the IIoT starting from the Industrie 4.0 strategy and proceeding to the Industrial Internet. We describe the IIoT reference architectures as they evolve from the ITU effort to the Industrial Internet Consortium. Finally, we describe some representative challenges in the evolution and implementation of IIoT focusing on the energy sector. As security and safety constitute a significant challenge in IIoT as well as in IoT, in general, we focus on this challenge in the following chapter. 5.2 Industrie 4.0 39 5.2 Industrie 4.0 Industrie 4.0 is a strategic initiative in Germany that targets to bring IoT technolo- gies to the manufacturing and production sectors [Ind14].The goal is to enable Germany to keep a leading role in manufacturing achieving efficient and low-cost production with flexible workflows. The means to achieve this goal is the wide- spread inclusion of cyber-physical systems in the manufacturing and production processes, in order to insert intelligence in the systems and processes, to enable their high connectivity and communication, and to achieve their coordination into more complex but flexible processes that lead to high-quality, low-cost products. Industrie 4.0 takes its name from the identification of the new, emerging industry as the fourth revolution of industrial production. It is widely accepted that industrial production to date has gone through three (3) revolutions. The first industrial revolu- tion, between the eighteenth and nineteenth century, is the one where mechanized production facilities were introduced in the production of goods and services, where the required energy was provided by water and steam. Electrical energy was intro- duced during the second revolution, which led to mass production, as electricity boosted productivity. In the post WWII era, the inclusion of electronics and soft- ware, i.e., industrial information technology, to the mechanical and electrical com- ponents led to the third revolution that enabled automation at high levels. Currently, many industrial stakeholders believe that we are at the verge of the next, the fourth, industrial revolution, through wide adoption and use of cyber-physical systems that leads not only to even higher levels of automation but enables mass customized manufacturing and production of goods and services, due to the flexibility offered by the easily programmable, configurable, and controllable manufacturing lines. The effort for Industrie 4.0 is based on the widespread deployment and use of computational and communication resources. The last two decades have been char- acterized by significant advances in high performance, low-power processors, mem- ories, and communication components that enable efficient processing and networking. These advances have brought significant processing capabilities to a large number of devices that are deployed to consumers or to the field. Smart con- sumer devices have become norm. Smartphones provide hundreds of applications and enable services ranging from identifying travel and transportation routes to mobile banking and health monitoring. Smart televisions combine and provide vari- ous types of entertainment and network services, from customized TV channel con- trol and management to Internet gaming and home device management. Smart home appliances monitor parameters, from environmental temperatures to water and energy consumption, enabling citizens to manage their homes efficiently and effectively leading to the required living quality while reducing operational cost at various fronts. The large basis of computational resources and connectivity becomes apparent by the published numbers of embedded processors and components that are cur- rently produced. According to [Ind14], the vast majority of produced processors, approximately 98%, are deployed in embedded systems. Deployed semiconductor 40 5 Industrial Internet of Things memory is also growing and expected to grow at 40% year over year in 2017 [Mic17]. Furthermore, the significant advances of wired and wireless networks in the last two decades have led to ubiquitous connectivity, approaching 100% in cities and towns, through different technologies. The available processing and communication basis leads to an evolving hierar- chy of embedded systems and services up to the level of the Internet of Things, Data, and Services. Examples of this evolution can be identified at several applica- tion areas. In transportation, for example, embedded systems are widespread con- trolling functions from car entertainment systems to car seat control. At this level, embedded processors are programmed to control specific, individual parameters, e.g., height and movement in car seats, based on user commands. However, embed- ded systems in cars are also networked, either within the car system or with the environment, providing networked embedded services; automatic toll payment is one of them where embedded systems in the car and the toll booths communicate with each other, in order to complete the electronic payment transaction of the toll passage. Such payment systems from several tolls, for example, can be further com- bined in a distributed system that enables traffic and toll management at a wider scale, leading to more effective transportation infrastructure that achieves lower waiting times and fuel costs for travelers as well as lower operational cost and, thus, higher income to transportation management authorities. One can even envision an even higher level of connectivity of such complex transportation systems to smart cities that combine transportation management with additional services, such as energy distribution, civil services, emergency services, etc., as required at different times, locations, and during special events. The advances of sensor technologies, in addition to the evolution of embedded systems and communication networks, make all these scenarios realistic. Importantly, sensors bridge the gap between the physical world and the digital world, providing increasingly rich information to digital systems and enabling intel- ligent control of systems and processes. In that respect, manufacturing and indus- trial automation has been traditionally employing IT technologies with sensors and electromechanical systems, leading the development and deployment of technolo- gies and concepts for intelligent control, systems, and services. Thus, the develop- ment of the Industrie 4.0 strategy and the related initiatives comes as a natural evolution step of industrial technologies influencing and being influenced by the advancement of consumer technologies of the Internet of Things. The smart factory concept embodies the goals of the Industrie 4.0 strategy to a large degree. The concept is based on the hierarchy of cyber-physical systems men- tioned above, where smart production systems are interconnected in a multilevel hierarchy to achieve a high degree of automation, targeting flexibility, efficiency, autonomy, resilience, safety, and low cost. Smart machines will be interconnected to establish smart plants, which, in turn, will be combined to provide smart facto- ries. Considering the typical components of manufacturing process, smart factories are targeted to automate efficiently all components and stages. Materials and resources will be managed and introduced in the process efficiently; production processing will be managed in real time minimizing the used resources for the 5.3 Industrial Internet of Things (IIoT) 41 p roducts and the operations, while reconfiguration and reorganization of production processes and customization of products will be feasible in real time and with safety for infrastructure and operators, minimizing environmental impact. Customers will be able to monitor the progress of the development of ordered products, while man- ufacturers will be optimizing their logistics chains. 5.3

Internet-of-Things (IoT) Systems Architectures, Algorithms PDF

Document Details

Tags

Related

Summary

Full Transcript

Upgrade to continue