Internet of Video Things: Next-Generation IoT With Visual Sensors PDF

6676 IEEE INTERNET OF THINGS JOURNAL, VOL. 7, NO. 8, AUGUST 2020 Internet of Video Things: Next-Generation IoT With Visual Sensors...

6676 IEEE INTERNET OF THINGS JOURNAL, VOL. 7, NO. 8, AUGUST 2020 Internet of Video Things: Next-Generation IoT With Visual Sensors Chang Wen Chen , Fellow, IEEE (Invited Paper) Abstract—The worldwide flourishing of the Internet of through a gateway device the data sink to the decision- Things (IoT) in the past decade has enabled numerous new appli- making center. The evolution of the sensing devices has also cations through the internetworking of a wide variety of devices undertaken fundamental transformation from scaler sensing and sensors. More recently, visual sensors have seen their con- siderable booming in IoT systems because they are capable of devices, such as simple temperature and proximity sensors, to providing richer and more versatile information. Internetworking slightly more complicated vector sensors, such as accelerom- of large-scale visual sensors has been named Internet of Video eters and gyroscope sensors, to the full multimedia sensors, Things (IoVT). IoVT has its own unique characteristics in including audio–visual cameras and even 3-D cameras and terms of sensing, transmission, storage, and analysis, which are camera arrays. Moreover, recent enormous advances in com- fundamentally different from the conventional IoT. These new characteristics of IoVT are expected to impose significant chal- puting and networking technologies have enabled the deploy- lenges to existing technical infrastructures. In this article, an ment of contemporary IoT systems consisting of networks overview of recent advances in various fronts of IoVT will be of multimedia and multimodal sensors for large geographical introduced and a broad range of technological and system chal- area monitoring for public security, intelligent transportation, lenges will be addressed. Several emerging IoVT applications will and smart city projects as well as for complex electromagnetic be discussed briefly to illustrate the potentials of IoVT in a broad range of practical scenarios. environments at the corporate manufacturing site for industrial IoT applications. Index Terms—Intelligent integration, Internet of Things (IoT), From its initial concept of IoT proposed by Kevin Ashton, Internet of Video Things (IoVT), IoVT applications, pervasive networking, smart visual signal analysis, visual communication back in 1999 to the contemporary research and development and networking, visual sensors. of IoT at its tremendous global scale, IoT have been grad- ually recognized and now broadly appreciated by almost all nations. It short yet drastic development over the past 20 I. I NTRODUCTION years, it was shown that IoT technologies have not only bene- HE DEVELOPMENT of Internet of Things (IoT) in the fited the well-developed countries in their endeavors to advance T past two decades has witnessed its vast potential in that the physical environments can be equipped with “smart” things this field but also greatly profited the developing and emerging economies in numerous new ways that were not considered embedded with contemporary information and communica- possible. However, there are still ample opportunities to tackle tion technologies for seamless cyber-physical interaction. Such the new challenges in the technology, context and application an IoT paradigm has grown from small and personal scale areas that emerge as the relevant fields in sensing, commu- applications to global scale coordinated endeavors. In the late nication, networking, and data analysis advance to their new 1990s, radio-frequency identification (RFID) was considered heights. New strategies will need to be developed to meet some prevailing devices to support IoT applications. of the new challenges that we have not observed before. However, after more than 20 years of evolution, modern In particular, contemporary IoT systems with visual sen- “things” are no longer limited to these personal objects we sors, as well as some full multimedia sensors, requires carry around, such as smart phones, tablets, digital cameras, much more sophisticated strategy based on their unique con- and wearable devices. Rather, modern things include large- straints imposed by the type of visual data as well as the scale smart things or sensors embedded in our environment, networks needed to transport such volumetric and continu- home, vehicle, building, or infrastructure that are connected ously generated data. First, unlike most conventional scaler sensor data that are usually highly structured, audio–visual Manuscript received April 7, 2020; revised June 1, 2020; accepted June sensor data are often unstructured and would need addi- 14, 2020. Date of publication June 30, 2020; date of current version August 12, 2020. This work was supported in part by the Key-Area Research tional processing in order to extract their “semantics” for and Development Program of Guangdong Province, China, under Grant information collection and decision making. It is this require- 2019B010155002; in part by National Natural Science Foundation of ment for visual data analytics that also provides opportunities China under Grant 91538203; and in part by U.S. NSF under Grant 1405594. The author is with Peng Cheng Laboratory, Shenzhen 518055, China, also for potential front-end processing onboard the IoT sensors. with the School of Science and Engineering, Chinese University of Hong Second, the volumetric and continuous generation of visual Kong, Shenzhen 518172, China, and also with the Department of Computer and multimedia data requires extremely high networking Science and Engineering, University at Buffalo, State University of New York, Buffalo, NY 14260 USA (e-mail: [email protected]). capacity for the timely Internet of Video Things (IoVT) appli- Digital Object Identifier 10.1109/JIOT.2020.3005727 cations. It would be ideal to enable the sensor front to perform 2327-4662  c 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information. Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY KURUKSHETRA. Downloaded on March 24,2022 at 11:57:45 UTC from IEEE Xplore. Restrictions apply.  CHEN: INTERNET OF VIDEO THINGS: NEXT-GENERATION IoT WITH VISUAL SENSORS 6677 appropriate compression/analysis of the sensor data for more In the advent of AI Era, the society is immersed by efficient transport of visual data. Third, it is also desired to the oceans of generated data, continuous creation of new integrate other types of sensor data to augment the visual sen- algorithms, and ever-increasing hardware and architectural sor data to achieve holistic ambient environment sensing and power. A significant portion of such data comes from vari- understanding. The fusion of multimodality sensor data shall ous IoT systems, in particular the IoVT systems. Traditional lead to more robust and more intelligent IoVT system to allow IoT data can be relatively easily handled with cloud-based the exploitation of the full potential of IoVT. approaches via offloading while analyzing visual sensor The IoT applications have penetrated into every aspect of data from IoVT face significant challenges because it is infea- the contemporary society, including mobile devices, medical sible and inefficient to transport all the data to the cloud servers and healthcare, automobile and transportation, home and pub- for processing. We shall need overwhelming computing and lic security, and industrial and manufacturing, to name a few. communication resources for transporting IoVT data from col- With its richer potential extracted from visual data to pro- lecting sites to the cloud centers. It becomes necessary to vide a full ambient environment, the new IoVT strategy will facilitate the IoVT nodes with onboard processing capabili- offer brand new solutions to advance human development and ties in order to extract proper information for more efficient evolution through the scientific and technological principles data transmission and analytics. Recent advances in several embedded in IoT. We expect that the IoVT paradigm will fur- related technological development trends have been driving ther change the lives of the people for a more powerful and the acceptance of IoVT paradigm as the next-generation IoT digitally smart future. platform capable of acquiring, transporting, and analyzing However, we also face significant technical challenges the visual data for much improved IoT application perfor- that we have not experienced before due to the introduc- mances. The characteristics of the current IoVT systems can tion of visual sensors at the frontend of IoVT systems. be summarized as follows. These challenges are rooted in the unique characteristics of 1) Significant Increase in Large-Scale Visual Sensor IoVT paradigm that will require brand new technical solu- Deployment: Visual data are collected everywhere nowa- tions. In this article, we shall address the following technical days. Surveillance cameras have been around every challenges. street corner across the world. Visual data can serve as 1) Onboard embedded visual data processing, storage, the foundation for smart cities, physical security, smart networking, and energy efficiency issues. transportation, and many other types of applications. 2) Reliable and adaptive IoVT networking capabilities to There are also increased number of cameras in retail meet the bandwidth-hungry application of visual sensor stores for customer analytics and directed advertising. data. Cameras also penetrated into more and more households 3) Integration of visual data compression and data analytics for security, senior care, baby care, and other intentions. for effective search and retrieval for a broad range of It was estimated a few years ago that the data gener- IoVT applications. ated by surveillance cameras alone have already reached 4) Balanced security and privacy issues for visual sensor thousands of petabytes. The visual data from IoVT data sharing and distributed cloud center processing. systems are rich in content and can open vast analytics 5) Building a standardized platform for moving visual opportunities. sensor data around the massive IoT networks for coor- 2) Powerful and Energy-Efficient Embedded Processing: In dination among different IoVT data owners. conventional IoT, sensors are connected to the servers via gateways or aggregators to stream sensed data in real time. With visual sensor data, the IoVT nodes generate ultrabig data volume intended for further data analytics. II. C HARACTERISTICS AND T ECHNICAL I SSUES OF I OVT To deal with such ultrabig data volume, onboard pro- The recent advances in IoVT have been made possi- cessing, also considered as ubiquitous computing and ble thanks to the technical achievements in multiple-related ambient intelligence, is a more efficient approach. research and development fronts, including several technolog- Rather than the traditional centralized approach, where ical trends that are changing the game today and will accelerate all the data analysis tasks for IoVT are executed on cloud the development of IoVT toward a more automated society. servers, it is now possible and becoming necessary to In the current era of artificial intelligence (AI) that influences embed computation on front end sensors and aggregators nearly every scientific and engineering disciplines, IoVT is with the recent advances in semiconductor technology. no exception. In fact, all components of the IoVT paradigm System-on-a-chip (SoC) technology shall play an impor- in sensing, transporting, and analyzing IoT data shall be sig- tant role in IoVT in that small chips can be employed at nificantly impacted by the contemporary AI penetration, in the end devices for embedded processing to help with terms of data, algorithm and computing power, the three most more efficient big data analytics at cloud servers. fundamental elements of AI. The seamless integration of AI 3) Explosive Rise of Connectivity via 5G/B5G Evolution: technologies with IoT has been initiated in the recent years From the wireless communication point of view, the IoT and it is called AIoT. The key to the success of IoVT will networking needs can be met by embedding the con- be governed by how smart generation and intelligent use of nectivity into the things that are capable of transmitting IoVT data can be achieved in real time. data and information from one node to another. Over the Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY KURUKSHETRA. Downloaded on March 24,2022 at 11:57:45 UTC from IEEE Xplore. Restrictions apply.  6678 IEEE INTERNET OF THINGS JOURNAL, VOL. 7, NO. 8, AUGUST 2020 years, various new applications of IoT have demanded the networking technologies to evolve to more sophisti- cated and mobile communications have been adopted for IoT. Contemporary smart IoT applications need much more than RFID types of connectivity and a suite of wireless networking technologies has already been developed. In particular, recent advances in 5G has been driving the IoT connectivity development in terms of higher bandwidth of 1000× of traffic, smaller device packaging for size reduction of IoT nodes, submillisec- ond latency for critical IoT applications, and higher capacity networking of millions of devices ,. The multiservice air interface of 5G is indeed capable of enhancing broadband performance as well as providing new levels of reliability, latency, and supported num- ber of users. Within 5G, the massive machine type Fig. 1. Illustration of three major technological building blocks of the communications (mMTC) usage scenario is specifically proposed IoVT system. designed for IoVT type of applications in which a mas- sive number of access attempts are common within 5G mMTC ,. execute embedded AI algorithms for new generation of 4) Rapid Development of the “Cloud” Computing and IoVT applications that often requires onboard processing Edge Computing: Cloud computing is providing access of visual data. to virtually infinite storage and computing capaci- Even with these impactful and inspiring technological ties, facilitating data processing and integration in advances, there are still major technological issues associated a mega scales. These cloud specific functionalities have with the ecological chain of the new IoVT paradigm. For become the engine for numerous new IoT applications the convenience of identifying the functionality of an IoVT that would use cloud to offload their needs for massive system, we can categorize the ecological chain of IoVT into processing of sensor data as well as for data sharing three major blocks, each with a well-defined function in data opportunities. However, it becomes difficult to meet sensing, information networking, and knowledge integration. the delay-sensitive and context-aware service require- The overall IoVT paradigm can be illustrated as in Fig. 1 ments of IoVT applications by using cloud computing shown below. These intertwined blocks are situated at differ- alone. Facing these challenges, computing paradigms ent stage of IoVT applications and the challenging technical are shifting in recent years from the centralized cloud issues associated with these stages shall be discussed in more computing to the distributed edge computing ,. detail in the next section. This is particularly useful for IoVT new paradigms in which the camera nodes are indeed capable of process- ing the stream data at the network edge and the IoVT A. IoVT Smart Sensing Issues nodes to provide timely and context-aware services. The Historically, IoT sensors are assumed to acquire scalar challenges in the seamless integration of IoVT with data, such as temperature and humidity level of a given space. cloud will the issue in power consumption so that vari- Such sensors may be sparsely placed and the duty cycle of ous energy-efficient schemes for IoT in different cloud the sensors are typically low so that they do not have a environments have been developed in recent years. high demand in communication and networking. For early 5) Miniaturization of IoVT Components: IoT nodes with IoT applications, RFID was the primary means of commu- the size of a grain of sand (∼1.0 cubic mm) which nication and only simple data aggregation among neighboring include battery unit, computing and memory unit, sen- IoT nodes is needed. When visual sensors are adopted for sor unit, and wireless communication unit, have been contemporary IoVT systems, the video data from IoVT sen- developed for various applications. For IoVT appli- sors are generated continuously in an all-weather and all-time cations, miniature visual sensors, that is, cameras in fashion. For a network of visual sensors, this produces megas- the size of a grain of salt, with a resolution at least cale sensor data that would need proper compression before 250×250 pixels or even higher, are already available. they are transmitted to the central sink for desired information Financially, the average cost of an IoT has come down extraction and analysis. to under $1.0 and is expected to drop even lower in The first technical issue at the smart IoVT sensing end is a couple years. Such a miniaturized IoVT component the development of IoVT specific video data compression suit- enables a new array of applications that connect wear- able for this type of data. Although the research in general able devices as IoVT nodes ,. In particular, in video compression has already become mature with a series addition to miniaturizing the power unit and wireless of international standards in MPEG and HEVC, they may communication unit, it is now possible to design the not be directly adapted for IoVT video compression because miniature deep learning accelerator unit to efficiently IoVT video streams are fundamentally different. In general, the Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY KURUKSHETRA. Downloaded on March 24,2022 at 11:57:45 UTC from IEEE Xplore. Restrictions apply.  CHEN: INTERNET OF VIDEO THINGS: NEXT-GENERATION IoT WITH VISUAL SENSORS 6679 compression of entertainment videos shall require high defi- the cloud center. The embedded frontend processing is also nition reproduction of virtually all pixels for pleasant viewing expected to generate proper visual features for potential search of video contents. In IoVT applications, it is the preserva- and retrieval by the IoVT users at the cloud centers. This tion of information of interests, rather than the value of each way, the sensing units at the frontend shall become capable of pixel, that counts as the ultimate goal of IoVT applications. understanding both context and contents of the sensed video Therefore, preserving the semantic contents and contexts of stream. This is different from a recent research in visual fog the IoVT data are the most important attribute for the IoVT computing strategy which is mainly focused on offload- video compression and sensor data processing. Novel video ing technologies making use of other devices connected to compression algorithms that are fundamentally different from front end sensors, the embedded visual processing proposed general MPEG and HEVC standards, are therefore needed in this article is focused on onboard processing of visual in order to best preserve the semantics and context of the data acquired by the IoVT frontend sensors. acquired IoVT sensor data during the compression process. One such recent effort is the development of a new video coding standard for machine communications. B. IoVT Pervasive Networking Issues The second technical issue at the smart IoVT sensing end Pervasive networking is critical for an IoVT system to suc- is the design of embedded processing onboard the sensor for cessfully transport timely sensor data from the networked integrated video analytics. In many IoVT applications, the fea- camera nodes to cloud center and to transmit command tures extracted from the video data is of paramount importance and control signals for IoVT nodes to take appropriate and those features actually constitute the semantics and con- actions. Contemporary IoVT systems have evolved from con- text of the sensor data. For a video compression algorithm necting initially locally installed visual sensors for isolated to preserve the semantics and context of the data as we have surveillance purpose. Historically, individual visual surveil- discussed early, the visual features need to be extracted first lance cameras store their data locally without connecting to via onboard analytics. In practice, both visual features and other neighboring cameras even though they may be monitor- the original video stream shall need to be transmitted together ing the same scene under observation. For conventional IoT to IoVT data sink for further processing. The extraction of networking, one key assumption is that the sensor data are gen- visual features onboard frontend sensor is the key operation erated with a low duty cycle and the demand for networking in smart sensing that an IoVT system distinguishes itself from has not been really pressing. For the emerging IoVT sce- the conventional IoT systems. Since IoVT data is typically narios, we are faced with unprecedented networking demand unstructured, sophisticated feature extraction algorithms will because these IoVT visual data are generated in an all- need to be designed to accomplish specific tasks under strin- weather and all-time fashion. Networking of these IoVT gent latency and energy efficiency constraints of the frontend nodes poses tremendous challenges in moving the high vol- visual sensors. ume of visual data seamlessly from the sensor end to the The third technical issue at the smart IoVT sensing end data sink, which will most likely be located at the cloud is the application of visual data aggregation for multiple center. frontend sensors to achieve collaborative compression and Without pervasive networking, the current IoVT system energy-efficient transmission. Unlike the traditional video will not be able to reach its potential in analyzing valuable compression in which only spatial and temporal redundan- networked visual information for many contemporary appli- cies of a single video stream are exploited, there exist possible cations, such as smart city and smart home projects. It is additional cross-sensor redundancy between neighboring IoVT not an understatement that IoVT camera nodes without perva- camera nodes whose field of view inherently overlaps sig- sive networking capability are like isolated information islands nificantly. Exploiting of overlapping views has been studied with very limited local usage. However, the path from IoVT before for wireless sensor networks when two neighboring nodes to the ultimate cloud data center will not be simple cameras are installed next to each other and a collaborative and direct and will depend on which networking type a par- coding and transmission scheme has been developed. The ticular IoVT system choose to operate with. We have moved additional processing power can be more than compensated significantly beyond conventional RFID networking means to by the energy saving achieved from the wireless transmission the contemporary 5G networks for much enhanced networking when less overall compressed data need to be transmitted to performances. Even with 5G mMTC usage scenario, an IoVT the data sink. The exploitation of cross-camera correlations in system still have the choices in applying computing, commu- video data is expected to benefit not only the energy-efficient nication, and caching (3C) strategy to the implementation of wireless transmission, but also the integrated analytics over pervasive networking specifics for the proper partition of tasks networked IoVT camera nodes for tracking objects of interests among computing, communication and caching. An optimal over multiple neighboring camera nodes. pervasive networking for IoVT may be implemented through It is clear that once we are able to resolve these technical the optimal partition of integrated 3C tasks for an overall issues at the frontend of an IoVT system, the sensing of IoVT system performance. data shall become more intelligent and shall provide smart The first technical issues for IoVT pervasive networking is visual information for transport and integration. The embedded the embedded design of radio access components for IoVT frontend processing is expected to help reducing the band- nodes to enable effective networking capable of high-speed width required for transport and the computational loading at transport of visual data. For wireless transmission of IoVT Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY KURUKSHETRA. Downloaded on March 24,2022 at 11:57:45 UTC from IEEE Xplore. Restrictions apply.  6680 IEEE INTERNET OF THINGS JOURNAL, VOL. 7, NO. 8, AUGUST 2020 visual data, we expect that the radio access component is specific IoVT task requirement in desired latency as well as integrated with the frontend visual sensors, such as cameras event detection and classification accuracies. unit. In the emerging 5G standard, mMTC usage scenario has These technical issues constitute the data transport chal- been well defined at the protocol level for potential IoVT lenges we are facing and should be resolved aggressively applications. However, it requires the IoVT system designer to ensure that there would be a high performance pervasive to have substantial innovations to create appropriate radio network behind the powerful IoVT sensors for moving the access hardware onboard the corresponding IoVT nodes for high volume visual sensor date around. These issues range pervasive networking. Once we have identified a specific appli- from hardware radio components to application-specific pro- cation scenario for an IoVT system, we can select proper radio tocol management, and to the system optimization strategy access modes in order to meet the requirements of transmit- that trades computing and caching for an improved networking ting visual sensor data in real time and with high fidelity. performance. With the desired pervasive networking capabil- Besides the video data transmission requirements, the selec- ity, the massively deployed IoVT sensors can be seamlessly tion of radio access components should also be constrained connected and their visual sensor data can be properly ana- by the energy efficiency of pervasive networking. Since the lyzed to achieve anticipated monitoring and decision-making networking requirements for IoVT can be quite different from functionalities. one application to another, it will be most effective to install multiradio access components on an mMTC device for a broad range of IoVT applications. The second technical issue for IoVT pervasive networking is C. IoVT Intelligent Integration Issues the development of application-specific intelligent networking A successful IoVT system will definitely go beyond the management protocols for transporting compressed video steps of sensing and networking as we have outlined earlier streams as well as associated data, such as features extracted in order to collect most desired visual data and transport to at the frontend to the destinations. The application-specific cloud center. The visual data sensed by IoVT frontend nodes, network performance parameters include latency, loss rate, as well as the information extracted during smart sensing and and potential jitters, which are unique for IoVT data traffics. pervasive networking, shall become very useful for backend An artificial intelligent (AI) module can be designed for the IoVT data users. As a result, how to most efficiently search management of networking flows to achieve two functionali- and retrieve from the IoVT data sets will become a critical ties in data classification and resource allocation. Various issue for any IoVT system. Unlike conventional IoT data that machine-learning (ML) approaches can be applied to train an are in the form of scalar representing clear physical mean- AI traffic classification module in order to achieve the best ing, IoVT visual data are mostly unstructured with semantics performance in transporting hybrid IoVT traffic to the destina- embedded in visual signal. Therefore, search and retrieval over tions of cloud centers. For resource allocation, a model-based such unstructured data sets is not a trivial task. Although approach can be adopted based on networking parameters, the research in general visual search and retrieval has been including traffic priority, networking route, bandwidth vari- well developed. These techniques cannot be applied to IoVT ation, and buffer fullness. Matching of IoVT traffic type and data search and retrival since the IoVT visual data often has QoS requirements with the dynamic networking status is the its own privacy issues since such data may contain facial and key to improving the pervasive networking performance of other personal images that require proper privacy protection. In a specific IoVT system. some IoVT systems, the collected data may also include nonvi- The third technical issue for IoVT pervasive networking is sual key information to supplement the visual data with proper the selection of intermediate networking layers for implement- context for complete characterization of the scenario. Such ing the 3C strategy to optimize for overall system performance. information may be of significant importance when designing It is true that the ultimate data sinks for IoVT applications search and retrieval algorithms. are the cloud centers. These cloud centers are usually located We often also need to integrate multiple IoVT systems to physically away from IoVT frontend nodes, thus requires acquire a whole picture of large-scale physical entity, such as pervasive networking to transport the visual sensor data for a metropolitan area. Each of these IoVT systems, for example, extensive computation on them. From the network architec- smart transportation, smart grids, smart healthcare, and smart ture point of view, there may exist several layers between home, has been designed and deployed independently. They the frontend nodes and the cloud centers and each layer has collect and analyze data without proper information sharing its own unique capability to support 3C. A recent develop- or interconnection among them. This phenomenon is regarded ment in edge-fog-cloud architecture is ideal to facilitate an as the isolated information islands and has been hindering the opportunity for exploiting the 3C strategy. In general, edge progress of large-scale IoVT applications. To make the best computing refers to the processing of IoVT data on the fron- use of IoVT data sets, a well-coordinated strategy to enable tend nodes while fog computing refers to the virtualized integrated search and retrieval and to construct a standard plat- intermediate platform located between the edge device and form for effective exchange of information between different the cloud centers. It is a challenging task to partition the IoVT systems needs to be developed. Without intelligent inte- computing and networking tasks among all three locations in gration of multiple IoVT systems and standardization of IoVT the network continuum to achieve an overall optimal utiliza- data exchange, the impact of IoVT will be limited within some tion of network resources. Such partition must depend on the stand-alone specific tasks. Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY KURUKSHETRA. Downloaded on March 24,2022 at 11:57:45 UTC from IEEE Xplore. Restrictions apply.  CHEN: INTERNET OF VIDEO THINGS: NEXT-GENERATION IoT WITH VISUAL SENSORS 6681 The first technical issue for IoVT intelligent integration is overall large-scale smart city initiative which is composed of the development of search and retrieval functionality for work- multiple IoVT applications designed for specific large-scale ing with networked visual sensor data. For IoVT data, the high-level decision-making purposes. search and retrieval is not trivial because the visual data are These high-level intelligent integration issues define how unstructured and needs the extraction of semantics from the well we are able to make best use of the collected IoVT data first and then the manipulation of semantic representa- sensor data at the overall system level. Upon coordinated col- tions. Assume that we have already extracted some features lecting of the precious visual data from IoVT sensor nodes, at the smart sensing stage. These features may be directly the basic function of search and retrieval allows the users to used when designing the search and retrieval algorithms for perform an in-depth examination in order to detect potentially IoVT data sets. The more challenging issue is the need to important events or to make meaningful correlations from his- search and retrieve information over multiple IoVT systems torical sensor data. Because of the privacy concerns on the designed and deployed independently. It is important for dif- visual signals that are easily identifiable, the issues in bal- ferent IoVT applications to develop a common platform in ancing security and privacy need to be properly resolved to which a standard search and retrieval algorithm is developed ensure that people and objects under security monitoring by that is common to those IoVT systems that require a proper IoVT can be guaranteed of privacy protection. To facilitate exchange of information via search and retrieval between the interoperability of different IoVT systems, a standardiza- them. To develop a more effective search and retrieval algo- tion at different levels of the IoVT system would empowers the rithm for application across multiple IoVT systems, nonvisual seamless exchange of the sensor data, the unified networking reference data, such as location information, may be used to protocol stack, and integrated data representation scheme for link different IoVT systems. large-scale IoVT applications. The second technical issue for IoVT intelligent integration is the trade-off design for IoVT security and privacy. Unlike con- III. E MERGING I OVT A PPLICATIONS ventional IoT data, the IoVT data usually contains an intuitive The new IoVT systems are collecting the unprecedented visual image of persons and objects that can easily identified volume of data for various applications. Because of the dis- by human viewers. This straightforward pictorial information tinct characteristics of visual sensor data, IoVT systems are poses serious privacy issue for deployment of IoVT systems. able to provide some unique insights that conventional IoT More importantly, the pictorial information is also heavily systems cannot offer. Such novel capabilities empower the used for designing IoVT applications for security monitoring IoVT systems to either penetrate into many new application and surveillance. How to balance between security and pri- domains or augment new dimensions for existing IoT appli- vacy in such a way that the objective of security monitoring cations. The acquired visual sensor data can be combined, is achieved without compromising the privacy of those per- analyzed, and interpreted using contemporary techniques, such sons and subjects who are not subject to security monitoring as predictive analytics, AI, and deep learning. The result- is a significant challenge for IoVT applications. IoVT visual ing knowledge, including identification of patterns and trends, data may be scrambled or encrypted for the purpose of pri- reveals new perspectives that have the potential to touch every vacy preservation. Then, developing an algorithm capable of aspect of our lives, from improving traffic management to processing such scrambled or encrypted visual data to detect combating crime, and from preventing disease to protecting persons or events of security concerns will be much more dif- the environment. ficult than developing an algorithm for original visual data. Currently, emerging IoVT systems are having a signifi- Privacy-preserving processing will become much more cru- cant impact on several major application domains, including cial when integrating multiple IoVT systems for coordinated medical and healthcare, mobile devices, automobile and traf- decision making. fic, robotics, and manufacturing, as illustrated in Fig. 2. In The third technical issue for IoVT intelligent integration is the following, we shall summarize the main attributes of these the standardization of the IoVT platform from data acquisition representative IoVT applications and elaborate the potential at the frontend visual data compression, to the data commu- challenges for them to take full advantage of visual sensor nication protocols in the middle of networking layers, and data when deploying IoVT for these applications. finally to the data sink at the cloud centers. At the frontend, both video compression for IoVT applications and feature extraction for semantic computation need to be standardized. A. Medical and Healthcare Applications This way the transport of visual data as well as the extracted The applications for medical and healthcare started even features can be easily made interoperable. In the middle of during the early years of IoT development. Over the years, data transport, the standardization shall make the selection of various types of medical or healthcare-relevant sensor data that networking layers and protocols suite possible to maximize can be collected via IoT systems to help medical and the overall pervasive networking performance. To facilitate the healthcare professionals to make a more informed decision. search and retrieval of IoVT data from data sink or cloud According to Infographic Design Team, the medical/health- center, data representation also needs to be standardized for related IoT applications will reach $117 billion by this year user friendly and effective operations. The standardization of (2020) with an average growth rate of 15% annually. data representation for different IoVT applications to enable Medical/health IoT devices are connected via WiFi or the intelligent integration of multiple IoVT systems for an machine-to-machine communications for transporting to cloud Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY KURUKSHETRA. Downloaded on March 24,2022 at 11:57:45 UTC from IEEE Xplore. Restrictions apply.  6682 IEEE INTERNET OF THINGS JOURNAL, VOL. 7, NO. 8, AUGUST 2020 area could form a large scale and dynamic IoVT system for specific applications. Smartphones are usually equipped with at least one cam- era for acquiring visual signals. Such an inherent capability makes it ideal for capturing images to be shared with other smartphone users. The processing capability of smartphone is usually able to process the acquired images before being trans- ported to IoVT systems for cloud storage for future search and retrieval tasks. As the smartphones are seamlessly con- nected via 4G and 5G networks to the IoVT cloud center, there is no need for deploying dedicated network for smartphone- based IoVT applications. For smartphone, there are several other sensors that can be adopted for IoT applications. These nonvisual sensor data acquired via smartphone can be supple- mented to visual data to provide contextual information about Fig. 2. Illustration of some emerging IoVT applications. how the visual data was acquired for better decision making. Furthermore, smartphones are not the only type of mobile device suitable for IoVT applications. Other type of mobile center for analysis and feedback to medical and healthcare devices, including tablets, laptops, and even moving vehicles, professionals. One prime example of the medical things is the can also be considered as sensor nodes in some contempo- monitoring of the patient with various sensors. It is believed rary IoVT applications. Among the new type of mobile that a continuous stream of patient-generated data (PGD) tells devices, flying drones will probably pose most significant chal- clinicians far more than a series of intermittent office visits lenge because of their speed and pattern of movement is much ever could. faster and more dramatic than other mobile devices ,. Visual sensors are becoming more effective in gauging the One major challenge in IoVT applications involving mobile personal behavior of subjects in their daily routines. Some phones is the seamless integration of IoVT functionalities of these visual sensors can simply be the smart phones that with the original smartphone functionality simply as a mobile most ordinary people now own, which may be used to take communication and networking device. pictures of their meal samples to monitor their dietary type and amount. Other visual sensor may be a network of cam- eras deployed at the senior citizen center to monitor their well beings, or IoT-enabled mobile devices for hospital’s radiol- C. Automobile and Traffic Applications ogy department to manage and share massive medical images There are two major categories of IoVT applications involv- for seamless patient care and even for teleradiology applica- ing cameras as the sensor nodes. One category has the cameras tions. Still other more dedicated visual sensors, such as video installed on the vehicles to monitor the environments sur- capsule endoscopy, may need to be explicitly developed for rounding the moving vehicles while the other category has collecting specific medically significant visual sensor data for the cameras as sensors nodes installed along the roadside performing diagnostic tasks. One major challenge in the appli- to monitor the intersections and critical locations for traffic cation of IoVT for healthcare is in the acceptance of constant intelligence. According to Gartner , more than 250 million monitoring of one’s daily behavior by the general population vehicles will be connected globally in which most of these with balanced privacy and security measures. vehicles will have one or more cameras installed to help mon- itor the external and internal environment for more efficient and safe navigation. By working with the intelligent trans- B. Mobile Device Applications portation infrastructures established by municipal governments The development of mobile phones has been going along for smart city project, the vehicles equipped with visual sen- a separate path from the IoT applications in the past two sors can contribute significantly to the overall perception of the decades. In recent years, the smart phones, with their enhanced road scenes, including vehicles, pedestrians, and both dynamic processing power to display, manipulate, transmit, and share and static environments. large imaging data sets, have become new favorite class of The cameras installed at the intersections and along crit- IoT sensor nodes, especially as IoVT sensor nodes to take ical locations of the roads have been networked to form advantage of their embedded image capturing and networking IoVT systems providing traffic intelligence for more efficient capabilities. In 2020, the number of smartphone users in the management of metropolitan transportation system. These net- world is 3.5 billion according to a recent estimate , which worked cameras can provide real-time visual information for translates to 45.12% of the world’s population owning a smart- integration at the traffic control center and possible interven- phone. Such a large scale of phone users—each may be treated tion in diverting vehicle movement well ahead of the traffic as an IoVT visual sensor node—when connected, shall form jamming location. More recently, these traffic monitoring the largest IoT network for an application. Even a small per- cameras nodes are augmented with embedded video ana- centage of these smartphone users without a local geographical lytic functions to extract semantics and contextual information Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY KURUKSHETRA. Downloaded on March 24,2022 at 11:57:45 UTC from IEEE Xplore. Restrictions apply.  CHEN: INTERNET OF VIDEO THINGS: NEXT-GENERATION IoT WITH VISUAL SENSORS 6683 to complement with the visual sensor data to achieve sub- capability can ensure instant and effective interaction between stantially improved information networking efficiency. The robots and service environments. One major challenge for cameras installed onboard the vehicle have been mostly used IoVT applications in robotics is the constant and 3-D update to monitor the interior and the immediate environments of of the working environments when both robots and the service an individual vehicle. However, new applications have been environment are equipped with cameras. developed to network onboard cameras with nearby vehi- cles to gather information beyond the view of one vehicle E. Industrial Manufacturing Applications allowing multiple vehicles to share perception data over The IoT technologies recently have been embraced as an vehicle-to-vehicle communications and collaboratively merge essential part of Industry 4.0 while the world is moving toward such data into a more complete traffic scene. One major digital transformation of manufacturing industry through the challenge in the IoVT application for automobile and traf- integration of massively deployed smart sensors into vari- fic monitoring is to develop a unified design of an integrated ous industrial process. Together with different industrial scale system that can combine the visual sensor data from both operational equipment, the sensors and actuators form a system onboard cameras and the cameras installed along the roadside capable of powerful computing and seamless networking under to achieve much needed information fusion. manufacturing environments. Such a system aims at automat- ing the industrial operations with reliable and agile control and D. Robotics Applications actuation based on the data acquired from massive IoT sensors. This class of IoT applications is called Industrial IoT (I-IoT) The IoT technologies have penetrated into many applica- and is expected to improve productivity, efficiency, safety, tion domains and have recently entered the robotics world. and intelligence of manufacturing systems. Because of its Historically, most IoT applications are focused on using unique environmental setting, I-IoT has several distinct char- Internet-connected devices with simple, onboard, passive sen- acteristics that are different from conventional IoT in terms of sors to monitor, and manage systems and processes. The special sensors deployed, fault-tolerant networking technolo- robotic systems usually augment the sensing and moni- gies adopted, and quality of service required, and real-time toring with actions and mobility. The synergies between control and actuations needed. IoT and robotics have been defined as Internet of Robotic Among various sensors deployed for I-IoT systems, visual Things (IoRT) , in which the robots are considered sensors shall play an increasingly important role beyond their as IoT node to perform not only sensor data acquisition but conventional applications in product line inspections. Because also sensor data fusion, embedded action computation, and of the growing affordability of machine vision sensors and manipulation execution. There are a broad range of application significantly more capable associated software and hardware domains have been identified for IoRT, including entertain- components, it is expected that IoVT applications with visual ment, healthcare, education, surveillance and culture. cameras shall be adopted timely for a broad range of I-IoT From its early days, a few classes of robots have already systems. Some of the recent applications focusing on been equipped with visual sensors or cameras to acquire envi- robotics, 3-D imaging, and ML. In particular, significant ronmental data for guidance, control, and navigation. Such researches have been carried out to develop smart software visually augmented IoRT systems are often able to perceive making use of the emerging AI technology and artificial neural the environments most efficiently and more accurately for an networks architectures. In addition to mostly visible imaging improved performance. solutions, nonvisible imaging solutions, including multispec- For IoVT applications in the robotics domain in which fast tral and infrared have also been explored. One successful action and manipulation based on data fusion from networked example of IoVT application in I-IoT is the embedded vision robots are usually required, low latency and broadband com- system capable of processing multiview videos for security munication are necessary. The arrival of 5G has made it pos- and smart process monitoring. One important challenge sible for smooth applications of IoVT principles to networked for adopting IoVT applications in the I-IoT environment is robotics. In particular, in the case of service robot applications for the machine vision industry to offer reliable, repeatable, when the robots are augmented with cameras, such new gen- and evolving standardized interfaces, such as generic interface eration service robots will be capable of obtaining enhanced for cameras (GenICams), for its products. Another crit- location information via IoVT-based simultaneous localization ical challenge is for the networking industry to provide high and mapping (SLAM). More importantly, powerful 5G high bandwidth and low latency communication links for moving volume and low latency networking ensures their instant sens- vast visual data from IoVT sensors around in order to monitor, ing and processing of visual data. In addition, service robot analyze, and most importantly, control the I-IoT operations in with visual sensors have the potential to acquire live emo- a seamless fashion. tional and mood of its client for more personalized service. When the environments of the service robots under opera- tion are installed with visual sensors, such an IoVT-enabled IV. S UMMARY AND L OOKING A HEAD environment will be able to provide more direct and realistic In this article, we have introduced an emerging class of IoT awareness of the service robots and will be able to acquire full systems consisting of massive visual sensors at their frontend. 3-D and dynamic spatial maps through IoVT sensors for more We have named this particular class of IoT systems as IoVT realistic rendering of the service environments. The new 5G and discussed their unique characteristics. The quick rise of Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY KURUKSHETRA. Downloaded on March 24,2022 at 11:57:45 UTC from IEEE Xplore. Restrictions apply.  6684 IEEE INTERNET OF THINGS JOURNAL, VOL. 7, NO. 8, AUGUST 2020 IoVT in the past few years has been propelled by the tech- Cloudview. Visual IoT: Where the IoT, Cloud and Big Data Come nological advances in the related fields, including large-scale Together. Accessed: Oct. 20, 2019. [Online]. Available: http://www.cloudview.co/viot deployment of visual sensors, powerful and energy-efficient S. Severiy et al., “M2M technologies: Enablers for a pervasive Internet embedded processors, explosive increase of bandwidth in 5G of Things,” in Proc. Eur. Conf. Netw. Commun. (EuCNC), 2014, pp. 1–5. networks, rapid development of cloud and edge computing, A. Nauman, Y. Ahmad, M. Amjad, Y. B. Zikria, M. K. Afzal, and S. W. Kim, “Multimedia Internet of Things: A comprehensive survey,” and miniaturization of IoVT components. Although these IEEE Access, vol. 8, pp. 8202–8250, 2020. progresses in the related fields have prepared the technolog- S.-Y. Chien et al., “Distributed computing in IoT: System-on-a-chip for ical environments ready for developing IoVT, we still face smart cameras as an example,” in Proc. 20th Asia South Pac. Design Autom. Conf., 2015, pp. 130–135. significant challenges at various stages of developing such I. B. F. de Almeida, L. L. Mendes, J. J. P. C. Rodrigues, and integrated system. We have also addressed the technical issues M. A. A. da Cruz, “5G waveforms for IoT applications,” IEEE in three primary phases: smart sensing, pervasive networking, Commun. Surveys Tuts., vol. 21, no. 3, pp. 2554–2567, 3rd Quart., 2019. and intelligent integration, which will need to be properly K. Nayudu. (2017). 5G Promises New Horizons for IoT Solutions. resolved in order to overcome key barriers in advancing the [Online]. Available: https://bridgera.com/5g-promises-new-horizons-for- IoVT systems to the next level. Despite numerous challenging iot/ technical issues, the IoVT continue to penetrate into various I-Scoop. (2017). 5G and IoT: The Mobile Broadband Future of IoT. [Online]. Available: https://www.i-scoop.eu/internet-of-things-guide/5g- emerging application domains. We have briefly discussed IoT iot/ and IoVT in five prominent application domains, including C. Bockelmann et al., “Towards massive connectivity support for scal- medical and healthcare, mobile devices, automobile and traf- able MMTC communications in 5G networks,” IEEE Access, vol. 6, pp. 28969–28992, 2018. fic, robotics, and industrial manufacturing, as well as specific J. Braga. (2019). The Internet of Small Things: New Applications in challenges in deploying IoVT in each of the five application Wearables and More. [Online]. Available: https://www.telit.com/blog/ domains. internet-of-small-things-miniaturization-of-iot/ N. Vokas. Next Generation Smart Miniature WiFi Modules Looking ahead, the ultimate objective for an IoT or an Accelerate IoT and M2M Econais Whitepaper. [Online]. Available: IoVT platform will become an intelligent self-learning system. https://www.digikey.com/en/articles/techzone/2015/feb/next-generation- IoVT nodes, such as visual cameras, smartphones, cars, home smart-miniature-wi-fi-modules-accelerate-iot-and-m2m A. R. Biswas and R. Giaffreda, “IoT and cloud convergence: monitoring devices, personal devices, all are considered as Opportunities and challenges,” in Proc. IEEE World Forum Internet data collectors that feed the massive data continuously to Things, 2014, pp. 375–376. future AI routines to be trained to acquire intelligence needed D. Kelaidonis et al., “A federated edge cloud-IoT architecture,” in Proc. EuCNC-EmergConc, 2016, pp. 230–234. for knowledge integration. Therefore, it is crucial for IoVT H. EL-Sayed et al., “Edge of things: The big picture on the system developers to augment the various components of the integration of edge, IoT and the cloud in a distributed com- current system with desired ML capabilities. A recent survey puting environment,” IEEE Access, vol. 6, pp. 1706–1717, 2017, doi: 10.1109/ACCESS.2017.2780087. article presented a variety of ML research activities carried out W. K. A. Hasan, Y. Ran, J. Agbinya, and G. Tian, “A survey of energy for I-IoT applications. We can also design and apply sim- efficient IoT network in cloud environment,” in Proc. Cybersecurity ilar ML principles to various IoVT applications. In particular, Cyberforensics Conf., 2019, pp. 13–21. ML can be applied to achieve smart sensing through embedded L. Chiariglione. Video Coding for Machines. [Online]. Available: https://mpeg.chiariglione.org/standards/exploration/video-coding- processing of frontend visual data to extract proper seman- machines tics and information for networking and integration. ML can M. Wu and C. W. Chen, “Collaborative image coding and transmis- also be applied to pervasive networking through an improved sion over wireless sensor networks,” EURASIP J. Appl. Signal Process., vol. 2007, Jan. 2007, Art. no. 070481, doi: 10.1155/2007/70481. recognition of wireless transmission environments for energy S.-W. Yang, O. Tickoo, and Y.-K. Chen, “A framework for visual fog efficient and low latency communication. ML will definitely computing,” in Proc. IEEE ISCAS, Baltimore, MD, USA, 2017, pp. 1–4. be crucial in the intelligent integration of IoVT data acquired K. Mikhaylov et al., “Energy efficiency of multi-radio massive machine-type communication (MR-MMTC): Applications, challenges, from heterogeneous sources through developing algorithms for and solutions,” IEEE Commun. Mag., vol. 57, no. 6, pp. 100–106, versatile search and retrieval, for balanced security and privacy, Jun. 2019. and for standardized knowledge fusion. A well-developed A. Rego, A. Canovas, J. M. Jimenez, and J. Lloret, “An intelligent system for video surveillance in IoT environments,” IEEE Access, vol. 6, IoVT paradigm will serve the humanity with much enhanced pp. 31580–31598, 2018. intelligence level throughout each block of the IoVT system. BankMyCell. (2020). How Many Smartphones Are in the World? [Online]. Available: https://www.bankmycell.com/blog/how-many- phones-are-in-the-world R EFERENCES Infographic Design Team. (2018). The Year of IoT, and Its Six Staggering Trends to Look for. [Online]. Available: K. Ashton, “That ‘Internet of Things’ thing,” RFID J., to be published. https://www.infographicdesignteam.com/blog/2018-iot-six-staggering- J. Lin, W. Yu, N. Zhang, X. Yang, H. Zhang, and W. Zhao, “A sur- trends-infographic/ vey on Internet of Things: Architecture, enabling technologies, security W. Ji, J. Xu, H. Qiao, M. Zhou, and B. Liang, “Visual IoT: Enabling and privacy, and applications,” IEEE Internet Things J., vol. 4, no. 5, Internet of Things visualization in smart cities,” IEEE Netw., vol. 33, pp. 1125–1142, Oct. 2017. no. 2, pp. 102–110, Mar./Apr. 2019. B. Marr. What Is the Artificial Intelligence of Things? When AI Meets M. Khan, K. Heurtefeux, A. Mohamed, K. A. Harras, and M. M. Hassan, IoT. Accessed: May 31, 2020. [Online]. Available: https://www.forbes. “Mobile target coverage and tracking on drone-be-gone UAV cyber- com/sites/bernardmarr/2019/12/20/what-is-the-artificial-intelligence-of- physical testbed,” IEEE Syst. J., vol. 12, no. 4, pp. 3485–3496, things-when-ai-meets-iot/#35993254b1fd Dec. 2018. V. Ganesan, Y. Ji, and M. Patel. Video Meets the Internet H. Liu, P. Reny, S. Jainz, M. Murady, M. Gruteser, and F. Bai, of Things. Accessed: Dec. 1, 2019. [Online]. Available: “FusionEye: Perception sharing for connected vehicles and its https://www.mckinsey.com/industries/technology-media-and- bandwidth-accuracy trade-offs,” in Proc. 16th Annu. IEEE Int. Conf. telecommunications/our-insights/video-meets-the-internet-of-things Sens. Commun. Netw. (SECON), 2019, pp. 1–9. Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY KURUKSHETRA. Downloaded on March 24,2022 at 11:57:45 UTC from IEEE Xplore. Restrictions apply.  CHEN: INTERNET OF VIDEO THINGS: NEXT-GENERATION IoT WITH VISUAL SENSORS 6685 P. P. Ray, “Internet of robotic things: Concept, technologies, and Chang Wen Chen (Fellow, IEEE) received the challenges,” IEEE Access, vol. 4, pp. 9489–9500, 2016. B.S. degree from the University of Science and ABI Research. The Internet of Robotic Things. [Online]. Available: Technology of China, Hefei, China, in 1983, the https://www.abiresearch.com/market-research/product/1019712-the-inter M.S.E.E. degree from the University of Southern net-of-robotic-things/ California, Los Angeles, CA, USA, in 1986, and A. Kamilaris and N. Botteghi. (Jan. 2020). The Penetration of Internet the Ph.D. degree from the University of Illinois at of Things in Robotics: Towards a Web of Robotic Things. [Online]. Urbana–Champaign, Champaign, IL, USA, in 1992. Available: https://arxiv.org/abs/2001.05514 He is currently a Presidential Chair Professor with H. Xu, W. Yu, D. Griffith, and N. Golmie, “A survey on industrial the Chinese University of Hong Kong, Shenzhen, Internet of Things: A cyber-physical systems perspective,” IEEE Access, China. He continues to serve as an Empire vol. 6, pp. 78238–78259, 2018. Innovation Professor of computer science and engi- A. Shikany. (Jul. 2017). Machine Vision’s Central Role in the neering with the University at Buffalo, State University of New York, Industrial Internet of Things Quality Magazine. [Online]. Available: Buffalo, NY, USA. He was Allen Henry Endow Chair Professor with the https://www.qualitymag.com/articles/94121-machine-visions-central-ro Florida Institute of Technology, Melbourne, FL, USA, from July 2003 to le-in-the-industrial-internet-of-things December 2007. He was on the faculty of electrical and computer engineer- T. Hussain, K. Muhammad, J. D. Ser, S. W. Baik, and ing with the University of Rochester, Rochester, NY, USA, from 1992 to 1996 V. H. C. de Albuquerque, “Intelligent embedded vision for sum- and on the faculty of electrical and computer engineering with the University marization of multiview videos in IIoT,” IEEE Trans. Ind. Informat., of Missouri–Columbia, Columbia, MO, USA, from 1996 to 2003. vol. 16, no. 4, pp. 2592–2602, Apr. 2020. Dr. Chen and his students have received nine (9) Best Paper Awards or Best Student Paper Awards. He has also received several research and pro- fessional achievement awards, including Sigma Xi Excellence in Graduate Research Mentoring Award in 2003, Alexander von Humboldt Research Award in 2009, the University at Buffalo Exceptional Scholar – Sustained Achievement Award in 2012, the State University of New York System Chancellor’s Award for Excellence in Scholarship and Creative Activities in 2016, and the Distinguished ECE Alumni Award from University of Illinois at Urbana-Champaign in 2019. He has been the Editor-in-Chief for IEEE T RANSACTIONS ON M ULTIMEDIA from January 2014 to December 2016. He has also served as the Editor-in-Chief for the IEEE T RANSACTIONS ON C IRCUITS AND S YSTEMS FOR V IDEO T ECHNOLOGY from January 2006 to December 2009. He has been an Editor for several major IEEE Transactions and Journals, including the P ROCEEDINGS OF IEEE, the IEEE J OURNAL OF S ELECTED A REAS IN C OMMUNICATIONS , and the IEEE J OURNAL OF E MERGING AND S ELECTED T OPICS IN C IRCUITS AND S YSTEMS. He has served as Conference Chair for several major IEEE, ACM and SPIE confer- ences related to multimedia video communications and signal processing. His research has been supported by NSF, DARPA, Air Force, NASA, Whitaker Foundation, Microsoft, Intel, Kodak, Huawei, and Technicolor. He is an SPIE Fellow since 2007. Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY KURUKSHETRA. Downloaded on March 24,2022 at 11:57:45 UTC from IEEE Xplore. Restrictions apply.