Emerging Technologies Overview

• Emerging Technology describes new technologies, or continuing development of existing ones
• Commonly refers to technologies currently developing, or expected to be available within 5-10 years
• Is reserved for technologies creating significant social or economic effects

Root Words

• Technology: From Greek "tekhnologia," meaning "systematic treatment of an art, craft, or technique"
• Evolution: From Latin "evolutio," meaning "an unrolling or opening," combined with "out" and "to roll"

Emerged Technologies

• Artificial Intelligence
• Blockchain: Growing, secured list of records for transactional data
• Augmented/Virtual Reality: Overlays computer-generated content
• Cloud Computing
• Angular and React
• DevOps: Software development and IT operations
• Internet of Things (IoT): Networking of smart objects
• Intelligent Apps (I-Apps)
• Big Data: Complex data sets difficult to process manually
• Robotic Processor Automation (RPA): Automates business processes

Introduction to the Industrial Revolution (IR)

• Period of major industrialization and innovation in the late 1700s and early 1800s
• Shift from tools to new energy sources like coal to power machines
• Started in England, leading to more efficient labor and productivity
• Manufacturing moved from homes/small shops to large factories.
• Cultural shifts resulted from people moving from rural to urban areas

American Industrial Revolution (IR)

• Commonly called the Second Industrial Revolution, between 1820 and 1870
• Industries like textile manufacturing, mining, glass making, and agriculture changed.
• 1st IR: Mechanization via water/steam power
• 2nd IR: Mass production and electricity
• 4th IR: Smart, autonomous systems fueled by data and machine learning build on the 3rd

Industrial Revolution Impacts

• Fundamentally transferred the world into modern society
• Key components include:
• Steam engine
• The age of science and mass production
• The rise of digital technology
• Smart and autonomous systems fueled by data and machine learning

Important Inventions of the Industrial Revolution

• Transportation: Steam engine, railroad, diesel engine, airplane
• Communication: Telegraph, Transatlantic Cable, Phonograph, Telephone
• Industry: Cotton Gin, Sewing Machine, Electric Lights

Historical Background (IR 1.0, IR 2.0, IR 3.0, IR 4.0)

• Began in Great Britain in the late 1770s and spread
• Belgium, France, and the German states followed England
• Triggered by the effects of the Agricultural Revolution.
• Agricultural Revolution increased food production

Types of Industries:

• Primary: Raw materials like mining, farming, and fishing
• Secondary: Manufacturing like cars and steel
• Tertiary: Service-based like teaching and nursing
• Quaternary: Research and development like IT

Industrial Revolution (IR 1.0)

• Transition to new manufacturing processes
• Started in the 1760s
• Transitions included from hand production -> machine and an increase in steam power

Industrial Revolution (IR 2.0)

• Also known as the Technological Revolution, began in the 1870s
• Involved improvements in manufacturing interchangeable parts
• Widespread use of telegraph and railroad systems
• Vastly improved movement of people and ideas
• New systems: electricity, telephones

Industrial Revolution (IR 3.0)

• Transition from mechanical/analog to digital electronics in the late 1950s
• Nicknamed the "Digital Revolution"
• Core factor: Mass production and widespread digital logic circuits
• Transformed traditional production and business communication techniques
• Proliferation of computers and digital records

Industrial Revolution (IR 4.0)

• Coined by Klaus Schwab in 2016
• Advancements in robotics, IoT, additive manufacturing, and autonomous vehicles
• Technologies are called cyber-physical systems
• Cyber-physical System: Controlled/monitored by computer-based algorithms integrated with the Internet
• Examples: CNC machines and AI

The Role of Data for Emerging Technology

• Data is a key strategic asset driving the future of science, tech, and the economy
• Presents challenges leading to innovation and economic opportunities
• Paradigm shifts are driven by understanding, exploring and utilizing data

Data and Debate

• Potential has triggered new debate about data-intensive discovery
• Focus involves core disciplines like computing, informatics, and statistics
• Also involves broad fields like business, social science, and health/medical science.

Enabling Devices & Network (Programmable Devices)

• Four basic kinds of devices in digital electronics: memory, microprocessors, logic, and networks
• Memory: stores random data
• Microprocessors: execute software instructions
• Logic: provides functions like interfacing, display, and control
• Network: interconnected devices for data sharing
• Example: Internet
• Refers to chips with FPLDs, CPLDs, and PLDs
• Also analog equivalents called field programmable analog arrays

Programmable Devices/Computer Connection

• Computers are programmable because they follow instructions

Service Enabling Devices (Network Related Equipment)

• Traditional channel service unit (CSU) and data service unit (DSU)
• Modems
• Routers
• Switches
• Conferencing equipment
• Network appliances (NIDs and SIDs) for intrusion detection
• Signature-Based Intrusion Detection System (SIDS) and Network-Based Intrusion Detection System (NIDS)
• Hosting equipment and servers

Human to Machine Interaction (HMI)

• Communication/Interaction between human and machine via a user interface
• Gestures offer a more natural control over machines

Human-Computer Interaction (HCI)

• Study of how people interact with computers and how computers should be developed
• Involves the User, the Computer, and their interaction

User Interaction with Computers

• Using hardware like displays through a graphical user interface
• Interacting with the software interface and I/O hardware

Importance of Human-Computer Interaction

• Aims to improve interaction between users and computers
• Make the experience user-friendly and receptive to needs
• Advantages include:
  • Simplicity
  • Increased efficiency
  • Easy deployment
  • Cost savings
  • Reduced solution design time

Disciplines Contributing to Human-Computer Interaction (HCI)

• Cognitive psychology: focus on limitations, information processing, performance, cooperative working
• Computer science: covers graphics, technology, prototyping, user interface
• Linguistics
• Engineering and design
• Artificial intelligence
• Human factors

Future Trends in Emerging Technologies

• Includes 5G Networks, Artificial Intelligence (AI), Autonomous Devices, Blockchain
• Augmented Analytics, Digital Twins, Enhanced Edge Computing and Immersive Experiences in Smart Spaces

High-Level Emerging Technologies

• Emerging technologies are increasingly influencing our lives
• Key tech include:
  • Chatbots
  • Virtual/augmented reality
  • Blockchain
  • Ephemeral apps(messages, photos disappear shortly after sharing)
  • Artificial Intelligence

Data Science

• Multi-disciplinary field extracting knowledge/insights from structured, semi-structured and structures data using scientific methods
• More than simply analyzing data.

Data

• Representation of facts, concepts, or instructions in a formalized manner
• Should be suitable for communication, interpretation, or processing
• Unprocessed facts and figures
• Represented using alphabets, digits, or special characters

Information

• Processed data used for decision-making
• Data processed into meaningful form
• Interpreted, organized, structured, and processed data.

Data Processing Cycle

• Restructuring or re-ordering data to add usefulness and values
• Basic steps: Input, processing, and output
• These steps constitute the data processing cycle

Data Processing Cycle Steps

• Input: Preparing data in a convenient form
  • depends on machine ex: audio signal can be hard disk
• Processing: Changing data for better use and calculation
  • example being interest deposit to a bank
• Output: Results of processing step
  • Example: payroll for employees

Data Types from Computer Programming Perspective

• Attribute of data telling the compiler/interpreter how the data is going to be used
• Includes:
• Integers (int): Whole numbers
• Booleans(bool): True or false
• Chars(char): Singular character
• Floating-point numbers(float): Real numbers
• Alphanumeric strings(string): Combination of characters' and numbers' storage

Data Types from Data Analytics Perspective

• Important to understand there are three common data structures: Structured, Semi-structured, and Unstructured

Structured Data

• Follows a pre-defined data model
• Straightforward to analyze
• Conforms to tabular format
• Relationships between rows and columns
• SQL databases and Excel files

Semi-structured Data

• Structured data with tags/markers delineating semantic elements and hierarchies
• A self-describing structure
• Formats: JSON and XML
• JavaScript Object Notation (JSON): interchangeable data format between web servers and clients
• Extensible Markup Language (XML): Markup language that provides rules to define any data

Unstructured Data

• Does not have a pre-defined model or organization
• Typically text-heavy, with data such as dates, numbers, and facts
• Irregularities and ambiguities make it difficult to understand with traditional programs
• Examples: Audio/video files, NoSQL databases

Metadata (Data about Data)

• Important part for Big Data analysis and solutions
• Provides more information about a specific set of data
• Used in Big Data solutions for initial analysis
• Examples: Date/location of a photo which can be categorized as structured

Data Value Chain

• Describes how information flows within a big data system
• How steps are needed to generate value and insights from the data

Key High-Level Activities

• Data Acquisition: Gathering, filtering, and cleaning data for analysis
• Data Analysis: Making raw data acquired more available to decision-makers
• Data Curation: Managing data over its life cycle to ensure the data quality requirements

High-Level Activities continued

• Data Storage: Persistence and management of data in a scalable way that will allow the fastest access
• Data Usage: Data-driven operations which need its analysis and business analysis integrating tools

Data Acquisition

• process of gathering, filtering, and cleaning data
• In preparation for data warehouse or other stores for analysis
• Big data challenge, needing infrastructure
• Must deliver low latency
• Handle high transaction volumes in distributed environment
• Support flexible and dynamic data structures

Data Analysis

• Making acquired data suitable for decision-making
• Exploring, transforming, and modeling data to highlight relevant data
• Synthesizing useful hidden in information with high potential
• Related areas: data mining, business, analytics, machine learning

Data Curation

• Active management of data over its life cycle
• Ensures data quality requirements for usage
• Categorized into activities: content creation, selection, transformation, validation, and preservation
• Performed by expert curators improving data accessibility & quality
• Known as scientific curators or data annotators, responsible for trustworthy, discoverable, accessible & reusable data

Data Storage

• Persistence and management of data in a scalable way
• Satisfies applications needing fast data access
• RDBMS has been a solution for almost 40 years
• ACID properties guarantee database transactions.
• NoSQL have scalability and alternative data models

Data Usage

• Data-driven activities needing data access
• Analysis and integrating tools
• Usage helps in business decision-making
• Enhances competitiveness, reduces costs, increases value

Basic Concepts of Big Data

• Non-traditional strategies and technologies
• Needed to gather, organize, process, and gain insights from large datasets
• Problem of working with data exceeding a single computer's capacity is not new
• Scale, value, and pervasiveness of this type of computing have expanded recently

Basic Info

• Big Data is for large, complex data that is difficult to manage
• "Large dataset" means that process or store with the common tool on computer
• Big data is characterized by 4Vs: Volume, Velocity, Variety, and Veracity

Characteristics of Big Data

• Volume (large amounts of data)
• Velocity (live streaming or in motion)
• Variety (different forms from diverse sources)
• Veracity (how accurate)

Clustered Computing

• Because of big data qualities, it’s inadequate for regular ones at most stages
• Clusters have a better storage and fit
• Combine smaller machines together
• Provides benefits: Resource, pooling, high availability, easy availability

Clustered Computing with Hadoop

• Solution for managing the cluster membership, coordinating resource sharing, and scheduling work on nodes
• Cluster membership and resource can be handled by Hadoop's YARN (Yet Another Resource Negotiator)
• The clustering often interfaces to process the data for the assembled software
• The involved machines will handle the distributed storage system with the data persistence

Hadoop and its Ecosystem

• Hadoop framework used for easier interactions with big data
• Distributed processing of datasets across computing clusters.
• Hadoop is based on Google's technical document
• Has four main properties
  • Economical
  • Reliable
  • Scalable
  • Flexible

Hadoop Properties

• Economical: Ordinary computers are used for data processing
• Reliable: Data copies are stored on different machines, resisting hardware failure
• Scalable: Easily grows horizontally and vertically with a few extra nodes
• Flexible: Can hold both structured and unstructured data and choose when to use it

Hadoop Components

• Includes HDFS: Hadoop Distributed File System
• YARN: Yet Another Resource Negotiator: allocates system resources
• MapReduce: Programming-based Data Processing, used to access big data in HDFS
• Spark: In-Memory data processing, overcoming limitations of Hadoop's external storage

Hadoop Components continued

• PIG, HIVE: Pig handles all structured and semi-structured data where as Hive handles structured
• HBase: NoSQL Database
• Mahout, Spark MLLib: Machine Learning algorithm libraries
• Solar, Lucene: Searching and Indexing
• Zookeeper: Managing cluster
• Oozie: Job Scheduling

Big Data Life Cycle with Hadoop: Ingesting and processing

• First step: Ingest
• Data is ingested from local files, systems or relations
• Sqoop - tool used to transfer data between hadoop systems
• It then goes from RDBMS to HDFS, with Flume

Big Data With Hadloop: storage, computing results

• Second step: Processings
• Then stage : is processed and stored data
• In the form of distributed data like HDFD & NoSQL data, HBAse
• After which is performed by spark and mapreduce
• Then third stage Analyze which Is for processing
• Then Pig and Hive can then Analyze this data
• Fourth stage involves Cloudera and Hue for access and full results