Chapter 1 Big Data PDF

Dr. Ali Alshdifat Data Science Department 2024 – 2025 Chapter 1: Big Data INDUSTRIAL REVOLUTIONS According to the World Economic Forum ” We stand on the brink of a technological revolution that will fundamentally alter the way we live, work, and relate to one another. In its scale, scope,...

Dr. Ali Alshdifat Data Science Department 2024 – 2025 Chapter 1: Big Data INDUSTRIAL REVOLUTIONS According to the World Economic Forum ” We stand on the brink of a technological revolution that will fundamentally alter the way we live, work, and relate to one another. In its scale, scope, and complexity, the transformation will be unlike anything humankind has experienced before” Industry 4.0 Enabling Technologies  What is a Data? Data is any set of characters that has been gathered and translated for some purpose, usually analysis. It can be any character, including text and numbers, pictures, sound, or video.  DATA is the NEW OIL! But do you have the capacity to refine it?  What is Digital Data? Digital data are discrete, discontinuous representations of information or work. Digital data is a binary language. Types of Digital Data 1.Unstructured Data 2. Semi Structured Data 3. Structured  Structured Data Refers to any data that resides in a fixed field within a record or file. Support ACID properties Structured data has the advantage of being easily entered, stored, queried and analyzed. Structured data represent only 5 to 10% of all informatics data.  Unstructured Data Unstructured data is all those things that can't be so readily classified and fit into a neat box. Unstructured data represent around 80% of data. Techniques: Data mining-Association rule, Regression analysis, Text mining, NLP etc.,  Semi Structured Data Semi-structured data is a cross between the two. It is a type of structured data, but lacks the strict data model structure. Semi-structured data is information that doesn’t reside in a relational database but that does have some organizational properties that make it easier to analyze.  Characteristic of Data Composition - What is the Structure, type and Nature of data? Condition - Can the data be used as it is or it needs to be cleansed? Context - Where this data is generated? Why? How sensitive this data? What are the events associated with this data?  What is Big Data? Collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.  What is Big Data? Cont.. The data is too big, moves too fast, or doesn’t fit the structures of your database architectures The scale, diversity, and complexity of the data require new architecture, techniques, algorithms, and analytics to manage it and extract value and hidden knowledge from it Big data is the realization of greater business intelligence by storing, processing, and analyzing data that was previously ignored due to the limitations of traditional data management technologies. Why Big Data? & what makes Big Data? Key enablers for the growth of “Big Data” are Increase of storage capacities Increase of processing power Availability of data Every day we create 2.5 quintillion bytes of data. 90% of the data in the world today has been created in the last two years.  Where does data come from? Data come from many quarters.  Science – Medical imaging, Sensor data, Genome sequencing, Weather data, Satellite feeds  Industry - Financial, Pharmaceutical, Manufacturing, Insurance, Online, retail  Legacy – Sales data, customer behavior, product databases, accounting data etc.,  System data – Log files, status feeds, activity stream, network messages, spam filters. Where does data come from? Cont.. Characteristics Of 'Big Data' 5V’s - Volume, Velocity, Variety, Veracity & Variability 7V’s  CHALLENGES More data = more storage space Data coming faster Needs to handle various data structure Agile business requirement Securing big data Data consistency & quality  What is the importance of Big Data? The importance of big data is how you utilize the data which you own. Data can be fetched from any source and analyze it to solve that enable us in terms of 1) Cost reductions 2) Time reductions 3) New product development and optimized offerings, and 4) Smart decision making.  What is the importance of Big Data? Cont.. Combination of big data with high-powered analytics, you can have great impact on your business strategy such as: 1) Finding the root cause of failures, issues and defects in real time operations. 2) Generating coupons at the point of sale seeing the customer’s habit of buying goods. 3) Recalculating entire risk portfolios in just minutes. 4) Detecting fraudulent behavior before it affects and risks your organization.  Who are the ones who use the Big Data Technology? Banking Government Education Health Care Manufacturing Retail  Storing Big Data Analyzing your data characteristics  Selecting data sources for analysis  Eliminating redundant data  Establishing the role of NoSQL Overview of Big Data stores  Data models: key value, graph, document,  column-family  Hadoop Distributed File System  HBase  Hive  Big Data Analytics It is the process of examining big data to uncover patterns, unearth trends, and find unknown correlations and other useful information to make faster and better decisions.  Why is big data analytics important? Big data analytics helps organizations harness their data and use it to identify new opportunities. That, in turn, leads to smarter business moves, more efficient operations, higher profits and happier customers.  Types of Analytics Business Intelligence Descriptive Analysis Predictive Analysis  Business intelligence (BI) It is a technology-driven process for analyzing data and presenting actionable information to help executives, managers and other corporate end users make informed business decisions.  Descriptive Analysis Descriptive statistics is the term given to the analysis of data that helps describe, show or summarize data in a meaningful way such that, for example, patterns might emerge from the data.  Predictive Analysis Predictive analytics is the branch of data mining concerned with the prediction of future probabilities and trends. The central element of predictive analytics is the predictor, a variable that can be measured for an individual or other entity to predict future behavior.  Predictive Analysis There is 2 types of predictive analytics: ◦ Supervised Supervised analytics is when we know the truth about something in the past Example: We have historical weather data. The temperature, humidity, cloud density and weather type (rain, cloudy, or sunny). Then we can predict today weather based on temp, humidity, and cloud density today ◦ Unsupervised Unsupervised is when we don’t know the truth about something in the past. The result is segment that we need to interpret Example: We want to do segmentation over the student based on the historical exam score, attendance, and late history.  Tools used in Big Data Where processing is hosted? Distributed Servers / Cloud (e.g. Amazon EC2) Where data is stored? Distributed Storage (e.g. Amazon S3) What is the programming model? Distributed Processing (e.g. MapReduce) How data is stored & indexed? High-performance schema-free databases (e.g. MongoDB) What operations are performed on data? Analytic / Semantic Processing Top Cloud Platforms in 2024  Top Big Data Technologies 1. Apache Hadoop Apache Hadoop is a java based free software framework that can effectively store large amount of data in a cluster. Hadoop Distributed File System (HDFS) is the storage system of Hadoop which splits big data and distribute across many nodes in a cluster. This also replicates data in a cluster thus providing high availability. It uses Map Reducing algorithm for processing.  Top Big Data Technologies Cont.. 2. NoSQL NoSQL (Not Only SQL)is used to handle unstructured data. NoSQL databases store unstructured data with no particular schema. NoSQL gives better performance in storing massive amount of data. There are many open-source NoSQL DBs available to analyse big Data.  Top Big Data Technologies Cont.. 3. Apache Spark Apache Spark is part of the Hadoop ecosystem, but its use has become so widespread that it deserves a category of its own. It is an engine for processing big data within Hadoop, and it's up to one hundred times faster than the standard Hadoop engine, Map Reduce.  Top Big Data Technologies Cont.. 4. R R, another open source project, is a programming language and software environment designed for working with statistics. Many popular integrated development environments (IDEs), including Eclipse and Visual Studio, support the language. Applications for Big Data Analytics  DATA SCIENTIST Data scientist/analyst is one of the trending and emerging job in the market