Big Data Introduction

Introduction to Big Data

• Big Data is an extremely large and diverse collection of structured, unstructured, and semi-structured data.
• Big Data datasets grow exponentially.
• Datasets are huge and complex in volume, velocity and variety.
• Traditional data management systems cannot store, process, and analyze Big Data.
• The amount/availability of data is growing because of digital technology advancements
• Connectivity, mobility, IoT (Internet of Things), and AI (Artificial Intelligence) spur the rapid growth of data.
• Big Data tools are emerging which are helping companies to collect, process, and analyze data at speeds that allow them to gain maximal value.

What is Big Data?

• There is no single standard definition.
• Describes data whose scale, diversity, and complexity require new architecture.
• Requires new techniques, algorithms, and analytics to manage it.
• Allows people to extract value and hidden knowledge.

Types of Data

• Structured
  • Has a defined data model, format, and structure.
  • Database is an example.
• Semi-Structured/Quasi-Structured/Unstructured
  • Textual data files showcase an apparent pattern that enables analysis: spreadsheets and XML files.
  • Textual data formats are erratic and complex to format even when using software tools: clickstream data.
  • Data has no inherent structure, stored as different types of files.
  • Text documents, PDFs, images, and videos serve as examples of the storage.

Characteristics of Big Data

• Volume (Scale)
  • Data volume has increased 44x between 2009-2020.
  • Data volume size went from 0.8 zettabytes to 35 zettabytes.
  • Data volume increases exponentially.
• Variety (Complexity)
  • Includes relational data (tables, transactions, legacy data), text data (web), semi-structured data (XML), and graph data.
  • Social networks, Semantic Web (RDF)
  • Streaming Data (Stream vs Static): data is only scanned only once.
  • A single application generates/collects many types of data.
  • Big public data is included, for example, online, weather, finance.
  • To extract knowledge, all types of data is linked together.
• Velocity (Speed)
  • Data is generated and processed fast.
  • Online Data Analytics
  • Late decisions = missed opportunities.
  • E-Promotions use current location/purchase history to send promotions quickly.
  • Healthcare monitoring uses sensors to monitor activities and the body.
  • Healthcare monitoring provides nearly immediate reaction to measurements.

4 V's of Big Data

• Volume - Refers to the amount of data.
• Velocity - Refers to the speed at which data is processed.
• Variety - Refers to the different types of data
• Veracity - Refers to the uncertainty of the data.

5/6 V's of Big Data

• Volume of data creates storage and analysis challenges.
• Velocity of rapidly changing data creates real-time analysis challenges.
• Variety of diverse data from numerous sources creates integration and analysis challenges.
• Variability of constantly changing meaning of data creates challenges in gathering and interpretation.
• Veracity refers to the varying quality/reliability of data; transforming and trusting data is challenging.
• Cost-effectiveness and business value.

Harnessing Big Data

• OLTP: Online Transaction Processing (DBMSs).
• OLAP: Online Analytical Processing (Data Warehousing).
• RTAP: Real-Time Analytics Processing (Big Data Architecture & Technology).

Generating/Consuming Data

• The model of generating/consuming data has changed.
• Old Model: few companies generate data, while others consume it
• New Model: all users generate and consume data.

What's Driving Big Data?

• Optimizations and predictive analytics drive big data.
• Complex statistical analysis.
• All types of data and many sources.
• Very large datasets, more real-time insights.
• Ad-hoc querying and reporting.
• Data mining techniques, structured data, typical sources.
• Small to mid-size datasets.

Challenges in Handling Big Data

• Needs a new architecture, algorithms, techniques.
• Technical skills are needed, experts in using the new technology and dealing with big data.