Big Data Overview and 5 Vs

Big Data Overview

• Data is crucial for decision-making in all business areas.
• By 2025, the world will generate 175 zettabytes (ZB) of data.
• In 2010, data generation was only 2 ZB.
• Daily, internet users generate approximately 2.5 million gigabytes (GB) of data.
• 90% of all data generated in the past two years.

5 Vs of Big Data

• Velocity: Data streams encompass batch, near real-time, real-time, and streamed data.
• Variety: Data encompasses structured, unstructured, and semi-structured formats.
• Volume: Data is measured in terabytes, records, transactions, tables, and files.
• Veracity: Data incorporates trustworthiness, authenticity, origin, reputation, and accountability.
• Value: Data includes statistical, event, correlation, and hypothetical information.

Data Sources

• Facebook
• Twitter (500,000 tweets per minute)
• Instagram (347,222 posts per minute)
• Internet of Things (IoT) sensors with 75 million connected devices.

Data Storage

• Less than 20% of global data is stored in relational databases (important for banking, hospitals, etc).
• 80% of global data is unstructured (text, images, videos).
• Storage occurs in Big Data Architectures, the cloud, and NoSQL databases.
• Specialized technologies are needed to process and analyze the massive scale of data that traditional databases cannot handle.

Data Storage Methods

• HDFS (Hadoop Distributed File System):

  • Divides data into small blocks (128MB or 256MB).
  • Distributes data across multiple servers for redundancy and fault tolerance.

• Data Lakes:

  • Centralized repositories for diverse raw data formats (structured, semi-structured, unstructured).
  • Useful when needing to store large volumes of data for potential future analysis.
  • Uses raw data as-is without transformation.

• NoSQL:

  • Flexible, fast, suitable for unstructured data.
  • Used when data is constantly evolving.

• Relational Databases (SQL):

  • High consistency and suited for well-structured data.
  • Necessary when data integrity is paramount.

This quiz covers the importance of data in business decision-making and the tremendous growth of data generation projected by 2025. It also explores the 5 Vs of Big Data: Velocity, Variety, Volume, Veracity, and Value, along with various data sources contributing to this massive influx. Test your knowledge on these critical concepts!