Big Data Overview

Big Data

• Data is crucial for decision-making across all business areas.
• Global data volume is projected to reach 175 zettabytes (ZB) by 2025 (1 ZB = 1 billion gigabytes).
• 90% of data has been generated in the last two years.
• 2.5 million gigabytes of data are generated daily.

The 5 Vs of Big Data

• Velocity: Batch, near real-time, real-time, and streaming data.
• Variety: Structured, unstructured, and semi-structured data.
• Volume: Large amounts of data (terabytes, records, transactions, etc.).
• Veracity: Trustworthiness, authenticity, origin, reputation, and accountability of data.
• Value: Statistical analysis, and correlations.

Data Sources

• Facebook, Twitter, Instagram
• IoT devices (75 million connected devices generate data).
• Less than 20% of global data resides in relational databases.
• 80% of data is unstructured (text, images, videos).
• Big data is stored in big data architectures, cloud, and NoSQL databases.

Big data storage

• Different technologies needed for storing, processing, and analyzing large volumes of data that cannot be handled by traditional databases.
• Hadoop Distributed File System (HDFS) used to handle large data volumes across multiple servers by dividing data into small blocks and distributing them across different nodes (servers) for high redundancy.
• Data Lakes: a centralized storage for a variety of data (structured, semi-structured and unstructured) is raw and stored as it's generated for long-term analysis.
• NoSQL databases are used for storing unstructured data.

Economic and Financial Data Sources

• Diverse sources for economic and financial data analysis.
• Descriptive analysis → summarize and describe a dataset (employment rate in Spain by age group).
• Trend analysis → analyze how data changes with time (Changes in employment rate in different regions of Spain in the last year).
• Comparative analysis → analyze differences between regions, groups, or variables (changes in unemployment rates by month in different regions).
• INE (Spanish National Statistics Institute) provides a wide range of data on the country's economy, demographics, and social aspects.
• Ministry of Economy, Trade, and Enterprise offers various financial data and statistics.

Relational Databases

• Store data related to one another. Data is structured in tables with rows and columns.
• Tables define an entity and each row a record.
• Relationships are stored using keys (primary and foreign) to link related data in different tables.
• Ensure data integrity and consistency.
• Efficient data retrieval.
• Reduce data redundancy.

Introduction to SQL

• Structured Query Language (SQL) is used for managing and manipulating relational databases.
• Provides powerful commands for creating, reading, updating, and deleting data.
• Widely used for data analysis, report generation, and query operations.

Microsoft Excel

• A tool that can function as a flat-file database.
• Stores data in a single table or sheet.
• Useful for small data analysis, rapid prototyping, simple queries and explorations of datasets.
• Data model for relationships between tables.
• Important consideration for data integrity.

NoSQL Databases

• Non-relational databases.
• Flexible schemas (schema-less).
• Used to handle unstructured or semi-structured data.
• Useful when dealing with large volumes of data or highly flexible schemas.
• Examples: MongoDB, Redis, Neo4j

PowerBI as a Database Tool

• Focuses on data analysis and visualization, not data storage.
• Connects data from various sources including SQL or Excel.
• Tool to create custom applications for data analysis and reporting.
• Tool for interactive visualization, reporting, and business intelligence with ease of use.

Explore the crucial aspects of big data, including its significance in decision-making and the 5 Vs that define its characteristics. Discover the sources and types of data generated in today's digital landscape and the implications for businesses as they navigate vast amounts of information.