Big Data Overview and Characteristics

Big Data

• Data is crucial for decision-making in all business areas
• By 2025, the world will generate 175 zettabytes (ZB) of data. In 2010 it was just 2ZB
• 90% of data were generated in the last 2 years
• Daily, internet users generate roughly 2,5 million gigabytes of data
• Key characteristics of big data are velocity, variety, volume, veracity and value

The 5 Vs of Big Data

• Velocity: Data gathered in real-time, near-real-time, batch, and streams
• Variety: Structured, semi-structured, and unstructured data
• Volume: Massive amounts of data including terabytes or petabytes
• Veracity: Trustworthiness, authenticity, source, reputation and accountability
• Value: Statistical methods, correlations, and hypothetical relations

Data Sources

• Main sources include Facebook, Twitter, Instagram, and the Internet of Things (IoT).
• Twitter averages 500,000 tweets per minute
• Instagram averages 347,222 posts per minute
• IoT devices generate data from 75 million connected devices.

Data Storage

• Less than 20% of data is stored in relational databases
• 80% is unstructured (text, images, video)
• Big Data storage utilizes Big Data Architectures, cloud storage, and NoSQL databases
• Modern storage solutions are needed because traditional database methods can't handle the volume of data

Data Analysis Methods

• Descriptive Analysis: Summarizing and describing datasets (e.g., unemployment rates)
• Trend Analysis: Analyzing how data changes over time (e.g., how employment changes monthly)
• Comparative Analysis: Analyzing differences between groups, regions, or variables (e.g., comparing unemployment rates across different Spanish communities)

Economic and Financial Data Sources

• Government entities like the INE (National Statistics Institute) provide statistical data on economics, demographics, and social aspects of a country
• Other sources include the Demography and population, economy, labor market, and companies and establishments statistics.
• The Spanish Ministry of Economy, Trade and Enterprise provides macroeconomic, public finance, labor market, financial system, and foreign trade statistics

Additional Data Sources

• Spanish Government
• Madrid Stock Market
• Spanish Bank
• Eurostat
• World Bank
• International Monetary Fund

Introduction to Databases

• Databases are crucial in today's digital world, facilitating efficient data management across industries like e-commerce, social media, banking and healthcare
• Before databases, data was stored on paper, magnetic tapes in books/files, electronic files and directories
• These older methods lacked integrity, security and inability to handle large volumes of data
• The Entity-Relationship model in the 1970s was standardized for managing databases
• Relational Database Management System (RDBMS) emerged with Oracle and eventually Microsoft SQL Server becoming standards
• 1990s led to NoSQL/Data mining and open-source databases with Big Data and cloud computing

Basic Concepts of Databases

• A database is a structured collection of interrelated data
• It is composed of tables with rows and columns where the data is stored
• Data can relate to people, products, orders and more
• Databases started as simple spreadsheets that evolved into complex organizations

Database Management Systems (DBMS)

• Software to handle creation, retrieval, updating, and maintenance of databases
• Interfaces between users and the database, ensuring data integrity and security
• Common DBMSs include Oracle Database, Microsoft SQL Server, and MySQL

SQL

• Structured Query Language (SQL) is a standardized language for managing and manipulating data in relational databases
• Handles large data volumes efficiently and is easy to learn

Data Types

• Different data types exist for storing various kinds of data (integers/characters, dates, large amounts of text, floating point numbers etc)

Data Analysis in Databases

• Database analysis use aggregations and filtering to derive insights
• Aggregation includes calculations like sums, average, counts, maximum or minimum values etc
• Filtering conditions allow data selection using criteria (e.g., select customers that order more than twice etc)
• Subqueries can be used in queries for more complex filtering, calculation, or results extraction

Database Design

• The process of defining data structures, storage mechanisms, and retrieval methods in a database system
• Crucial for efficiently storing, accessing and managing data
• Important factors that are included are schemes, normalization, physical implementation, and performance optimization

Excel as a Database Tool

• Excel serves as a flat-file database for small datasets, lacking complex relational structures found in SQL databases.
• Relationships between tables can be established and queries can be performed

NoSQL Databases

• Designed for large datasets and flexible schemas
• NoSQL databases come in different models (document stores, key-value stores, graph databases)
• They are good for storing and managing non-tabular data types such as images and social media feeds

PowerBI tools for Databases

• Provides a user-friendly interface, allowing both technical and non-technical users to interact with underlying data sources
• Supports report building and data visualization, and can query and analyze data from multiple sources
• Used for exploratory analysis and visualization rather than being a primary storage tool like SQL databases

Explore the essential aspects of big data, including its significance in decision-making across various business sectors. Learn about the five key characteristics of big data, known as the 5 Vs: velocity, variety, volume, veracity, and value, and understand the sources generating vast amounts of data daily.