Introduction to Big Data Concepts

Study Notes

Data consists of facts, numbers, words, sounds, or images that can be recorded and stored, serving as the foundation for information and knowledge derivation.
Raw, unprocessed data is the base, while processed and organized data is termed information; knowledge refers to the understanding derived from the information.

Structured Data: Highly organized, resembling spreadsheets with rows and columns; easy to search and analyze.
Unstructured Data: Lacks a predefined format; includes text documents, emails, social media posts, images, and videos; more complex to analyze.
Semi-structured Data: Falls between structured and unstructured; has some internal organization but does not conform to a strict format, such as JSON files with key-value pairs.

Big Data encompasses data generated frequently, in high volumes, and in various forms; defined by not only its size but also its variety and velocity.

Volume: Refers to the amount of data created; for instance, Facebook hosts over 250 billion images and grows daily.
Velocity: Indicates the speed at which data is generated; Twitter accounts for over 500 million tweets daily.
Variety: Pertains to different data types; Instagram generates diverse formats like photos, videos, and text.
Veracity: Relates to data reliability, quality, and accuracy; poor quality can lead to flawed insights and decisions.
Value: Reflects the importance and usefulness of data for deriving business insights and benefits; considered the most crucial "V" in a business context.

Data storage involves saving digital information in mediums, like hard drives or cloud services, for later access, management, and retrieval.
Multi-Temperature Storage:
- Hot Storage: Frequently accessed data needing fast read/write speeds; used for real-time applications.
- Warm Storage: Occasionally accessed data, which doesn't require rapid access, suitable for historical but relevant reporting.
- Cold Storage: Rarely accessed data kept for long-term retention, like archived historical records.

Data Lake: A flexible storage solution that accommodates both structured and unstructured data at scale, allowing raw data storage in its native format.
Data Warehouse: Structured storage optimized for analyzing large datasets, facilitating data extraction and reporting.
Data Mart: A subset of a data warehouse focusing on a specific business line or team function, providing specialized data access.