Understanding Hadoop and Big Data

Hadoop Lecture

• Hadoop is a framework for processing large datasets
• Key questions to answer include: why Hadoop, what is Hadoop, how to use Hadoop, and examples of Hadoop
• Big data is a collection of large and complex datasets that are difficult to process with traditional tools.

What is Big Data?

• Wikipedia defines big data as a large collection of data that is so large and complex that it's hard to process with traditional data management tools.

Data Creation Growth Projections

• Global data generated annually is increasing significantly year over year.

Who is Generating Big Data?

• Social media, user tracking & engagement, eCommerce, financial services, and real-time search generate big data.

Key Features of Big Data

• Volume: petabytes of data
• Velocity: large throughput, social media, sensor data
• Variety: structured, semi-structured, unstructured data
• Veracity: unclean, imprecise, unclear data

Philosophy to Scale for Big Data

• Divide and conquer approach is used

Distributed Processing

• Assigning tasks efficiently to workers is crucial.
• Task failures and result exchange between workers need solutions.
• Synchronization of distributed tasks is essential.

Big Data Storage

• Big data volumes are massive and storing PBs of data is challenging.
• Disk, hardware, and network failures are common.
• Probability of failures increases with the number of machines.

One Popular Solution: Hadoop

• Hadoop is a popular solution for big data.
• It features a cluster of computers to process large amounts of data.

Hadoop Offers

• Redundant, fault-tolerant data storage
• Parallel computation framework
• Job coordination
• Programmers do not need to worry about file location, task failure or data loss, or computational scaling.

Hadoop History

• Hadoop is an open-source implementation of Google File System (GFS) and MapReduce.
• Developed by Doug Cutting and Mike Cafarella in 2005.
• Donated to Apache in 2006.

Hadoop Stack

• Includes components like HDFS (Hadoop Distributed File System), MapReduce (distributed programming framework), Pig, Hive, and Cascading.

Hadoop Resources

• Links for documentation, tutorials, and guides are provided for further study.