Hadoop: Ecosystem and Tools

• This chapter introduces Hadoop, guiding principles, major components of its ecosystem, and tools used in homework labs.

Problems with Traditional Large-scale Systems

• Traditionally, computation has been processor-bound, with smaller amounts of data undergoing complex processing.
• The early solution of building bigger computers with faster processors/more memory was insufficient.
• Distributed systems use multiple machines for a single job, offering a better solution.
• Challenges with distributed systems include programming complexity, data synchronization, bandwidth limitations, and partial failures, all addressed by Hadoop.

What is Apache Hadoop?

• Apache Hadoop enables scalable and economical data storage, processing, and analysis, using industry-standard hardware to harness distributed and fault-tolerant capabilities.
• Hadoop's architecture, inspired by Google's technical documents, facilitates batch, search, SQL analytics, machine learning, and stream processing with workload management, data storage (filesystem, online NoSQL), and data integration.
• Common use cases include ETL, text mining, index building, graph creation, pattern recognition, collaborative filtering, prediction models, sentiment analysis, and risk assessment.
• The nature of the data involves volume, velocity, and variety.
• Traditionally, data is stored in a central location, and then copied to processors at runtime which is fine for limited amounts of data.
• The modern approach is to bring the program to the data rather than the data to the program.
• Hadoop distributes data when it is stored and runs computations where the data resides.
• Core Hadoop includes processing with Spark and MapReduce, resource management with YARN, and storage with HDFS.

Data Storage and Ingest

• Hadoop ingests data from many sources and formats including traditional data management systems, logs/machine-generated (event) data, and imported files.
• Data storage includes Hadoop Distributed File System (HDFS) as the storage layer, offering inexpensive, reliable storage for massive data amounts on industry-standard hardware.
• Data in HDFS is distributed during storage.
• Apache HBase is a NoSQL distributed database built on HDFS, scaling to support large data volumes and high throughput, with tables that can have thousands of columns.
• Data ingest tools include HDFS for direct file transfer, Apache Sqoop for high-speed imports to HDFS from relational databases, Apache Flume as a distributed service for ingesting streaming data, suited to event data from multiple systems or log files, and Kafka as a high throughput, scalable messaging system, enabling distributed reliable publish-subscribe functionality, and integrating with Flume and Spark Streaming.

Data Processing

• Spark is a large-scale data processing engine for general-purpose tasks which runs on Hadoop clusters and data in HDFS.
• Spark supports machine learning, business intelligence, streaming, and batch processing.
• Hadoop MapReduce is the original Hadoop framework that is Java-based, based on the MapReduce programming model.
• MapReduce is the core Hadoop processing engine, being the dominant technology but losing ground to Spark quickly. Other tools are still built using MapReduce code.
• MapReduce has extensive fault tolerance built into the framework.
• Apache Pig builds on Hadoop to offer high-level data processing.
• Pig is an alternative to writing low-level MapReduce code and is especially good at joining and transforming data.
• The Pig interpreter runs on the client machine, turns Pig Latin scripts into MapReduce or Spark jobs, and submits those jobs to a Hadoop cluster.

Data Analysis and Exploration

• Impala is a high-performance SQL engine running on Hadoop clusters with data in HDFS files, inspired by Google's Dremel project.
• Impala features low latency measured in milliseconds, making it ideal for interactive analysis.
• Impala supports a dialect of SQL, modeling data in HDFS as database tables, was developed by Cloudera, and is 100% open source under an Apache software license.
• Hive is an abstraction layer on top of Hadoop, that uses a SQL-like language called HiveQL, similar to Impala SQL, useful for data processing and ETL. Impala is preferred for ad hoc analytics.
• Hive executes queries using MapReduce with an early adopter version available for Spark.
• Cloudera Search facilitates interactive full-text search for data in a Hadoop cluster, allowing non-technical users to access the data.
• Cloudera Search enhances Apache Solr, integrating Solr with HDFS, MapReduce, HBase, and Flume, supports file formats widely used with Hadoop through its dynamic Web-based dashboard interface (Hue) and Apache Sentry-based security, being 100% open source.

Other Ecosystem Tools

• Hue is the Hadoop User Experience (UI) providing a Web front-end to Hadoop for actions like uploading and browsing data, querying tables in Impala and Hive, running Spark and Pig jobs/workflows, and search.
• Hue increases ease of use, is 100% open-source, was created by Cloudera and released under the Apache license.
• Oozie is a workflow engine for Hadoop jobs, defining dependencies between jobs.
• The Oozie server submits the jobs to the server in the correct sequence.
• Sentry provides fine-grained access control (authorization) to Hadoop ecosystem components like Impala, Hive, Cloudera Search, and HDFS.
• When used with Kerberos authentication, Sentry authorization ensures a secure cluster.

Introduction to the Homework Labs

• Homework labs in this course are based on a hypothetical scenario to provide a practical learning experience to practice the skills learned in the course.
• Loudacre Mobile is a (fictional) fast-growing wireless carrier, providing mobile service to customers throughout western USA.
• Loudacre needs to migrate their current infrastructure to Hadoop due to the size and velocity of data exceeding their ability to process/analyze it.
• Loudacre's data sources include MySQL databases with customer account data (name, address, phone numbers, devices), and Apache web server logs, HTML/XML files, and real-time device status/base station data.
• Instructions for the homework labs can be found in the Homework Labs.

Homework Lab Virtual Machine

• Homework uses a virtual machine environment, with training as the user and password.
• The VMs are pre-installed with Spark and CDH (Cloudera's Distribution, including Apache Hadoop), and other tools like Firefox, gedit, Emacs, Eclipse, and Maven for the homework labs.
• Training materials can be found at the location: ~/training_materials/dev1, with exercise and example script files.
• Homework Lab course data is stored in the default location, ~/training_materials/data.

Conclusion

• Hadoop is a framework for distributed storage and processing, and core Hadoop includes HDFS for storage and YARN for cluster resource management.
• The Hadoop ecosystem includes many components for ingesting, storing, processing, and modeling data, and for exploration and protection.