Hortonworks Data Platform (HDP)

• The presentation provides an introduction to Hortonworks Data Platform (HDP), focusing on data science foundations.
• Copyright IBM Corporation 2018.

Unit Objectives

• Describe the functions and features of HDP.
• List the IBM value-add components.
• Explain what IBM Watson Studio is.
• Briefly describe the purpose of each of the value-add components.

Hortonworks Data Platform (HDP)

• HDP is a platform designed for data-at-rest.
• It is a secure, enterprise-ready open-source Apache Hadoop distribution.
• HDP is built on a centralized architecture using YARN (Yet Another Resource Negotiator).
• HDP is open, central, interoperable, and enterprise-ready.

Data Workflow

• Sqoop is a tool designed for importing data from structured databases (like Db2, MySQL, Netezza, Oracle) and related Hadoop systems (Hive, HBase) into a Hadoop cluster.
• Sqoop is also capable of extracting data from Hadoop and exporting it to relational databases and enterprise data warehouses.
• Sqoop aids in offloading tasks such as ETL from an Enterprise Data Warehouse to Hadoop, offering lower costs and more efficient execution.
• Apache Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of streaming event data
• Flume helps aggregate data from various sources, manipulate this data, and then introduce it into a Hadoop environment.
• Flume functionality has now been superseded by HDF (Hortonworks Data Flow) and Apache Nifi.
• Apache Kafka enables building real-time data pipelines and streaming applications.
• Kafka is utilized in place of traditional message brokers like JMS and AMQP due to its higher throughput, reliability, and replication capabilities.
• Kafka can integrate with Hadoop tools such as Apache Storm, HBase, and Spark

Data Access

• Apache Hive is a data warehouse system built on top of Hadoop.
• Hive facilitates easy data summarization, ad-hoc queries, and the analysis of extensive datasets stored in Hadoop.
• Hive provides SQL capabilities on Hadoop.
• Hive offers a SQL interface, known as HiveQL or HQL, that enables easy querying of data
• It includes HCatalog, a global metadata management layer that exposes Hive table metadata to other Hadoop applications.
• Apache Pig is a platform for analyzing large data sets.
• Pig was designed for scripting a long series of data operations, making it suitable for ETL processes.
• Pig consists of a high-level language called Pig Latin, which was designed to simplify MapReduce programming.
• Pig’s infrastructure layer uses a compiler to produce MapReduce programs from Pig Latin code.
• The system optimizes and "translates" code into MapReduce, focusing on semantics rather than efficiency.
• Apache HBase is a distributed, scalable, big data store.
• Apache HBase is ideal when random, real-time read/write access to your Big Data is needed.
• HBase is designed to handle very large tables of data on clusters of commodity hardware.
• Modeled after Google's BigTable, HBase provides BigTable-like capabilities on top of Hadoop and HDFS.
• HBase is classified as a NoSQL datastore.
• Apache Accumulo is a sorted, distributed key/value store.
• Apache Accumulo provides robust, scalable data storage and retrieval.
• Accumulo is based on Google's BigTable and runs on YARN.
• Think of it as a "highly secure HBase."
• Apache Accumulo features server-side programming, scalability, cell-based access control, and stability.
• Apache Phoenix enables OLTP and operational analytics in Hadoop.
• Phoenix is ideal for low latency applications by combining SQL and JDBC APIs with ACID transaction capabilities, and flexibility using HBase as its backing store.
• Phoenix is essentially SQL for NoSQL.
• Phoenix is fully integrated with other Hadoop products like Spark, Hive, Pig, Flume, and MapReduce.
• Apache Storm: is an open-source distributed real-time computation system.
• Apache Storm is Fast, Scalable, and Fault-tolerant.
• Apache Storm is used to process large volumes of high-velocity data.
• Apache Storm is useful when milliseconds of latency matter, and Spark isn't fast enough.
• Apache Storm has been benchmarked at over a million tuples processed per second per node.
• Apache Solr is a fast, open-source enterprise search platform.
• Apache Solr is built on the Apache Lucene Java search library.
• Apache Solr enables full-text indexing and search.
• Apache Solr uses REST-like HTTP/XML and JSON APIs to make it easy to use with various programming languages.
• Apache Spark is a fast and general engine for large-scale data processing.
• Spark runs programs faster than MapReduce in memory.
• With Spark, applications can be written quickly with Java, Scala, Python, and R.
• Spark can combine SQL, streaming, and complex analytics.
• Spark runs on various environments and can access diverse data sources like Hadoop, Mesos, standalone, cloud, HDFS, Cassandra, HBase, S3, etc.
• Apache Druid is a high-performance, column-oriented, distributed data store for Interactive sub-second queries.
• Apache Druid uses a Unique architecture that enables rapid multi-dimensional filtering, ad-hoc attribute groupings, and extremely fast aggregations
• Apache Druid handles Real-time streams

Data Lifecycle and Governance

• Apache Falcon is a framework for managing the data lifecycle in Hadoop clusters, providing a data governance engine.
• Falcon defines, schedules, and monitors data management policies.
• Hadoop admins can use Falcon to centrally define the pipelines.
• Falcon uses these definitions to auto-generate workflows in Oozie.
• Falcon helps address enterprise challenges related to Hadoop data replication, business continuity, and lineage tracing by deploying a framework for data management and processing.
• Apache Atlas is a scalable and extensible set of core foundational governance services, that enables enterprises to meet the compliance requirements of Hadoop.
• Apache Atlas is designed to exchange metadata with other tools and processes within and outside of Hadoop.
• Offers integration with the whole enterprise data ecosystem.
• Atlas has the following features: Data Classification, Centralized Auditing, Centralized Lineage, Security & Policy Engine.

Security

• Apache Ranger is a centralized security framework.
• Apache Ranger is designed to enable, monitor, and manage comprehensive data security across the Hadoop platform.
• Apache Ranger manages fine-grained access control over Hadoop data access components such as Apache Hive and Apache HBase.
• Using the Ranger console, manage policies for access to files, folders, databases, tables, or columns with ease.
• Policies can be set for individual users or groups.
• Policies are enforced within Hadoop.
• Apache Knox offers a REST API and Application Gateway for the Apache Hadoop Ecosystem and provides perimeter security for Hadoop clusters.
• It serves as a single access point for all REST interactions with Apache Hadoop clusters.
• Knox integrates with prevalent SSO and identity management systems.
• Knox simplifies Hadoop security for users who access cluster data and execute jobs.

Operations

• Apache Ambari provisions, manages, and monitors Apache Hadoop clusters.
• Apache Ambari provides access to its Restful APIs
• Apache Ambari supports application developers and system integrators to easily integrate Hadoop provisioning, management, and monitoring capabilities to their own applications.
• Cloudbreak provisions and manages Apache Hadoop clusters in the cloud.
• It helps automate launching of elastic Hadoop clusters.
• This tool is policy-based and autoscaling on major cloud infrastructure platforms, including: Microsoft Azure, Amazon Web Services, Google Cloud Platform, OpenStack, Platforms that support Docker container.
• Apache ZooKeeper is a centralized service.
• Apache ZooKeeper maintains configuration information, naming, providing distributed synchronization, and providing group services.
• Services are used in some form or another by distributed applications & saves time so you don't have to develop your own.
• ZooKeeper is fast, reliable, simple, and ordered.
• Distributed applications can use ZooKeeper to store and mediate updates to important configuration information.
• Apache Oozie manages Apache Hadoop jobs, java based workflow scheduler system .
• Oozie Workflow jobs are Directed Acyclical Graphs (DAGs) of actions.
• Its integrated with the Hadoop stack (YARN)
• Supports Hadoop jobs for MapReduce, Pig, Hive, and Sqoop.

Tools

• Apache Zeppelin is a Web-based notebook.
• Zeppelin enables data-driven, interactive data analytics and collaborative documents.
• Documents can contain SparkSQL, SQL, Scala, Python, JDBC connection, and much more.
• Zeppelin is easy for both end-users and data scientists to work with.
• Notebooks combine code samples, source data, descriptive markup, result sets, and rich visualizations in one place.
• Ambari web interface includes a built-in set of views that are pre-built for convenience, that increase ease-of-use.
• It includes views for Hive, Pig, Tez, Capacity Scheduler, File, and HDFS.
• The Ambari Views Framework allow developers to create new user interface components that plug into the Ambari Web UI

IBM Value-Add Components

• Big SQL is SQL on Hadoop.
• Big Replicate provides active-active data replication for Hadoop across supported environments and hybrid deployments.
• BigQuality and BigIntegrate is Information Server can now be used with Hadoop.
• IBM InfoSphere Big Match is a Probabilistic Matching Engine (PME) running natively within Hadoop for Customer Data Matching

Big SQL

• Big SQL builds on the Apache Hive foundation and integrates with the Hive metastore.
• Instead of MapReduce, it uses: a powerful native C/C++ MPP engine
• It offers a view on your data residing in the Hadoop FileSystem.
• It supports Modern SQL:2011 capabilities.
• The same SQL can be used on your warehouse data with little or no modifications.

Big Replicate

• Provides active-active data replication for Hadoop and replicates data automatically with guaranteed consistency.
• It provides SDK to extend Big Replicate replication to virtually any data source.
• Patented distributed coordination engine enables Guaranteed data consistency across any number of sites at any distance and minimized RTO/RPO
• It is totally non-invasive: No modification to source code & Easy to turn on/off
• Provides data integration features of Information Server (BigIntegrate) and BigQuality which provides data quality features of Information Server.
• It is used with Hadoop, enable understanding, cleansing, monitoring, transforming, and enabling data delivery

IBM InfoSphere Big Match for Hadoop

• This is a Probabilistic Matching Engine (PME).
• It runs natively within Hadoop for Customer Data Matching

Watson Studio

• Watson Studio (formerly Data Science Experience (DSX)) is a collaborative platform for data scientists.
• It is built on open-source components and IBM added value.
• Watson Studio is available in the cloud or on-premises.