Big Data and Analytics Overview

Big Data

• Big data refers to datasets that are too large or complex for traditional databases to handle efficiently.
• Rise of big data is due to increased data generation from various sources, including sensors, devices, social media, and online transactions.

Big Data Analytics

• Big data analytics involves using advanced techniques to analyze large and diverse datasets, including structured, semi-structured, and unstructured data.
• These techniques include text analytics, machine learning, predictive analytics, data mining, statistics, and natural language processing.
• The analysis of big data enables better and faster decision-making by providing insights from previously inaccessible data.
• It also helps in risk management, product development, and improving customer experiences.

Structured, Unstructured, and Semi-structured Data

• Structured data is organized in a predefined format, like databases.
• Unstructured data does not have a predefined format, like text documents, images, and videos.
• Semi-structured data falls between structured and unstructured, with some degree of organization, like XML or JSON.

Parallel Processing and Distributed Storage

• Challenges of big data include single central storage, serial processing, one input/output, and difficulty handling unstructured data.
• Solutions include distributed storage, parallel processing, multiple inputs/processors, and the ability to process all data types.

Phases of Big Data Analytics

• Data Acquisition: Collection and gathering data from various sources.
• Data Preparation: Cleaning, transforming, and integrating data for analysis.
• Data Analysis: Applying analytical techniques to extract insights and patterns.
• Data Visualization: Presenting findings in a clear and understandable manner.
• Data Communication: Sharing insights and recommendations with stakeholders.

Types of Data Analytics

• Descriptive Analytics: Summarizes past data to understand what happened.
• Diagnostic Analytics: Explores why something happened to identify the root causes.
• Predictive Analytics: Forecasts future trends and outcomes.
• Prescriptive Analytics: Recommends actions to take based on predictions.

Hadoop

• An open-source framework for storing and processing large datasets in distributed clusters.
• Handles structured and unstructured data, providing flexibility for data management.
• Three main components: Hadoop Distributed File System (HDFS), Hadoop MapReduce, and Hadoop YARN.

Hadoop HDFS

• HDFS stores data in a distributed manner across multiple machines (data nodes).
• A master node (name node) manages the data nodes and stores metadata.
• Designed for handling large datasets efficiently on commodity hardware.
• Highly fault-tolerant, meaning data is replicated across multiple nodes to prevent data loss.

Hadoop MapReduce

• The processing component of Hadoop, which distributes data processing across multiple nodes.
• Utilizes a map phase to prepare data for analysis and a reduce phase to aggregate results.

Hadoop YARN

• Manages resources in Hadoop clusters, ensuring efficient job scheduling and allocation.
• Acts like an operating system for Hadoop, managing file systems and cluster resources.

Hadoop Ecosystem

• Includes HDFS, YARN, MapReduce, Spark, Pig, HIVE, HBase, Mahout, Spark MLLib, Solar, Lucene, Zookeeper, and Oozie.

Pig

• Provides a platform for structuring data flow, processing, and analyzing large datasets using a query language (Pig Latin).
• Runs on Pig Runtime, similar to Java running on JVM.
• Offers ease of programming and optimization.

HIVE

• A data warehouse software facilitating querying and managing large datasets in distributed storage.
• Allows defining schemas for data in HDFS.
• Supports various data formats, including XML, JSON, and compressed files.
• Uses SQL-like query language (HQL) for easy data manipulation.

Mahout

• Enables machine learning capabilities in Hadoop.
• Provides algorithms for tasks like collaborative filtering, clustering, and classification.
• Allows users to invoke specific algorithms through its own libraries.

Apache Spark

• A data processing engine that offers in-memory processing for faster performance.
• Handles batch processing, real-time processing, graph conversions, and visualization.
• Offers optimization compared to earlier technologies.

This quiz explores the essential concepts of big data, its significance, and the techniques utilized in big data analytics. It covers different types of data, including structured, semi-structured, and unstructured, and highlights the benefits of advanced analytics in decision-making processes.