Big Data Analytics and Warehousing

Introduction to Big Data and Data Warehousing

• Big data refers to large and complex data sets that cannot be processed by traditional data processing software and databases.
• Big data can be structured, semi-structured, or non-structured.
• Various operations like analysis, manipulation, and changes are performed on big data, and then it is used by companies for intelligent decision making.

Data Warehousing

• A data warehouse is a collection of data from various heterogeneous sources.
• It is the main component of the business intelligence system where analysis and management of data are done to improve decision making.
• It involves the process of extraction, loading, and transformation for providing data for analysis.

Big Data vs Data Warehouse

• Big data refers to large and complex data sets, while a data warehouse is a collection of data from various sources.

Characteristics of Big Data

• Volume: Refers to the huge set of data, which is complex to process further for extracting valuable information.
• Velocity: Refers to the speed at which companies receive, store, and manage data.
• Variety: Refers to the diversity and range of different data types, including unstructured data, semi-structured data, and raw data.
• Veracity: Refers to the accuracy, meaningfulness, and confirmation of true data.
• Value: Refers to the potential value of big data, which comes from insight discovery and pattern recognition that lead to more effective operations, stronger customer relationships, and other clear and quantifiable business benefits.

Types of Data

• Structured Data: Data that is in the format of a relational database and is structured properly in rows and columns.
• Unstructured Data: Data that includes various types of data, such as audio, video, XML files, and does not have a proper format.
• Semi-structured Data: Data that is partially structured and mixed with unstructured data.

Data Warehouse Architecture and Design

• Top-Down Approach: A data warehouse architecture that involves storing data in a central repository and then creating data marts.
• Bottom-Up Approach: A data warehouse architecture that involves creating data marts and then integrating them into a data warehouse.

Data Warehouse Components

• External Sources: Sources from where data is collected, including structured, semi-structured, and unstructured data.
• Stage Area: Where data is extracted, transformed, and loaded into a data warehouse.
• Data Warehouse: A central repository that stores meta data and actual data.
• Data Marts: Store information of a particular function of an organization, which is handled by a single authority.
• Data Mining: The practice of analyzing big data present in a data warehouse.

Big Data Technologies

• Hadoop Ecosystem: A platform that provides various services to solve big data problems, including Apache projects and commercial tools and solutions.
• Apache Spark: An open-source analytics engine used for big data workloads, which can handle both batches and real-time analytics and data processing workloads.
• NoSQL: A database management approach that can accommodate a wide variety of data models, including key-value, document, columnar, and graph formats.