Data Warehouse characteristics and design

• Data warehousing is the first topic

Data Lakes

• These are locations used to store all types of raw data, whether structured or unstructured

Data Warehouse (DW)

• A data warehouse involves is a organized and clean dataset
• Data warehouses help organizations in making informed decisions

Data Marts

• Data marts are subsets of data warehouses

• They are business-domain specific, such as HR, Operations, and Finance

• Heterogeneous information sources are a problem due to different interfaces, data representations, and potential for duplication and inconsistency

• Data management in large enterprises faces issues like vertical fragmentation of informational systems

Systems Used in Organizations: Operational vs. Informational

• Operational systems focus on day-to-day operations with current, real-time, high-speed transactional processing

• Informational systems are for decision-making and analysis, using historical, summarized data with complex queries

• Operational systems are typically used by operational staff, while analysts and decision-makers use informational systems

• Examples of operational systems include POS, ERP, and inventory systems, which require immediate response times and highly structured, detailed data

• Informational systems include BI tools and data warehouses, which are less time-sensitive and use aggregated, multidimensional data

• The goal is unified acess to data through collection, combination, integrated views, uniform user interfaces, and sharing capabilities

The Warehousing Approach

• Involves integrating information in advance and storing it for direct querying and analysis

Advantages of Warehousing

• High query performance
• Does not interfere with local processing at sources
• Enables complex queries at the warehouse
• Involves copying information to the warehouse
• Allows modification, annotation, summarization, and restructuring of data
• Provides security without auditing
• Stores historical information
• OLTP, or online transaction processing, occurs at information sources

Other Considerations

• A query-driven approach is suitable for rapidly changing information and sources, truly vast amounts of data from numerous sources, and clients with unpredictable data needs

• Barry Devlin from IBM says that a data warehouse is a complete store of data, obtained from a variety of sources, available to end-users in a business context

• Bill Inmon coined the term "data warehouse" in 1990

• Query-driven approaches directly collect answers from sources

• The data warehouse data is subject-oriented, integrated, time-variant, and non-volatile, supporting management's decision-making process

• A data warehouse is a repository containing cleaned, integrated, and reconciled data from various sources

Data Warehouse Characteristics

• Involves a stored collection of diverse data

• Solves data integration problems by acting as a single repository of information

• Is subject-oriented and organized by subject (not application)

• Intended for analysis and data mining

• Features a user interface designed for executives

• Involves large data volumes (GB, TB) that are non-volatile and historical

• Includes the importance of time attributes

• All transactions at places are included

• Warehouse queries are long and complex that contains data that is summarized, historical and reconciled that are mainly reads

• Standard databases focus on mostly updates, many small transactions, current snapshots and are raw

• Data warehousing market insights project a global market size of $51.18 billion by 2028

• Snowflake holds the largest market share in data warehousing with 3,174 domains

• Approximately 2.5 quintillion bytes of data are created daily

Types of Data

• Business data represents the meaning of collected data
• Real-time data is the ultimate source of all business data
• Reconciled data represents harmonized information
• Derived data is information created from existing datasets
• Metadata describes the meaning of the data

Metadata Types

• Build-time
• Control
• Usage

Data as a product

• Data has intrinsic meaning, it can also be produced for its own reasons
• An example is the content of a textbook

Data Warehouse Architectures

• Single-layer: every data element is stored once
• Two-layer: a structure composed of of real-time plus derived data
• Three-layer: transforming data from real-time to derived forms

Architecture Comparison

• Single-Tier: Small retail tracking with sales in excel

• Two-Tier: Marketing agency that analyzes campaign data through moderate analytics

• Three-Tier: Multinational retailer that analyzes global sales data with advanced analytics

• Data warehousing distinct issues contain topics on how to get information into a warehouse and what to do with the data once it is in the warehouse

Data Warehouse Issues

• Warehouse design
• Extraction
• Monitoring
• Integration
• Warehousing specification and Maintenance
• Optimizations

Data Extraction

• Deals with various source types like relational databases and flat files
• Tools include replication tools, dump files, created reports, and wrappers

First-Issue Considerations

• It must be determined if warehouse uses relational or multi-dimensional data model must be used to store data
• Source-type include legacy, relational, and hierarchical structured data
• Flat files can be semistructured

How to maintain the Data Warehouse

• A warehouse must be kept current with data source changes, and also detect updates in the sources.
• A wrapper converts data and queries from one data model to another and extends query capabilities for sources with limited capabilities

Wrapper Generation Types

• Hard coding for each source

• Automatic wrapper generation

• Source-specific adapters translates different sources of data by use of interfaces

• Cooperation between cooperative and uncooperative sources will effect the way data is translated

Data Management (Standard v Non Standard Interactions)

• Standard interactions has many tools and standard sources such as commercial DBMSs, or ODBC-compliant systems.
• The data is transferred through extraction or replication
• Non-standard interactions requires semi structured sources with only small schemas which requires the use of data transformations for translation

Data Transformation

• Format uniformity
• Byte ordering
• String termination
• Internal layout

Monitors

• Monitors act as detectors for changes of interest
• It propagates integration through triggers and comparing data

Data integration

• A process of receiving data (changes) from multiple wrappers/monitors
• It resolves inconsistencies, eliminates duplicates, and integrates with any existing data
• Fetch more data from sources

Data Cleansing

• Remove duplicate tuples and detect incorrect data

Topic 2: Digital Data Warehousing

• Digital technology generates, stores, and processes data with positive and non-positive states
• Digital can provide value by executing user experience or building foundational cababilities

Data

• Data is information translated for efficient movement or processing, especially into binary digital form

• Raw data is data in its more basic digital format

• Digital data is numeric codes (0,1) stored in computer systems and software

• Systems convert this to human-readable information

Digital Data Management

• Activities involves digital data collection, storing, and providing acess to information
• Goal: Provide secure user access
• Key components include optimized storage, security, backups, data retention policies, and tools for data transfer
• The organization of data provides easy acess to the information for data

Data Classification Features

• Homogeneity
• Clarity
• Stability

Data Types

• Discrete: Data with separate values
• Nature: Countable values
• Values: Fixed, specific values
• Precision: Finite amounts
• Statistics: Bar charts and Pie charts

Digital Data Management Classifications

• 80%: Unstructured Data
• 10%: Semi-structured Data
• 10%: Structured Data

How to Manage Data

• Unstructured: Data that doesnt conform with common formats
• Structured: Use graph-based data models and schemas for better readability
• Schema-less

Digital Data Platforms

• Digital data platforms helps retrieve large volumes of data
• It provides secure and efficient functions that can allocate memory and storage resources
• DMPs help increase data security and enhance system performances

DMP (Data-Management Platform) Benefits

• Brings multiple data resources into one platform to make data analyzation more cohesive
• Helps find specific audiances and customers

Examples of DMPs

• Google Marketing Platform
• Nielsen DMP
• Oracle BlueKai DMP

Managing Data

• Creates acess through all data tiers that store private data

• Acesses data through multiple clouds while providing high availability data

• Archives data with schedule retention

• A database model contains constraints to manage data acess

• Defines a structure that will support the analytical needs for analyzation

Data Model Types

• Relational
• Hierarchical
• Network
• Object-oriented database
• Object-relational
• Hierarchical
• Network

Relational Model

• Data model sorts data into tables or relation which contains columns and rows

Hierarchical Model

• Organizes data into tree like data where each record is linked to a root

Network Data Model

• Builds linked records onto sets based on mathamatical set theory

Object Model

• Object-oriented model contains reasuable software elements with their features
• Its also known as a multimedia database

Object Relational Model

• Combines the simpliticy of advance functionalities from common models with a table structure that can have different language configurations

Explore data warehouse characteristics, design principles, and key functionalities. Understand the advantages and disadvantages of using data warehouses. Learn about components of digital data management systems.