Business Intelligence and Data Mining

Business Intelligence (BI)

• BI consists of techniques and tools transforming raw data into meaningful info for business analysis
• BI includes applications and technologies for data gathering, analysis, and access to improve business decisions

Data Mining

• Data Mining explores large data quantities using statistical, technical, and business knowledge
• Exploratory Data Analysis (EDA) analyzes data to formulate testable hypotheses, complementing conventional statistical tools
• Data Science has fundamental principles to guide knowledge extraction from data
• Data mining extracts knowledge from data using technologies adhering to Data Science principles

Key Metrics

• Churn Rate = Subscribers Lost / Total Subscribers
• Duration = 1 / Churn Rate
• ARPU (Average Revenue Per User) = (Total Subscribers * Price Per Subscriber) / Total Subscribers
• Total ARPU = (Total Subscribers * Price Per Subscriber) Per Plan / Total Subscribers across all Plans
• Monthly Recurring Revenue = Total Subscribers * Price Per Subscriber
• Annual Run Rate = Monthly Recurring Revenue * 12
• Net Customer Lifetime Value (CLV) = (ARPU * Profit Margin * Average Duration) - Subscriber Acquisition Cost (SAC)
• Net CLV Ratio = Net CLV / SAC

OLAP vs. OLTP

• Online Analytical Processing (OLAP) is for Analytics
• OLAP uses queries aggregating large amounts of detailed data to find overall trends
• Online Transaction Processing (OLTP) is for Transactions

Business Intelligence as an Improvement Tool

• Business Intelligence improves how a business functions, approaches, and uses data
• It entails integrating information streams into an enterprise-wide data set
• BI uses modeling, statistical analysis, and data mining
• BI provides historical, current, and predictive views of the business
• The goal is data-driven decision-making

CRISP-DM

• The Cross-Industry Standard Process for Data Mining (CRISP-DM) provides a framework to structure data analytics problems
• CRISP-DM is a structured method for planning and executing data mining or data science projects
• Business Understanding focuses on understanding the objectives and requirements of a project from a business perspective.
• Data Understanding involves collecting, describing, exploring, and verifying the quality of the data.
• Data Preparation involves selecting, cleaning, constructing, integrating, and formatting data.
• Data Modeling entails selecting modeling techniques, generating test designs, building models, and assessing models.

Data Strategy

• Data Strategy aligns an organization around creating value from data, improving decision-making, and increasing efficiency
• Key elements include understanding data needs based on business strategy, data acquisition, management, reliability, and utilization

Major Influences on Data Strategy

• Data Governance and Stewardship involves authority, control, and shared decision-making over data asset management
• Data Architecture involves designing blueprints to meet enterprise data needs, guiding integration, and aligning investments
• Data Modeling and Design involves discovering, analyzing, and representing data requirements through iterative models
• Data Storage and Operations includes designing, implementing, and supporting stored data to maximize its value
• Data Security requires executing security policies for authentication, authorization, access control, and auditing
• Data Integration and Interoperability focuses on moving and consolidating data within and between systems to ensure communication
• Document and Content Management requires controlling the capture, storage, access, and use of data outside relational databases
• Reference and Master Data includes managing reconciled and integrated data for enterprise-wide sharing
• Data Warehousing and Business Intelligence plans and manages integrated data systems for reporting, querying, and analysis
• Metadata Management refers to activities to process, maintain, integrate, secure, audit, and govern data
• Data Quality Management requires planning and controlling activities to ensure data is fit for use

Phases of a Data Project

• Phase 1: Business Understanding; determine the client's objectives and assess the current situation
• Phase 2: Data Understanding; collect initial data, describe data, explore it, and verify its quality
• Phase 3: Data Preparation; select data, clean it, construct new data, integrate, and transform values

Data Modeling and Analytical Maturity

• Phase 4: Data Modeling; select a technique, generate a test design, build a model, and assess the model
• Analytical Maturity includes Descriptive, Diagnostic, Predictive, and Prescriptive analysis

Data Operations and Lifecycle

• Agile Development means to be iterative, flexible, and rationalized
• DevOps uses infrastructure as code for automatic provisioning
• Lean Development applies statistical process control and real-time measurements
• Data Lifecycle concerns various stages and processes applied to data as it moves through its lifecycle

Data Warehousing

• Multidimensional Data Models are for data analysis rather than online transactions
• Key warehouse components include Cubes (hypercubes), Dimensions (context for grouping), and Facts (business measures)
• A Star Schema has a fact table surrounded by dimension tables
• Snowflake Schemas refine star schemas by normalizing dimensions
• Fact Constellations have multiple fact tables sharing common dimensions

Properties and ETL

• A Data Warehouse is Subject-oriented, Integrated, Time-variant, and Non-volatile
• ETL (Extract-Transform-Load) extracts data transforms it, then loads it into the warehouse.
• The Dimensional Design Process involves selecting a business process, declaring granularity, choosing dimensions, and identifying facts

Data Warehouse Rationale

• A Data Warehouse requires a separate database due to performance needs for both transactional and analytical systems
• Data Warehouses contain historical data, consolidate data from different sources, and reconcile inconsistent data representations

Typical OLAP operations

• Operations include roll-up (summarize), drill-down (reverse of roll-up), slice and dice (project and select), and pivot (rotate)
• Other operations include drill-across (multiple fact tables) and drill-through (bottom level to relational tables with SQL)

Data Pipelines and Latency

• ETL transforms data before loading, while ELT loads data and then transforms it
• Latency is the delay between data creation and its availability for analysis

Types of Pipelines

• Batch Processing moves data in scheduled chunks resulting in high latency
• Real-Time/Streaming processes and delivers data immediately leading to low latency
• Lower latency leads to faster decision-making but increases complexity and cost
• Pipelines enable ETL, a critical step in integrating data into a warehouse
• Pipelines integrate data from multiple sources and ensure time variance by storing historical context

Data Integration and Governance

• Data Integration requires that data is moved and consolidated across systems
• Interoperability ensures that pipelines can communicate with diverse systems
• Pipelines should maximize data value, use dimensional design, and ensure data quality through cleaning and validation
• Scalability is achieved by architecting pipelines to handle growing data volumes

Data Governance and Stewardship

• Pipelines call for applying data security policies and metadata management for auditing
• Data pipelines support business intelligence by delivering clean, integrated data to dashboards and analytics tools
• They enable analytical maturity and align with data strategy by ensuring reliable data flows for decision-making
• Data governance is the exercise of authority, control, and shared decision-making over data assets

Data Governance Objectives

• Data governance manages data quality, security, and availability throughout its lifecycle
• It establishes frameworks to maintain high-quality, reliable data
• Data Security requires protecting data from unauthorized access and ensuring compliance
• Data Availability ensures that data is accessible when needed for decision-making
• Compliance adheres to internal policies and legal requirements

Core Components

• Core Components include establishing policies and procedures for data handling
• Data Stewardship implements data governance policies managing integrity and quality
• Infrastructure and Technology establishes systems and tools to support data governance
• Cross-Organizational Management facilitates access to relevant data across different business units while ensuring compliance
• Data governance plays a pivotal role in BI ensuring that data is accurate, secure, and aligned with organizational policies enhancing reliability

Data Stewardship

• Data stewardship implements governance practices by overseeing the management of data assets
• Responsibilities include managing quality by ensuring data meets standards for accuracy and consistency
• Data Usage is also monitored across the organization to guarantee compliance with governance policies
• Facilitating Data Democratization means to let users access relevant data while maintaining security
• Data stewardship focuses on executing data governance policies effectively within the organization

Data Governance Benefits

• Data Governance results in improved decision-making, high-quality data leads to better strategic decisions at all organizational levels
• Regulatory Compliance helps organizations adhere to data privacy laws and regulations
• Enhanced Trust in Data establishes a culture of accountability, fostering trust among stakeholders

Other Key Areas

• Other Key Areas include data literacy, analytical maturity, data life cycle, data privacy, exploratory data analysis, visualization, and interpretation

Data Storage

• An Operational Data Store (ODS) is an integrated database of operational data containing current or near-term data
• A Data Mart provides data for prepared analysis
• BI architecture includes a core component Data Warehouse that has a central repository for integrated enterprise data
• ETL extracts data from sources, transforms it, and loads it into the warehouse
• Multidimensional Models have a cube structure with dimensions and facts

Data Areas

• Data Warehousing includes a large repository of integrated data for specific data analysis
• Online Analytical Processing aggregates large amounts of detailed data to find trends
• Data Mining (semi-)discovers unknown knowledge in large databases

Schema Types

• Star Schema has a central fact table linked to denormalized dimension tables optimized for querying
• Snowflake Schemas have normalized dimensions to reduce redundancy
• Fact Constellations have multiple fact tables sharing conformed dimensions

Data Warehouse Characteristics

• Data warehousing is subject-oriented, integrated, time-variant, and non-volatile to support management's decision-making
• Design involves selecting a business process, declaring grain, choosing dimensions, and identifying facts
• Operations include Roll-Up, Drill-Down, Slice & Dice, and Pivot
• Data warehouses enable historical, integrated analysis, and OLAP operations allow users to explore data at varying granularities

Data Handling Steps

• Data Generation creates new data from various sources
• Data Collection gathers data from identified sources
• Data Processing transforms collected data into a usable format
• Data Storage stores data appropriately for future use

Key Data Considerations

• Key Considerations include Data Sources, Data Format, and Data Volume/Velocity
• Each stage; data should be handled with data technology & operations support and data governance in mind
• Each phase requires legislative, judicial, & executive functions
• The overall goal: Ensure data is reliable, secure, compliant, and readily available for analysis

Analyzing Data

• Analysis: Examine processed data to glean insights, patterns, and trends.
• Business Understanding (Phase 1): Determine Data Science Goal and Produce Project Plan
• Exploratory Data Analysis: Summarize key data and uncover hidden patterns in the data
• Data Modelling: Select Modelling Technique, Generate Test Design, Build Model, Assess Model, and Choose an Algorithm

Visualizing Data

• Visualization: Represent data and analysis results in a visual format
• Your Viz must always tell the audience something!
• Understand the audience and their needs is crucial.
• Data Exploration (Phase 2.3): Visualization is used during data exploration to identify patterns and relationships.

Interpreting Data

• Interpretation: Deriving meaning from the results and visualization and translating them into actionable insights.
• Aspects: Drawing Conclusions, Make Recommendations, and Use Communication Effectively
• Data strategy includes the approach to help create the necessary alignment across the org
• Data strategy is needed to create more value from data and goes beyond simply collecting data
• Key goals: Improved decision making, increased efficiency, and value creation

Key Data Decisions

• Data Acquisition: How will the organization obtain the necessary data?
• Data Management: How will the data be stored and maintained over time?
• Data Reliability: Ensuring the data is accurate, consistent, and trustworthy is crucial for making sound decisions.
• Data Utilization: How will the data be used for analysis, reporting, and other business intelligence purposes?

Data Stewardship Functions

• Understanding that data modeling is discovering and analyzing data requirements
• Data modeling will represent those in a precise form called a data model
• Data Warehousing: Designed for data analysis, not online transaction processing (OLTP).
• Data analysis includes the exploration using various dimensions, contexts, metrics and relational databases.

Database Operations

• Data operations are used to describe key aspects for relational databases.
• These databases offer integration, time-variants, versions, consistencies, and are often non-volatile.
• Dimensions are often the main way data is stored across these databases.