Data Acquisition and Preparation

Data Acquisition

• Data is gathered and extracted from enterprise source systems, cloud-based applications, and external sources
• Data understanding requires knowledge of its origin, completeness, collection points, handling, processing and quality issues

Data Preparation

• It is defined as set of processes to gather data from diverse sources and transform it according to business and technical rules
• Data preparation stages it for conversion into useful information
• Reformatting converts data from multiple systems into a common format, needing schema and column definitions, also known as a data dictionary
• Consolidation standardizes data definitions
• Validation tests data against predefined business rules of querying
• Transformation converts data into business information and uses business rules, algorithms, filters, and creating associations
• Cleansing analyzes data for quality, consistency, and addresses issues
• The final step involves storing the processed data for further use

Data Preparation Challenges

• Project delays and cost overruns result from underestimating time and resources
• Variations in format, differing rules, and collection rates presents a challenge
• Operational defects, human errors, manual steps, and environmental instability introduce data quality issues
• Skewed information arises through biased transformation and aggregation involving defects, errors, and incorrect algorithms
• Incomplete understanding of source data leads to suboptimal data models and may cause missed relationships
• The failure to bring data from multiple sources to a consistent definition inhibits joining or comparing it

Data Cleansing Techniques

• Validity checks correct invalid formats and values like syntax errors, typos, and white space
• Relevance checks detect and remove irrelevant data that is corrupted, inaccurate
• Duplicate removal finds and resolves duplicate information, such as events recorded by two sources, the same events that is processed twice, or duplicate customer address
• Consistency checks identify contradictory or incompatible values
• Assess data quality using summary statistics to detect range, mean, distribution, unique values, and outliers during data profiling
• Visualization uses statistical methods to detect unexpected or irregular values
• Missing values can be addressed using one of three methods, by discarding, imputing, or flagging
• Data normalization rescales data values into a range from 0 to 1 for normally distributed data
• Standardizing names and addresses are examples of data cleansing
• Linking personal and business accounts of family members under a household grouping is considered an example of data cleansing

Data Wrangling

• Data wrangling a.k.a. data munging involves aggregation, summarization, and enrichment of data with the use of business intelligence (BI) tools
• BI tools selected influence data wrangling processes
• Data wrangling is an iterative process that can involve profiling transformations
• Data is gathered from different formats, filter the smaller part of the data set and make analytical subsets
• Data is examined and relationships are uncovered by profiling
• Transforming data from a source schema to the target BI tool schema for the target is a aspect of data wrangling
• Advanced cleaning may involve cleaning performed earlier during the data preparation phases
• Enriching data involves using business transformations, joining multiple datasets, and creating aggregations

Data Profiling Questions

• Questions for understanding data include:
  • What's in the data
  • What is its quality
  • Is it complete and unique
  • Are there problem records or anomalies
  • What is the data distribution
  • What are the ranges of values

Types of Data Profiling

• Individual values profiling involve checking the validity of individual record fields like data formats, value ranges, and semantic checks for contextual relevance
• Set-based profiling involves understanding value distributions and checking validity across multiple records, including analysis of numeric, categorical, geospatial, and date-time fields

Data Wrangling Transformations

• An example of data wrangling involves changing a dataset's structure through intra-record actions (changing the order of fields within a record, to combining fields into complex structures), and inter-record actions
• Granularity transformations involves actions like aggregations, and pivots shift records into fields

Data Wrangling Methods

• Cleansing missing values fixes irregularities in the dataset primarily by handling missing (or NULL) values through one of the following methods:
  • Discarding records, calculate using average of statistical methods of median value or keep and add a flag
• Cleansing invalid or inconsistent data involves correcting inconsistent data and overwriting the original values and marking values as invalid
• Duplicate record removal, reconciliation of record is an aspect of data wrangling

Dataset Structure Types

• Rectangular datasets are arranged in a basic database matrix.
• Jagged datasets vary in record length
• Heterogeneous datasets feature differing structures in a single dataset

Transformations and Data Enrichment

• Data is enriched with multiple datasets by joins, unions, and metadata enrichment from data-related information
• Computing new values derives new data for existing data and reduce the number of categories through categorization

Metadata Elements

• Data structure refers to the records and field encodings
• Granularity refers to the depth or number of entities represented by a data record.
• Accuracy refers to the quality, consistency of data
• Temporality refers the time sensitivity of the dataset and impact dataset quality
• Scope of data the represent attributes of a dataset

Data Temporality Questions

• Questions to ask about data temporality includes the followings:
  • When the data was collected
  • When records collected or measured
  • Are there any timestamps associated with the data
  • If records were modified
  • At what point did the data turn stale

Data Scope Questions

• Questions to ask about data scope includes the fallowing:
  • What features of records captured and accessible through name or position
  • Does dataset have missing records
  • Multiple records are the same or does the data require data deduplication
  • Heterogeneous record included

Data Publishing

• Data publishing involves storing refined datasets into the target analytics platform including the logic and scripts needed to generate the datasets

Predictive Modeling Process

• The predictive model process involves the following steps:
  • Exploring data for anomalies, and patterns
  • Formulating a hypothesis
  • Training machine learning models
  • Evaluating the model performance
  • Deploying the best performing model to production

Data Decay

• Data decay occurs when predictive model size, activity, quantity decreases gradually, declines from condition, or has fallen into ruins
• Decay occurs because relationships between behaviour is changing, new data becomes available, data becomes unavailable, or organizations objective change

Monitoring & Maintenance

• Monitoring model performance involves alerts for model decay and alerts for when effectiveness declines
• This step requires documentation, maintenance, and updated documentations and requires to tune and optimize

Analytics Maturity

• Business intelligence (BI) evolves through five levels:
  • Unaware: where spreadsheets dominate
  • Opportunistic: there are inconsistent data
  • Standards: Business executives became BI champions
  • Enterprise: where the technology is advanced to the point that research methods begin
  • Transformative: The company drives the company forward and industry transformation

Analytics Success Data

• In 2020, 80% of AI projects did not scale
• Only 20% of analytic insights will deliver business outcomes through 2022
• 87% data science projects will not be deployed as production

Analytics Maturity Factors

• Maturity in analytics relies on key factors: people, culture, organization, process, data quality, and technology
• Executive support, data governance drives maturity

Failure Reasons

• A lack of business value, over reliance of software leads of the failure of projects
• Expertise needs to be adequate, data must properly organized and analyzed, analytics properly incorporated