Data Science Principles: Data Preparation

Data Consistency and Quality

• Systematic errors can occur due to issues with data feeds from sensors, leading to invalid, incorrect, or missing data values.
• Surveys and visualization of data can help identify outliers, skewness, and inconsistencies.

Data Preparation Key Activities

• Leverage data visualization tools to gain an overview of the data and detect outliers/skewness.
• Review data to ensure calculations remained consistent within columns or across tables for a given data field.
• Assess the granularity of the data, the range of values, and the level of aggregation of the data.
• Check for consistency of data distribution over time.
• Examine the consistency of state or country abbreviations used in geospatial datasets.
• Check if data is standardized or normalized, and if units used are consistent (e.g., metric units).

Data Gap Analysis

• Compare available data with required datasets to identify gaps.
• Identify additional data sources that can be leveraged, such as social media data for sentiment analysis.

Data Conditioning

• Data transformation involves cleaning, normalizing, and performing transformations on the data.
• Data integration involves joining or merging different datasets.
• Feature selection involves deciding which aspects of datasets to analyze or discard.

ETL vs ELT Approach

• ETL (Extract, Transform, Load) approach: data is extracted, transformed, and then loaded into a centralized database.
• ELT (Extract, Load, Transform) approach: data is extracted, loaded into a centralized database, and then transformed.

Dataset Inventory

• A dataset inventory is a structured and organized record of all datasets available within an organization.
• It helps in identifying available data sources and gaps in data.