Data Collection and Processing Quiz

Data Collection and Processing

• Data is gathered from various sources: web browsers, smartphones, search engines, the internet, banking transactions, and gaming activities.
• Data collection is conducted by government agencies, pharmaceutical companies, consumer product companies, large retailers (e.g., big box stores), and credit card companies.
• Big Data infrastructure encompasses the processes of collection, ingestion, preparation, computation, and presentation.
• A data source provides raw data for analysis, reporting, and other applications. Sources include structured and unstructured data: databases, APIs, flat files, cloud storage, data warehouses, spreadsheets.

Data Capture Methods

• Data capturing involves collecting and entering data from diverse sources into a system.
• Manual data entry: Humans input data into systems (spreadsheets, databases).
• Automated data collection: Sensors, IoT devices, APIs, surveys, forms, and point-of-sale systems.
• Mobile data capture: Utilization of mobile devices for on-site data gathering.
• Data capturing methods must adhere to accuracy, privacy, security, and integration standards.

Data Processing Steps

• Data processing encompasses collection, cleaning, transformation, storage, and analysis.

Data Presentation

• Data presentation effectively displays data for insightful understanding, showcasing clarity, engagement, context, interactivity, and storytelling.

Data Warehouses vs. Data Lakes

• Data warehouses consolidate, clean, and organize incoming data into a consistent schema, optimizing analysis.
• Data lakes store raw data in its original format, allowing flexible selection and organization.

Data Infrastructure Models

• On-premises: Total management by the user, encompassing networking, storage, servers, virtualization, operating systems, middleware, runtime, data, and applications (e.g., private data centers).
• SaaS (Software as a Service): Vendor-managed systems, where users access applications through the internet (e.g., Gmail, Microsoft Office 365).
• Serverless: Cloud service providers manage servers, leveraging services like AWS, Azure, and GCP.

Data Treatment: Data Lake Zones

• Transient zone: Data ingestion, tagging, and cataloging for addition to the data lake.
• Raw zone: Metadata application, protecting sensitive attributes, and data identification.
• Trusted zone: Data quality and validation assessments, ensuring accuracy.
• Refined zone: Data enrichment and automated workflow processes.

Data Treatment: Data Lake Steps

• Transient loading zone: Ingestion and storage from various sources (e.g., streaming, file data, relational data) without transformations.
• Raw data zone: Unaltered data storage, applying tokenization.
• Refined data zone: Data integration for uniformity.
• Trusted data zone: Provision of reliable, high-quality data.
• Discovery sandbox: Support for experimental analysis and exploration.

Data Consumers

• Consumer systems: Data Catalog, data preparation tools, data visualization, and external connectors.
• Business analytics researchers and data scientists.

Technology Overview

• Data collection technologies from applications and IoT devices.
• Third-party ingestion systems (e.g., MQTT).
• Data preparation and computation within the data lake.
• Data transfer to data warehouses.
• Data presentation.

Other relevant data sources and treatments

• Data sources include OLTP/ODS, Enterprise Data Warehouse, logs, cloud services, streaming, and file data.
• Data sources like streaming, file data, and relational data are pertinent.