EMR Cluster Concepts Quiz

Data Characteristics

• Structured data is organized in a defined manner, schema, rows, and columns. Found in relational databases, CSV files, and Excel spreadsheets. Easily queryable.
• Unstructured data has no predefined format or schema. Examples include text files without fixed formats, video/audio files, images, emails, and Word docs. Not easily queryable without preprocessing.
• Semi-structured data is more organized than unstructured but lacks the rigid structure of relational data. Examples include XML, JSON, email headers, or log files with varied formats.

Properties of Data

• Volume: Amount/size of data generated, collected, and processed.
• Velocity: Speed at which new data is generated, collected, and processed. High velocity data requires real-time processing capabilities.
• Variety: Different types, structures, and sources of data.

Data Repositories

• Data warehouses are centralized repositories optimized for complex queries and analysis. Data is structured, cleaned, transformed, and loaded (ETL). Optimized for read-heavy operations.
• Data lakes are storage repositories holding vast amounts of raw data in native formats. No data preprocessing, supports batch, real-time, and stream processing. Can be queried for data transformations or exploration.
• Data lakehouses combine the features of data warehouses and data lakes. They support both structured and unstructured data.

Data Pipelines

• Extract: Retrieves raw data from source systems, ensuring data integrity during extraction. Can be done in real-time or batches.
• Transform: Converts data into a suitable format for the target. Could include data cleaning, enrichment, format changes, and handling missing values.
• Load: Moves data into the target data warehouse or repository, ensuring data integrity during loading.

Data Sources and Formats

• JDBC (Java Database Connectivity): A platform-independent way to connect java apps to databases.
• ODBC (Open Database Connectivity): A platform-independent API for connecting apps to databases using SQL.
• CSV (Comma Separated Values): A simple data format for storage of small to medium datasets.
• JSON (JavaScript Object Notation): A human-readable data format used often in data interchange between web servers and web clients.
• Avro: A binary file format (used in big data/real-time processing applications) including schema and data
• Parquet: Columnar storage format for large datasets - good for analyzing specific columns

Data Modeling

• Star Schema: A simple schema with a central fact table and connected dimension tables. Good for reporting and dashboards.
• Snowflake Schema: A more complex version of a star schema with further normalized dimension tables. Better for large datasets and complex queries.
• Flat Schema: A single table for all data, no normalization. Great for basic apps where data is small & relationships are minimal.
• Relational Schema: Multiple normalized tables related via foreign keys. Good for transactional apps.

Data Validation & Profiling

• Completeness: Checking for missing/null values.
• Consistency: Validation across data sources for possible differences.
• Accuracy: Comparison with trusted sources & known rules, sanity checks.
• Integrity: Validating the data integrity with relationship validations.

DynamoDB

• A NoSQL database service offering fast & predictable performance with seamless scalability, fault tolerance, and high availability.
• Fully managed, highly available with replication across multiple AZs.
• Scales horizontally to massive workloads.
• Millions of requests per second, trillions of rows, and 100s of TB of storage.
• Use for fast & consistent data retrieval, event-driven applications.
• Contains tables with primary keys that must be decided at creation.
• Data is stored in tables in rows. Partition & sort keys are used to efficiently organize data.
• Choose the right type of table (standard vs. provisioned) for your workload.

Data Pipelines

• Data pipelines automate data transfer and transformations between data stores.
• Extract, transform, load (ETL) processes are crucial for data quality and reliability.
• The process needs to be automated for efficiency.

#Data Validation and Analysis

• Identify discrepancies and enforce data integrity across different data sources and periods.
• Check for null or missing values, and inconsistencies in data entries.

Other concepts

• Data Mesh: A federated governance structure for data products within a domain.

Miscellaneous

• Schema Evolution: Adapt and change the schema of datasets over time while maintaining backward compatibility.

Data Pipelines

• Extract, transform, load (ETL): Data extraction, transformation and loading within data pipelines.

Additional Services/Concepts

• DataSync: A managed service for moving large amounts of data to and from on-premises storage & AWS services. Good for batch or scheduled data transfer.
• Snow Family: Highly secure, portable devices that let use collect & process data at the edge for transfer to AWS.
• App Flow: Enables secure data transfer between cloud & SaaS apps & can be scheduled or run on demand.
• AWS Backup: Centrally manages and automates backups across many AWS services.
• Application Discovery: Provides data about on-premise servers to plan easier migration to AWS.
• Application Migration: Converts physical/virtual/cloud servers to run on AWS.
• DynamoDB Accelerator (DAX): A fully managed, high-performance accelerator for DynamoDB allowing microsecond read latency, compatible with existing DynamoDB APIs.
• Timestream: An AWS database service for time series data designed for high throughput and scalable querying.
• Redshift: A fully managed, petabyte-scale data warehouse with columnar storage, optimized for analytical workloads.
• CloudFront: A content delivery network used for faster and more efficient data delivery across the world.
• CloudWatch: A monitoring service for collecting and tracking metrics, logs and events regarding resources and applications.
• CloudTrail: A service that logs all API calls and all activity that can be archived for security analysis.
• Kinesis: Records are received & can be continuously read by consumers
• Kinesis Data Analytics: Provides a managed compute service for running real-time analytics on Kinesis Data Streams.
• Glue: Enables the extraction of schemas, building ETL pipelines, and managing data lakes.
• Glue Catalog: Centralized metadata repository for storing data schemas or metadata.
• Glue Studio: A visual interface for designing, building, and running ETL workflows with Glue services.
• Glue DataBrew: A visual interface for cleaning and preparing large datasets using transformations.
• Glue Workflows: Allow you to design and automate ETL in a visual interface using DAGs.
• Athena: A serverless query service that can read data from S3 directly using Structured Query Language.
• EMR: A fully managed Hadoop framework offering flexible deployments and scaling for large-scale data processing workloads. Supports batch & real-time data processing.
• EFS: A file system service that stores data on objects without requiring management of compute resources.
• EKS: A managed Kubernetes service that provides the flexibility and scalability needed by containerized applications. Highly available & scalable.
• MSK: A fully managed Apache Kafka service for processing streaming data.
• Secrets Manager: A managed service for storing sensitive information like API keys, passwords, and databases credentials.
• WAF: Protects web applications from common web exploits.
• Shield: A managed service that protects your applications from denial-of-service (DoS) attacks.