DynamoDB and Athena Quiz

Data Characteristics

• Structured data is organized in a defined manner or schema, found in relational databases. It's easily queryable and organized in rows and columns. Examples include database tables, CSV files, and Excel spreadsheets.
• Unstructured data lacks a predefined structure or schema, making it difficult to query without pre-processing. It comes in various formats, such as text files without a fixed format, video/audio files, images, emails, and word documents.
• Semi-structured data is not as organized as structured data but has some structure, using tags, hierarchies, or other patterns. It's more flexible than structured but less chaotic than unstructured. Examples include XML, JSON, email headers, and log files.
• Key properties of data include volume (amount of data), velocity (speed at which data is generated), and variety (different types and sources of data).

Data Repositories

• Data warehouses are centralized repositories designed for complex queries and analysis. Data is stored in structured format, cleaned, transformed, and loaded (ETL). Often uses star or snowflake schema. Optimized for read-heavy operations. Example: Amazon Redshift.
• Data lakes are repositories holding vast amounts of raw data in native, unpreprocessed format. They support batch, real-time, and stream processing. Example: Amazon S3.
• Data Lakehouses combine the best of Data Warehouses and Data Lakes. They support both structured and unstructured data.
• Data Mesh is an approach where individual teams own data products in a given domain with various uses.

Data Pipelines

• Extract raw data from source systems, ensuring data integrity.
• Transform data into suitable formats, handle missing values, and do transformations.
• Load data into the target data warehouse or repository, ensuring data integrity during loading.
• Data pipeline tasks are done automatically in a reliable manner, using services like AWS Glue.

Data Sources and Formats

• JDBC- Java Database Connectivity—Platform independent, language dependent.
• ODBC — Object Database Connectivity - Platform dependent, Language independent.
• Common data formats: CSV, JSON, Avro, Parquet. Each has its own use cases (e.g., CSV for small datasets, Avro for big data).

Data Modeling

• Star Schema: Fact tables and dimension tables which branch.
• Data Lineage: Visualization of data flow from source to destination.
• Schema Evolution: Adaptability of schema to changing requirements.

Data Validation and Profiling

• Check completeness (missing values), consistency (cross-field validation), accuracy (comparing with trusted sources), and integrity (validating against rules/standards)

DynamoDB

• A fully managed NoSQL database service for key-value and document storage.
• Provides fast and predictable performance with seamless scalability.
• Supports various data types (scalar, document, set).
• Offers low-cost and auto-scaling options.

Read/Write Capacity Modes

• Manage table capacity for read/write throughput.
• Data stored in partitions (copies of data on servers).
• WCUs and RCUs—Specify read/write units.
• Modes include provisioned (predefined read/write) and eventually consistent (less costly).

Basic API calls

• CRUD operations for data manipulation in DynamoDB.
• Conditional writes (e.g., PutItem).
• Reading data (e.g., GetItem, Scan).
• Deleting data (e.g. DeleteItem).
• Batch operations to optimize operations.

Indexes

• Local Secondary Indexes (LSIs) —Create alternative sort keys to the base table for data that can be more easily and quickly queried. Uses WCUs/RCUs of main table.
• Global Secondary Indexes (GSIs) - Alternative primary keys that speed up queries on non-key attributes.

DynamoDB Accelerator (DAX)

• A fully managed cache for DynamoDB.
• Provides high-speed access to frequently read data by using in-memory caches.

Stream records

• Data retention can last up to 24 hours.
• Options to choose what is written to stream (metadata, new image or older image of items).

Data Sources and Formats (continued)

• Avro, Parquet, and other data formats, each with attributes and use cases relevant to different data types.

Miscellaneous

• Data modeling (star, snowflake, etc.) relevant details, like primary and foreign keys.
• Data validation and profiling: Checking for completeness, consistency, accuracy and integrity.
• Data sampling techniques like random, stratified, and systematic sampling.

Relational Database Service (RDS) :

• Relational database service frequently used for smaller data volumes and managed backups, software patching
• Full ACID compliance, so data consistency and reliability are guaranteed
• Suitable for transactional apps and multi-user environments.

RDS Read Replicas

• A feature that replicates the primary to other locations within the same or different regions
• Optimized for scaling read-heavy workloads, which allows the main database to manage writing workload more efficiently and improves performance

Aurora

• A fully managed MySQL and PostgreSQL-compatible database service. ○ Offers high performance.
• Offers compatibility with various data types.

DocumentDB

• NoSQL database compatible with MongoDB.
• Good for storing JSON data.
• Offers automatic scaling, high availability, and replication.

MemoryDB for Redis

• In-memory database
• Optimized for fast read/write performance and used in Microservices architectures to minimize latency.

Keyspaces

• A managed Apache Cassandra-compatible database service.
• Allows you to define the structure of your data.

Neptune

• Fully managed graph database for complex and highly interconnected data
• Enables real time querying and analysis on graph data.

Timestream

• A time series database • Suitable for processing large amounts of time-stamped data.

Redshift

• A massively parallel processing (MPP) database service. • Optimized for high performance data warehousing use cases
• Ideal for supporting complex analytical work and data warehousing modernization.

Importing/Exporting Data

• COPY command to efficiently load large amounts of data into Redshift.
• Use UNLOAD command to unload data from a Redshift table.

Workload Management (WLM)

• Workload Management in AWS. • Allow prioritization of certain types of queries by placing them into queues to avoid overloading the system

Serverless

• Auto scaling for workloads, pay-only usage, and easier management.
• Suitable when workloads are unpredictable or variable - high activity sometimes, lower activity other times.

Data Sharing

• Enables secure sharing of data across Redshift clusters.

Other Tools/Features

• Query Performance Insights dashboards for understanding query performance.
• Redshift Advisor, to make suggestions about how to perform analytical queries more efficiently.
• Windows functions.

Materialized Views

• Pre-calculated results for efficient querying or retrieving pre-calculated result, useful for complex reports, can improve reporting performance

Miscellaneous (continued)

• Specific details on retry mechanisms/handling exceptions and issues or common tasks.

DataSync

• Transfers large datasets between on-premises and AWS storage.
• Useful for disaster recovery or for data lakes use cases.

Snow Family

• Highly secure and portable devices for moving large amounts of data.
• Use cases include data migration/data warehousing.

Edge Computing Services

• Process data near the edge instead of routing it to the cloud.
• Use cases include real-time analytics, media streaming, and machine learning at the Edge.

Transfer Family

• Fully managed service for file transfers in AWS. • Supports FTP, FTPS, and SFTP protocols.

EC2

• Virtual servers • Manual provisioning or auto-scaling.

Lambda

• Serverless compute service.
• Ideal for running code in response to events.
• Use cases: processing data from different sources, building microservices architecture and event-driven apps.

Integration

• Different services within AWS integrate well with each other.

SQS

• A secure, fully managed message queue service for asynchronous messaging that supports exactly once delivery for both messages and records.
• A good choice for decoupling applications and buffering high volumes of requests.

FIFO Queue

• In FIFO SQS is meant for message ordering - exactly once delivery with FIFO ordering.

SNS

• A fully managed pub-sub messaging service. ○ Good for asynchronous messaging (push-based instead of pull)

Step Functions

• A workflow orchestrator • Coordinate AWS services and can have steps for processes such as deployment or ETL pipelines.

AppFlow

• A fully managed integration service.
• Transfers data between different SaaS apps and AWS services.

EventBridge

• A managed event bus where data and messages are routed to designated apps/services.
• Useful for enabling event-driven architectures.

Amazon Managed Workflows for Apache Airflow (MWAA)

• Fully managed service to perform tasks based on Python-defined workflows (DAGs).
• Can scale and manage airflow tasks.

General

• Principles of least privilege and best practices

CloudTrail

• Logging and security service • For auditing, compliance, and analyzing operational events in AWS

CloudWatch

• Tracking and monitoring data from AWS services, applications and apps. • Ideal for performance monitoring, dashboards/notifications, and debugging

Cost Explorer, Budgets

• Provides high level analysis on costs. • Budgets can set limits that prevent spending from going above a certain level, with notifications when this happens

CloudFormation

• IaC service to manage and deploy AWS resources through templates

Secrets Manager

• For securely storing and managing sensitive data, such as passwords or API keys.
• Data is encrypted and kept securely.

WAF

• Security service to protect web applications.

Shield

• Protects against Distributed Denial of Service (DDOS) attacks.

Networking

• Essential concepts of VPCs, Subnets, Route tables, Network ACLs and Security groups are covered.
• Concepts of PrivateLink, Direct Connect (DX), Internet Gateway, NAT Gateway are covered for secure communication/networks.

Data

• Structured, Unstructured, Semi-structured data.

Key Management Service (KMS)

• Manage encryption keys for data stored in various AWS services

Key Types

• Symmetric/asymmetric keys

Macie

• Automates the discovery of sensitive data in AWS environment, mostly S3 data.

Additional Concepts

• AWS services like EFS, S3, Lambda, DynamoDB, Glue and their use cases