Caching Strategies in Software Applications

A way to run SQL queries directly on data stored in S3 without loading it.

Formats that can be queried directly including CSV, JSON, ORC, Parquet, and Avro.

Pay as you go model; only successful queries are charged, failed ones are not.

Methods to manage access to S3 data including IAM, ACLs, and bucket policies.

Situations where Athena should not be used, such as for high-format reports or ETL processes.

A unique identifier used in database tables, must be defined at creation.

The HASH part of the primary key, must be unique and diverse for data distribution.

The RANGE part of the primary key, used in combination with the partition key to maintain uniqueness.

Measurement of throughput for reads; varies by item size and consistency type.

Measurement of throughput for writes; calculated based on item size.

Ensures correct data is read immediately after a write; uses more RCUs.

Allows for stale data; default read type that is quicker but less reliable.

User-specified capacity for read/write operations; requires planning ahead.

Prevents applications from exceeding their capacity limits to maintain stability.

Temporary exceedance of throughput limits, useful during traffic spikes.

Data is written to the cache whenever it is updated in the database, ensuring synchronization.

Minimizes cache misses and offers consistent performance.

Writes are slower and may cache rarely accessed data.

Manages the cache explicitly, checking it before fetching data from the database.

Flexible and allows custom logic for cache population and eviction.

Requires app-level logic to manage the cache and risks cache misses.

Cache retrieves data from the database for the app when it's not present in cache.

Data is written to the cache first, then asynchronously to the database later.

A fully managed service for near real-time data streaming to various destinations.

A unit of scalability and parallelism in Kinesis, with a 2MB/sec throughput limit.

The process of saving progress in Kinesis Clients to resume processing later.

A feature allowing each consumer to receive 2MB/s of throughput per shard, improving delivery speed.

A compute service that allows running code in response to events or triggers without managing servers.

Temporary storage of data in Firehose before sending to the destination based on size/time.

A library that simplifies consuming data from Kinesis and managing state across multiple consumers.

Converting or modifying data formats in real-time before storage.

Applications that read and process records from Kinesis streams.

The configuration of how much data accumulates in Firehose before flushing it out.

A system allowing S3 to be used like HDFS, supporting persistent storage after cluster termination.

Disk storage connected locally, ideal for temporary data like buffers and caches.

EBS volumes are used with EMR and are deleted upon cluster termination; attached during cluster launch only.

Automatically manages resources, determining node count and capacity for EMR jobs without manual intervention.

Pre-initialized capacity needs to be at least 10% higher than the memory requested for drivers and executors.

Service for streaming data, made of shards which are sequences of records ordered by arrival time.

A shard contains an ordered sequence of records and can be scaled by provisioning more shards beforehand.

Kinesis capacity mode that auto-scales based on observed throughput without pre-provisioning shards.

The process of adding or merging shards in Kinesis to adjust stream capacity, which is not done in parallel.

Producer retries can create duplicate data; fixing requires unique record IDs to prevent repeats on the consumer side.

Consumer applications that can read the same data multiple times without causing adverse effects.

Access control via IAM, encryption in transit using HTTPS, and KMS for encryption at rest for Kinesis data.

Data in Kinesis can be retained for 1 to 365 days, programmable for reprocessing or replaying data later.

Once data is added to Kinesis, it cannot be deleted, ensuring ordered and reliable data flows.

Organized units for users, teams, and applications that control query access and track costs.

Policies that define permissions for access control in AWS services.

A centralized repository for storing metadata of data assets with fine-grained access.

An interactive environment for executing queries and collaborating on data analysis.

Combines Spark's distributed computing with serverless features for analyzing large datasets.

A distributed processing framework designed for big data analytics.

Storing frequently accessed data in memory for faster retrieval.

Real-time data processing framework integrated with services like Kinesis.

A library for machine learning algorithms capable of large-scale processing.

Spark’s API for graph processing and analysis.

SQL command to create a new table based on existing data through selection.

Extract, Transform, Load - processes to prepare data for analysis.

Interactive computing environments for code writing, running, and visualization.

A system that records changes to files and allows tracking revisions over time.

A record of previously run queries within a workgroup or environment.