Cloud Storage and Database Concepts

Cloud Storage and Database

• Cloud storage offers various types, including block storage, file storage, and object storage.
• Block storage organizes data as blocks, and file storage manages data through files.
• Object storage stores data in virtual containers (objects) with metadata.

Storage Characteristics

• Durability: Measured by expected data loss. Options include single-site, two-site, and multi-site copies, offering 99.99% to 99.9999% durability.
• Availability: Measured by expected downtime. Includes site redundancy and recovery measures.
• Security: Security measures for data at rest and in transit.
• Cost: Storage cost varies per unit (e.g., $/GB).
• Scalability: Upward storage capacity and user base growth.
• Performance: Performance metrics like flexibility, storage size, and bandwidth per unit (e.g., $/GB).
• Integration: Ability to interact via API or with other services.

Block Storage

• Instance storage is temporary and attached to host hardware, ideal for storing information subject to change or replication across instances.
• SSDs and HDDs, various types with specific characteristics (e.g., general purpose SSD).
• Includes snapshots for incremental point-in-time copies.

File Storage

• Network Attached Storage (NAS) and File Systems (like AWS Elastic File System or Google Filestore) provide file storage.
• Suitable for scenarios needing high I/O operations and low latency access.
• Offers high performance suitable for databases and other performance-sensitive services.

Object Storage

• Offers web accessibility through APIs or HTTPS.
• Highly durable (e.g., 99.9999999% design) and limitlessly scalable.
• Multiple data tiers to match workload and data lifecycle rules.
• Suitable for static website hosting, security, compliance, and audit capabilities.
• Standard storage pricing varies based on region (e.g., $0.023/GB).

Database Concepts

• A database is a structured collection of information for easy access, management, and updates.
• Databases consist of a storage pool and a management system.
• Different types of databases include Relational, Key-Value, Document, Wide Column and Graph Databases.

ACID Properties in DBMS

• Atomicity: Transactions either complete entirely or not at all.
• Consistency: Databases maintain consistency before and after transactions, strong and eventual.
• Isolation: Multiple transactions execute independently without interference.
• Durability: Successful transactions persist even if the system crashes.

DB Services

• Database services like Amazon RDS (Relational Database Service) manage databases on EC2.
• Managed services simplify tasks like scaling, backups, and patches, allowing you to focus on application optimization.

Relational Database

• Highly structured data stores, designed to prevent anomalies while offering query languages to manipulate data.
• Support ACID transactions and typically use SQL.
• Cloud-managed relational databases exist (e.g., AWS Aurora, PostgreSQL).
• Use pre-configured VMs, auto backups, REST APIs, high scalability with CPU and memory.

Global Transactional Database

• Fully managed Relational database services used in distributed systems needing high availability (e.g., financial trading).
• Support strong consistency, horizontal scalability, and high availability (e.g., 99.999%).
• GCP Cloud Spanner is a relevant and applicable service.

DataWarehouse

• Relational databases optimized for storing and querying large datasets.
• Supports fast queries for large volumes of data, with batch and streaming job support.
• Tools like BigQuery and Redshift are common choices.

Non-Relational Database

• Databases that don't rely on the tabular schema of rows and columns, optimizing for particular data types.
• These include document, graph, key-value, and wide-column models.

Key-Value Database

• Stores data as key-value pairs, suitable for session stores and shopping carts. Offers horizontal scalability compared to traditional models.
• DynamoDB is a common example of this type.

Document Database

• Flexible and semi-structured, storing documents with unique keys and supporting content-based retrieval and modification.
• Used in content management systems, e-commerce platforms, and similar applications.
• Suitable for handling semi-structured data.

Wide-Column Database

• Tables where columns can vary within rows, and treated as key-value stores.
• Ideal for applications requiring flexibility and handling large quantities of data, like financial data, IoT data, and reporting systems.
• AWS Keyspaces and Bigtable are common examples.

Memory/Cache databases

• In-memory data stores designed for ultra-fast access to copies of data, improving performance by eliminating data retrieval from slower storage mediums like hard drives.
• They help in decreasing network cost and improving responsiveness in applications.
• AWS ElastiCache and Google Cloud Memorystore are services focused on caching and memory-based data stores.

Redis vs Memcached

• Redis provides persistence and supports complex data structures (Hashes, lists, sets etc).
• Memcached is designed purely as a caching solution.

Database Summary

• Various database services are available spanning from Relational Databases to NoSQL databases.
• Key factors when choosing the right database are the consistency needs, scalability requirements, type of data, and the architecture of the application.

Database Services in AWS and GCP

• Both AWS and GCP offer various database and storage services categorized as, Relational, Key-Value, Document, Graph, Wide-Column, and special in-memory storage mediums like Caches.

Assignment

• Choose a non-relational database, create an instance, and test simple operations.
• List 3 interesting technical points about the chosen database.
• Design a storage and database structure for 4 projects.