System Design - Scalability: Sharding Part 3

Re-sharding Strategies

• Proactive re-sharding: anticipate growth and performance issues by periodically evaluating data distribution and redistributing before issues arise (e.g., Facebook)
• Reactive re-sharding: trigger re-sharding in response to detected imbalances or performance degradation (e.g., online retail platform on Black Friday)

Zero-Downtime Re-sharding Strategies

• Dual-writes: temporarily write to both old and new shard configurations while reading from the old until migration is complete (e.g., Twitter)
• Phase-based approach: migrate in phases (e.g., start with less critical data) to reduce risk and ensure stability
• Blue-green deployments: maintain two parallel environments (old and new shards) to switch traffic seamlessly after the new configuration is fully tested (e.g., Netflix)

Zero-Downtime Migrations

• Incremental data copying: gradually copy data from old shards to new ones in small batches to minimize impact (e.g., LinkedIn)
• Consistency checks: continuously validate data consistency between old and new shards during migration (e.g., Amazon)
• Versioning: implement schema versioning to handle changes without breaking existing queries (e.g., Google Cloud Spanner)
• Operational best practices:
  • Monitoring and rollback plans: continuously monitor migration process and have rollback procedures in place for quick recovery if issues arise (e.g., Uber)
  • Staged migrations: conduct migrations during low-traffic periods and in stages to mitigate risk (e.g., eBay)

Managing Distributed Transactions

• Two-Phase Commit (2PC) protocol:
  • Ensure atomicity by dividing transaction into two phases: prepare and commit
  • Phase 1: each node votes to commit or abort the transaction
  • Phase 2: if all nodes vote to commit, transaction is committed; otherwise, it is aborted (e.g., financial institutions)
• Challenges with 2PC:
  • Blocking nature: participants may be blocked waiting for a response, impacting system performance
  • Coordinator failure: failure of the transaction coordinator can lead to uncertainty
• Alternative approaches:
  • Three-Phase Commit (3PC): adds an extra phase to 2PC to mitigate the blocking problem, but increases complexity and latency (e.g., large-scale retail companies)
  • Eventual consistency models: use compensating transactions to resolve inconsistencies over time, suitable for less critical operations (e.g., social media platforms)

Ensuring High Availability

• Replication strategies:
  • Synchronous replication: ensures immediate consistency by waiting for all replicas to acknowledge writes, but may introduce latency (e.g., banking systems)
  • Asynchronous replication: provides higher performance by not waiting for all replicas, at the cost of potential temporary inconsistencies (e.g., e-commerce websites)
• Redundancy and failover mechanisms:
  • Primary-replica configuration: designate one primary shard for writes and multiple replicas for reads, with automated failover to a replica if the primary fails (e.g., Amazon DynamoDB)
  • Multi-data center deployment: deploy shards across multiple data centers to ensure availability during regional outages (e.g., Google Cloud Spanner)

Dealing with Cross-Shard Queries

• Challenges with cross-shard queries:
  • Complexity: queries involving multiple shards are more complex to design and optimize
  • Latency: increased latency due to data being fetched from multiple sources
• Techniques to optimize cross-shard queries:
  • Distributed SQL engines: use distributed SQL query engines (e.g., Apache Calcite, Google F1) to optimize and execute queries across shards
  • Query routing: implement query routers to direct parts of the query to the appropriate shards and aggregate results (e.g., Facebook)
  • Pre-aggregation: pre-compute and store aggregates to reduce the need for cross-shard queries in real-time (e.g., Twitter)

Modern Technologies and Frameworks

• Sharding support in modern databases:
  • MongoDB: offers built-in support for sharding with automated data distribution and balancing (e.g., Craigslist)
  • Cassandra: uses partition keys to distribute data across nodes in a cluster, supporting large-scale sharding (e.g., Netflix)
  • CockroachDB: a distributed SQL database that automatically shards and replicates data across nodes (e.g., DoorDash)
• Frameworks and tools:
  • Vitess: an open-source sharding middleware for MySQL, providing scaling and sharding capabilities for large databases (e.g., YouTube)
  • Apache HBase: a distributed database that supports sharding through region servers and automatic splitting (e.g., Pinterest)
  • Gizzard: a sharding framework that provides APIs for managing data distribution and routing across shards (e.g., Twitter)