MapReduce and Distributed Systems Overview

MapReduce Example: Word Count

• The reducer function sums values for each key.
• Example output: ("Hello", 3), ("world", 2)
• Final output is a list of words and their counts.

MapReduce Example: Maximum Temperature

• Problem: Find the maximum temperature for each year.
• Input: A dataset with Date (YYYY-MM-DD) & Temperature.
• Map Phase: Extracts year and temperature, emits (Year, Temperature) pairs.
• Shuffle and Sort: Groups pairs with same key (Year).
• Reduce Phase: Processes grouped pairs and finds the maximum value for each year.

Distributed Systems: Why Not Make a System Distributed?

• Increased Complexity: Designing, implementing, and managing a distributed system is more complex than centralized systems.
• Increased Latency and Synchronization: Network communication can introduce latency and maintaining consistency across nodes can be challenging.
• Increased Security Risks: Distributing resources increases the attack surface, requiring careful management of security protocols and access controls.

Distributed Systems: Challenges

• Communication: Reliable communication between nodes is difficult due to network failures, latency, and bandwidth constraints.
• Synchronization: Synchronizing clocks, data, and operations across distributed nodes is complex.
• Consistency: It's challenging to achieve data consistency.
• Fault Tolerance: Detecting and recovering from node failures is essential for maintaining system reliability.

Client-Server Example: Online Payments

• Client interacts with the server to make payments.
• Server processes payments and interacts with other services (e.g., payment gateways).
• Communication between client and server is essential.

RPC in Enterprise Systems

• Service-oriented Architecture (SOA) / Microservices: Splitting a large software application into multiple services that communicate via RPC.
• Interoperability: Different services can be implemented in different languages.
• Interface Definition Language (IDL): Provides a language-independent API specification for communication between services.

Interface Definition Language (IDL)

• IDL defines the interface between a client and server in RPC systems.
• Allows different programming languages and systems to communicate.
• Useful in distributed systems and RPC frameworks (e.g., gRPC, CORBA, Thrift).

Key Features of IDL

• Language Agnostic: IDL allows different languages to communicate by defining a common interface.
• Protocol Agnostic: IDL doesn't specify the communication method, only the structure of messages and services.
• Service and Data Type Definition: IDL specifies the operations (methods) that can be called remotely and the data types (messages) exchanged.

Why Use IDL?

• Enables communication between services implemented in different programming languages.
• Provides a consistent interface for developers.
• Simplifies the development of distributed systems.

Models of Distributed Systems

• Fundamental problems arise in distributed systems due to decentralized control, unreliable networks, and the possibility of failures.
• Two widely recognized problems are:
  • The Two Generals Problem
  • The Byzantine Generals Problem

The Two Generals Problem

• Illustrates the challenges of achieving coordination and reliable communication between two parties over an unreliable network.
• It is considered unsolvable in its most general form due to the impossibility of guaranteeing message delivery and achieving consensus.

The Two Generals Problem - Explanation

• Two generals must decide whether to attack or retreat.
• They communicate through unreliable messengers.
• If a response is not received, the sender is unsure if the message was received.

Summary: The Two Generals Problem

• The Two Generals Problem involves two parties trying to coordinate an action.
• They communicate over an unreliable channel where messages can be lost.
• The goal is to reach an agreement, but message delivery cannot be guaranteed, making absolute certainty impossible.
• It demonstrates that reliable communication and agreement are impossible over an unreliable channel.

The Two Generals Problem - Applied

• Applying the problem to real-world scenarios:
  • Two parties trying to coordinate over a network connection.
  • Systems relying on distributed databases.

Byzantine Generals Problem

• Deals with achieving consensus in the presence of faulty or malicious components (Byzantine faults).
• Introduced by Leslie Lamport, Robert Shostak, and Marshall Pease in 1982.
• Models systems with participants who might behave incorrectly or maliciously, yet still need to reach an agreement.

The Byzantine Empire (650 CE)

• "Byzantine" has long been used for "excessively complicated, bureaucratic, devious".

Byzantine Generals Problem - Overview

• A group of generals must decide on a common strategy (e.g., attack or retreat).
• Some generals may be traitors (Byzantine generals) who send conflicting or false messages.
• The challenge is for the loyal generals to reach a consensus despite the presence of traitors.

This quiz explores key concepts in MapReduce, including examples like Word Count and Maximum Temperature. It also highlights challenges in building distributed systems such as complexity and security risks. Test your understanding of these critical topics in computer science.