Understanding MPI Communication and Node Types

Distributed Computing Era

• Mass adoption of global networks enabled distributed computing on a larger scale.
• Peer-to-peer networks promoted decentralization, increased resilience, and scalability by sharing resources directly.

Distributed Computing Types

• Cloud Computing: introduced virtualization, allowing remote access to computing resources on-demand, offering scalability, flexibility, and cost-efficiency.
• Distributed Computing Clusters: multiple computers interconnected, working together to process tasks in parallel, improving performance.
• Grid Computing: geographically distributed resources connected over a network to solve large-scale problems by utilizing idle resources.
• Distributed Databases: data distributed across multiple nodes, improving scalability and fault tolerance; includes sharded databases and NoSQL databases.

Distributed Computing Models

• Message Passing Model: achieves parallelism by having multiple processes co-operate on the same task, communicating through message passing.
• Actor Model: not mentioned in the provided text.

Message Passing Interface (MPI)

• MPI: standardized means of exchanging messages between multiple computers running a parallel program across distributed memory.
• MPI Features:
  • Standardization: widely accepted industry standard.
  • Portability: implemented for many distributed memory architectures.
  • Speed: optimized for hardware.
  • Functionality: designed for high performance on massively parallel machines and clusters.
  • Availability: various implementations available, including open-source and commercial.
• MPI Implementations:
  • OpenMPI and MPICH are popular open-source implementations.
  • Microsoft has an open-source implementation for Windows called MSMPI.

Message Passing Programming

• SIMD/SPMD: Single-Instruction-Multiple-Data (SIMD) or Single-Program-Multiple-Data (SPMD) model, where all processes run an identical copy of the same program.
• Process Identification: processes belong to one or more groups called "communicators" (communication channels) and can be identified by a unique number within each communicator, called rank.
• Message Communication:
  • Messages typically contain sender ID, receiver ID, data type, number of data items, data, and message type identifier.
  • Sending a message can be either synchronous or asynchronous.
  • Receives are usually synchronous.

Data Communication in MPI

• Data Communication: like email exchange, where one process sends a copy of the data to another process, and the other process receives it.
• Communication Requirements:
  • Sender needs to know receiver's process rank, data type, and user-defined "tag" for the message.
  • Receiver might need to know sender's process rank, data type, and user-defined "tag" of the message.