Introduction to MPI

• Parallel Algorithms and Programming - Introduction to MPI
• MPI stands for “Message Passing Interface”

Outline

• Message Passing Interface (MPI)
• Example: Hello MPI world
• Point-to-Point communication
• Blocking send and receive
• Variants of send and receive
• Non-blocking send and receive
• Status, Test & Probe
• Eager vs. Rendezvous protocol
• Brief wrap-up

MPI - Efficient Message Passing

• MPI is a standard for communication in HPC centers
• The website is https://www.mpi-forum.org/
• MPI exists for a variety of languages including: C/C++, Fortran, Julia, and Python
• This course will use C++

MPI - API

• MPI describes the API (application programming interface) which supports efficient communication
• MPI is not the implementation, there are different implementations, like Open MPI, MPICH and vendor-specific MPIs
• HPC centers provide a library/header file with a communications standard and special interconnects optimized for HPC tasks

MPI Setup

• Before you can use MPI in a program, include the mpi.h header file

#include <mpi.h>

• Before using MPI, initialize it with MPI_Init
• After using MPI, shut it down with MPI_Finalize

Compiling MPI Programs

• Since MPI is an API, programs need compiled with special flags
• MPI uses a special compiler wrapper and hides those flags
• When using C++ code, replace the C++ compiler with "mpicxx" to compile a file:

$ mpicxx hello_mpi_world.cpp -o hello_mpi_world

• The above command uses a specific compiler, sets the include path to include mpi.h, and links the MPI library during the linking step
• That “mpicxx” is platform specific, which could be “mpiCC”

Running MPI Programs

• MPI programs need to be executed with mpirun
• You need to specify the number of parallel processes to run, such as:

$ mpirun -n 4 ./hello_mpi_world [possible arguments to the program]

• That command starts the executable with 4 parallel processes and "mpirun" stops after all 4 parallel processes stop running.
• Running MPI programs on larger compute clusters would be more challenging
• You need to write a batch jobfile to describe how the job looks as a scheduler gets the job file to decide when to execute it

MPI Communicators

• An MPI communicator is a handle that describes sets of MPI processes which can communicate via a communicator
• MPI provides two default communicators which include
  • MPI_COMM_WORLD: All MPI processes of the current application
  • MPI_COMM_SELF: Contains only own process

// Provided by mpi.h
MPI_Comm MPI_COMM_WORLD;
MPI_Comm MPI_COMM_SELF;

• Each communicator also assigns each MPI process a unique rank

MPI Size and Rank

• The size of MPI_COMM_WORLD matches the number of MPI processes used to run the MPI program

int MPI_Comm_size(MPI_Comm comm, int *size)

• An MPI process has a unique rank within the communicator so that it belongs to the communicator: 0 < R < size

int MPI_Comm_rank(MPI_Comm comm, int *rank)

MPI Example - Hello World

• It initializes MPI, gets rank within MPI_COMM_WORLD, and prints rank
• It gets the number of ranks in the communicator, then shuts down MPI

Point-to-point Communication

• Blocking send and receive
• Variants of send and receive
• Non-blocking send and receive
• Status, Test & Probe
• Eager vs. Rendezvous protocol

Sending MPI Messages

• The MPI_Send command sends data from one rank to another

int MPI_Send(
    const void* buffer,   // pointer to buffer
    int count,            // number of elements of particular type
    MPI_Datatype datatype,  // type of elements in buffer
    int dest,             // destination rank
    int tag,              // tag for send (must match tag for receive)
    MPI_Comm comm         // communicator to use
)

MPI Datatype

• Predefined types include: MPI_INT, MPI_CHAR, MPI_FLOAT, and MPI_DOUBLE
• Datatype describes the data type of elements that should be sent
• It's important rather than sending a chunk of binary data to handle different storage formats such as little-endian or big-endian
• Custom datatypes can also be created

MPI Message Parameters

• buffer points to the start of data block that should be sent
• datatype refers to the type of elements in buffer
• count then represents the number of elements in the buffer and is of the type specified in datatype
• dest is the destination rank to send the data to
• tag is an identifier of the message and can be used at receiving rank to receive messages with a matching tag
• comm is the MPI communicator to use

Sending Example

const char *hello_world = "HELLO WORLD";
MPI_Send(
    hello_world,               // pointer to buffer
    strlen(hello_world)+1,     // number of elements of particular type
    MPI_BYTE,                  // type of elements in buffer
    3,                         // destination rank
    123,                       // tag for send (must match tag for receive)
    MPI_COMM_WORLD             // communicator to use
);

Receiving MPI Messages

• The MPI_Recv command receives data from a particular rank, such as:

int MPI_Recv(
    void* recv_buffer,    // pointer to buffer to store data
    int count,            // max capacity of elements in buffer
    MPI_Datatype datatype,  // type of elements in buffer
    int source,           // destination rank
    int tag,              // tag for send (must match tag for receive)
    MPI_Comm comm,        // communicator to use
    MPI_status *status    // status information of message
)

• buffer, count, datatype are the same as before, but buffer points to the data location to write data to “source"
• source is the number of the rank to receive data from or MPI_ANY_SOURCE to receive data from any source
• "tag" is an identifier of message, only messages that were sent with the matching tag are processed
• MPI_ANY_TAG can be specified to get messages independent to the tag
• comm is the MPI communicator to use
• “status” gives more information about message (discussed later)

wildcards

• When receiving messages MPI_ANY_SOURCE and MPI_ANY_TAG should be avoided!
• Always try to clearly specify from which source you're expecting information because there are only a few cases where they are useful

MPI_INT Data Type

• The types include:
  • MPI_CHAR - signed char
  • MPI_SHORT - signed short int
  • MPI_INT - signed int
  • MPI_LONG - signed long int
  • MPI_UNSIGNED_CHAR - unsigned char
  • MPI_UNSIGNED SHORT - unsigned short int
  • MPI_UNSIGNED - unsigned int
  • MPI_UNSIGNED_LONG - unsigned long int
  • MPI_FLOAT - float
  • MPI_DOUBLE - double
  • MPI_LONG_DOUBLE - long double
  • MPI_BYTE - 1 Byte
  • MPI_PACKED - see MPI_Pack()

MPI_Send - Different Variants

• MPI_Ssend: Synchronous send, which only returns if the receive started
  • It is most straight-forward way to send data
• MPI_Bsend: Buffered send, where the user needs to specify a buffer
  • Buffer is used to temporarily store the data to be sent
  • It may complete before receive started
• MPI_Send: Can either be either MPI_Ssend OR MPI_Bsend
  • Note, this is implementation depending because the internal buffer used if MPI_Bsend is used
• MPI_Rsend: Ready send assumes that receiver is already ready

Ssend vs. Bsend

• Subtle difference
  • Ssend waits until the receive started, which also ensures that source data can be overwritten after function returns as the message started to be processed by the recipient
  • Bsend waits until the message is copied to buffer: then the function can return since the source data can be overwritten
    • This means nothing more than the message is on the way, nor anything else
• The memory chunk to be sent can be overwritten in both cases
• See also https://iamsorush.com/posts/mpi-send-types/ for more examples

MPI_Isend Non-Blocking Message

• MPI variants: MPI_Isend, MPI_Issend, MPI_Ibsend, MPI_Irsend which work similar to previous versions

int MPI_Isend(
    const void* buffer,   // source buffer
    int count,            // number of elements to send
    MPI_Datatype datatype,  // type of elements
    int dest,             // destination rank
    int tag,              // tag to use
    MPI_Comm comm,        // communicator
    MPI_Request *request  // request handler
)

• This function returns non-blocking right away, and the buffer may not be modified until send was successful

MPI_Request Data Structure

• MPI_Request is a data structure that is filled in by operations.

MPI_Request mpi_request;
int MPI_Isend(
    &mpi_request // request handle
)

• MPI_Request
  • Needs to be writable until the operation is finished
  • Can only be free'd after the operation finished
• Can be used to inquire about the status of the operation

Status Information

• Status information can be requested, MPI_status structure contains directly accessible information:
  • status.MPI_SOURCE: Source rank to receive data from
  • status.MPI_TAG: Tag of the message
  • status.MPI_ERROR: Error information
• It contains size information which needs to be queried.

int MPI_Get_count(
    const MPI_Status *status, // status information
    MPI_Datatype datatype,     // datatype of the elements
    int *count  // number of elements for given datatype
)

• Count is not the number of bytes for all data, but the number of elements and depends on the data type

Testing Request Status

• A non-blocking action.

int MPI_Test(
    MPI_Request *request,   // input: pointer to request handler
    int *flag,               // output: flag showing whether request finished
    MPI_Status *status       // output: status information (optional)
)

• it allows to see whether a request is finished
  • 0: False
  • 1 or all other values: True
• “status” can be optionally requested

Message Probing

• Messages may be tested for availability with MPI_Probe:

int MPI_Probe (
    int source,      // source rank
    int tag,         // tag of message
    MPI_Comm comm,   // communicator to use
    MPI_Status *status  // status
)

• This function blocks until a message is available
• The status can be used to get information about the message size
• MPI_Iprobe is non-blocking

Console output of MPI

• MPI does not guarantee anything about the order of the console output messages, because it is not defined in MPI!
• This only worked for our particular MPI implementation, so the only way to ensure a particular order of printing messages is to receive the messages into 0-th rank and print them with the 0-th rank in the required order
• For this, all messages should be sent to Rank 0

Communication Protocols

• There are two different protocols of communicating messages, like Eager and Rendezvous

Eager Protocol

• Data is sent assuming the receiver can directly store it, but if the receiver isn't ready, it introduces delay
• Works well for small messages less than 64kB
• Might lead to overhead for large messages

Message Passing Interface (MPI) Terminology

• MPI process is an instance of a program to use MPI
• Group is a set of MPI processes
• Communicator is an ordered list of MPI processes in group
• Rank is a unique id of MPI process in an ordered list of communicator