Pthreads in High Performance Computing

Pthreads and High-Performance Computing

• Pthreads is a library for developing parallel applications using shared memory.
• It assumes a POSIX-compliant operating system as its base.
• The library can be embedded in any programming language, usually C.
• Threads are created by invoking functions from the library.

Compiling and Executing Pthreads Programs

• To compile a Pthreads program, include the library headers with #include .
• Use the linker option -lpthread to link the program.
• Example compilation command: gcc -lpthread hello.c -o hello.

Pthread API for Creating and Joining Threads

• The pthread_create function creates a new thread:
  • pthread_t* thread_p: thread object reference
  • const pthread_attr_t* attr_p: creation attributes (can be NULL)
  • void* (*start_routine)(void*): function to execute
  • void* arg_p: function argument
• The generic function header for the start_routine function is: void* start_routine(void* args_p);

Pthread API for Joining Threads

• The pthread_join function waits for a thread to terminate:
  • pthread_t thread: thread to wait for
  • void** ret_val_p: return value from the thread

Example Incremental Application

• The example application has three global variables:
  • long long n: number of iterations
  • long long thread_count: number of threads
  • long long sum: global sum value
• The Increment function is executed by each thread:
  • It calculates its range of iterations based on its rank and the total number of iterations
  • It prints its thread number and range of iterations
  • It increments the global sum value within its range of iterations

Condition Variables

• A condition variable is a structure that allows threads to suspend execution until a certain event or condition occurs.
• When the event occurs, a signaling mechanism will "wake up" the locked threads to continue execution.
• A condition variable must be associated with a mutual exclusion (mutex) mechanism.

Condition Variables API

• pthread_cond_wait(cv, mt) function waits for the event to occur, and has a combined and atomic function of unlocking the mutex, waiting for the signal, and locking the mutex.
• pthread_cond_signal(cv) function signals the occurrence of the event for any locked thread.
• pthread_cond_broadcast(cv) function signals the occurrence of the event for all locked threads.
• pthread_cond_t is the type of auxiliary structure for condition variables.
• pthread_cond_init and pthread_cond_destroy functions are used to initialize and terminate condition variables respectively.

Condition Variables Usage Pattern

• The pattern solves the problem of synchronization between threads without using active waiting.
• Threads invoke the wait() function, which only returns when the event is signaled.
• All threads update the global state with a mutex and block.
• The last thread that detects the event or condition signals to all using broadcast().

Barrier Example using Condition Variables

• The barrier example demonstrates synchronization between threads using condition variables.
• The pseudocode uses a mutex and a condition variable to synchronize threads.
• When all threads reach the barrier, they wait until all threads have reached it, and then they can proceed.

Producer-Consumer Example

• The producer-consumer problem is an example of synchronization between threads.
• A producer thread generates items for a buffer with limited size, and a consumer thread extracts items from the buffer.
• The producer can only execute if the buffer has space available, and the consumer can only execute if there is at least one item in the buffer.

Producer-Consumer Unbounded Buffer Example

• The example uses two condition variables: notEmptyforConsumer and notFullforProducer.
• The producer thread waits on notFullforProducer if the buffer is full, and signals on notEmptyforConsumer when an item is produced.
• The consumer thread waits on notEmptyforConsumer if the buffer is empty, and signals on notFullforProducer when an item is consumed.

Thread Execution Ordering

• The mutual exclusion mechanism ensures that only one thread can execute code that manipulates a shared resource at a time.
• Some problems are based on the ordering of thread execution, such as producer-consumer, reader-writer, and barrier problems.

Thread Synchronization Problems

• Producer-consumer: producer produces resources, and consumer consumes resources from a shared buffer.
• Reader-writer: readers read data, and writers change data, with restrictions on concurrent access.
• Barrier: threads invoke a barrier function, which blocks until all threads have reached it, and then unblocks for all threads.

MPI Basics

• MPI (Message Passing Interface) is a standard for parallel computing that allows processes to exchange data among themselves.
• MPI is based on the message-switching paradigm, where processes only send and receive messages to and from each other without sharing memory.
• The MPI standard defines an API for processes to exchange data among themselves.

MPI Initialization and Finalization

• MPI_Init(int *argc, char ***argv) is the function that must be invoked before any other MPI function in the program.
• It receives the address of the main function parameters, or NULL.
• MPI_Finalize() is the function that terminates the MPI library in the process, after which no other MPI functions can be invoked.

Communication Groups

• MPI_Comm_rank(MPI_Comm comm, int *rank) returns a process identifier within the process set.
• MPI_Comm_size(MPI_Comm comm, int *size) returns the size of the process set.
• MPI_COMM_WORLD is a constant representing the set of all processes in an execution.

Point-to-Point Communication

• Communication between processes happens by sending and receiving messages.
• Each process executes a different part of the same code, through "if"s.
• Each process is identified by its rank value.
• A process executes a function to send: "Send", and another process executes a function to receive: "Recv".

MPI Send and Receive

• MPI_Send(const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) is used to send a message.
• MPI_Recv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Status *status) is used to receive a message.

MPI DataTypes

• MPI datatypes are used to specify the type of data being sent or received.

MPI Count

• MPI_Get_count(const MPI_Status *status, MPI_Datatype datatype, int *count) returns the number of elements received by the last message.

Synchronization Model

• The Send function can have a synchronous or deferred sending behavior.
• The Recv function is always synchronous, blocking until it receives a message.
• If the next message received does not match the reception parameters, the program may block!

Ssend Function

• MPI_Ssend is similar to MPI_Send, but it is always synchronous and blocking, only ending when the message reaches the destination.

Collectives

• Collectives are used for group communication, where multiple processes exchange data.
• Examples of collectives include MPI_Barrier, MPI_Bcast, and MPI_Reduce.

MPI Barrier

• MPI_Barrier(MPI_Comm comm) is used to synchronize processes within a communication group.
• All processes in the group must call this function before any process can proceed.

MPI Broadcast

• MPI_Bcast(void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm) is used to broadcast a message from one process to all other processes in the group.
• The root process sends the data, and all other processes receive the data.

MPI Reduce

• MPI_Reduce(const void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) is used to collect a value from all processes and apply an aggregation function.
• The result is merged in the root process.

MPI Compilation and Execution

• MPI programs can be compiled using the mpicc wrapper around the system compiler.
• The program can be executed using the mpirun command with the -n option to specify the number of processes.