Lecture-Cap8-Part1-MIMD Systems and Multiprocessor Architecture

Study Notes

MIMD architecture is related to multiprocessors, where different instructions can work on different data in parallel.
Multiprocessors are computers consisting of tightly coupled processors, whose coordination and usage are controlled by a single operating system.
Processors in a multiprocessor system share memory through a shared address space.

Multiple processors in a multiprocessor system can share:
- Cache memory, main memory, and I/O system
- Main memory and I/O system
- I/O system
- Nothing, communicating through networks
Avoiding bottlenecks in the architecture is important, and the answer depends on project requirements.
Multiprocessors can be classified in terms of their memory organization: symmetric (shared-memory) multiprocessors - SMP and distributed shared memory - DSM.

SMP are centralized shared-memory multiprocessors with approximately 32 cores or less.
All processors have equal access to a single centralized memory, with uniform access time - UMA.
SMP use large caches and many buses to avoid bottlenecks.
Each processor has its own local caches, and the shared cache is accessed through a bus.

DSM organization counts on a larger number of processor cores (e.g., 16 to 64).
Distributed memory is used to increase bandwidth and reduce access latency.
Communication among processors becomes more complex, requiring more software effort to take advantage of increased memory bandwidth.
I/O system is also distributed, and each node can be a small distributed system with centralized memory.
DSM has nonuniform memory access - NUMA, i.e., access time depends on the location of a data word in memory.

SMP follows UMA, where processors share a single memory and have uniform access times to the centralized memory.
DSM follows NUMA, where processors share the same address space but not necessarily the same physical memory.

SMP Communication Model:
- Uses threads and fork-join model.
- Implicit communication through memory access.
- OpenMP is an application programming interface for SMP.
DSM Communication Model:
- Uses message passing model.
- Explicit communication through message passing.
- MPI (Message Passing Interface) is a library project implementing a message passing interface.