Computer Science Quiz: Parallelism in CS

Study Notes

Parallelism in Computer Science

Parallelism is a technique used to improve the performance and efficiency of computer systems by executing multiple tasks or processes simultaneously.

Types of Parallelism:

Data Parallelism: parallel processing of multiple data elements using the same operation.
Task Parallelism: parallel execution of multiple tasks or processes.
Pipelining: parallel processing of multiple tasks or processes in a sequence.

Parallel Computing Architectures:

Symmetric Multiprocessing (SMP): multiple processors share a common memory and operate as a single system.
Distributed Computing: multiple computers or nodes work together to achieve a common goal.
Cluster Computing: a group of computers or nodes work together to achieve a common goal, often used in high-performance computing.

Parallelism in Programming:

Parallel Programming Models: models that allow developers to write parallel code, such as:
- Shared Memory Model: multiple threads share a common memory space.
- Message Passing Model: threads communicate with each other by passing messages.
- Data Parallel Model: parallel processing of multiple data elements using the same operation.
Parallel Programming Languages: languages that support parallel programming, such as:
- OpenMP: an API for parallel programming in C, C++, and Fortran.
- MPI (Message Passing Interface): a standard for message passing in parallel computing.

Challenges and Limitations:

Synchronization: coordinating access to shared resources to avoid conflicts and errors.
Communication Overhead: the time and resources required for threads or processes to communicate with each other.
Load Balancing: distributing workload evenly among processors or nodes to achieve optimal performance.
Scalability: the ability of a parallel system to increase performance as the number of processors or nodes increases.

Parallelism in Computer Science

Types of Parallelism:

Data parallelism involves processing multiple data elements using the same operation.
Task parallelism involves executing multiple tasks or processes simultaneously.
Pipelining involves processing multiple tasks or processes in a sequence.

Parallel Computing Architectures:

Symmetric Multiprocessing (SMP) features multiple processors sharing a common memory and operating as a single system.
Distributed Computing involves multiple computers or nodes working together to achieve a common goal.
Cluster Computing involves a group of computers or nodes working together to achieve a common goal, often used in high-performance computing.

Parallelism in Programming:

Parallel programming models allow developers to write parallel code, including:
- Shared Memory Model, where multiple threads share a common memory space.
- Message Passing Model, where threads communicate by passing messages.
- Data Parallel Model, which involves parallel processing of multiple data elements using the same operation.
Parallel programming languages, including:
- OpenMP, an API for parallel programming in C, C++, and Fortran.
- MPI (Message Passing Interface), a standard for message passing in parallel computing.

Challenges and Limitations:

Synchronization is crucial to coordinate access to shared resources and avoid conflicts and errors.
Communication Overhead refers to the time and resources required for threads or processes to communicate with each other.
Load Balancing is essential to distribute workload evenly among processors or nodes to achieve optimal performance.
Scalability is the ability of a parallel system to increase performance as the number of processors or nodes increases.