PDP Chapter 1: Introduction to Parallel & Distributed Processing

Introduction to Parallel & Distributed Processing

• Parallel computing is motivated by the need for efficient processing as computing power increases.
• Moore's Law indicates computing power doubles approximately every 18 months, driving reliance on parallelism.
• Performance gaps exist between processor and memory speeds, necessitating intermediate solutions.

Computational Power Dynamics

• The rate of processor speed increase surpasses memory speed, leading to bottlenecks in computation.
• High-end processors have seen a 40% per year clock rate increase, while DRAM access times improve by only 10% annually.
• Parallel platforms mitigate memory bottlenecks through larger aggregate caches and increased bandwidth.

Data Communication and Parallelism

• The Internet is envisioned as a large-scale parallel/distributed computing environment, especially for data mining.
• Infeasibility of central data collection in large datasets necessitates the use of parallel computing.

Importance of Learning Parallel Applications

• Effective parallel application development is complex and resource-intensive, involving coordination of concurrent tasks and non-standardized environments.
• Standardization efforts are underway to extend the lifecycle of parallel applications and improve development efficiency.

Scope of Parallel Computing

• Engineering and Design: Utilized for optimizing circuits and structures, and designing MEMS and NEMS.
• Scientific Applications: Includes bioinformatics, aiding in drug development, and understanding biological processes.
• Commercial Applications: Employed in web/database servers, handling large transaction volumes, especially in finance sectors.
• Computer Systems: In cryptography and embedded systems, parallel computing is crucial for distributed control algorithms.

Course Structure and Focus

• The course encompasses problem-solving using parallel computers and emphasizes algorithms and metrics suitable for various architectures.
• Programming techniques explore widely used paradigms such as MPI and POSIX threads across parallel platforms.

Key Concepts in Parallel Processing

• Processes vs. Processors: Processes represent logical tasks, while processors are physical computing units. They usually maintain a one-to-one correspondence in parallel algorithms.
• Programs: Defined as sequences of instructions crafted to perform specific tasks.

Processor Performance Metrics

• CPU performance is expressed through equations correlating CPU time to clock cycles, instruction count, and clock cycle time.
• The performance of a processor is linked to three factors: clock cycle time, cycles per instruction (CPI), and the instruction count.
• Key metrics include:
  • CPI: Lower values indicate better performance.
  • Instruction Count: Refers to the total number of executed commands.

Quantitative Principles of Computer Design

• CPU time can be calculated using the formula:
  • CPU time = Instruction count × Cycles per instruction × Clock cycle time
• Performance fundamentally hinges on the interplay between instruction set architecture, compiler technology, and hardware efficiency.

Conclusion

• Mastery of parallel and distributed processing principles is essential for tackling contemporary computing challenges across diverse application areas.

Explore the fundamentals of parallel and distributed processing, focusing on the implications of Moore's Law and the dynamics of computational power. Understand how parallelism addresses bottlenecks between processor and memory speeds, and its role in data communication on the Internet. This quiz will test your knowledge on these essential concepts.