High Performance Computing (HPC) Lecture PDF

High Performance Computing (HPC) Dr. Maha Dessokey My Background and Contact Details  Dr. Maha Dessokey  Researcher at Electronics Research Institute, HPC and Big Data Lab since 2007  Worked at ERI scientific cloud center of excellence since 2015  Interests: High Performance Computing, Clou...

High Performance Computing (HPC) Dr. Maha Dessokey My Background and Contact Details  Dr. Maha Dessokey  Researcher at Electronics Research Institute, HPC and Big Data Lab since 2007  Worked at ERI scientific cloud center of excellence since 2015  Interests: High Performance Computing, Cloud Computing and Big Data Engineering  Contact: [email protected] Resources  Lectures  Text Book Foster, Ian. Designing and Building Parallel Programs: Concepts and Tools for Parallel Software Engineering. 1st ed. Pearson, May 24, 2019.  Notes  Agenda  What is HPC?  Key Drivers of HPC  Application Areas of HPC in Science & Engineering  Parallel Processing  Flynn's Classical Taxonomy  Performance Metrics What is HPC? High Performance Computing most generally refers to the practice of aggregating computing power in a way that delivers much higher performance than one could get out of a typical desktop computer or workstation in order to solve large problems in science, engineering, or business.” Key Drivers of HPC Growth in data generation Key Drivers of HPC (Contd.)  Growth in data generation  Need for complex simulations and modeling in fields like climate science, physics, and bioinformatics drive the need for greater computing power.  Single-core processors can not be made that have enough resource for the simulations needed.  Making processors with faster clock speeds is difficult due to cost and power/heat limitations  Expensive to put huge memory on a single processor Serial computing Only one instruction may execute at any moment in time Parallel Computing  Solution: parallel computing –a form of computation in which many calculations are carried out simultaneously, operating on the principle that large problems can often be divided into smaller ones, which are then solved concurrently ("in parallel”).“ http://en.wikipedia.org/wiki/Parallel_computing Application Areas of HPC in Science & Engineering HPC in Science  Space Science Applications in Astrophysics and Astronomy  Earth Science Applications in understanding Physical Properties of Geological Structures, Water Resource Modelling, Seismic Exploration  Atmospheric Science Applications in Climate and Weather Forecasting, Air Quality HPC in Science (Contd.)  Life Science Applications in Drug Designing, Genome Sequencing, Protein Folding  Nuclear Science Applications in Nuclear Power, Nuclear Medicine (cancer etc.), Defence  Nano Science Applications in Semiconductor Physics, Microfabrication, Molecular Biology, Exploration of New Materials HPC in Engineering  Crash Simulation Applications in Automobile and Mechanical Engineering  Aerodynamics Simulation & Aircraft Designing Applications in Aeronautics and Mechanical Engineering  Structural Analysis  Applications in Civil Engineering and Architecture HPC in Multimedia and Animation  Increased Complexity of Content  High Resolution: Demand for 4K, 8K, and beyond requires significant processing power.  Complex Effects: Advanced visual effects (VFX) and rendering techniques necessitate heavy computations.  Real-Time Rendering  Interactive Experiences: HPC enables real-time rendering for gaming and virtual reality (VR) applications.  Simulation: Realistic physics simulations for animations and gaming environments.  Large Data Processing  Managing and processing massive datasets (e.g., video files, 3D models). HPC Applications Parallel Processing  A Quick Review- The von Neumann Architecture Model  Stored-program and Data in memory unit  Fetch and Decode instructions, Get data, Execute and Store back the results  Flynn's Classical Taxonomy  It is all still von Neumann: Data and Program in Memory  Based on the where the parallelism comes from: data and/or instructions (functional)  Multiplicity of data and instructions streams-- Categories = {single instr.(SI), multiple instr.(MI)} X {single data(SD), multiple data(MD)} SISD– Single Instruction Single Data SIMD– Single Instruction Multiple Data MISD- Multiple Instructions Single Data MIMD- Multiple Instruction Multiple Data Flynn's Classical Taxonomy (Contd.) SISD-- This is the uniprocessor architecture (Sequential) Flynn's Classical Taxonomy (Contd.) SIMD– Parallelism Comes from Data Flynn's Classical Taxonomy (Contd.) MISD --» Systolic array like and pipeline like Flynn's Classical Taxonomy (Contd.) MIMD (Shared Memory)– Parallelism Comes from Data and Instructions Flynn's Classical Taxonomy (Contd.) MIMD (Distributed memory)– Parallelism Comes from Data and Instructions  Pipelining  Example of pipelining pipeline processor of 3 stages 4 tasks each has 3 subtasks each performed by a stage  Takes 6 clocks instead of 12 Types of Parallelism  Data Parallelism -- many data items can be processed in the same manner at the same time (aka fine grain parallelism)  Functional Parallelism -- program has different independent modules that can execute simultaneously (coarse grain parallelism)  Overlapped/Temporal Parallelism -- program has a sequence of tasks that can be executed in an overlapped fashion. The most important form of this overlapped parallelism is PIPELINING. Performance Issues  Overhead in making the sequential application parallel Redundancy,  Interprocessor communications  Imbalance  Ratio of sequential part to parallel part in the application Performance Metrics Speedup factor ”S”-- Ratio of completion time on one processor to completion time on the n-processor system S = Sequential Time/ Parallel Time Efficiency “E” = Useful Parallel time/ Overall parallel time E= S/n Throughput-- Amount of work done per unit time MIPS and problems MFLOPS and problems Application related measures-- e.g. number of particle interactions per unit time Wall clock completion time Performance Metrics - Example Example: Consider Y = (a*b) + (c/d) + e, which is represented by the following dependence graph/ scedule assuming a 2-processor system Summary  What is HPC? Why we need HPC?  Application Areas of HPC in Science & Engineering  Flynn's Classical Taxonomy  Performance Metrics