Intro to Parallel Computing

• The simultaneous growth in big data availability and simultaneous Internet users places pressure on computing tasks.
• Parallel and distributed computing are used across algorithms, computer architecture, networks, operating systems, and software engineering.
• The early 21st century saw rapid growth in multiprocessor design and strategies for faster complex application execution.
• Parallel and distributed computing use: Concurrency, mutual exclusion, consistency in state/memory manipulation, message-passing, and shared-memory models.

Introduction to Parallel Computing

• Parallel and distributed computing are important high-performance computing models.
• Both are designed for performing many calculations by breaking down processes into parallel tasks.
• They differ in structure, function, and utilization.

Parallel Computing

• Parallel computing involves multiple processors performing tasks simultaneously.
• Memory in parallel systems can be shared or distributed.
• Parallel computing increases concurrency, saving time and money.
• Examples: Blockchains, Smartphones, Laptop computers, Internet of Things, Artifical intelligence and machine learning, Space Shuttle and Supercomputers

Serial Computing

• Traditionally, software has been written for serial computation.
• A problem is broken into a discrete series of instructions.
• Instructions are executed sequentially, one after another.
• Execution occurs on a single processor.
• Only one instruction can execute at any moment in time.
• Serial computation refers to processes carried out sequentially.
• Each operation or instruction is executed in order, with the next starting only after the previous one is completed.
• This contrasts with parallel computation, which executes multiple tasks/computations simultaneously for faster processing.

Key Characteristics of Serial Computation

• Sequential Execution: In serial computation, one instruction is executed at a time with the CPU moving to the next only after completion.
• The time taken to execute a program depends on the number of operations and the time required for each.
• Simplicity: Serial computation is straightforward without managing concurrent tasks or synchronization.
• Serial computation is used in scenarios where tasks are inherently dependent, making parallel execution inefficient/unnecessary.
• Limited Performance: Serial computation may not fully utilize modern multi-core processors.
• Serial computation can become slow with increasing problem size, especially for large-scale tasks like scientific simulations, image processing, or data analysis.
• Hardware Dependence: Serial computation performance is determined by the processor's clock speed and instruction set efficiency.
• Deterministic: Serial computation's outcome is predictable since it follows a single execution path without concurrency issues.
• Each step is completed before the next begins.

Examples of Serial Computation

• Basic Mathematical Calculations: Calculating a sum of numbers serially involves adding them in order, (i.e., 1 + 2 = 3, then 3 + 3 = 6, then 6 + 4 = 10).
• Simple Programs: It is serial computation if a program prints a sequence of statements or processes data step by step.
• Sorting Algorithms: Certain sorting algorithms operate sequentially, processing pairs of elements until the list is sorted.
• Algorithm Execution in Single-Core Processors: Any computation on a single-core processor is serial, as only one task can be processed at a time.

Advantages of Serial Computation

• Simplicity: Serial algorithms are easier to implement and debug compared to parallel algorithms.
• Predictability: Serial computation's deterministic behavior is desirable in some cases.
• No Need for Synchronization: Since operations are performed one at a time there is no need to synchronizing multiple processes or threads.

Disadvantages of Serial Computation

• Serial computation can be slow for large-scale tasks as it uses one processing unit.
• Underutilization of Modern Hardware: Serial computation often fails to leverage the capabilities of multi-core processors.
• Scalability Issues: The time grows linearly with problem size, making serial computation inefficient for large data processing.

Comparison with Parallel Computation

• Serial Computation: Involves one task at a time, sequentially.
• Parallel Computation: Multiple tasks are executed simultaneously across multiple processing units.
• Parallel computation divides tasks and processes them concurrently for performance gains.
• Serial computation is simple and predictable; however, it is often less efficient for large or complex problems.
• Modern computing increasingly uses parallel processing to leverage multi-core processors and distributed environments.
• Serial computation still plays a key role in simpler or inherently sequential tasks.

Parallel Computing

• Parallel computing uses multiple compute resources simultaneously to solve a computational problem.
• A problem is broken into discrete parts that can be solved concurrently.
• Each part is broken down to a series of instructions.
• Instructions from each part execute on different processors simultaneously.
• Overall control/coordination mechanism.

Computational Problem Requirements for Parallel Computing

• It should be broken into pieces of work, which can be solved simultaneously.
• Execute multiple program instructions at any moment in time.
• It should be solved in less time with multiple compute resources than a single one.

Compute Resources for Parallel Computing

• A single computer with multiple processors or cores.
• An arbitrary number of such computers connected by a network.

Parallel Computers

• Virtually all stand-alone computers today are parallel from a hardware perspective.
• Multiple functional units (L1 cache, L2 cache, branch, prefetch, decode, floating-point, graphics processing (GPU), integer, etc.).
• Multiple execution units/cores.
• Multiple hardware threads.
• Networks connect multiple stand-alone computers (nodes) to make larger parallel computer clusters.
• Each compute node is a multi-processor parallel computer in itself.
• Multiple compute nodes are networked via an InfiniBand network.
• Special purpose nodes, also multi-processor, are used for other purposes.
• Most large parallel computers/supercomputers are clusters of hardware produced.

Advantages of Parallel Computing

• Increased Speed: Calculations are executed concurrently, reducing the computation time for large-scale problems.
• Efficient Use of Resources: Full advantage is taken of processing units, maximizing the machine's computational power.
• Scalability: The more processors built into the system, the more complex problems can be solved in a short time.
• Improved Performance for Complex Tasks: Well-suited for activities like number simulation, scientific analysis, modeling, and data processing.

Disadvantages of Parallel Computing

• Complexity in Programming: Organizing tasks in a parallel manner is challenging.
• Synchronization Issues: Interactions of processors during concurrent operations can result in problem areas.
• Hardware Costs: Implementation may require multi-core processors, which can be costly.

Why Use Parallel Computing?

• The real world is complex and interrelated.
• Parallel computing is better suited for modeling, simulating, and understanding complex, real-world phenomena.
• Save time and/or money.
• Solve larger / more complex problems.
• Provide concurrency.
• Take advantage of non-local resources.
• Better use of underlying parallel hardware.
• It is the future.

Save Time and/or Money

• Adding more resources is theorized to potentially shorten completion time while saving possible costs.
• Parallel computers can be built with cheap components.

Solve Larger / More Complex Problems

• Serial programs may require petaflops/petabytes of computing resources to solve larger and more complex problems.
• An example is web search engines that use databases processing millions of transactions every second.

Provide Concurrency

• Multiple compute resources can do many things simultaneously.
• Collaborative networks offers a global venue where people from around the world can conduct work "virtually."

Take Advantage of Non-Local Resources

• Wide area networks use compute resources, and local resources are used if there are scarce or insufficient resources.
• SETI@home has over 1.7 million users in nearly every country.

Make Better Use of Underlying Parallel Hardware

• Parallel software is intended for parallel hardware with multiple cores, threads, etc.
• Serial programs can "waste" potential computing power on modern computers.
• The trends are being driven toward faster networks, and computers with multiprocessors architectures, which show that parallelism is the future of computing.
• Supercomputer performance has increased 500,000x.
• Exascale Computing is the next race.
• Exaflop = 1018 calculations per second
• Example: US DOE Exascale Computing Project: https://www.exascaleproject.org

Who Is Using Parallel Computing?

• Parallel computing models difficulties in science and engineering:
• Science and Engineering: Atmosphere, Earth, Environment, Physics, Bioscience, Chemistry, Geology, Mechanical Engineering, Electrical Engineering, Computer Science, and Defense.
• Commercial applications drive computer development due to demands for processing large data amounts.
• Industrial and Commercial: Big Data, AI, Oil Exploration, Web Search Engines, Medical Diagnosis, Pharmaceutical Design, Financial Modeling, International Corporations, Graphics, and multimedia Technologies.
• Parallel computing is used in science, engineering and global applications.

Von Neumann Computer Architecture

• Named after John von Neumann, it was the design for an electronic computer.
• Defined in 1945.
• A "stored-program computer" keeps program data in electronic memory, differing from the "hard wiring" in previous computers.
• Virtually all computers follow this basic design.

Components

• Memory (RAM)
• Control Unit (CPU)
• Arithmetic Logic Unit (ALU)
• Input/Output

Function

• Random access memory stores all program data.
• Control units fetches the data, decodes, and then coordinates tasks.
• Arithmetic units perform basic operations.
• input/output works as interface for users.

Flynn's Classical Taxonomy

• Classifies parallel computers with Instruction and Data Stream.
• These can be Single or Multiple.
• 4 Classifications: SISD, SIMD, MISD, MIMD.
• Single Instruction, Single Data (SISD)
  • Serial computer.
  • Single Instruction: Only one instruction stream being acted on by the CPU during one clock cycle.
  • Single Data: Only one data stream is input during any one clock cycle.
  • Deterministic execution (refers to the predictable and repeatable nature of program execution).
  • The oldest type of computer.
• Single Instruction, Multiple Data (SIMD)
  • A type of parallel computer
  • Single Instruction: All processing units execute the same instruction at any given clock cycle
  • Multiple Data: Each processing unit can operate on a different data element
  • Well suited for problems such as graphics/image processing.
  • Synchronous (lockstep) and deterministic execution
  • Examples:
    • Processor Arrays: Thinking Machines CM-2, MasPar MP-1 & MP-2, ILLIAC IV
    • Vector Pipelines: IBM 9000, Cray X-MP, Y-MP & C90, Fujitsu VP, NEC SX-2, Hitachi S820, ETA10
• Multiple Instruction, Single Data (MISD)
  • A type of parallel computer
  • Multiple Instruction: Each processing unit operates on the data independently via separate instruction streams.
  • Single Data: A single data stream is fed into multiple processing units.
  • Few actual examples of parallel computers found.
  • Uses:
    • frequency filters operating on a single signal stream
    • cryptography algorithms attempting to crack coded messages.
• Multiple Instruction, Multiple Data (MIMD)
  • A type of parallel computer.
  • Multiple Instruction: Every processor may execute with a different instruction stream.
  • Multiple Data: Every processor may be working with a different data stream.
  • Execution can be synchronous or asynchronous, as well as deterministic or non-deterministic.
  • The most common and modern parallel computer falls into this category.

Common Parallel Terminology

• CPU: Has one or more cores.
• Node: The standalone components networked to form a supercomputer.
• Task: A program or program-like set of instructions.
• Pipelining: Processor units perform different steps for a task.
• Shared Memory: Architecture where all processors have direct access and can directly address and access where memory is.
• Symmetric Multi-Processor (SMP): shared memory hardware, where multiple processors share a single address space and have equal access
• Distributed Memory: It refers to network based memory access for physical memory that is uncommon and tasks only logically "see" local machine.
• Communications: Parallel tasks exchange data.
• Synchronization: Coordinates parallel tasks.
• Computational Granularity: Quantitative/qualitative measure of the computation-to-communication ratio.
• Coarse: when larger amounts of computational work are done between communication events
• Fine: when relatively small amounts of computational work are done between communication events
• Observed Speedup: Observed speedup of a code which has been parallelized, it is a indicator for parallel program
• Parallel Overhead: Execution time that is unique to parallel tasks.
• Massively Parallel: Hardware with several processing elements.
• Embarrassingly (IDEALY) Parallel: Independent tasks needing minimal coordination
• Scalability: A parallel system's to demonstrate proportionate increase in parallel speedup.

Limits and Costs of Parallel Programming

• Speedup has the hardware properties, and app algorithms.
• Amdahl's Law
  • Speedup = P/ 1 - P
    • P = fraction of code that can be parallelized.
  • Introducing processors:
    • P = 0: speedup = 1.
    • P = 1: speedup = to infinity (in theory).
    • P = 50%: MAX speedup = 2.

Parallel Programming with Variable Processors

• P = the percent of code that can be paralleled.
  • more scalability.

Complexity

• parallel apps to be complex than serial
• measured by programmer time in
• designing, coding, debugging, tuning and development cycles

Portability

• Common APIs are MPI, OpenMP and PoSIX threads
  • has been standardized
  • hardware makes characterizations to be versatile

Resource Requirements

• to process the time , and more CPu
• needs more code and memory than serial
• costs performance that setting up the parallel envionments

Scalability

• types
  • strong scailing - as processors - goal is to solve the problem with same faster - weak scaling - processor are added, problem size is proportion to processors used to goal run larger - run longer and same proessors Concrruency - the real life problems and applications
• concurrency - when events happening at differe times
• parallel - CPU cores are at the same times
• communications network card - amount of meamry available
  • clocking

Latency

• the effecienvcy of perfoming the algorithm and system

Types of Latency in Parallel Computing:

      -  communication  - data transfer
           - causes: network, , bandlimited
     2  synchronization - coordination
            - causes : lock contention
      memory - time taken to access memory resurces
            - causes : contention , memory heiartchy

Mitigate Tech

 1. comminuition   
      -patterns, mini,e data change
     -  fester
      -overal  computation
 3 syn
      -  point
    -   coordination
    -  exucurtion
 -optimize memory

Real Application

• High computeing - latancy imrpove perfomrance

Memory architecture

• multi - processory to sngle other,

• sharinf is fast and uniform proximiatu dsadavanage lacks scaility

• Distrubted processorts w/ local - no concept - fabric used for data thanfer Advantgnes

  • scales wiht # of os , memory, can acess without interference, effectiveness.

Chapter 4 programming style/models

• there are a lot to help
• share memory / threads
• data parallel
• hybrid
• spMD singal program multip Data
• MPMD mutpile propgram multiple Data
• share memory distribution of distribution of , not speficiationt and

shared memory model

process Interaction

• sharing to r/w and processes the memory Implementations
  • os compiler and to support sharing to posix support shmglet...to help -memory distributed, but can be a available to wikipdeida to shmem Threads - process multple path can run together

Implementation

-threads to subs - source

• help compile _ hardware implment to their versions

Posiixh

    - langauges are not use only linc,  they have explicate prgrams and details

Distributd memory to help by message passing - it helps each process

The models -taks use the momry, same machine , exachange data

• From prgram pesrive is help, the sub rouintmes all From wiki. Porg

SPMd vs MIMd

single proghram multipleData, multipropgrammultiple DAta is high ,and have prhrams and are all copy all multipal all task diffrernt dat

Hybrid

• that  comunationtion  by multi prgraog models , mpi,  and opend
  

• compumtiontion intensitve on - on comuuinction happens

     model helps by multi  models