Parallel Processing Architectures Quiz

Parallel Processing

• Parallel processing is the execution of concurrent events in computing to achieve faster computational speed.
• The goal is to speed up processing and increase throughput (amount of processing in a given time).
• Hardware increases with parallel processing, resulting in higher system costs.
• Technological advancements have reduced hardware costs, making parallel processing more feasible.

Parallel Processing Levels of Complexity

• Lower level: Serial shift registers versus parallel load registers.
• Higher level: Multiple functional units performing identical or different operations simultaneously.

Parallel Processing Classification

• Classification can be based on processor organization, interconnection structure, or information flow.
• One classification, by Flynn, categorizes systems based on instruction and data streams.

Parallel Processing Streams

• The normal computer operation fetches instructions from memory and executes in the processor.
• Instructions flow as an instruction stream.
• Data operations form a data stream.
• Parallel processing can occur in either or both streams.

Flynn's Classification

• SISD (Single Instruction Stream, Single Data Stream): A single processor with a single instruction stream operating on a single data stream. Includes sequential execution and may have internal parallel capabilities achieved through multiple functional units or pipelining.
• SIMD (Single Instruction Stream, Multiple Data Stream): Multiple processors execute the same instruction concurrently on different data items. Common in modern GPUs, vector processors, and array processors, often used extensively in image and digital signal processing.
• MISD (Multiple Instruction Stream, Single Data Stream): Multiple processors work on a single data stream, employing different instruction streams concurrently. This structure is theoretical with limited practical applications.
• MIMD (Multiple Instruction Stream, Multiple Data Stream): Multiple processors execute different instruction streams concurrently on different data items. This is the prevalent structure in most multiprocessor and multicomputer systems.

Single Instruction Stream, Single Data Stream (SISD)

• A single processor handles data from a single memory address.
• A single instruction is performed on data.
• Pipelining can be used for parallelism, but only one instruction is executed at a time.
• Single-processor systems are SISD.
• Advantages: Relatively inexpensive, low power consumption.
• Disadvantages: Limited processing speed due to single core.
• Uses: Microcontrollers and older mainframes.

Single Instruction Stream, Multiple Data Stream (SIMD)

• Multiple processing units execute the same instruction on distinct data.
• All processors receive the same instruction from a common control unit.
• Instructions can be executed sequentially or in parallel, leveraging pipelining.
• Advantages: Highly efficient for identical operations on large datasets.
• Disadvantages: Limited to specific applications.
• Uses: GPUs, scientific processing.

Multiple Instruction Stream, Single Data Stream (MISD)

• Multiple processors operate on a single data stream, each with its own set of instructions.
• A theoretical construct, no practical implementations.

Multiple Instruction Stream, Multiple Data Stream (MIMD)

• Multiple processors perform operations on several data elements independently or as a part of a shared memory space.
• Multiple instruction streams are executed simultaneously on different data streams.
• A primary organization for multi-processor and multi-computer systems.
• Advantages: Suitable for multitasking.
• Disadvantages: Complex architecture, often costly.
• Uses: Modern PCs, laptops, and smartphones.

Pipelining

• A technique for overlapping instructions in execution.
• An instruction can start processing before the prior instruction is fully complete.
• It speeds up operation in an otherwise non-parallel execution.

Laundry Example

• A practical, relatable example for understanding pipelining in action.
• Sequential washing leads to a much longer overall time than if the wash, dry, and fold tasks are overlapped (pipelined).

Balanced Pipeline

• Ideal pipeline where all stages have equal duration.
• With equal stage durations, tasks are completed far more quickly than the sum of each segment's duration.
• The slowest stage dictates the overall completion time of the task.

Pipelining Terminology

• Latency: The time taken for an instruction to complete.
• Throughput: The number of instructions completed per second.
• Clock cycle: Each computation step (like a clock tick).
• Processor cycle: Time for an instruction to progress one step in a pipeline.
• CPI (Clock Cycles Per Instruction): A measure of the processor's efficiency in executing instructions.

General Considerations for Pipelines

• Operations divisible into similar sub-operations lend themselves to pipelining.
• Pipeline segments (stages) are usually separated by registers that hold intermediate results.
• Clock cycles synchronized to ensure sequential, but overlapped, execution.
• Pipeline structure has potential to provide substantial speedup relative to a non-pipelined approach. But there are factors that can affect the speed achieved in a pipeline architecture.

Considerations for Pipelined Implementation

• Actual execution time is rarely equal to theoretical maximum speed up factors.
• Differences in time to compute a suboperation amongst stages will cause a lower performance than the theoretically maximum theoretical speedup.
• Non-ideal circuit designs may also affect speedup.
• Specific areas of computer design which leverage pipeline organization including arithmetic pipelines and instruction pipelines.