Memory Hierarchy and Cache

• The arrangement of various types of storage in a computing system is based on access speed.

Memory Hierarchy Structure

• Computer storage is organized according to response time
• There are two types: primary/internal and secondary/external
• Primary/internal memory can be directly accessed by the processor
• Secondary/external memory can be accessed by the processor via I/O Modules

Cache Memory

• It stands as the fastest memory type and sits closest to the CPU.
• This memory stores frequently accessed data and instructions, making their retrieval faster than from main memory.

Three Levels of Cache Memory

• L1 Cache (Level 1): Smallest and closest to the CPU, embedded within the CPU chip, and stores the most frequently accessed data
• L2 Cache (Level 2): Larger and slower than L1, located on the CPU chip or connected via a high-speed bus, stores less frequently accessed data than L1 but more than L3
• L3 Cache (Level 3): Largest and slowest, shared by multiple CPU cores, located on the motherboard, and stores less frequently accessed data than L2 but more than RAM

Cache Memory Mapping Techniques

• Direct Mapping is know as cache collisions
• Uses Associative Mapping which allows any memory block to be placed in any cache block
• Uses Associative Mapping which combines the advantages of direct and fully associative

Cache Replacement Algorithms

• First-In, First-Out (FIFO) which replaces the oldest memory block in the cache
• Least Recently Used (LRU) which replaces the memory block that was accessed least recently
• Random Replacement (RR) algorithm randomly selects a memory block for eviction

Advantages of Cache Memory

• Faster Data Access
• Reduced Memory Traffic
• Improved Overall Performance
• Enhanced Power Efficiency:
• Flexibility in System Design

Disadvantages of Cache Memory

• Increased Complexity
• Limited Storage Capacity
• Increased Cost
• Cache Coherence Overhead

Main Memory

• Primary memory of a computer system
• Slower than cache memory but faster than secondary storage

Main Memory Characteristics

• Directly Accessible by CPU which allows for rapid data retrieval and processing
• Volatile, losing its data when power is off
• High Speed is crucial for CPU operations
• Limited Capacity compared to secondary storage

Main Memory Types

• Dynamic Random Access Memory (RAM) is the type of RAM used in modern computers
• Static Random Access Memory (SRAM) uses latches to store data and doesn't need to be periodically refreshed

Main Memory Role

• When programs launch, instructions and data load into main memory for program execution
• It provides data storage for CPU-active data
• Facilitates communication with secondary storage by acting as temporary area for data transfer

Main Memory vs Secondary Memory

• Primary memory is faster, has less capacity, and is more volatile than secondary memory

Secondary Storage

• Enables long-term storage for data exceeding main memory's capacity
• Slower than main memory, but provides larger capacity

What is Secondary Storage?

• Non-volatile data persists when device is powered off
• Uses less frequent access where data is accessed less often than primary storage (RAM)
• Lower Cost and more affordable compared to primary storage per unit of storage
• Larger Capacity and stores significantly more data than primary storage

Why Use Secondary Storage:

• Enables crucial backup and disaster recovery to ensure data recovery in case of hardware failure, software errors, or natural disasters
• Archiving for preserving legal, regulatory, or historical data
• Storage of organization-stored data that is not accessed frequently but still needs to be readily available

Types of Secondary Storage

• Hard Disk Drives (HDDs)
• Solid-State Drives (SSDs)
• Cloud Storage
• Optical Discs (CDs, DVDs, Blu-ray)

Importance of the Memory Hierarchy

• Memory hierarchies play a crucial role in optimizing computer performance
• By using faster, smaller and more expensive primary storage and slower, cheaper secondary storage, access time is reduced
• Leads to faster instruction, execution, and data manipulation

Registers

• Smallest and fastest form of memory, existing directly inside the CPU
• Store small amounts of frequently accessed data

Register Characteristics

• High Speed with fast access times measured in CPU cycles
• Small Capacity holds a limited, fixed amount of data usually 8, 16, 32, or 64 bits
• High Cost per Bit due to the high speed and specialized technology

Importance of Registers

• Enables fast data access
• Enables complex operations quickly and efficiently for the CUP
• Enables CPU to decode and execute instructions rapidly which leading to faster program execution

Types of Registers

• DR (Data Register) holds memory operand
• AR (Address Register) holds address for memory
• AC (Accumulator) is the Processor Register
• IR (Instruction Register) holds Instruction Codes
• PC (Program Counter) holds Address of instruction
• TR (Temporary Register) holds temporary data
• INPR (Input Register) holds input character
• OUTR (Output Register) holds output character

Magnetic Disks

• High storage capacity and both sequential and random access
• Cost-effective bulk data storage for various applications

Disk Performance Parameters

• Seek time refers to the time required for the read/write head to reach the desired track
• Rotational Latency refers to the time it takes for the desired sector to rotate under the read/write head
• Data Transfer Rate is the amount of data under the read/write
• Controller overhead is the overhead imposed by the disk controller

Magnetic Disk Advantages

• Uses quick Access Time which is suitable for applications demanding immediate data access, such as databases
• Uses Large Storage Capacity offering higher storage capacities at a lower cost
• Are Cost-Effective, often more affordable than SSDs and offer attractive budget option

Magnetic Disk Disadvantages

• Offers Reliability which is highly reliable for long-term data preservation
• Uses slower Speed which exhibits slower read and write speeds, making them less suitable requiring fast data access
• Requires a large Physical Space taking up more physical space than SSDs
• Are Prone to Failure where data can degrade

Magnetic Disk Applications

• Enterprise Storage: Used in storing databases and applications.
• Backup Storage: create backups of data
• Archiving: Used in data archiving
• Server Architectures: Used in cloud computing, virtualization

Magnetic Tape

• Defined as a sequential data storage medium that consists of a long plastic ribbon coated with magnetic material
• For archival data and backup purposes

Magnetic Tape Applications

• Audio: used in audio recording and playback devices.
• Video where video formats such as VHS and Betamax rely on magnetic tape
• Computer Data for data archiving and exchange

IBM's Role in Magnetic Tape

• IBM invented the current standard and in the early 1950's created reliable and high-speed tape drive
• In the 1980s IBM transitioned from large reels to compact, cartridge based tapes.

Example of a Modern Computer Memory Hierarchy Organization

• Registers: Immediate use of holding data and instructions
• Cache: multiple levels of cache
• L1 Cache: built directly into the CPU and stores data currently being used
• L2 Cache: motherboard located and stores instructions likely to be used soon
• L3 Cache: shared by multiple CPU cores and stores data that might be needed in the near future
• Main Memory (RAM): operating system for programs
• HDD or SSD: applications and user files
• Tertiary storage: external backup storage

How the memory hierarchy works

• Functions based on the principle of locality of reference
• Cache checks memory first, then CPU retrieves data, otherwise uses main memory

Important of the Memory Hierarchy

• Speeds up data access
• Increases capacity memory and data
• Increases optimization of cost

Real World Example of Memory Hierarchy:

• Your Hard drive is storing the photo you're working on
• The photo is loaded into ram
• Portions from the photo are copied into the CPU's Cache

Cache Design Fundamentals

• Serves as an important tool for today's modern technology
• Bridges the speed gap between the processor and main memory

Cache Design Parameters

• Cache Capacity is defining how much data can be stored
• Block Size is the amount of data transferred between main memory and cache
• Associativity is Determines the amount of locations
• Access Time defines how much time it takes to retrieve cache data

Cache Performance Metrics

• Cache Hits are successful access to the requested cache
• Cache Misses happen when you can't access the cache which results in slower access to memory
• AMAT is calculated as: Cache Hit Rate x Hit Latency + (1 - Cache Hit Rate) x Miss Penalty

Cache Performance Example:

• Serve Product Images to Customers
• Cache system is 1GB of size
• Website will use the following metrics:
• Cache Hit Rate
• Lower Cache Latency.
• Percentage Miss rate

Factors Influencing Cache Performance

• Cache Size: A larger cache enables the ability to store more data
• Associativity: Conflict is reduced but is higher in complexity
• Replacement Policy: Defines evicted data, commonly uses LRU, LFU and Random Replacement.
• Block Size: Optimizes spatial locality, affecting the reduction of misses
• Stack Distance Profiling: Provides information into misses and temporary re-use.

Types of Cache Misses

• Compulsory that occurs on the first block to access the cache
• Conflict that occurs on eviction leading to storage
• Capacity that occurs when the working directory exceeds access cache
• Coherence happens hen a line is validated over a multi-access
• Coverage is when the directory can't track a limit of capacity
• System related that happens in system alters like memory, interrupts, etc.

How to Measure Cache Performance

• Bench Marking is the evaluation of CPU and STREAM
• Simulation the models to create access for data
• Profiling analyzes the VTune causes.
• Instrumentation access for data and statistics

Cache Strategies to help data access

• Choosing Cache Size and Associativity to based on the work load of specific sources/loads
• Selecting a policy of access for data
• Optimizing, aligning cache data and improving the local vicinity.

Replacement and Write Policies

• Managing how data is accessed, handled and managed.

Common Policies Include

• Least Recently Used that isn't for a long period, or what is likely is to be used again
• First in First Out to evict the data
• Least Frequently used or the one needed the least

Writing a Policy

• Helps the access of system memory and modifications
• Write through or data written through the memory and cache
• Allocate and set it for operations
• System Access helps choose from access writing

Virtual Memory

• It provides management for efficient memory
• A memory that enables functions and access for data
• Allocates spaces, has protection against vulnerabilities
• Address spaces even with limited memory.

Benefits of Virtual Memory

• Increased multi-programming
• Simplified Memory Allocation
• Enhanced security through Isolation
• Has large address spaces

Address Translation

• Converts or assigns address' for virtual use
• Facilitate data with organized systems

Page Tables and Page Table entries

• Has protection and its own PTE's
• PFN virtual pages
• Includes all controls and permissions

Page Faults

• Systems are instructed to handle the faults by available access
• Maps updating through new access
• Can recover if the system fails

Translation Lookaside Buffers

• Accelerate translated address and stores used entries
• Avoid costing from walking through the memory process
• Effects what the performance and access is

Design Impact

• Reduces affects and improves data but complexity is more difficult
• TLB's associative best flexibility needs additional hardware
• Needs to add and access from the TLB list
• Performance to the computer relies on the machine type.

Virtual Memory and Interactions

• Work to provide efficiency on the system and what instructions they receive
• Physical access or TLB data system
• Transfers requests to data.

Considerations to remember

• Stalling is one important effect on stall speed and the over all affect
• Improves through association and performance.
• Effective pages need to be accessed in order improve the process.