Computer Memory Systems

Memory Systems

• The performance gap is measured as the difference in time between processor memory requests and the latency of a DRAM access.
• It is not possible to make all memory on a computer from the same CPU technology because of its high cost and volatility.
• A lot of information needs to persist even after the machine is powered off, which requires non-volatile memory.

Principle of Locality

• Programs tend to reuse both data and instructions they have used recently, which is the most important program property regularly exploited.
• The rule of thumb is that a program spends 90% of its execution time in only 10% of the code, which is also applicable to data, but not as strong as to code instructions.
• It is possible to predict, with reasonable accuracy, what instructions and data a given program will use in the near future based on its accesses in the recent past.

Memory Hierarchy

• The visible memory levels range from the lowest level to the processor, with size decreasing from TB to Bytes, speed increasing from µs to ps, and cost increasing.
• The hierarchy includes main memory, secondary memory, and cache memory.

Cache Memory

• Cache stores copies of regions from the main memory that are frequently used by the CPU for a short time period.
• Cache is transparent to programmers, even in assembly language, and is crucial for multiprocessors to control access and keep cache coherence.
• Cache is divided into data cache and instruction cache, and can be further divided into hierarchical cache levels (e.g., L1, L2, L3).
• Cache hit occurs when the processor finds a requested data in the cache, while cache miss occurs when it does not find the data item it needs in the cache.

Cache Memory Organization

• Cache can be organized as direct-mapped, fully associative, or set associative.
• In direct-mapped cache, hardware decisions are simplified, and there is no choice for block replacement.
• In fully associative or set associative cache, there are three main strategies for block replacement: random, least recently used (LRU), and others.

Cache Memory Operation

• The levels of memory hierarchy are divided into blocks (or lines), and every access request is presented first to the highest level.
• If the requested data is found in the highest level, access to the lower level is avoided; otherwise, the requested word (actually a block containing the word) is fetched from the lower to the higher level.

Cache Memory Questions

• Four questions about cache memory hierarchy:
  1. Block offset calculation: [4:0] offset = (M) = 5 bits.
  2. Block identification: using tags and block offset to identify cache blocks.
  3. Block replacement: strategies for replacing blocks in cache memory, including direct-mapped, fully associative, and set associative.
  4. Cache organization: cache organization and its impact on cache performance.

Memory Hierarchy

• Fast memory is more expensive, so a memory hierarchy is organized into several levels, each smaller, faster, and more expensive per byte than the next lower level.
• The goal of the memory hierarchy is to provide a memory system with a cost/byte that is almost as low as the cheapest level of memory, and a speed almost as fast as the fastest level.

Inclusion Property

• In most cases, the data contained in a lower level are a superset of the next higher level.
• Inclusion property is always required for the lowest level of the hierarchy, such as main memory in the case of caches and secondary storage (disk or flash) in the case of virtual memory.

Cache Memory

• Cache memory is divided into blocks with a fixed number of memory words.
• The cache address is divided into two parts: block address (tag) and where the block can be placed in the upper level.

Block Placement

• There are three categories of cache organization: fully associative, direct mapped, and set associative.
• Fully associative cache allows a block to be placed anywhere in the cache, with the place usually defined by the replacing policy.
• Direct mapped cache means each block has only one place it can appear in the cache, with the mapping given by (Block Address) mod (# of Blocks in Cache).
• Set associative cache means a block can be placed in a restricted set of places in the cache, with the block first mapped onto a set and then placed anywhere within that set according to (Block Address) mod (# of Sets in Cache).

Block Identification

• Block identification is related to finding a block in the upper level if it is present.
• It involves dividing the cache address into two parts: block address (tag) and where the block can be placed in the upper level.