Information Theory Basics

• Information theory quantifies information flow across tasks and measures human information processing efficiency.
• Information reduces uncertainty about the state of the world.

Transmission of Information

• Humans are information channels.
• H represents information at different points:
  • Hs is stimulus information.
  • HR is response information.
  • HT is information transmitted by the operator.
  • HL is information lost by the operator.
  • HSR is information dispersion.
  • Noise is non-relevant information.
• In an ideal human information channel, no information is lost (HS = HT = HR).
• More realistically, human channels have noise and information loss, meaning HT < HS or even HT = 0.

Units and Quantity of Information

• Bits are the units of information ("binary digit"), with a value of 0 or 1.
• A bit is the amount of information in response to a two-alternative-forced-choice (2AFC) question.
• Signal detection is a 1-bit choice of signal presence or absence.
• The quantity of information depends on the number of possible events (N), the likelihood of those events (P), and sequential constraints (context).
• Sequential constraints consider contingent probabilities (Pi/X, the probability of event i given context X).

Calculating Information

• Assuming one needs to know the current weather, with four possibilities (clear, cloudy, rainy and snowy).
• The quantity of that information is the average minimum number of true-false (yes-no) questions needed.
• Two possible states: raining or not-raining, assuming that the base probability of rain is 0.5 or 50%.
  • N = 2
  • Hs = log2(2) = 1 bit of information
• Four possibilities: Raining, Overcast, Partly Cloudy, & Clear assuming that the base probability of each is 0.25 or 25%.
  • N = 4
  • Hs = log2(4) = 2 bits of information
• Rare events convey more information, while likely events convey less

Equations and Examples

• Total stimulus information (Hs) in bits: Hs = log2 N. For N equal probability events:
  • Have = ( Σ HSi ) / N = HS.
• For N events with unequal probabilities:
  • Have = Σ [ Pi log2 (1/Pi) ], a "weighted average" based on probability.
• Average information, Have, decreases when event probabilities are unequal.
• Formulas using contingent probabilities:
  • HS = log2 (1 / (Pi/X)).
  • Have = Σ [(Pi/X) log2 (1 / (Pi/X))].

Information Redundancy

• Happens when events aren't equally probable or have sequential constraints
• The percentage loss of information is: % redundancy = [1 - (Have / Hmax)] * 100
• Making probabilities unequal/adding sequential constraints shrinks information.
• In English, not all letters occur with equal frequency, and sequential constraints exist.
• In an English language example redundancy is (1 - 1.5/4.7)*100% = 68%

Human Information Channels Quantified

• For a group of events, HS and HR are determined using the equation for Have
• HT is determined by: HT = HS + HR - HSR
• HSR represents the dispersion of stimulus-response relationships

Determining HSR and HT

• Compute the average of values within the matrix.
• Dispersion = count the number of squares that something happens.
• HSR is determined by log2 of the number of squares.
• HT = HS + HR – HSR

Limitations of Information Theory

• HT reflects the consistency of stimulus-response mappings, not accuracy or appropriateness.
• HT doesn't account for the size of an error.