Connectionism and Real Neurons

Connectionism

• Connectionist computation is also known as parallel distributed processing.
• It is also known as artificial neural network (ANN) models.
• Connectionist models are neurally inspired, according to Rumelhart (1989).
• Connectionist models simulate the computational architecture of brains on a computer.
• It is useful to consider whether connectionist models reflect how the brain processes information.
• There is no evidence of backpropagation in the brain.
• Algorithms should be carried out to simulate the cognitive architecture of brains.
• The simulation is functional and algorithmic, not implementational.
• Marr's algorithmic level involves how information is transferred between inputs and outputs.

Real Neurons

• Real neurons pass electrical-chemical signals to other neurons through synapses along their axons.
• Dendrites receive synaptic input signals from axons.
• Signals fired by a neuron's synapses is either excitatory or inhibitory.
  • Excitatory signals promote firing
  • Inhibitory signals inhibit firing.
• The strength of the signal at the dendrite equals the total input received by a neuron.
  • The strength is the value of the incoming signal * the corresponding weight.
• Excitatory signals from synapses have positive weights, while inhibitory signals have negative weights.
• If the total input exceeds the threshold, the neuron will fire.

Artificial Neurons

• Artificial neurons in connectionist models receive input signals from many units/nodes.
• Each input (I) is assigned a weight (W) between -1 and 1
  • This is to simulate the excitatory/inhibitory signals from the synapses.
• For each input, the activation value (WI) is the input value multiplied by the weight (W).
• The level of activation of each neuron is the sum of the activation values of all the inputs.
• A neuron transmits an output signal if the level of activation reaches the threshold.

Output Signals

• Output signals will be transmitted to the next neuron.
• This depends on the output function
  • Linear output: signal increases in direct proportion to total input
  • Threshold linear: no output signal until threshold is reached, then linear
  • Sigmoid: roughly linear between threshold and max firing rate, similar to real neurons
  • Binary threshold: zero output below threshold, max output above threshold (on-off switch)

Network Layers

• Single layer networks exist, but there are also multilayer networks.

Computational Theory of Mind (CTM)

• CTM proposes that mental processes are computational.
• The mind should be seen as software executed by the hardware of the brain
• Mental states are internal symbols that are contentful and representational.
• The formal conception of computation (FCC) states that computations are only sensitive to the syntax of internal symbols.

PSS vs Connectionism

• A comparison exists between PSS (Physical Symbol System) and Connectionism
• Rumelhart (1989) said that all knowledge is in the connections (Textbook, p. 204).
• In a conventional programmable PSS, information is explicitly stored in the states of units and symbols being manipulated.
• In a connectionist model, information is implicitly stored in the structure of the device that carries out the task.
• Information is distributed across the entire network in the patterns of connections between weights/strengths of the connections.
• The connectionist approach says that information processing relies on the interactions among the large numbers of processing units. -Parallelism refers many processing elements carry out computations simultaneously.
  • Information processing is implemented by huge numbers of processing elements, realized by the patterns of connections between the units.
  • There can be different constraints for different processing elements.
• In a PSS, algorithmic rules have serial steps.
• Parallelism implies fast processing and graceful degradation
  • High processing speed includes large numbers of steps and each step takes time.
• Parallelism leads to steps processed simultaneously, saving time.
  • Graceful degradation means damage in some processing elements will not interfere with other processing elements since they work in parallel.
• An ANN learns by changing weights
  • This happens through the development of new connections
  • Loss of existing connections, and/or modification of the connection strengths that already exist
• Learning is successful when the configuration of the network allows it to produce the right outputs.
• A single layer network follows, the delta rule/perceptron learning rule/the perceptron convergence rule.
  • The amount of learning is proportional to the difference between the actual activation and the target activation.
    • δω = ε * (τ₁ - a₁) * o₁ (Textbook, p. 209) where amount of learning.
• Multilayer networks do not know the target activation levels of the units in the hidden layers and cannot calculate the difference in activation levels.
  • Backpropagation is a generalized version of the delta rule.
  • Information is transmitted forwards, with error signals propagated backwards throughout the network.
• Weights are modified on the "second to last layer" and then sent back through another layer.

Programming

• In a PSS, the information carried by the symbols is distinct from the algorithmic rules about how to manipulate those symbols.
• In a connectionist model, connectionist models do not follow programmed rules
• To change the activity of a physical symbol system, the algorithmic rules need to be reprogrammed.
• To change the activity of an artificial neural network, point out where it's going wrong and it should automatically adjust itself.
• In an ANN, for each node (unit):
  • The output signal equals total input signal.
    • The threshold is 1.
    • The node will fire when the total input signal is greater than or equal to the threshold.

AN Gate

• A logic gate that returns true only if all inputs are true.

OR Gate

• A logic gate that returns false if all inputs are false.

XOR Gate

• XOR Gate, the "exclusive or," returns true if and only if one of the inputs are true, and the other is false.
• The XOR problem “cannot be solved by networks that lack hidden units” (Textbook, p. 213).
• This problem is solvable by adding another hidden unit.
  • This hidden unit will take the value 1 if and only if the first two are both 1.

Rumelhart's Examples

• Constraint satisfaction problems exist.
• Pattern Recognition is another Rumelhart example.
• Automatic, similarity-based generalization is also a thing.
  • ANNs can solve constraint-satisfaction problems very well. Constraints and Interactions
• Mutually interacting constraints emerge when feature A is present, feature B is expected (not) to be present.
• One constraint is that each vertex has a single interpretation.
• The other constraint is that the interpretation of any vertex is constrained by the interpretations of its neighbors.
• Constraints can be mathematically translated into positive or negative connections between the units. Each Unit
• Each unit has three positively connected neighbors.
• Each unit has two negatively connected competitors.
• Each unit has one positive input from the stimulus (p. 210).
• Some constraints must be followed.
• The degree of constraint satisfaction/goodness happens when weights modify the system.
• The system will change from satisfying less constraints to satisfying more.
• Stable State
• Eventually, one subsystem will have all its units.
• Other units will have all its units (competitors) not activated.
• Hidden layers, automatically employed learning rules, and configured pattern connections are involved with Pattern recognition
• In the recognition response, patterns become stronger if they are familiar/previously stored.
• The input pattern becomes weaker if unfamiliar.
• Input familiar patterns can be filled in and “assimilated".

Similarity-Based Generalization

Conventional PSS-based Al programs could not modify themselves or adapt to their environments. ANN can: “similarities among patterns are directly represented along with the patterns themselves in the connection weights. ANN can do this in such a way that similar patterns have similar effects” (Textbook, p. 212, emphasis added). Rumelhart's proposal says ANNs automatically make correct similarity-based generalization about the environment.

• Similar input patterns are mapped to similar output patterns.
• Some hidden layers serve a similar function to “internal representations".
• The XOR problem exists when the added hidden unit and the weights function to "represent" a constraint from the environment
• For example, when the two input stimuli are both present, the output pattern should be 0.
• The learning procedure involves two stages
• The ANN receives an input.
• After a period of time, certain units of the ANN receive target values.
• If target values are actually attained, then the weights between those units and their input units remain unchanged
• Otherwise, the weights are modified (hidden layer!).
• In backpropagation, this learning procedure starts from the output units.

Classical vs Connectionism

• Implementational levels differentiate classical architectures from connectionism
• Arguments against Classical cognitive architecture depend on properties that are not intrinsic to Classical architecture.
• Classical architectures implement on current computers, but "need not be true of such architectures when differently (e.g., neurally) implemented".
• ANNs are alternative to PSS on the algorithmic level (Slides #3-4)!
• All ANNs are implemented on computers with the von Neumann architecture.