The Hebb rule

The use of (1) which is an algorithm or recipe for fixing the weights without adapting to a training set may appear to run counter to the ideas being promoted in the connectionist cause. It is possible, however, to view the prescription in (1) as a short-cut to a process of adaptation which would take place if we were to obey the following training algorithm

1. present the components of one of the patterns to be stored at the outputs of the corresponding nodes of the net.
2. If two nodes have the same value then make a small positive increment to the inter-node weight. If they have opposite values then make a small negative decrement to the weight.

where, as usual is a rate constant and . It is clear that the storage prescription is just the result of adding all the weight changes that would accumulate under this scheme if enough pattern presentations were made. The rule in (2) is one of a family of rules known as Hebb rules after D. O. Hebb. The characteristic of such a rule is that it involves the product of a pair of node activations or outputs.

As a variation, suppose we had used the usual Boolean representation for the components so that is 0 or 1 The Hebb rule would now be . Interpreting this, we have that the change in weight is only ever positive and only occurs if both nodes are firing (output `1'). This is, in fact closer to the original rule proposed by Hebb (1949) in a neurophysiological context. In his book The Organization of behaviour Hebb postulated that

When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased.