In a second important paper [Hopfield, 1984] Hopfield introduced
a variant of the discrete time model discussed so far which uses
nodes described by their rate of change of activation. This kind
of node was discussed in the last part of lecture 1 but we review
it here. Denote the sum of excitation from other nodes for the
j node by so that
then the rate of change of activation is given by
here, k and c are constant. The first term (if positive)
will tend to make the activation increase while the second term is a
decay term (see lecture 1). The output is then just the sigmoid
of the activation as usual. Hopfield also introduced the possibility
of external input at this stage and a variable threshold.
In the previous TLU model, the possible states of an N node net are just the corners of the N-dimensional hypercube. In the new model, because the outputs can take any values between 0 and 1, the possible states include now, the interior of the hypercube. Hopfield defined an energy function for the new network and showed that if the inputs and thresholds were set to zero, as in the TLU discrete time model, and if the sigmoid was quite `steep', then the energy minima were confined to regions close to the corners of the hypercube and these corresponded to the energy minima of the old model.
There, however, are two advantages of the new model. The first, is that the use of the sigmoid and time integration make more contact possible with real biological neurons. The second is that it is possible to build the new neurons out of simple, readily available hardware. In fact, Hopfield writes the equation for the dynamics - eqn (4) - as if it were built from an operational amplifier and resistor network. This kind of circuit was the basis of several implementations - see for example Graf et al. (1987).