Memory in Neural Networks and Glasses

The thesis tries and models a neural network in a way which, at essential points, is biologically realistic. In a biological context, the changes of the synapses of the neural network are most often described by what is called `Hebb's learning rule'. On careful analysis it is, in fact, nothing but a qualitative statement allowing for as many as 81 different interpretations, all of which differ with respect to the question of how the synapses change as a function of dendritic input and axonal output. In the second chapter of the thesis, a more detailed analysis of Hebb's postulate shows that of these 81 possible interpretations only two survive, if a simple, biologically plausible, argument is used.
In the second part of chapter 2, a physical derivation is given of Hebb's rule. Starting point for this derivation is an expression for the biochemical energy needed to change a synapse. Then Hebb's rule is found by requiring that changes of the synapses take place in an economical way, so that a minimal amount of energy is spent. The derivation of the Hebb rule as performed is `better' than any other derivation that can be encountered in the literature, where the Hebb rule is derived from a `cost function', since such a cost function is construed in such a way that it yields the `right', i.e., 'guessed to be right' Hebb rule one is after. It is comparable to the way the Einstein equations can be found from an action principle: the action then is chosen in such a way that the Einstein equations follow from it. Therefore, any derivation based on a cost function or an action is nothing more than postulating the final result in a disguised form. This is in contrast to our derivation of the Hebb rule, which starts from a biologically plausible assumption of economic use of the available biochemical energy in the physically ever changing system called brain.
All Hebb rules ever derived in the literature from a cost function did not
ever lead to the Hebb rule derived from the principle of minimal use of energy used in the present thesis. Apparently, the Hebb rule which the author of the thesis considers to be the actual rule, was considered to be inprobable by all his predecessors.
The particular Hebb rule used in the thesis will be referred to as the `energy saving Hebb rule', in order to distinguish it from other rules found and used in the existing literature. In the technical terms of the subject, the energy saving Hebb rule can be stated to be a local, mixed Hebbian--Anti-Hebbian learning rule, which is asymmetric with respect to pre- and post-synaptic activity. The mathematically derived energy saving learning rule, as a surprise, encompasses precisely the two rules which one was left with after the careful but naive inspection mentioned above of the 81 a priori possible interpretations of Hebb's original postulate, an inspection that was based on biological plausibility only.
The energy saving learning rule as found in the thesis still is a quantitative prescription, albeit that there remain two unknown parameters, not easily accessible within the chosen theoretical framework. These parameters are: an overall factor $\eta$ (the so-called learning rate), and a constant $\kappa$ (the so-called margin parameter). Setting apart these two fparameters, the energy saving learning rule can be characterized as quantitative, whereas Hebb's original learning rule certainly is qualitative only.
In the third and last part of chapter 2 an analytical formula for the strengths $w_{ij}$ of the synaptic connections is derived, supposing that the synaptic
adaptation takes place according to the energy saving learning rule.
The learning procedure used in this derivation is known to yield a result
which happens to coincide with what essentially is the method that in the mathematical literature goes under the name of `pseudo-inverse solution'. However, for a biological neural net the original formulation of this
well-known method of the pseudo-inverse could not be used directly. In the appendix of chapter 2 a modification of the method of the pseudo-inverse
is be presented which overcomes all problems resulting from biologically induced particularities of the problem.
In the following two chapters, chapters 3 and 4, the study of chapter 2 is deepened. Firstly, it is taken into account that not only ideal, unperturbed,
patterns are learned by the neural net: realistic input data are noisy, i.e., the set of input patterns is enlarged. Secondly, one tries to get some grip
and understanding of the margin parameter $\kappa$ and the learning rate $\eta$ occurring in the energy saving learning rule.
In chapter 3, the following question is asked, and answered: what are the most suitable values to be assigned to the weights of a neural network. The result turns out to be a generalization of the expression for the weights after a learning process with the energy saving learning rule of chapter 2. This is unexpected, because it was found independently of any learning rule, but it certainly is an encouraging surprise. Apparently, the biological learning rule leads, after repeated application, to the result that one should find from a mathematical point of view.
In chapter 4 learning in a neural network is considered again, just as in chapter 2, but now for noisy instead of fixed patterns. Via a Discrete Time
Master Equation, the the process of adaptation of the synapses is described theoretically, using the energy saving learning rule. The final result is an
expression for the weights $w_{ij}$, which, for unperturbed input, reduces to the expressions for $w_{ij}$ found in chapter 2, as it should.
For noisy input there is a slight difference with the mathematical result of chapter 2. This is due to the fact that, in an actual neural network, the
synapses will always change a little bit, so that fixed patterns have a non-fixed representation in the mind. The mathematical approach of chapter 2 was not yet devised to take into account this subtle, time-dependent effect.
The research of the chapters 2-4 of the thesis, which treat biological neural networks, can be summarized by stating that Hebb's postulate has been derived ---for the first time, and half a century after its formulation--- from a physical principle involving only economy of consumption of energy. The resulting Hebb rule, which was called 'energy saving rule' is much more precise, and almost quantitative, in contrast to its 1949 predecessor.
In the final chapter 5 there is a switch from neural networks to related problems, namely the theory of spin-glasses. Both neural networks and spin-glasses possess a highly complex phase-space, which can be studied in many different ways. One of them is the so-called method of damage spreading, in which two identical systems with a different initial spin-configuration, are followed in time. It is shown that in a specific model, the spherical p-spin model, there exists a critical temperature which separates two different dynamical regimes. Moreover, a theoretical explanation is presented for the observed behavior of this spin-system. Although this chapter is less, or perhaps even totally non-biological, the concept of damage spreading is a useful tool to gain more insight in complex systems, among which biological neural networks.