HAL Id: jpa-00211062
https://hal.archives-ouvertes.fr/jpa-00211062
Submitted on 1 Jan 1989
HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.
Dynamical phase transitions in short-ranged and long-ranged neural network models
K.E. Kürten
To cite this version:
K.E. Kürten. Dynamical phase transitions in short-ranged and long-ranged neural network mod- els. Journal de Physique, 1989, 50 (17), pp.2313-2323. �10.1051/jphys:0198900500170231300�. �jpa- 00211062�
Dynamical phase
transitions inshort-ranged
andlong-ranged
neural network models
K. E. Kürten
Institut fur Neuroinformatik, Ruhr-Universitdt Bochum, Universitdtsstr. 150, 4630 Bochum, F.R.G.
Institut fur Theoretische Physik, Zülpicher Str. 77, 5000 Köln, F.R.G.
(Reçu le 22 mai 1989, accepté le 25 mai 1989)
Résumé. 2014 Nous comparons la période des cycles limites, l’évolution temporelle de la distance entre deux configurations initiales et la distribution d’activité locale pour des automates cellulaires en dimension finie et infinie. Nous complétons le critère de la distance de Hamming,
un outil puissant et bien établi pour la localisation des paramètres critiques, par des critères
incorporant des quantités telles que la longueur moyenne des modes cycliques et la distribution d’activité locale qui, chacune, peuvent distinguer entre les phases. A cause de leur structure spatialement plus corrélée, les modèles de réseaux avec interaction de courte portée ont un comportement beaucoup plus ordonné que leurs équivalents avec portée infinie.
Abstract. 2014 The period of limit cycles, the time evolution of the distance between two initial
configurations and the local activity distribution are compared in finite- and infinite dimension threshold automata. The Hamming distance criterium, well established as a powerful tool for the
localization of critical parameters, is complemented by criteria incorporating quantities as the
average length of the cyclic modes as well as the local activity distribution which both can
distinguish between two phases : an ordered phase and a chaotic phase. Due to their strongly correlated spatial structure short-ranged lattice models exhibit by far more ordered behavior than their infinite-range counterparts.
Classification
Physics Abstracts
87.30 - 75.10H - 64.60
1. Introduction.
It is well known that the dynamical properties and the underlying mathematical structure of disordered cellular automata are a suitable instrument to simulate a variety of complex
behaviors found in physical, biological and computational systems [1]. Analysis of general
features of their behavior may therefore yield general principles and universal results on the emergence of many complex phenomena. This paper concentrates on a modification of a class of threshold automata of the McCulloch and Pitts [2] type widely used in the description of
neural network models. Diluted and randomly connected infinite-range models, where each
unit is allowed to have an infinite range of interaction, have extensively been studied in recent
years. Unexpectedly, it turned out that in contrast to totally connected and symmetric models
those are exactly soluble in the thermodynamic limit (nu oo ) [3]. On the other hand, for models based on geometrically correlated couplings such as nearest-neighbor or distance dependent « local » interactions [4] to date no analytical solutions have been found. Thus, the
effects of spatial structures, a subject of a series of recent studies of a modified Kauffman model [5] have to be explored by computer simulations.
Article published online by EDP Sciences and available at http://dx.doi.org/10.1051/jphys:0198900500170231300
Though infinite range models are more attuned to biological reality [6] the actual interest in various aspects of parallel distributed processing models gives finite dimension studies their
own right. The prospects of easy software and direct hardware realization of short-ranged
models capable to display reasonable cognitive behavior, particularly in the field of computer vision, suggest strongly practical nearest-neighbor implementations in « neural computers » with parallel architectures. Here, as well as in biological applications, the investigation of stability against minor changes of network parameters is fundamental.
2. Formalism.
2.1 THE MODEL IN INFINITE DIMENSION. - In general, cellular automata are considered as
discrete dynamical systems consisting of N interacting formal « binary units » comprising the
network. The state variable of unit i at discrete time steps t denoted as ai (t ) taking either the
value + 1 or - 1 is determined from an arbitrary Boolean transition function fi of
K input variables, where the connectivity parameter K denotes an integer between one and
N so that each unit receives exactly K input lines and on the average each unit are assigned
K output lines. The time evolution for each unit of the system is then given by the
N independently chosen functions fi of the K input variables uj¡(1)’ ..., UjK(i) as follows
The randomness entering the model is twofold : the K input variables to each unit as well as
the transition functions fi are chosen at random. Specified at time t = 0, they remain fixed for
ever and the units update their state in parallel according to the values of their input units at
the previous time step. We will refer to this prescription as the infinite dimension limit, since
the interaction range of each unit is not necessarily local but can be infinite. The quantity
T may be interpreted as the propagation time for signal transmission from one unit to another.
In the Kauffman model the functions fi are determined by randomly chosen arbitrary
Boolean functions of randomly chosen K distinct sites, whereas in models of threshold automata and neural networks, the functions fi can be defined by the prescription
where j only runs over the K inputs of cell i. The coupling coefficients cij usually not being symmetric are sampled randomly from a given distribution p (Cij) so that a fraction
q of the set of the nonzero couplings is taken negative (inhibitory), the remainder positive (excitatory). Throughout this study the value of the network parameter q will be chosen as one half and without loss of generality the distribution of the coupling coefficients is supposed
to be uniform in the interval [-1,1 ]. According to (2) each unit undergoes a threshold test at
clocked time steps with a threshold value h so that the state of the system at time
t is uniquely determined by the state vector u (t). The dynamics is parallel, synchronous and fully deterministic.
2.2 THE MODEL IN FINITE DIMENSION. - The « mean field » neural network model described in section 2.1 can be easily modified to a finite dimensional one by injecting some two-
dimensional spatial structure. One route to generalize the model is to let the generating
entities reside on the sites of a regular lattice so that each site represents a cell, a spin or a
formal neuron. A suitable neighborhood of each cell can then be defined by the three nearest neighbors on a honeycomb lattice (K = 3 ) or by the four orthogonal neighbors on a square lattice (K = 4), the so called Von Neumann neighborhood. One way to increase the
connectivity to K = 5 is to add a dependence on the cell itself so that the value of the updated
state depends also on the current state of the central cell. Then, the input cells defined in (1)
are nearest neighbors of cell i on a regular lattice, the transition functions fi are the usual
threshold functions of a linear combination of the weighted nearest neighbor states and the
threshold h.
The biological relevance of some models based on nearest-neighbor interactions seems
somewhat questionable. Little is known for example about the anatomy and physiology of large portions of the cerebral cortex, though Golgi-stained nerve tissues in various cortical
areas suggest that local neural organization favors the radial direction. Thus, locally distance- dependent interactions are claimed to be meaningful in a simplified « mean anatomy » description [8] of spatio-temporal activity in the cerebral cortex modelling the propagation of
excitations within « homogeneous » cellular layers. However, to be more precise, even parts of biological brains are extended systems and it is well known that they possess far more than nearest neighbor connections. In fact, biological genes, lymphocytes and neurons have an
infinite interaction range which has been taken into account in the genuine Kauffman model
[9] and in the pioneering work of McCulloch and Pitts [2], though the biological reality could
be in general intermediate. In this sense, cells rigorously fixed to the sites of regular lattices
contradict biological findings, since spatial structures or even spatial distances do not seem to be most relevant for the functioning of biological networks. However, from its very origins, theory of cellular automata was intended to model both, living systems and machines, and
with respect to useful practical implementations, an interaction range limited to a near
neighborhood finds extensive application to image processing and visual pattern recognition [7]. On the other hand, an interaction range over the whole spectrum of inputs, biologically as
unrealistic as exclusively short-ranged interactions, leads to totally connected networks which
serve successfully as content addressable memories [10].
3. Periods.
Cycling behavior is a direct consequence of the finite state set and the deterministic nature of
our model. Since there exist only 2N different configurations in phase-space, the state of a
finite system will inevitably evolve through a transient phase in a deterministic manner
towards an attractor, either a limit cycle or a fixed point. Locked into a cyclic mode, the same
set of states is repeated indefinitely, in the same order. Some aspects of cycling phenomena
are of neurobiological and genetic interest. In neural network models limit cycles have often
been considered as model analogues of active short-term memory traces or thoughts [11],
whereas in an evolutionary model for cell differentiation Kauffman relates the total number of cyclic modes to the highly limited number of different cell types in living organisms [9].
Since cycling activity can also be interpreted as the reponse of the network to extemal stimuli
represented by the initial configuration systematic investigation of the nature of the attractors
is crucial for the problem of visual pattern recognition, processed by the dynamical time
evolution. We are especially concerned with the length of the periods of the accessible limit
cycles, the distribution of their periods and their relative stabilities. A range of experiments is presented with a view to compare the cycling properties of two fundamental classes of cellular automata in order to find any prominent trends in their behavior as various gross parameters of the network are varied.
For the infinite dimension limit, where K distinct input cells are randomly chosen among the N cells of the system, the length of the periods of the attractors have been studied by computer simulations [11, 12]. Two regimes have been found : an ordered phase, where the
average period Lp (N ) increases with a power of the total number of cells N
and a chaotic phase, where the length of the period Lp (N ) varies exponentially with the
system size N
The coefficients a and J3 depend on the connectivity parameter K, the threshold h and the distribution of the coupling coefficients. Note that even for almost identical initial conditions different attractors can be reached depending on the complexity of the basins of attraction. This effect is quite common in the chaotic phase characterized by an extreme sensitivity to the initial condition. On the contrary, in the ordered phase two almost identical
initial conditions usually evolve to the same attractor which makes the ordered phase quite
stable against local damages. Note that these features are very reminiscent of the occurrence
of chaos in continuous time models [13]. Figure la provides information about the
Fig. 1. - a) Distribution of the limit cycle length L for N = 49 cells with K = 3 randomly interacting neighbors for 100 000 specimen nets. Each net was tested with one randomly chosen initial condition. b)
Same for the honeycomb lattice.
distribution of the limit cycle lengths in the infinite-range model. Strongly skewed to shorter lengths, the even-valued period lengths are qualitatively comparable with a 1/L distribution apart from local deviations at the smallest cycle lengths. This observation is also consistent for the odd-valued cycle lengths.
As for the honeycomb lattice model figure 1b shows that the odd-valued cycle lengths essentially disappear and multiples of period lengths 4 become dominant. Figure 2 shows the
average length of the cyclic modes for the honeycomb lattice (K = 3 ) and the two square lattice models (K = 4 and K = 5 ), respectively. As has been verified numerically the
variation of the mean cycle length is exponential for K = 4 and K = 5, whereas for
K = 3 the increase seems to be linear.
As will be demonstrated by a stability analysis in the next section the honeycomb lattice (K = 3) clearly does not exhibit chaos, even at zero threshold, where the highest degree of
Fig. 2. - Mean cycle length as a function of N for (a) the honeycomb lattice model K = 3 and (b) the
two square lattice versions K = 4 and K = 5 (from below) at zero threshold.
disorder is to be expected. Moreover, computer simulations of a modified Kauffman model
on a two-dimensional honeycomb lattice are in accord with these results [7]. This finding
stands in marked contrast to the corresponding infinite-range model with randomly chosen neighbors which exhibits a sharp phase transition from an ordered behavior for K * 2 to a
chaotic behavior for K > 2 [3, 4]. There, the prevalence of the chaotic phase has been well established by analytical results in the thermodynamic limit [12]. In the case of the square lattice with four nearest-neighbor interactions (K = 4 ) large fluctuations indicate that our
system seems to be slightly above the critical value of K, the average period already increasing exponentially. It is interesting to note that for various thresholds in the K = 5 lattice model the variation of the mean cycle length turns out to be always linear in the ordered phase. This
stands in contrast to infinite range models and a nearest-neighbor Kauffman modification where usually a small power law increase has been found [5, 12], except at a critical value,
where the increase turned out to be also linear [12].
4. Stability.
According to the disturbance imposed on a system there are quite a few practical definitions for the stability of an attractor being proportional to the size of its basin of attraction. A disturbance might consist of incorporating various sorts of « noise » affecting the transition function in (1) at some arbitrarily selected time step in the course of a limit cycle. An
alternative less elaborate approach because of its technical simplicity is to invert the state
variable of a single cell at the initial condition. One can go further to investigate stability
under simultaneous flips of the states of two or more cells. In order to study how a single or microscopic damage spreads through a system and affects its dynamical evolution it is convenient to introduce a useful metric, the normalized Hamming distance H(t), the complement of the spin-glass « overlap » order parameter, which represents the fraction of cells having different settings in two states (T (1)(t) and (T (2)(t) :
The fundamental question is how the distance H(t ) after a sufficiently long time depends on
the initial distance H(t = 0 ). Only if an arbitrarily small initial normalized Hamming distance H(t = 0) eventually vanishes for large N and sufficiently large times t, the system is in its ordered phase, whereas in the chaotic phase the time evolution of (5) for almost identical initial configurations will obviously approach a finite limit. Hence, arbitrarily small
differences in the initial condition grow into differences of a substantial size in the chaotic
phase completely erasing the memory of the initial state and giving rise to an unpredictable long-time evolution of a formally completely deterministic system. In practice, one simulates
two identical systems simultaneously except that the initial conditions differ by a small
fraction of the state variables. One can then compare the time evolution of the Hamming
distance (5) between the states of both systems at each time step.
For the infinite range model it has been shown analytically in the thermodynamic limit that
in the ordered phase a small initial distance decreases exponentially to zero [14]. Moreover,
for K 2, H = 0 is the only attractive fixed point so that the final distance vanishes
independent of the initial condition. In the chaotic phase the fixed point H = 0 loses its stability and two close initial conditions diverge exponentially fast in phase space, and
H(t ) eventually approaches an attractive fixed point H:o 0. At the critical point the asymptotic time behavior is spécial : H(t) either remains constant or follows an inverse- t law [14]. It should be stressed however, that in order to derive exact results in the
thermodynamic limit it is crucial that the connectivity parameter K is finite and that the
synaptic weights and the input cells are chosen at random so that there are neither correlations between the input cells nor between their weights.
For our two-dimensional lattice model computer simulations reveal that the time evolution of the Hamming distance H(t) behaves fundamentally different the final distance
H(oo ) depending strongly on the initial distance H(t = 0 ).
In marked contrast to the mean field model, even in the ordered phase the Hamming
distance does not decrease but increases linearly for H(t = 0) :0 0. Figure 3 shows the
Hamming distance H (t ) after t = 500 time steps as a function of the initial distance
H(t = 0) for K = 3, K = 4 and K = 5 lattice models. Extrapolation reveals that for K = 3 H (t ) vanishes for H(t = 0) --+ 0, whereas for K = 4 and K = 5 H(t ) approaches a
finite value. Since H(t ) remains strictly zero for H(0 ) = 0 by definition, there is a discontinuity in H(t) at t = 0 for K = 4 and K = 5. Notice that the ratio of the final and the initial distance in the ordered phase can be interpreted as a susceptibility [5] reminiscent of
phase transitions occurring in ferromagnets. There H(t = 0) plays a similar role as the
magnetic field and H (t = oo ) the role of the magnetization. As has already been
demonstrated in the modified Kauffman model [5] order and disorder can be detected by the dependence of H(oo ) on H(t = 0 ). As frequently found in biological systems [15] increasing connectivity implies a higher degree of disorder clearly seen by the increase of H(oo ) with increasing K.
Fig. 3. - Final nonnormalized distance H(t = 500 ) versus initial distance H(t = 0) for (a) the honeycomb lattice (K = 3), and square lattices (K = 4 (b) and K = 5 (c)) averaged over 500 specimen
nets for N = 1 225 cells.
5. Average activity.
Another useful quantity playing a prominent role in the microscopic description of the
network dynamics is the global activity level [11] at time t defined as
The activity level, the normalized Hamming distance (5) between the state variable
u (t) and the state o-0 = (- 1, - 1, ..., - 1 ), represents the fractional number of cells being
active at time t. This quantity is a scalar not specifying which cells of the network are active,
whereas the local activity a i i of each cell i averaged over a sufficiently large time
T defined as
gives a microscopic insight into the dynamics of the network and affects directly the dynamical
evolution of the macroscopic variable a (t). The average local activities ai then define the
probability distribution P (a ) [16] specified as
Since P (c ) can be related to a power spectrum its degree of continuity represents a suitable
measure for the degree of ordered and disordered behavior. The power spectrum widely used
in the analysis of continuous time dynamics, qualitatively different in different phases [13], is
considered as a convenient tool for estimating the level of disorder. Similarily, as has been
observed for the Kauffman model, P ( a ) is expected to consist essentially of discrete lines at rational values in the ordered phase, whereas in the chaotic phase cells with larger periods or
even quasi aperiodic behavior cause the appearance of broad continuous parts in the spectrum
[16]. Thus, also the distribution of the average activities can be considered as a candidate to
classify regular and chaotic motion. In practice, one starts with a random initial condition and
measures for each cell i the activity ai i averaged over a sufficiently long time for a large system. Figures 4a-4c show the local activity distribution for the above discussed lattice models being symmetric at zero threshold.
Fig. 4. - Local activity distribution P (a ) for (a) the honeycomb lattice (K = 3 ), (b), (c) square lattices
(K = 4 and K = 5) and (d) for the infinite range model (K = 5). All simulations have been performed
with a total number of N = 29 561 cells.
Fig. 4 (continued).
For all lattice models studied so far we find main concentrations of the local activities at 1/2, 1/4, 1/3, 1/8, ... besides their symmetric counterparts reflecting the existence of relatively
short periods. Note that the heights of the peaks at the value one half are several magnitudes larger than illustrated in figures 3a-3c. Their heights are drastically reduced with increasing connectivity, accompanied by the appearance of more numerous and smaller concentrations at rational numbers comparable with the bifurcation route to chaos by an (infinite) cascade of period doublings [13]. This phenomenon might shed some light on the marked predominance
of the even-valued lengths of the cyclic modes. The activity spectrum for the honeycomb
lattice shows the most discrete structure, whereas for the square lattice (K = 4 ) a trend to a
continuous part with still a pronounced peaked structure is to be seen. For the square lattice
(K = 5 ) there appear more cells with a quasi aperiodic motion contributing any local activity
value to the distribution showing a transition to an even more continuous structure. In
general, approaching a phase transition the peaked structure will gradually disappear. Note
that the disappearance tums out to be continuous which makes it extremely difficult to
localize a critical point. For comparison we add the activity spectrum for the infinite range model with K = 5, exhibiting strongly chaotic behavior. Here we see a totally continuous
Gaussian like probability around the mean activity level one half. Obviously figure 4a and figure 4d illustrate convincingly a marked difference characterizing the two phases, whereas
the figures also indicate that the criterium for our K = 4 and K = 5 lattice models is less
precise.
6. Conclusion.
We have suggested several ways of analyzing the chaotic and the ordered phase. All criteria indicate irregular motion by a qualitative change of an order parameter. The time evolution of the Hamming distance gives reliable information even for the localization of critical parameters. The distribution of the local activities is technically easiest to handle but gives only clear answers far away from the critical region comparable with power spectra analysis in
continuous time models. The variation of the mean length of the periods with the total number of cells is a quite reasonable instrument, though very elaborate, especially in the
critical region and difficult to realize in the chaotic region where the periods are very long.
Comparing short-ranged and long-ranged models we find the expected shift in the critical parameters. The time evolutions of the Hamming distance behave qualitatively different, especially in the ordered phase. In marked contrast to the mean field model also characteristics of cycling activity are quite different in the ordered phase : in our short-ranged
model the variation of the mean cycle length turns out to follow a much stronger power law.
In general, lattice models show a weaker tendency to disordered behavior due to their
strongly correlated structure substantially reducing randomness in the network connectivity.
A variety of easier software and hardware realizations and higher fault tolerance with respect
to stability against minor damage makes them an interesting object for a broad range of useful
applications.
Acknowledgments.
It is a pleasure to thank J. W. Clark, M. Husson, U. Keller, H. A. Mallot, M. L. Ristig,
M. Schreckenberg, W. von Seelen, D. Stauffer and G. Weisbuch for helpful and illuminating
discussions. Funding for this work by the Deutsche Forschungsgemeinschaft under grant Nr. Se 251/30-1 is gratefully acknowledged.
