Storing cycles in Hopfield-type neural networks

Abstract

The storage of pattern sequences is one of the most important tasks in both biological and artificial intelligence systems. Clarifying the underlying mathematical principles for both the storage and retrieval of pattern sequences in neural networks is fundamental for understanding the generation of rhythmic movements in animal nervous systems, as well as for designing electrical circuits to produce and control rhythmic output. In this dissertation, we investigate algebraic structures of binary cyclic patterns (or for short cycles) and study relations between these structures and the topology and dynamics of Hopfield-type networks with and without delay constructed from cyclic patterns using the pseudoinverse learning rule. A cycle defined by a binary matrix Σ is called admissible, if a connectivity matrix J satisfying the cycle's transition conditions exists. We show that Σ is admissible, if and only if its discrete Fourier transform contains exactly r = rank(Σ) nonzero columns. Based on the decomposition of the rows of Σ into disjoint subsets corresponding to loops, where a loop is defined by the set of all cyclic permutations of a row, cycles are classified as simple cycles, and separable or inseparable composite cycles. Simple cycles contain rows from one loop only, and the network topology is a feed-forward chain with feedback to one neuron if the loop-vectors in Σ are cyclic permutations of each other. For special cases this topology simplifies to a ring with only one feedback. Composite cycles contain rows from at least two disjoint loops, and the neurons corresponding to the loop-vectors in Σ from the same loop are identified with a cluster. Networks constructed from separable composite cycles decompose into completely isolated clusters. For inseparable composite cycles at least two clusters are connected, and the cluster-connectivity is related to the intersections of the spaces spanned by the loop-vectors of the clusters. The remainder of this thesis deals with the dynamics of Hopfield-type networks with connectivities constructed from admissible cycles. In this approach, the connectivity is composed of two contributions, C0J0 and C1J, where the matrix J0 serves to store cycle's patterns as fixed points and the matrix J induces the transitions between the cycle's patterns. Delayed couplings are associated with the transition matrix J. An admissible cycle is called strongly retrievable if for appropriate initial data the network dynamics undergoes a persistent oscillation in accordance with cycle's transition conditions. An admissible cycle is called weakly retrievable if for any M there exists a sufficiently large delay time τ such that at least M consecutive patterns are retrieved. When the Hamming distance between successive cycle-patterns is greater than one, the sign-changes in the network dynamics occur asynchronously, leading to the occurrence of intermediate patterns that are not contained in the cycle-matrix. We call the time-intervals with these intermediate patterns misalignment intervals and introduce a novel method to analyze the lengths of these intervals, which is referred to as Misalignment Length Analysis (MLA). Using this method, intermediate patterns are determined and for a special class of cycles a recurrence relation for successive misalignment intervals is established. In addition, a class of cycles, related to properties of the intermediate patterns, is identified which can be proved to be weakly retrievable in the case C0 = 0 and for sufficiently large values of the gain scaling parameter, λ, of the sigmoid coupling function. More generally, we also prove that for a given J constructed from a preselected cycle in that class, all other cycles satisfying the transition conditions associated with J are weakly retrievable as well. These results provide an analytic explanation for the long-lasting transient oscillations observed recently in simulations of cooperative Hopfield-type networks with delays. For general values of C0, C1, λ, we perform a linear stability analysis and give a complete description of all possible bifurcations of the trivial solution for networks constructed from admissible cycles. Numerically we illustrate that, depending on the structural features of a cycle, admissible cycles are stored and retrieved either as attracting limit cycles or as long-lasting transient oscillations. Moreover, if the cycle is revealed as attracting limit cycle, this limit cycle is created in a Hopf bifurcation from the trivial solution, and the transition from fixed point attractors to the attracting limit cycle is established through multiple saddle-nodes on limit cycle bifurcations. Lastly, simulations showing successfully retrieved cycles in continuous-time Hopfield-type networks and in networks of spiking neurons exhibiting up-down states are presented, which strongly suggests that the results of the study presented in this dissertation can be extended to more complicated networks.