Implementation of an Associative Memory Using a Restricted Hopfield Network

Authors: Tet H. Yeap

Abstract:

An analog restricted Hopfield Network is presented in this paper. It consists of two layers of nodes, visible and hidden nodes, connected by directional weighted paths forming a bipartite graph with no intralayer connection. An energy or Lyapunov function was derived to show that the proposed network will converge to stable states. By introducing hidden nodes, the proposed network can be trained to store patterns and has increased memory capacity. Training to be an associative memory, simulation results show that the associative memory performs better than a classical Hopfield network by being able to perform better memory recall when the input is noisy.

Keywords: Associative memory, Hopfield network, Lyapunov function, Restricted Hopfield network.

References:

[1] Hopfield, J.J., Neural networks and physical systems with emergent collective computational abilities, Proc. of the National Academy of Sciences, USA, Vol. 79, 1982, 2554-2558.
[2] Hopfield, J.J. and Tank D.W., Neural computation of decisions in optimization problems, Biological Cybernetics, Vol. 52, 1985, 141-152.
[3] Hopfield, J.J. and Tank D.W., Computing with neural circuits: model, Science, New Series, Vol. 233, No. 4764 (Aug. 8, 1986), 625-633.
[4] McEliece, R.J., Posner, E.C., Rodemich, E.R., and Venkatesh, S.S., The capacity of Hopfield associative memory, IEEE Transaction on Information Theory, Vol. IT-33, No. 4, July 1987, 461-482.
[5] Storkey, A.J. and Valabregue, R., The basins of attraction of a new Hopfield learning rule, Neural Networks, Vol. 12, 1999, 869-876.
[6] Gardner, E., Maximum storage capacity in neural networks, Europhys. Lett., Vol. 4, 1987, 481-485.
[7] Gardner, E., The space of interactions in neural network models, J . Phys. A : Math Gen., Vol. 21, 1988, 257-270.
[8] Gardner, E. and Derrida, B., Optimal storage properties of neural network models, J . Phys. A : Math Gen., Vol. 21, 1988, 271-84.
[9] Salakhutdinov, R. and Hinton, G., Restricted Boltzmann Machines, Proc. of the 12th International Confe-rence on Artificial Intelligence and Statistics (AISTATS), 2009, Clearwater Beach, Florida, USA.
[10] Sutskever, I., Hinton, G., and Taylor, G., The recurrent temporal restricted Boltzmann machine, Proc. of Neural Information Processing Systems, 2008.
[11] Salakhutdinov, R. R., Mnih, A., and Hinton, G. E., Restricted Boltzmann machines for collaborative filtering, Proceedings of the International Conference on Machine Learning, Vol. 24, 791-798., ACM.
[12] Spall, J.C., Multivariate stochastic spproximation using simultaneous perturbation gradient approximation, IEEE Transaction on Automatic Control, Vol. 37, No. 3, March 1992, 333-341.
[13] Spall, J.C., An overview of the simultaneous perturbation method for efficient optimization, Johns Hopkins Apl Technical Digest, Vol. 19, No. 4, 1998, 482-492.