- Pre-Wiring and Pre-Training: What does a neural network need to learn truly general identity rules?
- CEUR Workshop Proceedings
- Article number
- Number of pages
- Document type
- Interfacultary Research Institutes
- Institute for Logic, Language and Computation (ILLC)
In an influential paper, Marcus et al.  claimed that connectionist models cannot account for human success at learning tasks that involved generalization of abstract knowledge such as grammatical rules. This claim triggered a heated debate, centered mostly around variants of the Simple Recurrent Network model [Elman, 1990]. In our work, we revisit this unresolved debate and analyze the underlying issues from a different perspective. We argue that, in order to simulate human-like learning of grammatical rules, a neural network model should not be used as a tabula rasa, but rather, the initial wiring of the neural connections and the experience acquired prior to the actual task should be incorporated into the model. We present two methods that aim to provide such initial state: a manipulation of the initial connections of the network in a cognitively plausible manner (concretely, by implementing a “delay-line” memory), and a pre-training algorithm that incrementally challenges the network with novel stimuli. We implement such techniques in an Echo State Network [Jaeger, 2001], and we show that only when combining both techniques the ESN is able to learn truly general identity rules.
- Final publisher version
- Proceedings title: CoCo 2016 : Cognitive Computation : proceedings of the Workshop on Cognitive Computation: Integrating neural
and symbolic approaches 2016, co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016)
: Barcelona, Spain, December 9, 2016
Place of publication: Aachen
Editors: T.R. Besold, A. Bordes, A. d'Avila Garcez, G. Wayne.
If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Ask the Library, or send a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You will be contacted as soon as possible.