logo Idiap Research Institute        
 [BibTeX] [Marc21]
Weight Initialization for High Order and Multilayer Perceptrons
Type of publication: Conference paper
Citation: Thimm-94.2
Booktitle: Proceedings of the '94 SIPAR--Workshop on Parallel and Distributed Computing
Year: 1994
Month: 10
Organization: SI Group for Parallel Systems
Address: Institute of Informatics University, P\'erolles, Chemin du Mus\'ee 3, CH-1700 Fribourg, Switzerland
Abstract: Proper weight initialization is one of the most important prerequisites for fast convergence of feed-forward neural networks like high order and multilayer perceptrons. In order to determine the optimal value of the initial weight variance (or range,',','), which is a important parameter of random weight initialization methods for high order perceptrons, a wide range of experiments (more than $200,000$ simulations) was performed, using seven different data sets, three weight distributions, three activation functions, and several network orders. The results of these experiments are compared to weight initialization techniques for multilayer perceptrons, which leads to the proposal of a suitable weight initialization method for high order perceptrons. Experiments over a large range of initial weight variances are performed (more than $20,000$ simulations) for multilayer perceptrons and compared to weight initialization methods proposed by other authors. The results of this comparison are justified by sufficiently small confidence intervals.
Userfields: ipdmembership={neuron learning}, ipdxref={Article:thimm-97.1.bib, TechReport:thimm-94.4.bib},
Keywords: activation function, comparison of weight initialization methods, connectionism, high(er) order neural network, high(er) order perceptron, initial weight, initial weight distribution, interconnection strength, learning rate, multilayer perceptron, neural computation, neural network, neurocomputing, optimization, sigma-pi connection, weight initialization
Projects Idiap
Authors Thimm, Georg
Fiesler, Emile
Editors Aguilar, Marc
Added by: [UNK]
Total mark: 0