Weight Initialization for High Order and Multilayer Perceptrons
| Type of publication: | Conference paper |
| Citation: | Thimm-94.2 |
| Booktitle: | Proceedings of the '94 SIPAR--Workshop on Parallel and Distributed Computing |
| Year: | 1994 |
| Month: | 10 |
| Organization: | SI Group for Parallel Systems |
| Address: | Institute of Informatics University, P\'erolles, Chemin du Mus\'ee 3, CH-1700 Fribourg, Switzerland |
| Abstract: | Proper weight initialization is one of the most important prerequisites for fast convergence of feed-forward neural networks like high order and multilayer perceptrons. In order to determine the optimal value of the initial weight variance (or range,',','), which is a important parameter of random weight initialization methods for high order perceptrons, a wide range of experiments (more than $200,000$ simulations) was performed, using seven different data sets, three weight distributions, three activation functions, and several network orders. The results of these experiments are compared to weight initialization techniques for multilayer perceptrons, which leads to the proposal of a suitable weight initialization method for high order perceptrons. Experiments over a large range of initial weight variances are performed (more than $20,000$ simulations) for multilayer perceptrons and compared to weight initialization methods proposed by other authors. The results of this comparison are justified by sufficiently small confidence intervals. |
| Userfields: | ipdmembership={neuron learning}, ipdxref={Article:thimm-97.1.bib, TechReport:thimm-94.4.bib}, |
| Keywords: | activation function, comparison of weight initialization methods, connectionism, high(er) order neural network, high(er) order perceptron, initial weight, initial weight distribution, interconnection strength, learning rate, multilayer perceptron, neural computation, neural network, neurocomputing, optimization, sigma-pi connection, weight initialization |
| Projects |
Idiap |
| Authors | |
| Editors | |
| Added by: | [UNK] |
| Total mark: | 0 |
|
Attachments
|
|
|
Notes
|
|
|
|
|