logo Idiap Research Institute        
 [BibTeX] [Marc21]
Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model
Type of publication: Idiap-RR
Citation: Bengio03
Number: Idiap-RR-35-2003
Year: 2003
Institution: IDIAP
Abstract: Previous work on statistical language modeling has shown that it is possible to train a feed-forward neural network to approximate probabilities over sequences of words, resulting in significant error reduction when compared to standard baseline models. However, in order to train the model on the maximum likelihood criterion, one has to make, for each example, as many network passes as there are words in the vocabulary. We introduce adaptive importance sampling as a way to accelerate training of the model. We show that a very significant speed-up can be obtained on standard problems.
Userfields: ipdmembership={learning},
Projects Idiap
Authors Bengio, Yoshua
Sen├ęcal, Jean-S├ębastien
Added by: [UNK]
Total mark: 0
  • rr-03-35.pdf
  • rr-03-35.ps.gz