Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model
| Type of publication: | Idiap-RR |
| Citation: | Bengio03 |
| Number: | Idiap-RR-35-2003 |
| Year: | 2003 |
| Institution: | IDIAP |
| Abstract: | Previous work on statistical language modeling has shown that it is possible to train a feed-forward neural network to approximate probabilities over sequences of words, resulting in significant error reduction when compared to standard baseline models. However, in order to train the model on the maximum likelihood criterion, one has to make, for each example, as many network passes as there are words in the vocabulary. We introduce adaptive importance sampling as a way to accelerate training of the model. We show that a very significant speed-up can be obtained on standard problems. |
| Userfields: | ipdmembership={learning}, |
| Keywords: | |
| Projects |
Idiap |
| Authors | |
| Added by: | [UNK] |
| Total mark: | 0 |
|
Attachments
|
|
|
|
|
Notes
|
|
|
|
|
