Sampling from multiple discrete items with diverse probability is relatively slow. When applying the noise contrastive estimation method, the intuitive sampling method becomes a bottleneck for the model’s training. So we apply the alias method for generating samples from the given noise distribution.
Generate Samples
if you want to generate many samples from the distribution, and don’t want specific tokens to be appear.
- generate 1 sample, check whether this sample is not equal to your specific token. then loop though this step, until you get as much samples as you want. (slow)
- generate 3 times more samples and remove the specific tokens in this list. (relatively faster)