Photo by

Perplexity: Evaluating a Language Model

We have a serial of \(m\) sentences: \(s_1,s_2,\cdots,s_m\)

We could look at the probability under our model \(\prod_{i=1}^m{p(s_i)}\). Or more conveniently, the log probability:

\[\log \prod_{i=1}^m{p(s_i)}=\sum_{i=1}^m{\log p(s_i)}\]

where \(p(s_i)\) is the probability of sentence \(s_i\).

In fact, the usual evaluation measure is perplexity:

\[PPL=2^{-l}\] \[l=\frac{1}{M}\sum_{i=1}^m{\log p(s_i)}\]

and \(M\) is the total number of words in the test data.


Given words \(x_1,\cdots,x_t\), a language model products the following word’s probability \(x_{t+1}\) by:

\[P(x_{t+1}=v_j|x_t\cdots,x_1)=\hat y_j^t\]

where \(v_j\) is a word in the vocabulary.

The predicted output vector \(\hat y^t \in \mathbb{R}^{V}\) is a probability distribution over the vocabulary, and we optimize the cross-entropy loss:

\[\mathcal{L}^t(\theta)=CE(y^t,\hat y^t)=-\sum_{i=1}^{|V|}{y_i^t\log \hat y_i^t}\]

where \(y^t\) is the one-hot vector corresponding to the target word. This is a point-wise loss, and we sum the cross-entropy loss across all examples in a sequence, across all sequences in the dataset in order to evaluate model performance.

The relationship between Cross-Entropy and PPL

\[PPL^t=\frac{1}{P(x_{t+1}^{pred}=x_{t+1}|x_t\cdots,x_1)}=\frac{1}{\sum_{j=1}^V {y_j^t\cdot \hat y_j^t}}\]

which is the inverse probability of the correct word, according to the model distribution \(P\).

suppose \(y_i^t\) is the only nonzero element of \(y^t\). Then, note that:

\[CE(y^t,\hat y^t)=-\log \hat y_i^t=\log\frac{1}{\hat y_i^t}\] \[PP(y^t,\hat y^t)=\frac{1}{\hat y_i^t}\]

Then, it follows that:

\[CE(y^t,\hat y^t)=\log PP(y^t,\hat y^t)\]

In fact, minimizing the arthemtic mean of the cross-entropy is identical to minimizing the geometric mean of the perplexity. If the model predictions are completely random, \(\mathbb{E}[\hat y_i^t]=\frac{1}{V}\), and the expected cross-entropies are \(\log V\), (\(\log 10000\approx 9.21\))