Language Modeling with nn

3.38

29.50

1.83

<p>3.60</p> Signup and view all the answers

<p>The log-softmax function isn't applied here due to the later use of CrossEntropyLoss, which requires the inputs to be unnormalized logits.</p> Signup and view all the answers

<p>The shape of the src tensor is [seq_len, batch_size].</p> Signup and view all the answers

<p>The positional encodings inject some information about the relative or absolute position of the tokens in the sequence.</p> Signup and view all the answers

<p>Batching enables more parallelizable processing.</p> Signup and view all the answers

<p>The transformer model has proven to be superior in quality for many sequence-to-sequence tasks while being more parallelizable compared to RNNs.</p> Signup and view all the answers

<p>The square attention mask is required because the self-attention layers are only allowed to attend the earlier positions in the sequence. It masks any tokens on the future positions.</p> Signup and view all the answers

<p>The output of the model is passed through a linear layer to output unnormalized logits. These logits are then used to calculate the probability distribution using a softmax function.</p> Signup and view all the answers