Machine Learning Training Data and Optimization

Training Data and Batching

• The training data used for the models is the standard WMT 2014 English-German dataset (4.5 million sentence pairs) for English-German and the larger WMT 2014 English-French dataset (36M sentences) for English-French.
• Byte-pair encoding was used to encode sentences, resulting in a shared vocabulary of roughly 37,000 tokens for English-German and 32,000 word-piece vocabulary for English-French.
• Sentence pairs were batched together based on approximate sentence length, with each batch containing approximately 25,000 source tokens and 25,000 target tokens.

Hardware and Schedule

• The models were trained on one machine equipped with 8 NVIDIA P100 GPUs.
• The training process for the base models took approximately 0.4 seconds per step and required 12 hours (100,000 steps) of training time.
• Larger models required 1.0 seconds per step and were trained for 3.5 days (300,000 steps).

Optimizer

• The Adam optimizer was used for training, with parameters: β1 = 0.9, β2 = 0.98, and ϵ = 10−9.
• The learning rate was dynamically adjusted during training using the formula: lrate = d−0.5model · min(step_num−0.5, step_num · warmup_steps−1.5)
• The learning rate was increased linearly for the first 4,000 training steps (warmup_steps) and then decreased proportionally to the inverse square root of the step number.

Other Training Aspects

• Beam search was used for decoding, but checkpoint averaging was not implemented.
• Experiments varying the number of attention heads, attention key and value dimensions, and other hyperparameters were conducted to evaluate model performance.
• Results indicate that increasing the number of attention heads beyond a certain point leads to reduced quality.
• Decreasing the attention key size negatively affects model quality, suggesting that more sophisticated compatibility functions may be beneficial.
• Larger models generally perform better, dropout helps prevent overfitting, and learned positional embeddings show similar results to sinusoidal positional encoding.