18 Questions
In the context of neural word embeddings, what was the main idea behind word2vec?
To compute probabilities of context words for a central word and adjust vectors to increase these probabilities.
What is the term that has become almost synonymous with 'embeddings' or 'neural embeddings'?
Word2Vec
How did Mikolov et.al contribute to the idea of neural word embeddings?
Expanded and improved the idea in their 2013 paper 'word2vec'.
What is the iterative method used in Word2Vec for learning word embeddings?
Computing probabilities of context words and adjusting vectors.
Which paper first presented the idea of Neural Probabilistic Language Model?
'A Neural Probabilistic Language Model' by Bengio et.al
What is the primary task in building a Neural Network for Word2Vec according to the text?
Predict a nearby word given a central word.
What is the primary purpose of the Output layer in word2vec?
To calculate the loss for better word embeddings
Which of the following statements about the training process of word2vec is true?
The training process involves adjusting the matrix values with gradient descent
What is the role of the context window size in word2vec?
It determines the range of words to consider for each target word
What happens to the Output layer after the training process in word2vec?
It is discarded
What is the purpose of the dot product and sigmoid function in word2vec?
To determine the similarity between the target word and context vectors
What is the typical range for the dimensionality of word embeddings in word2vec?
Between 100 and 1,000
What is the primary purpose of the hidden layer in word2vec?
To convert one-hot representations of words into dense vectors
How does the Continuous Bag of Words (CBOW) model try to predict a target word?
By using the sum of vector embeddings of surrounding words
Why does the Skip-gram model work better in practice than the CBOW model?
It scales better with larger datasets
What is the role of the input layer in word2vec?
To convert input data into one-hot representations
How is the vocabulary of words built in word2vec?
By extracting all unique words from the training documents
What type of neural network is used in word2vec?
Feedforward Neural Network (FNN)
Study Notes
Neural Word Embeddings
- Neural word embeddings, also known as word embeddings, are a way to learn relationships between words using neural networks.
- The concept was first introduced by Bengio et al. in their 2003 paper "A Neural Probabilistic Language Model".
- Mikolov et al. further expanded and improved the idea in their 2013 paper "word2vec", which has become synonymous with neural embeddings.
Word2Vec
- Word2Vec is an iterative method that takes a huge text corpus and moves a sliding window over the text to learn word relationships.
- The method computes probabilities of context words for a central word and adjusts vectors to increase these probabilities.
- Word2Vec learns word embeddings that know viable surrounding words.
Word2Vec Architecture
- Step 1: The input layer converts words to one-hot representations to feed into the neural network.
- Step 2: The hidden layer has a number of neurons equal to the desired dimension of the word representations.
- Step 3: The output layer is a softmax layer with 10,000 neurons, used to calculate the loss and generate better word embeddings.
Word2Vec Training
- Training involves sliding a window of size 2C+1 over the training corpus, shifting it by one word each time.
- For each target word, the dot product of its vector and context vector is computed and passed through a sigmoid function.
- Positive examples have an expected output of 1, while negative examples have an expected output close to 0.
- The process is repeated multiple times to train the model.
Word2Vec Results
- Hyper-parameter settings include word embedding dimensionality (between 100 and 1,000) and context window size.
- There are two models: Continuous Bag of Words (CBOW) and Skip-gram.
- CBOW predicts a target word given a context of words, while Skip-gram predicts surrounding words given a target word.
Test your knowledge about neural word embeddings and their development through papers like 'A Neural Probabilistic Language Model' by Bengio et.al and 'word2vec' by Mikolov et.al. Learn about how neural networks can be used to understand the relationships among words.
Make Your Own Quizzes and Flashcards
Convert your notes into interactive study material.
Get started for free
