Text Data Preprocessing and Tokenization

To convert text into numerical vectors directly

To remove all punctuation marks from the text

To break up the words in the text into individual tokens (correct)

To combine words into phrases for better analysis

Tokenization to form individual tokens (correct)

Stemming and lemmatization of words

Removing all stopwords from the text

Converting text into numerical vectors

The length of each token in the text data

The position of each token in the original text

The set of all unique tokens occurring in the data (correct)

The frequency of each token in the text data