Feature Encoding for Structured Data

Study Notes

Quiz on Feature Encoding for Structured Data

Overfitting: Overfitting occurs when a model learns the characteristics of the training data too well, including the noise, leading to poor performance on new, unseen data. It is not from a model being too simple (underfitting).
Overfitting (cont.): Overfitting is when the model has learned the training data's characteristics so significantly that it cannot generalize well to other, unseen data. This happens when the model is unnecessarily complex, fitting the training data's noise instead of its underlying patterns.
Overfitting (cont.): Overfitting doesn't result from high regularization. Instead, it occurs because a complex model focuses on noise from training data rather than the underlying pattern in data.
N-grams: Given a dataset with V vocabulary and n-gram length, there are in theory Vⁿ possible n-grams. (this answer pertains to question 2)
Sequence Length and N-grams: In a sequence of length L, the total number of n-grams is L - (n - 1). (This answer pertains to question 3)
N-grams in Multiple Sequences: Given N sequences and a total of T tokens, with <S> and </S> tokens at the start/end, the total possible n-grams are T + N(3-n). (answer for question 4).
Image Patches: With an image of size W x H, and patches of size m x n, (W - m + 1)(H - n + 1) patches can be created. (answer for question 5)
Bigram Features in Conversation Tree: With V possible words, V² possible bigram features can exist in a conversation tree. (answer for question 6)
Bigram Features from Parent-Reply: For a parent message with A tokens and a reply with B tokens, (A -1) * (B - 1) bigram features can be found using both the parent and response text. (answer for question 7)
Sequence Encoding (Vectors): Encoding a sequence as a vector of word indices (like [42, 11, 2, 7, 11, 2]) can be done correctly with linear models. (answer for question 8)