NLP with NLTK, Stopwords and Stemming

• NLTK (Natural Language Toolkit) is imported
• stopwords and word_tokenize are imported from nltk.corpus and nltk.tokenize, respectively
• PorterStemmer is imported from nltk.stem
• TfidfVectorizer is imported from sklearn.feature_extraction.text
• cosine_similarity is imported from sklearn.metrics.pairwise

NLTK Data

• The code downloads the 'stopwords' and 'punkt' packages from NLTK
• Stopwords are a list of commonly used words in a language that are often removed from text during natural language processing tasks
• Punkt is a pre-trained model for tokenizing sentences

Documents Initialized

• A list of documents is created, with each document being a string related to natural language processing, information retrieval, text mining, and Python
• The query "Python natural language processing" is defined as a string

Stop Words and Stemming

• A set of English stop words is created using stopwords.words('english')
• A PorterStemmer is initialized for stemming words

Text Preprocessing

• A preprocess_text function is defined to process text by:
  • Converting it to lowercase
  • Tokenizing the text into words
  • Stemming each word using the Porter Stemmer, after checking if the word is alphanumeric and not in the stop words
  • Joining the processed words back into a string

Document Preprocessing

• The preprocess_text function is applied to each document in the documents list and the query
• The preprocessed documents are stored in the preprocessed_documents list, while the preprocessed query is stored in the preprocessed_query variable

TF-IDF Vectorization

• A TfidfVectorizer is initialized
• The vectorizer is used to fit and transform the preprocessed query and documents into a TF-IDF matrix
• TF-IDF (Term Frequency-Inverse Document Frequency) is a numerical statistic that reflects how important a word is to a document in a collection or corpus

Cosine Similarity Calculation

• Cosine similarity is calculated between the TF-IDF vector of the preprocessed query and the TF-IDF vectors of the preprocessed documents
• Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space that measures the cosine of the angle between them

Document Sorting

• The documents are sorted based on their cosine similarity scores
• A list of tuples is created, containing each document and its corresponding cosine similarity score
• This list is then sorted in reverse order based on the similarity scores
• The top 3 relevant documents are printed, along with their similarity scores to the given query
  • Similarity: 0.50 - The nltk library in Python provides tools for natural language processing tasks
  • Similarity: 0.45 - Natural language processing is a subfield of artificial intelligence
  • Similarity: 0.30 - Python is a popular programming language for data science