Data Science — NLP

Text preprocessing

import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

# Tokenize
tokens = word_tokenize("Hello world!")

# Remove stopwords
stop_words = set(stopwords.words('english'))
filtered = [w for w in tokens if w.lower() not in stop_words]

Bag of words

from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(["Hello world", "World peace"])
print(X.toarray())

TF-IDF

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(["Hello world", "World peace"])

Sentiment analysis

from textblob import TextBlob

text = "This movie is great!"
blob = TextBlob(text)
print(f"Sentiment: {blob.sentiment}")

Word embeddings

import gensim.downloader as api

model = api.load('word2vec-google-news-300')
similar = model.most_similar('king')
print(similar)

Mini Practice

1. Preprocess text
2. Create BoW features
3. Calculate TF-IDF
4. Analyze sentiment

Up Next

Continue with Deep Learning - Neural networks.