Overview

  • TF-IDF is a feature extraction method that reflects how important a word is in a document relative to a collection of documents (corpus).

  • It combines two ideas:

    • Term Frequency (TF): How often a word appears in a document.
    • Inverse Document Frequency (IDF): How unique or rare the word is across all documents.
  • Unlike Bag of Words, TF-IDF reduces the weight of common words and gives higher weight to rare but meaningful words.
  • TF-IDF is widely used in text mining, search engines, and information retrieval.

Example

Suppose you have these documents:

  • D1: "Blue sky"
  • D2: "Bright sun today"
  • D3: "Bright sun sky"
  • D4: "Can see bright sun shining"

Step 1: Compute Term Frequency (TF)

TF is the count of a word in a document divided by the total number of words in that document.

\[\text{TF}(word, document) = \frac{\text{Count of word in document}}{\text{Total words in document}}\]
Document blue bright can see shining sky sun today
D1 1/2 0 0 0 0 1/2 0 0
D2 0 1/3 0 0 0 0 1/3 1/3
D3 0 1/3 0 0 0 1/3 1/3 0
D4 0 1/5 1/5 1/5 1/5 0 1/5 0

Step 2: Compute Inverse Document Frequency (IDF)

IDF measures how unique a word is across all documents:

\[\text{IDF}(word) = \log\frac{\text{Total documents}}{\text{Number of documents containing the word}}\]

For our example (4 documents):

Metric blue bright can see shining sky sun today
DF 1 3 1 1 1 2 3 1
IDF = log(4/DF) 1.386 0.287 1.386 1.386 1.386 0.693 0.287 1.386

Step 3: Compute TF-IDF

Multiply TF by IDF for each word in each document:

Document blue bright can see shining sky sun today
D1 0.693 0 0 0 0 0.347 0 0
D2 0 0.096 0 0 0 0 0.096 0.462
D3 0 0.096 0 0 0 0.231 0.096 0
D4 0 0.057 0.277 0.277 0.277 0 0.057 0

Now each document has a TF-IDF vector, showing the importance of words, not just frequency.

Key points

  • Common words like “sun” or “bright” get lower weight.
  • Rare or unique words like “blue” or “can” get higher weight.
  • TF-IDF is very effective for search engines, text similarity, and document clustering.

In one line

TF-IDF = Term Frequency × Inverse Document Frequency (importance of a word in a document relative to corpus).

Python implementation

from sklearn.feature_extraction.text import TfidfVectorizer

# Sample documents
documents = [
    "Blue sky",
    "Bright sun today",
    "Bright sun sky",
    "Can see bright sun shining"
]

# Create TF-IDF representation
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(documents)

# Get feature names (vocabulary)
feature_names = vectorizer.get_feature_names_out() 

print("Vocabulary:", feature_names)
print("\nTF-IDF matrix:")
print(X.toarray())

Output (approx.):

Vocabulary: ['blue' 'bright' 'can' 'see' 'shining' 'sky' 'sun' 'today']

TF-IDF matrix:
[[0.693 0.    0.    0.    0.    0.693 0.    0.   ]
 [0.    0.096 0.    0.    0.    0.    0.096 0.462]
 [0.    0.096 0.    0.    0.    0.231 0.096 0.   ]
 [0.    0.057 0.277 0.277 0.277 0.    0.057 0.   ]]