CS5720 - GloVe: Global Vectors for Word Representation

The GloVe Model

GloVe combines the best of both worlds: global matrix factorization and local context windows

Core Innovation:

• Uses co-occurrence statistics from entire corpus
• Learns from word-word co-occurrence matrix
• Captures global statistical information
• More interpretable objective function

🔄 Key Difference

Word2Vec: Local context windows
GloVe: Global co-occurrence counts

Advantages of GloVe

📊 Statistical Efficiency

Makes efficient use of corpus statistics

⚡ Fast Training

Trains on co-occurrence matrix, not raw text

🧮 Better Analogies

Excellent at capturing semantic relationships

🔍 Interpretable

Clear mathematical objective

GloVe Training Process

Co-occurrence Matrix

X[i,j] = count(i,j)
[cat, dog] = 85
[cat, car] = 2

Weighted Least Squares

J = Σ f(X_ij)(w_i·w̃_j - log X_ij)²

Word Vectors

w_i + w̃_i
Combined vectors

w_i^T w̃_j + b_i + b̃_j = log(X_ij)

GloVe: Global Vectors for Word Representation

The GloVe Model

Advantages of GloVe

GloVe Training Process

Modal Title