Ch 1 Dense VectorsDesign

The single most important primitive in semantic search: what a dense embedding is and why it beats sparse keyword vectors for meaning.

Core concepts

  • Sparse vs. dense. Sparse vectors (bag-of-words, TF-IDF, BM25) are huge, mostly zeros, and match on exact tokens. Dense vectors are short (e.g. 384–1536 dims), fully populated, and encode meaning learned from data.
  • Distributional hypothesis. Words appearing in similar contexts have similar meaning — the principle every embedding model exploits.
  • Word2Vec & the analogy trick. Static one-vector-per-word embeddings where king − man + woman ≈ queen. Great intuition, but no context sensitivity.
  • Contextual embeddings. Transformers (BERT family) recompute a token’s vector from its neighbours, so “bank” differs in “river bank” vs “bank account”.
  • Similarity metrics. Cosine similarity, dot product, and Euclidean distance — how “closeness” is measured in vector space.
Sparse (TF-IDF / BM25)[0,0,3,0,0,…,1,0]~30k dims, mostly 0matches exact wordsDense (embedding)[0.21, -0.9, 0.4, …]~384–1536 dimsmatches meaning
Sparse = lexical match; dense = semantic match. Modern systems often combine both (hybrid search).

What you must master

  • Explain the difference between sparse and dense vectors and when each wins Level 1
  • Describe the distributional hypothesis and how Word2Vec learns from context Level 1
  • Generate embeddings and compute cosine similarity between two texts in code Level 2
  • Choose an embedding dimensionality & similarity metric for a use case Level 3

Architect’s lens

Embedding choice is a foundational, hard-to-reverse decision. Dimensionality drives storage and query cost in the vector DB; the metric must match how the model was trained (most modern models are trained for cosine). Know when hybrid (dense + sparse) search is worth the complexity — e.g. domains heavy in exact identifiers, codes, or rare product names.