Embeddings, Vector DBs & RAG
A concept map distilling the key ideas needed to build a Retrieval-Augmented Generation (RAG) application — from turning text into vectors, to searching those vectors at scale, to assembling a working pipeline that grounds an LLM's answers in your own data.
1. Embeddings Foundations
How raw text becomes a vector that captures meaning.
An embedding is a list of numbers (a vector) that represents a piece of text — a word, sentence, or document — such that texts with similar meaning end up close together in vector space. This relies on the distributional hypothesis: words that appear in similar contexts tend to have similar meanings.
Word2Vec learns one fixed vector per word by predicting neighboring words. The famous result is that relationships become arithmetic: king − man + woman ≈ queen.
Word2Vec gives every word one vector regardless of context — "bank" (river) and "bank" (money) get the same embedding. Transformers use self-attention so each word's vector is recomputed based on surrounding words, producing contextualembeddings. This is what modern embedding models (used for RAG) are built on.
Key takeaways
- Embeddings turn text into vectors so "similar meaning" becomes "close in space" (cosine/dot-product distance).
- Word2Vec proved meaning can be captured with simple vector arithmetic, but one vector per word ignores context.
- Transformer-based embedding models produce contextual vectors — the foundation of today's embedding APIs (used for search and RAG).
2. Similarity Search & Vector Databases Retrieval
Once everything is a vector, how do you find the closest ones — fast, at scale?
Brute-force search (compare a query to every stored vector) is exact but scales linearly — too slow for millions/billions of vectors. Approximate Nearest Neighbor (ANN)algorithms trade a small amount of accuracy for massive speed gains, which is what makes production vector search possible.
| Index type | Idea | Trade-off |
|---|---|---|
| Flat | Brute-force exact search | Accurate but slow at scale |
| IVF (Inverted File) | Cluster vectors, search only nearby clusters | Much faster, small accuracy loss |
| PQ (Product Quantization) | Compress vectors into compact codes | Huge memory savings, more accuracy loss |
| IVF+PQ | Combine clustering + compression | Best scalability, tunable accuracy |
HNSW (Hierarchical Navigable Small World) builds a multi-layer graph: the top layer has few long-range links for fast coarse jumps, lower layers have many short-range links for fine-grained search — similar in spirit to a "skip list" over vector space.
Key takeaways
- Vector databases exist to make ANN search fast, persistent, filterable, and scalable — FAISS is a library of ANN algorithms; vector DBs (Chroma, Pinecone, etc.) wrap similar algorithms with storage, metadata filtering, and APIs.
- IVF and PQ trade some accuracy for speed/memory; HNSW trades some accuracy for very fast graph-based traversal — most vector DBs default to an HNSW-style index.
- Choosing an index is a trade-off between recall (accuracy), latency, and memory footprint.
3. Chunking Strategies Data Prep
Before embedding, documents must be split into retrievable pieces — how you split matters as much as how you search.
Embedding models have limited context windows, and one embedding per giant document loses detail. Chunk size directly affects retrieval quality: too large → noisy/imprecise matches; too small → missing surrounding context needed to answer a question.
Key takeaways
- Chunk size and overlap are tunable hyperparameters of a RAG system, not an afterthought.
- Recursive/semantic splitting (by sentence, paragraph, heading) generally retrieves more coherent context than naive fixed-size splitting.
- Overlap between chunks helps avoid cutting key information exactly at a boundary.
4. Building RAG Applications Application
Combining embeddings + vector search into an application that grounds LLM answers in your data.
Ingest documents → chunk → embed each chunk → store vectors (+ metadata) in a vector DB → at query time, embed the user's question → retrieve the most similar chunks → insert them into the LLM prompt as context → generate a grounded answer.
- Query expansion: rewrite or generate multiple variants of the user's question to retrieve a broader, more relevant set of chunks.
- Re-ranking: retrieve a larger candidate set with the vector DB, then use a more precise (often cross-encoder) model to re-order results by true relevance.
- Metadata filtering: narrow the search space using structured filters (date, source, tags) alongside vector similarity.
Key takeaways
- RAG's job is to fetch the right context so the LLM doesn't have to rely purely on what it memorized during training.
- Naive top-k similarity search is a starting point — production systems layer query expansion, re-ranking, and filtering on top.
- Retrieval quality is the ceiling on answer quality: garbage retrieved context leads to garbage-grounded answers.
5. NLP for Semantic Search Pinecone Learn · Ch. 1–13
Condensed notes from Pinecone's "NLP for Semantic Search" course — how sentence embeddings and retrievers are actually built, trained, and adapted to new domains. See the full structured lesson track for the architect-focused deep dive.
Sparse vectors (TF-IDF/BM25-style) represent syntax — mostly zeros with a few "hit" positions for matching words. Dense vectors represent semantics — every dimension carries learned meaning, produced by a neural net (Word2Vec, sentence transformers, DPR, CLIP).
- Sentence transformers (e.g.
all-mpnet-base-v2) embed whole sentences for similarity/clustering/search. - DPR (Dense Passage Retriever) uses two encoders (question + context) trained so a question vector lands close to the vector of the passage that answers it — the basis of retrieval for Q&A.
- CLIP extends the same idea across modalities, embedding images and text captions into one shared space.
Plain BERT only produces token-level vectors; comparing sentences required an expensivecross-encoder (both sentences fed together, one inference per pair — impractical at scale: ~65 hours to cluster 10K sentences). SBERT (2019) fixed this with asiamese architecture: one BERT processes each sentence independently, then mean-pooling turns its token vectors into a single sentence vector that can be pre-computed, stored, and compared with cosine similarity in milliseconds (~5 seconds to embed + ~0.01s to compare 10K sentences).
Newer models (MPNet, RoBERTa-based, trained on 1B+ pairs) now clearly outperform the original SBERT.
The original SBERT training recipe: fine-tune on NLI datasets (SNLI + MNLI, ~943K premise/hypothesispairs labeled entailment/neutral/contradiction). Each sentence is mean-pooled into vectors uand v, concatenated as (u, v, |u-v|), and fed into a feedforward classifier optimized with softmax (cross-entropy) loss over the 3 labels. This pulls entailment-pair vectors together and contradiction-pair vectors apart. Softmax loss is now largely superseded by MNR loss.
Source: Training Sentence Transformers the OG Way (with Softmax Loss)
MNR loss only needs positive (anchor, positive) pairs — e.g. entailment pairs from NLI, dropping neutral/contradiction rows. Within a training batch, every other positive in the batch is treated as an implicit negative for the current anchor, so a batch of size N yields N−1 negatives per anchor "for free" — no manual negative mining needed. Models trained with MNR loss clearly outperform softmax-loss and vanilla-BERT baselines on STS benchmarks, and the sentence-transformerslibrary makes it a one-line loss function (losses.MultipleNegativesRankingLoss).
Source: Next-Gen Sentence Embeddings with Multiple Negatives Ranking Loss
Goal: map equivalent sentences in different languages ("I love plants" / "amo le piante") to the same region of vector space. Since labeled cross-lingual similarity data is scarce, the practical approach ismultilingual knowledge distillation: a fine-tuned monolingual teacher model (e.g. English SBERT) produces target embeddings; a multilingual student model (e.g. XLM-RoBERTa) is trained with MSE loss to reproduce those same vectors for translated sentence pairs (using parallel corpora like TED talk translations). Pretrained multilingual models (e.g. paraphrase-multilingual-mpnet-base-v2) usually make training your own unnecessary.
Source: Tomayto, Tomahto, Transformer: Multilingual Sentence Transformers
When there's no labeled data at all (e.g. a niche domain or low-resource language), TSDAE(Transformer-based Sequential Denoising Auto-Encoder) trains on raw text alone. Sentences are "damaged" (tokens deleted, ~60% deletion ratio performed best), encoded into a single sentence vector (using the [CLS] token), and a decoder tries to reconstruct the original sentence from just that one vector — unlike MLM, the decoder only sees the compressed sentence vector, forcing it to be information-rich. TSDAE clearly underperforms supervised methods (NLI/MNR) but is far better than an untrained model, and is one of the few options when labels genuinely don't exist.
ODQA lets you ask natural-language questions over a large, unstructured corpus. Three flavors:
- Extractive (open-book): retriever finds relevant contexts (vector DB + retriever model), a reader model extracts the answer span from those contexts. Most reliable for factual, specific answers.
- Abstractive open-book: retrieved contexts are fed into a seq2seq generator (BART/T5) that writes an answer combining retrieved facts with its own internal knowledge — better suited to more open-ended/opinion-style questions.
- Abstractive closed-book: no retrieval step at all — a pure generative model answers from what it memorized during pretraining; simplest but least grounded/reliable, and scales with model size.
The retriever is arguably the most critical ODQA component — a poor retriever guarantees a poor final answer, while even a weak reader can still be somewhat useful given good contexts. Retrievers encode questions and contexts into the same vector space (e.g. a sentence transformer fine-tuned on SQuAD question/context pairs with MNR loss), store context vectors in a vector database, and at query time embed the question to retrieve the top-k most similar contexts. Evaluation uses information-retrieval metrics like mAP@K rather than simple accuracy.
A reader (e.g. BertForQuestionAnswering) takes a question + retrieved context and predicts a span — start and end token positions in the context — rather than generating free text. Training needs question+context inputs and start/end position labels; "no answer" cases are handled by pointing both positions to position 0. Evaluated with ROUGE (or exact-match/F1); scores look artificially low if the dataset contains ambiguous "unanswerable" questions humans would also struggle with.
Sentence transformers (bi-encoders) need lots of labeled pairs, but labeled data is often scarce in-domain, while cross-encoders (full-attention over both sentences at once) reach high accuracy with far less data — at the cost of not scaling to large-scale search. AugSBERTexploits this: (1) fine-tune a cross-encoder on the small labeled ("gold") dataset, (2) generate new unlabeled sentence pairs via random sampling, (3) label them with the cross-encoder to build a "silver" dataset, then (4) fine-tune the bi-encoder on gold + silver combined. Reported gains: up to 6% in-domain, up to 37% for domain-adaptation tasks.
Extends AugSBERT to cross-domain transfer: train the cross-encoder on a labeled source domain, then use it to pseudo-label an unlabeled target domain, and fine-tune the bi-encoder on those target-domain pseudo-labels. Before investing in training, n-gram (Jaccard) overlap between source and target datasets gives a cheap early signal of how well transfer is likely to work — bigger domain gaps are harder to bridge, and results confirm that source/target similarity predicts transfer performance better than raw source-domain cross-encoder accuracy alone.
Built for asymmetric semantic search (short query, long passage — e.g. "How do I tie my shoelaces?" vs. a paragraph explaining it). Given only unlabeled passages, a T5query-generation model synthesizes plausible queries for each passage, creating synthetic (query, passage) pairs. These pairs then fine-tune a bi-encoder with MNR loss exactly as if they were real labeled data. Generated queries are noisy (T5 is general-purpose), so GenQ works best when some domain overlap with the query-generation model's training data exists.
GPL combines and improves on the previous techniques into a three-step domain-adaptation pipeline that needs nothing more than unstructured target-domain text:
- Query generation: same T5-based approach as GenQ — generate a query for each unlabeled passage.
- Negative mining: embed passages with an existing retriever, store them in a vector database (Pinecone), and for each generated query retrieve top-k similar-but-wrong passages as hard negatives.
- Pseudo-labeling: a cross-encoder scores
sim(query, positive)andsim(query, negative); the difference between these scores becomes a soft "margin" label.
The bi-encoder is then fine-tuned with Margin MSE loss to reproduce that margin — a soft, continuous training signal (rather than binary similar/dissimilar), which is what gives GPL an edge over GenQ for adapting retrievers to a brand-new domain purely from raw text.
Source: Domain Adaptation with Generative Pseudo-Labeling (GPL)
Key takeaways
- Sentence transformers turned expensive cross-encoder comparisons into cheap, pre-computable vector search — the foundation of every retriever in this series.
- Loss function matters: MNR loss > softmax loss for fine-tuning bi-encoders, and needs only positive pairs (negatives come free from the batch).
- ODQA is a three-part pipeline — vector DB, retriever, reader — and retriever quality is the ceiling on the whole system, same lesson as RAG in general.
- When labels are scarce or missing entirely, there's a ladder of techniques: AugSBERT (in-domain augmentation) → domain transfer (cross-domain pseudo-labeling) → GenQ (synthetic queries) → GPL (synthetic queries + mined negatives + soft margin labels) → TSDAE (fully unsupervised, weakest but requires zero labeled data).
- Multilingual embeddings are usually best obtained from existing pretrained models; train your own only via knowledge distillation when no suitable model covers your target languages.
6. End-to-End Architecture Blueprint
Click or hover a stage to see how it ties back to the concepts above.
7. Cheat Sheet Recap
One-glance recap, organized by source.
Embeddings (Cohere, Jay Alammar, Hugging Face)
- Text → vector, similar meaning → close vectors
- Word2Vec: static vectors, vector arithmetic works
- Transformers: contextual vectors via self-attention
FAISS Missing Manual / HNSW (Pinecone)
- Brute force doesn't scale → use ANN
- Flat = exact, IVF = clustered, PQ = compressed
- HNSW = layered graph for fast approximate search
Chunking Strategies (Pinecone)
- Chunk size/overlap is a tunable design choice
- Recursive/semantic splitting beats naive fixed-size
- Overlap prevents losing context at boundaries
RAG & Vector DB Apps (DeepLearning.AI)
- Pipeline: chunk → embed → store → retrieve → augment → generate
- Advanced retrieval: query expansion, re-ranking, metadata filters
- Retrieval quality caps final answer quality
8. RAG on AWS Reference Architecture
Mapping the same pipeline onto managed AWS services. Click or hover a stage for details.
Embedded search (FAISS / LanceDB in Lambda) vs. a serverless vector service
Cost level is relative/directional (check the AWS Pricing Calculator for exact numbers).
| Option | Cost | Pros | Cons |
|---|---|---|---|
| FAISS in Lambda | $ Low pay-per-invocation, no idle cost, no separate store | Battle-tested ANN library; index loads straight into memory; no network hop to a separate DB — lowest latency for small/medium corpora | Whole index must fit in Lambda memory and be (re)loaded on cold start; you own serialization to S3 and rebuild-on-update logic; vector-only, no built-in metadata filtering |
| LanceDB in Lambda | $ Low pay-per-invocation, disk/S3-backed, no idle cost | Disk-backed columnar format (mmap from /tmp or S3) avoids loading the full index into RAM; native SQL-like metadata filtering combined with vector search; append-only writes with versioning, no full rebuild needed | Newer project, smaller ecosystem/community than FAISS; still bound by Lambda's /tmp size and per-invocation compute limits |
| OpenSearch Serverless (NextGen) | $$ Low–Medium scale-to-zero OCUs, but still a standing service + storage bill | True hybrid (BM25 + vector) search, rich range/metadata filtering, HNSW/IVF tuning, mature and widely recognized skill | Overkill at low/bursty volume — you're paying for a managed search service instead of a per-call function; more moving parts (collection, index mappings, IAM) to stand up and operate |
| Managed vector DB (Pinecone, etc.) | $$$ Medium+ often a minimum monthly floor | Fully managed scaling, replication, and ops; good for high-QPS or rapidly growing corpora | Idle cost even with near-zero traffic; another vendor/API to integrate; unnecessary complexity for a small, low-frequency batch job |
When to choose embedded search vs. a serverless service
- Choose embedded (FAISS/LanceDB in Lambda) when: volume is small/bursty (e.g., ~30 JDs/day matched against a resume corpus in the thousands), you want pay-per-invocation cost with zero idle spend, and the goal is to showcase hands-on understanding of embeddings, ANN index types, and the cost/latency trade-offs of rolling your own retrieval layer.
- Prefer LanceDB over FAISS specifically when you need metadata filtering (seniority, location) combined with vector search, or resumes arrive incrementally and you want append/versioning instead of rebuilding the whole index on every update.
- Prefer FAISS over LanceDB specifically when the corpus is small enough to fully fit in Lambda memory, you want the most mature/battle-tested library, and filtering can be handled outside the index (e.g., pre-filter candidates in application code).
- Move to a serverless/managed vector service (OpenSearch Serverless, Pinecone, etc.) when the corpus grows into the millions of vectors, query volume becomes high/steady enough that cold starts and per-invocation index loads hurt, or you need built-in hybrid (BM25 + vector) search and enterprise-grade relevance tuning that would otherwise have to be hand-rolled.
- Embedding model either way: Cohere Embed v3 via Bedrock (asymmetric
search_query/search_documentmodes fit the JD→resume relationship), or Titan Text Embeddings v2 to stay fully within Bedrock.