Ch 10 Data Augmentation with BERT — AugSBERTApply

Using a slow, accurate cross-encoder to label data for a fast bi-encoder.

Core concepts

  • The scarcity problem. Small labelled datasets give weak bi-encoders.
  • AugSBERT recipe. Train a cross-encoder on the small labelled set → use it to label many new sentence pairs (silver data) → train the bi-encoder on the enlarged set.
  • Pair sampling. Which unlabeled pairs to score matters (random, BM25, kNN sampling) to avoid a flood of trivial negatives.

What you must master

  • Explain the cross-encoder-as-labeler (“silver data”) idea Level 1
  • Implement the AugSBERT train→label→train loop Level 2
  • Choose a pair-sampling strategy that yields informative pairs Level 2

Architect’s lens

A pragmatic pattern: spend a strong-but-slow model’s accuracy once, offline, to cheaply expand training data for a fast model you’ll serve at scale. This “distill accuracy into speed” mindset recurs across Gen AI system design (e.g. using a big LLM to generate fine-tuning data for a small one).