Ch 3 Training with Softmax Loss (original SBERT)Apply

The historical first recipe for turning BERT into a sentence embedder using labelled NLI data.

Core concepts

  • NLI datasets. Sentence pairs labelled entailment / neutral / contradiction (SNLI, MNLI) provide supervision for “do these mean related things?”.
  • Siamese network. Two encoders sharing weights embed each sentence; the two vectors (u, v) plus their difference |u−v| feed a softmax classifier over the three labels.
  • Softmax / classification loss. Training the classifier reshapes the embedding space so that semantically related sentences land near each other.
  • Known weakness. Softmax loss is now considered a relatively weak objective — later chapters (MNR) produce better embeddings.
Sentence ASentence B“A man eats.”“Someone dines.”BERT encoderBERT encodershared weightsuvconcat(u, v, |u−v|)softmax classifier → entailment / neutral / contradiction
Siamese SBERT: shared encoder embeds each sentence, and concat(u, v, |u−v|) feeds a 3-way softmax trained on NLI labels.

Under the hood: the 3-way softmax classifier here is trained with PyTorch’s CrossEntropyLoss — see Chapter 3.5 for a goal/input/target/code comparison of CrossEntropyLoss in Softmax loss vs. MNR loss.

What you must master

  • Explain what NLI data is and how entailment labels supervise similarity Level 1
  • Describe the siamese/softmax architecture at a block level Level 1
  • Fine-tune an SBERT model with softmax loss on a labelled pair dataset Level 2
  • Judge when softmax loss is “good enough” vs. when to reach for MNR Level 2

Architect’s lens

This is your baseline mental model for supervised embedding fine-tuning. The key architectural takeaway: the training objective shapes the geometry of the space your vector DB will search. Knowing softmax is dated tells you to prefer the stronger objectives that follow — don’t ship a weak embedder just because a tutorial used it.