I often find myself going down rabbit holes. Today’s rabbit hole was Kaggle. I started clicking through starter notebooks for NLP competitions. The kind where people predict which chatbot response a human would prefer, or classify text into categories.

After the third or fourth notebook, something clicked. They all follow the same playbook. Nobody starts from scratch. They grab a pretrained model, wire up a standard architecture, get a baseline running as fast as possible, and then the real work begins. Swapping loss functions, tweaking how vectors are pooled, trying different training tricks. The architecture is a template. The wins come from knowing which knobs to turn.

That got me thinking. The building blocks underneath these notebooks are worth writing down. Not as theory, but as a reference you can actually use when you’re reading competition code and wondering “wait, what is a bi-encoder again?” So I’m starting a series. This is the first entry.

Encoders: How text becomes numbers

You can’t feed raw text into a neural network. You need to turn it into numbers first. An encoder does exactly this. It reads text and produces a vector (just a list of numbers) that captures the meaning.

"I love this app" → Encoder → [0.2, -0.8, 0.5, ..., 0.1]  (768 numbers)

This vector is called an embedding. Similar texts produce similar vectors. Think of it like coordinates on a map. “I love this app” and “This app is great” would end up close together, while “stock market crash” would be far away.

Bi-Encoder: two encoders, working independently

“Bi” just means two. That’s it. Two encoders running in parallel on two different inputs.

Text A → Encoder → Vector A
Text B → Encoder → Vector B

Usually it’s the same model used twice, but the inputs are processed independently. Text A doesn’t know about Text B during encoding. Why does this matter? Because you can pre-compute vectors for a million documents ahead of time, store them, and at search time only compute the query vector once. This is how fast search works.

Cross-Encoder: slower but smarter

Instead of encoding separately, you feed both texts together as one input.

"[CLS] Text A [SEP] Text B" → Single Encoder → score

The model sees both texts at the same time, so it can compare them word-by-word internally. More accurate, but much slower. You can’t pre-compute anything because every pair needs a fresh pass through the model.

Bi-Encoder:    fast,  less accurate  → good for searching millions of docs
Cross-Encoder: slow,  more accurate  → good for re-ranking top 10 results

In practice, many systems use both. Bi-encoder to quickly find the top 100 candidates, cross-encoder to re-rank and pick the best 10.

Classification heads: making decisions

After the encoder gives you a vector, you need to actually decide something with it. The layer that makes the decision is called the head.

Softmax turns raw scores into probabilities that sum to 1. Use this when the categories are mutually exclusive. It’s either spam or not spam, not both.

Raw scores:    [2.0, 1.0, 0.5]
After softmax: [0.59, 0.24, 0.17]   ← sums to 1.0

Sigmoid turns each score independently into a 0 to 1 probability. They don’t need to sum to 1. Use this when multiple labels can be true at once. A photo can be “outdoor” AND “sunny” AND “beach” all at once.

An easy way to remember:

Softmax:  "pick one"  → this email is Inbox OR Spam OR Promotions
Sigmoid:  "pick any"  → this email gets tags: Important + Work + Urgent

Loss functions: how the model learns

A loss function measures how wrong the model is. Training is just minimizing this number. There are different flavors depending on what you’re trying to teach the model.

Cross-Entropy Loss is the standard for classification. Measures how far your predicted probabilities are from the true label. If the model says “90% class A” and it really is class A, loss is low. If the model says “10% class A” and it really is class A, loss is high. Simple.

Contrastive Loss teaches the model that similar things should have similar vectors and different things should have different vectors. You give it pairs and it pushes vectors closer or apart. This is how embedding models learn.

Triplet Loss is the same idea but with three items: an anchor, a positive, and a negative. The anchor should be closer to the positive than to the negative.

BEFORE training:
    Anchor -------- Positive
    Anchor ---- Negative        ← negative is closer, bad

AFTER training:
    Anchor -- Positive
    Anchor --------------- Negative   ← fixed

This is how face recognition works. “Is this the same person?” becomes a distance calculation.

Ranking Loss says given two items, the model should score the better one higher. This is what powers search ranking and the reward models in RLHF.

These form a nice progression:

Cross-Entropy:  "classify this into a bucket"
Contrastive:    "are these two similar or different?"
Triplet:        "which of these two is closer to this reference?"
Ranking:        "which of these should score higher?"

Distance metrics: measuring similarity

When you have two vectors, how do you measure “how similar” they are?

Euclidean distance is the straight-line distance. Like measuring with a ruler. Smaller means more similar.

Cosine similarity measures the angle between vectors and ignores magnitude. This is what embedding search systems use, and for good reason. Two sentences can mean the same thing but have different lengths. Cosine doesn’t care about vector length, only direction.

 1.0 = identical direction (same meaning)
 0.0 = perpendicular (unrelated)
-1.0 = opposite direction (opposite meaning)

Training tricks

Fine-tuning is when you take a model that already learned language from billions of words and continue training it on your small dataset. Like hiring an experienced chef and teaching them your restaurant’s menu. Much faster than training someone from scratch.

Label smoothing is when instead of telling the model “I’m 100% sure this is class A”, you say “I’m 98% sure.” Soft labels like [0.98, 0.01, 0.01] instead of hard labels like [1.0, 0.0, 0.0]. This prevents the model from becoming overconfident and keeps its predictions calibrated.

Architectural patterns

Siamese Network is when the same model processes two or more inputs with shared weights. Named after Siamese twins. It’s not a specific model. It’s a pattern you can apply to any model.

Pooling is needed because transformers output one vector per token. But for classification you need one vector for the whole text. How do you collapse it?

• [CLS] token: BERT adds a special token at the start whose vector becomes the summary
• Average Pooling: average all token vectors together (most stable, very common in competition code)
• Max Pooling: take the max value across all tokens for each dimension (captures strongest signals)

Putting it all together

Now here’s the fun part. Remember those Kaggle notebooks I was reading? Here’s what they actually look like once you know the vocabulary:

Bi-encoder with shared weights (Siamese) processes each input separately → average pooling collapses token embeddings into one vector → concatenate the vectors → dense layer + softmax head for classification → trained with cross-entropy loss and label smoothing.

Every piece is a building block from this post. The competition is about choosing the right combination and fine-tuning it well.

I’ll keep adding to this series as I work through more notebooks. Next up, I want to dig into what makes DeBERTa the go-to model for so many NLP competitions.

TLDR:

Every Kaggle NLP notebook follows the same pattern. Pretrained encoder, pooling layer, classification head, loss function. The building blocks are:

1. Encoders turn text into vectors. Bi-encoders are fast (pre-compute vectors), cross-encoders are accurate (see both texts at once).
2. Softmax picks one class, sigmoid picks many.
3. Loss functions go from classification (cross-entropy) to comparison (contrastive, triplet, ranking).
4. Cosine similarity is how you measure whether two vectors mean the same thing.
5. Fine-tuning a pretrained model beats training from scratch. Label smoothing keeps it honest.

The architecture is a template. The wins come from knowing which knobs to turn.

Numbers between 0 and 1

I often think about what kind of problems people would have faced before inventing things and how the inventions would have solved them. In pursuit of this knowledge, I pondered how math would have been before the invention of real numbers. What kind of gaps were they having in their models? That’s when I came to an understanding that irrational numbers are natural and part of our existence.

There are infinite numbers between 0 and 1. But there are two sets of infinities within this range,

1. Numbers that you get from dividing any one integer by another integer. Eg. Splitting a pizza between 5 friends, everyone gets 20% or 0.2 pizza. We all studied how to get the fraction of any real in school like 0.42069.
2. Numbers that contradict type 1 . Eg. 1.41421356…, 3.14159265358979323846…, 2.71828182845904523536… The ‘…’ simply means the number goes on without repeating itself in a pattern and we cannot get a fraction out of this. But why would we care about a number that goes on infinitely? more importantly, Can I split a pizza between 5 friends using these numbers? It cannot! But it can help you find something better. It can tell you how big of a cheese crust you will get in your pizza. How you ask!

Our options for measuring the length of a circle?

1. Take a thread, and carefully hold it throughout the out edge of the pizza. Mark the point where we surround the pizza.
2. Take a ruler, wash it first, then measure the diameter of the pizza. Find the circumference (ie. Cheese crust length) by using 2$\pi$r.

You guessed it right, the 3.14159265358979323846… that we have seen in the example is $\pi$, an irrational number. This number is the ratio between the length that you measured with the ruler and the length that you measured with the thread. So, if you know the diameter of any pizza, you can just compute how many inches of cheese crust you will get in that pizza.

Oh wait, $\pi$ is never ending and non-repeating, how to decide when to stop

It all depends on the problem we are solving and the accuracy we need. In the last millennia, people have come up with a lot of cool ways to accurately generate this ratio. In our use cases, we can stop whenever we feel the accuracy is good enough for us. In schools, we would have only used the 3.14 approximation of $\pi$.

So the ratio of its diameter to the circumference of a circle that exists in nature is an Irrational number. Likewise, we can find irrational numbers in many real-world models.

More examples

1. Take a slide that we enjoy in parks, for argument’s sake, let’s assume the stairs and floor as two adjacent sides of a square. Then the slide will be the hypotenuse of the square. We want to know how much smooth steel we have to buy for the slide. We can use help from Pythagoras to find that $slide^2$ = $4^2$ + $4^2$, ie slide = 4*$\sqrt{2}$

   For any square with side ‘x’, the diagonal will always be ‘x’*$\sqrt{2}$.

2. In continuous exponential growth or exponential decay, the number 2.71828… also known as Euler’s number plays a vital role. It is being used in modeling population growth, compound interest, and radioactive decay.

3. Golden ratio $\phi$ = 1.161803… . The ratio which we see in the branching of trees, spiral pattern in sea shells, arrangement of leaves, and the Human body is also an irrational number.

TLDR:

Irrational numbers are real numbers that cannot be expressed as a fraction or ratio of two integers. They are non-repeating, non-terminating decimal numbers. We might wonder what are the use cases or real-world scenarios where we might need/use them. Irrational numbers exist in nature.

1. Pi (π): Pi is approximately 3.14159… and is the ratio of a circle’s circumference to its diameter.
2. The Square Root of 2 (√2): The square root of 2 is approximately 1.41421… is used for calculating the diagonal length of a square.
3. Euler’s Number (e) = 2.71828… used in modeling population growth, compound interest, and radioactive decay. and so much more.

This is the importance of irrational numbers. They fill in the gaps and let us model the world in a consistent, smooth and simple way.