Vectors, Dot Products, and the Shape of Meaning

Getting your Trinity Audio player ready…

Vectors, Dot Products, and the Shape of Meaning
A layman’s tour of the math inside modern language models

1 Why bother with math at all?

When you ask your phone “What’s the weather?” you trigger an avalanche of arithmetic hidden deep inside the tiny chip. Language models cannot “see” letters or hear syllables; they only manipulate numbers. The central trick is to turn words into numbers in such a way that arithmetic on those numbers preserves something about meaning. That trick relies on two ideas:

A vector—a list of numbers that behaves like an arrow in space.
A dot product—a single arithmetic step that measures how two arrows line up.

Together, vectors and dot products let a computer treat “cat,” “kitten,” and “feline” as near-neighbors while keeping “cat” far from “thermodynamics.” The entire neural network—the artificial brain we call a model—is just a vast factory for creating, stretching, tilting, and comparing these arrows. Its levers and gears are the weights and biases learned during training.

This essay walks through that factory in plain language. We will start with vectors, visit the dot product, peek inside the matrices that house weights and biases, and finally see how the whole assembly line converts a question from you into an answer from the model.

2 From words to coordinates

Imagine sending a package. You give the courier a street address—a set of numbers (house number, ZIP code, maybe GPS coordinates). The courier does not care about your living-room décor; they only care that the numbers guide them to the right doorstep.

Language models do something similar. They assign every word or sub-word piece (called a token) an address in a fictional semantic city with hundreds or even thousands of dimensions instead of just two. A token’s address is its vector. For today’s most capable models those vectors often have 1,024 or 2,048 coordinates.

At first those addresses are random. During training the model repeatedly writes a token, checks whether it landed in the “right neighborhood,” and nudges the address until similar tokens become neighbors. Over millions of rounds, verbs drift toward verbs, plurals toward plurals, and “Paris” sidles up beside “London.” The precise neighborhoods are determined not by a human cartographer but by the model’s experience of vast text corpora.

3 What does “multi-dimensional” feel like?

Two dimensions are easy: north–south, east–west. Three add height. Beyond that, human imagination stalls. Yet a familiar everyday object—a spreadsheet—offers a clue. Each row in a 1,000-column spreadsheet is a point in 1,000-dimensional space; each column is a coordinate. Your bank’s customer database lives in hundreds of dimensions (balance, credit score, birthday, …). We never “see” that space, but we work with it by adding, subtracting, averaging, and sorting columns. Language vectors live in an invisible spreadsheet exactly like that.

Why so many dimensions? Because meaning is messy. Any small set of axes—say happy–sad, formal–casual, or abstract–concrete—captures only a sliver of nuance. High-dimensional space acts like a huge filing cabinet where each drawer can store one subtle facet: plural-singular, masculine-feminine, musical-not-musical, and so on. The cabinet does not come labeled; training gradually scrawls ad-hoc labels on drawers as the model learns.

4 The dot product: a single yes-or-no number

Take two arrows on a piece of paper. If they point in exactly the same direction, their dot product is large and positive. If they are perpendicular, the dot product is zero. If they point opposite ways, the dot product is large and negative.

Formally, you multiply matching coordinates and add the results. For two 1,024-length vectors that is 1,024 multiplications and 1,023 additions—nothing more exotic than middle-school arithmetic.

Why is this useful?

Similarity test. The closer two meanings are, the larger their dot product—just like two arrows pointing the same way.
Projection. The dot product also tells you how much of one idea lies along the direction of another. For example, a “spaghetti” vector may have a sizeable component along a “food” direction but almost none along a “planet” direction.

Computers adore the dot product because it is fast, parallel-friendly, and easy to differentiate (a boon for learning). Modern GPUs can compute billions of dot products per second.

5 Bundles of arrows: matrices

If a vector is a single arrow, a matrix is a bundle—rows and columns of numbers, each column often holding one arrow. Multiplying two matrices is, at heart, performing countless dot products between rows of the first matrix and columns of the second. That is why you often hear “matrix math” and “dot product” in the same breath: the former is built from industrial-scale repetition of the latter.

Inside a language model, matrices serve two main roles:

Embeddings. Turning token IDs into their starting vectors.
Transformations. Rotating, stretching, and mixing vectors as they flow through layers.

Both rely on numbers called weights. A weight is simply one cell in a matrix. Adjust every weight, and you reshape the whole vector bundle—a bit like bending the metal teeth of a gear to make it spin differently.

6 Weights: the adjustable knobs

Picture an old-fashioned equalizer on a stereo: a row of sliders controls bass, midrange, treble. During a concert a sound engineer nudges sliders up or down until the mix feels right.

Weights are the sliders of a neural network. Each weight says, “When coordinate i of this input meets coordinate j of that input, amplify it this much.” Early in training the sliders are near random: noise blasts from the speakers. Training gently lowers one slider, boosts another, back and forth, until the overall sound—the model’s predictions—matches human-produced text.

Because a single weight’s impact is tiny, language models use millions or billions of weights to attain subtlety. Yet the underlying adjustment principle remains as simple as sliding a volume knob.

7 Biases: the hidden dials

If weights are sliders, biases are the “master volume” knobs that shift everything up or down before the music leaves the mixer. Mathematically, a bias adds a constant to a vector coordinate after the dot products are done. That small nudge can make all the difference between an activation that stays silent and one that fires.

Biases also help the model learn faster: they give each neuron (think mini-function inside the network) freedom to decide when zero input should already count as “something.” The combination weighted sum + bias is the canonical building block of neural computation.

8 Training: correcting mistakes with back-prop

How does the model decide which sliders and knobs to tweak? Through an algorithm called back-propagation:

Forward pass. Feed text in, watch the model’s guess for the next word.
Loss calculation. Measure how wrong that guess is (e.g., “probability of the real next word was only 5%”).
Backward pass. Use calculus to trace which weights and biases contributed to the error and by how much.
Update. Nudge each culprit weight a hair in the direction that would have made the guess less wrong.

Repeat on billions of sentences and the model gradually sculpts a space where sensible prose occupies the low-error valleys. In a literal sense, training carves the hills and valleys of the high-dimensional landscape. Weights are the chisels and picks; biases are the fine-grit sandpaper.

9 The Transformer: dot products at industrial scale

The architecture running today’s models is called the Transformer. Its signature move is attention, implemented almost entirely with dot products:

Every token creates three vectors—query, key, and value—by multiplying the token vector with learned weight matrices.
The model computes the dot product of each query with every key in the sentence. Big dot product ⇒ high similarity.
Those dot products are turned into attention scores (after a softmax probability step).
Each token’s output is a weighted blend of the value vectors, where the weights are those attention scores.

Thus, a Transformer literally decides “which other words to pay attention to” by comparing arrows with the dot product and then remixing them with—once again—weights.

After attention comes a feed-forward network, two more weight matrices separated by a nonlinear squish. It, too, boils down to matrix multiplications dotted with biases. Stack dozens of these layers end-to-end and you have GPT-style depth.

10 Why the dot product and not some fancier trick?

Because the dot product hits a sweet spot:

Linearity. Linear functions are easy to compose, fast to differentiate, and stable during training.
Hardware synergy. Graphics processors were born to crunch dot products for 3-D games; AI piggybacks on that engineering.
Interpretability. Even researchers can plot similarity heatmaps from dot products and intuit why one word attends to another.

More complex similarity metrics exist, but none deliver the same trio of speed, stability, and simplicity at billion-scale.

11 High-dimensional intuition hacks

Although we cannot picture 1,024-dimensional space, we can animate small slices:

PCA or t-SNE plots project vectors down to two dimensions, revealing clumps of verbs, adjectives, or animal names.
Cosine similarity (normalized dot product) graphs show which emotions cluster around “joy,” “grief,” or “anger.”
Vector arithmetic creates word puzzles: vector(“king”) – vector(“man”) + vector(“woman”) ≈ vector(“queen”). Each subtraction or addition is itself built from dot products lurking inside the matrix operations.

These glimpses assure us the invisible landscape is not random—it bears the fingerprints of grammar and semantics.

12 Where weights live after training

You might picture a trained model as a static sculpture. In reality it is a giant table of numbers stored on disk: one file for weights, one for biases, plus metadata. Loading the model into memory is like reheating a frozen meal; inference (answering a prompt) is like tasting it. No weights change during inference—dot products and matrix multiplications run forward only. Training is the cooking phase; inference is serving.

Because weights and biases fully define the model, they are precious intellectual property. Compressing them (“quantization”) or pruning unimportant ones saves hardware without losing flavor—akin to trimming fat off meat.

13 Everyday examples of dot-product magic

Autocorrect. Your phone compares the typed vector stream with vectors of known words; a high dot product between “thw” and “the” triggers a suggestion.
Search engines. Query vectors dot-product against billions of document vectors; higher scores rank higher.
Music recommendation. User-preference vectors and song-feature vectors meet in a dot product; the bigger the number, the more the algorithm thinks you will like the song.

Each application stores meaning in weights and biases tailored to its data, but the underlying math is identical.

14 Pitfalls: when arrows mislead

Dot-product similarity can also reinforce stereotypes: if the training text over-associates certain professions with one gender, those vectors drift closer, embedding bias into the space. Researchers explore debiasing by adjusting vectors or retraining with curated text, yet the challenge remains active.

Moreover, high-dimensional intuition sometimes fails. Two random points in a thousand-dimensional cube are almost always far apart, so models must be careful not to treat everything as unrelated noise. Clever initialization schemes and normalization layers help keep distances meaningful.

15 Conclusion: the algebra of understanding

Strip away the jargon, and a modern language model is a symphony of dot products orchestrated by learned weights and biases. Vectors store addresses for ideas; the dot product measures closeness; matrices full of weights bend the addresses into ever-richer spaces; biases fine-tune timing and volume. Back-propagation slowly tunes every knob until the symphony sounds like coherent language.

That is why talking to ChatGPT is less like querying a rule-book and more like strolling through a continent of meaning shaped by arithmetic. Each sentence you type launches a caravan of vectors across that landscape, guided by weights that whisper, “This way lies the most plausible reply.”

So next time your phone autocompletes a message, picture billions of tiny arrows lining up, one dot product at a time, to translate your fleeting thought into text on a screen. Math—humble multiplication and addition—has become the loom on which modern language is woven.

Do you like this personality?

Deep research

Create image

ChatGPT can make mistakes. Check important info.