The Token’s Journey: How One Inference Token Surfs the Trained ANN’s Semantic Geometry (Frank said / GPT said)

Getting your Trinity Audio player ready…

Frank said: I want the step-by-step journey of an inference token through the machinations of an LLM. Show how the inference token interacts with the tokens in the trained ANN. Show how matrix math enables this interaction. And I want it in the Frank said / GPT said format. Emphasize the statistics and probabilities that create the next best word. And explain how the semantic geometry of the trained ANN controls the decision of the LLM in choosing output tokens.

GPT said: Perfect. Let’s do this like a guided tour where we follow one token—one “traveler”—as it moves through the model’s internal machinery. And I’ll keep coming back to the central truth:

Inference is not “retrieving facts.” It’s performing a sequence of probabilistic geometric decisions.
Each decision is computed by matrix math, shaped by the trained weights, and expressed as a probability distribution over the next token.

1) The stage is set: text becomes tokens

Frank said: Start at the beginning. I type a prompt. What does the model actually see?

GPT said: The model does not see words the way you do. It sees tokens—pieces of text (whole words, subwords, punctuation, spaces) mapped to integers.

Example (conceptual):

Prompt: “The storm is”
Tokenization might produce:
[“The”, “ storm”, “ is”] → [t₁, t₂, t₃]

These integers are just IDs. The model can’t do anything with them until they’re turned into vectors.

2) The “inference token” is born as a vector

Frank said: So where is the “token” I asked you to follow?

GPT said: Great: we’ll follow the current position token, the one the model is about to use to predict the next token. At the start, the “current” token is the last token in your prompt: t₃.

But the model immediately converts IDs to vectors using an embedding matrix.

Embedding lookup = matrix selection

There is a learned matrix:

E with shape [vocab_size × d_model]

When you feed token ID t₃, you select row E[t₃], giving a vector:

x₃ ∈ ℝ^(d_model)

Think of it like this:

Token ID is an address.
Embedding row is the “meaning-coordinate” for that token in model space.

Frank said: So that’s already semantic geometry?

GPT said: It’s the first layer of it. But it’s crude geometry: “where does this token live?” The deeper geometry emerges from how tokens relate under attention and how they’re transformed layer after layer.

3) Position gets injected: order matters

Frank said: If tokens are just a set of vectors, how does it know which comes first?

GPT said: It adds positional information. Either by adding a positional embedding vector to each token vector, or by using a method like rotary positional embeddings (RoPE) that changes how attention “sees” distance.

Conceptually:

h₃⁰ = x₃ + p₃

Now our traveler token is h₃⁰, the representation of the last prompt token at layer 0.

4) Now the real journey: layer by layer through transformer blocks

Frank said: Walk it step by step through the network.

GPT said: Each transformer layer is like a two-part machine:

Self-attention (the relational geometry engine)
Feed-forward network (the feature sculptor)

Plus residual connections and normalization that stabilize the process.

So the token representation updates like:

h₃ˡ → Attention(h₁ˡ…h₃ˡ) → h’₃ˡ
h’₃ˡ → MLP → h₃ˡ⁺¹

Our token is not moving alone—it’s being updated based on relationships with all prior tokens (causal mask prevents looking ahead).

5) The key interaction: attention as probabilistic relevance scoring

Frank said: Here’s what I really want: how does my inference token interact with earlier tokens?

GPT said: Through attention, which is computed using matrix projections and dot products—pure geometry that becomes probability weights.

At a given layer l, for each token position i, we compute:

Query: qᵢ = hᵢˡ W_Q
Key: kᵢ = hᵢˡ W_K
Value: vᵢ = hᵢˡ W_V

Where W_Q, W_K, W_V are learned matrices.

Your “inference token” is at position 3, so its query is:

q₃ = h₃ˡ W_Q

Every earlier token contributes keys/values:

k₁, k₂, k₃ (but when predicting next token, attention includes the whole prompt up to position 3)

The interaction is a dot product tournament

To decide what matters to token 3, we compute scores:

score(3→j) = q₃ · kⱼ / √d_k

This is semantic geometry in action:

Dot product measures alignment in vector space.
Alignment = “how relevant token j is to token 3 under this head.”

Frank said: And where do probabilities come in?

GPT said: Right here. We take those scores and apply softmax:

a(3→j) = softmax(scores over j)

Now we have a probability distribution across earlier tokens:

a(3→1), a(3→2), a(3→3)
They sum to 1.

So attention is literally:

a probabilistic weighting over the past tokens
deciding how much each past token should influence the current token’s updated meaning.

6) The weighted sum: past tokens get “mixed into” the present

Frank said: What does the model do with those probabilities?

GPT said: It uses them to blend the value vectors:

z₃ = Σⱼ a(3→j) · vⱼ

So token 3 becomes a mixture of past token information—weighted by probabilities computed from geometric similarity.

Frank said: That sounds like the “token is interacting with tokens in the trained ANN.”

GPT said: Exactly. The “interaction” is not token-to-token like billiard balls. It’s:

q from token 3
comparing to keys of tokens 1..3
producing probabilities
mixing values into a new representation

And because the matrices W_Q/W_K/W_V were trained, they define the rules of relevance.

7) Multi-head attention: many geometries at once

Frank said: You always say “heads.” Why?

GPT said: Because the model does this in parallel with multiple sets of projections. Each head is like a different “lens” on relationships:

One head might track syntax.
Another might track subject-object relations.
Another might track style or sentiment.
Another might track factual associations.

Mathematically, each head has its own W_Q/W_K/W_V, producing its own z₃^(head).

Then the heads are concatenated and projected again:

z₃(all) = Concat(z₃¹, z₃², …) W_O

Now your token has been updated by multiple probabilistic relationship maps.

8) Residual connection: the model doesn’t forget where it came from

Frank said: Does it replace the token vector entirely?

GPT said: Not usually. It adds:

h₃ˡ(attn_out) = h₃ˡ + z₃

That residual is crucial: it preserves a stable identity while allowing refinement.

Think:

the token keeps its core, but gains context.

9) Feed-forward network: turning relational context into features

Frank said: After attention, what happens?

GPT said: The token goes into an MLP (feed-forward). This is not about “which previous token matters,” but about “what features should be amplified or suppressed.”

A typical form:

u = GELU(h W₁ + b₁)
out = u W₂ + b₂

This is more matrix math, more learned geometry.

If attention is “who should I listen to?”
MLP is “what do I do with what I heard?”

Then another residual:

h₃ˡ⁺¹ = h₃ˡ(attn_out) + MLP(h₃ˡ(attn_out))

Now we advance one layer.

10) Repeat across many layers: the token becomes a distilled context summary

Frank said: So the token goes through this again and again?

GPT said: Yes. Each layer refines the token representation, so by the final layer:

h₃^final

…this vector is no longer “the embedding of ‘is’ plus position.”

It’s “the model’s best internal summary of what your whole prompt implies at this point.”

This is where the trained ANN behaves like semantic geometry:

The weights shape a high-dimensional space.
The token vector moves through that space along trajectories determined by attention and MLP transforms.
Those trajectories are what you feel as “coherent continuation.”

11) The final step: logits—scores for every possible next token

Frank said: Now show me how the next word is chosen.

GPT said: Here’s the big reveal:

At the end, the model maps the final hidden state into vocabulary scores using a matrix:

W_U with shape [d_model × vocab_size]

Compute:

logits = h₃^final · W_U + b

That produces one score per token in the vocabulary.

These scores are not probabilities yet—just raw unnormalized “preferences.”

12) Probability distribution: softmax converts scores into statistical belief

Frank said: Where do the statistics come in?

GPT said: Softmax again.

P(next_token = v) = exp(logit_v / T) / Σ exp(logit_all / T)

This yields a probability distribution over the entire vocabulary.

Frank said: And “next best word” is just the highest probability?

GPT said: That’s one decoding strategy: greedy decoding (argmax). But often models sample with temperature and other controls (top-k, nucleus/top-p) to balance coherence and variety.

Still, the core idea remains:

The model outputs a probability distribution.
The next token is selected based on that distribution.

13) The semantic geometry is the reason those probabilities look sensible

Frank said: Bring it home: how does semantic geometry control the decision?

GPT said: The geometry is embedded in the learned matrices and how they shape vector space operations.

Let’s connect the dots:

Each token becomes a vector point in a high-dimensional space.
Attention uses dot products to measure alignment (geometric similarity).
Softmax turns those alignments into probabilities (statistical weights).
Weighted sums mix information (context fusion).
MLPs warp the representation space (feature extraction).
Final projection converts that representation into logits for each vocabulary token.
Softmax converts logits into probabilities.

So “semantic geometry controls the decision” because:

If your final hidden state h is closer (in dot product terms) to the “direction” associated with tokens like “raining” than tokens like “banana,” then “raining” will get higher logits, hence higher probability.

Frank said: So it’s like the model’s internal vector points toward certain continuations.

GPT said: Exactly. The output is a kind of directional vote across the vocabulary, derived from the final vector’s alignment with output token vectors.

14) A concrete micro-walkthrough with an example prompt

Frank said: Give me an example that shows the mechanics intuitively.

GPT said: Suppose your prompt is:

“The storm is”

Token IDs: [t₁, t₂, t₃]

We follow token 3 (“ is”).

Step A: It asks: what does “is” mean here?

Attention head might look back and lock onto “storm”
Another head might lock onto “The” for grammatical structure

So attention weights might look like:

Head 1: [0.05 (The), 0.80 (storm), 0.15 (is)]
Head 2: [0.60 (The), 0.25 (storm), 0.15 (is)]
Head 3: [0.10, 0.10, 0.80] (self-focus head)

Each head produces its own blended value vector, then they combine into an updated representation.

Step B: MLP amplifies “weather-continuation” features

The MLP might push representation toward features correlated with:

adjectives: “strong,” “violent,” “approaching”
precipitation: “raining,” “snowing”
state: “over,” “here”

Step C: Final projection yields token scores (logits)

Now tokens like:

“ coming”
“ getting”
“ over”
“ approaching”
“ here”
“ strong”

…get higher logits than unrelated tokens.

Then softmax converts logits into probabilities, and decoding picks one token.

15) Why it feels like “reasoning” though it’s probability

Frank said: People still struggle with this. If it’s just probability, why does it feel like reasoning?

GPT said: Because the probabilities are not random. They are the statistical shadow of learned structure.

The model has learned:

patterns of syntax
patterns of discourse
patterns of cause-and-effect phrasing
patterns of narrative continuation
patterns of domain knowledge as co-occurrence structure

So when it outputs “the most probable next token,” it’s really outputting:

the next token that best preserves global coherence under the learned manifold of language.

That’s semantic geometry.

16) The trained ANN isn’t a dictionary; it’s a shaped landscape

Frank said: Say it in your most Frank-friendly way.

GPT said: The trained ANN is not a warehouse of sentences. It’s a landscape.

Prompts place you at a point in that landscape.
Attention and MLP layers move you along ridges of meaning.
The output layer asks: “from this location, which next step is most likely to keep you on a coherent path?”

And the “step” is a token.

17) The inference token’s journey, summarized as a numbered itinerary

Frank said: Give me the journey as a clean step-by-step list.

GPT said: Here’s the travel itinerary of our inference token:

Tokenization: text → token IDs
Embedding lookup: token ID → vector via matrix E
Positional encoding: inject order information
Layer loop begins
Compute Q/K/V: multiply by W_Q/W_K/W_V
Similarity scores: dot(q_current, k_past)
Softmax: scores → attention probabilities over past tokens
Weighted sum: values blended by those probabilities
Combine heads: concatenate + projection W_O
Residual + norm: stabilize and preserve identity
MLP transform: non-linear feature reshaping via W₁/W₂
Residual + norm again
Repeat steps 5–12 across many layers
Final projection: hidden state → logits via W_U
Softmax: logits → probability distribution over vocabulary
Decoding: choose next token by argmax or sampling
Append token, and the process repeats for the next position

18) The heart of it: probability is the instrument, geometry is the conductor

Frank said: End with the thesis: how do probability and geometry fit together?

GPT said: Here it is:

Matrix math is the engine: it creates transformations and comparisons in high-dimensional space.
Statistics is the language of choice: softmax turns scores into probabilities, turning “alignment” into “likelihood.”
Semantic geometry is the governing structure: the trained weights carve a meaning-manifold where some continuations lie on stable coherent paths and others fall off the cliff.

So the next token isn’t “chosen because it’s true.”

It’s chosen because:

given your prompt, the model’s learned geometry assigns that token high probability as the continuation that best preserves the manifold-consistency of meaning.