Category: Uncategorized
-
MAC, dot product, and matrix multiply – frank said/gpt said
Frank-said: I keep seeing these three terms together in LLM discussions: MAC, dot product, and matrix multiply. They seem related, but they are not the same thing. Define them cleanly. GPT-said: Good. The cleanest way to understand them is as a hierarchy of scale. A MAC is the smallest arithmetic work unit.A dot product is…
-
Where the Dot Products Actually LiveA third Frank-said / GPT-said sequel on scalar dot products, matrix multiplies, and how each transformer equation expands into MAC counts
Frank-said:Redo it, but make it sharper. I want the focus to be brutally clear: where exactly do the dot products live inside the equations of one transformer block? I want to see the difference between a scalar dot product and a matrix multiply. I want to see how a matrix multiply is really a giant…
-
Inside One Transformer BlockA deeper Frank-said / GPT-said sequel on tensor shapes, attention equations, and the MLP path in plain English
Frank-said:All right, let’s go one level deeper. I do not just want the broad story anymore. I want to walk inside a single transformer block and actually look at the tensor shapes as the prompt flows through it. I want to see the algebra, but I also want it translated into plain English so the…
-
Why the Math Load of an LLM Is Bigger Than It First LooksA Frank-said / GPT-said dialogue on matrix shapes, dot products, MACs vs FLOPs, and why MLP cost often dominates attention during prefill
Frank-said:When people talk about an LLM doing “dot products,” I think the phrase sounds deceptively simple. It makes the process sound like the model is just doing one kind of little comparison over and over again. But the more I look at it, the more I think that hides the real picture. A dot product…
-
One Token Through One Transformer Block – A Frank-said / GPT-said numerical walk through with toy values
Frank said: Do that. I want to actually watch one token go through the machinery with numbers, even if they are toy numbers. GPT said: Good. We will use tiny fake numbers so the math stays visible. Real models might use: But here we will use: That lets us see the whole thing without drowning…
-
Two Attention Heads, Real Tensor Shapes, One Transformer Block – A deeper Frank-said / GPT-said walkthrough with explicit shapes and a more realistic flow
Frank said: Do that. GPT said: Good. Now we will move from the tiny cartoon version to a more realistic transformer-style example. Not giant real-model scale, but realistic enough to show: We will still use toy numbers, but the structure will now look much more like what actually happens. 1. The setup GPT said: Let’s…