The Geometry of Intelligence – A lecture

Getting your Trinity Audio player ready…

Welcome, everyone. Please take your seats. Today, we are going to dive into the architecture of Large Language Models—but we are going to look at them through a fundamentally different lens than you might be used to.

The Hook: The Geometry of Intelligence

Imagine you are trying to understand a sprawling, massive metropolis. If you want to know how the city functions, you wouldn’t just count the number of bricks in the buildings. Instead, you would look at the traffic, the major highways, the transit routes, and the detours.

For years, we have measured LLMs by counting the “bricks”—the parameter count. But an LLM is actually a machine for moving, reshaping, amplifying, suppressing, and recombining vectors in high-dimensional space. Today, we are going to stop counting parameters and start interrogating the directional economy of these models. We will explore how four mathematical concepts—PCA, SVD, LoRA, and Attention-Head Subspaces—allow us to understand and engineer the true “directional geometry” that powers modern AI.

Core Concepts & Real-World Analogies

To understand this directional economy, we have four distinct tools. They are cousins, not twins; they all live in the world of vector geometry but ask very different questions.

1. PCA (Principal Component Analysis): The Map of the Traffic PCA asks: Along which directions does a cloud of data spread out the most?. It studies the geometry of the representations themselves.

The Analogy: Think of PCA as looking at a flock of birds in the sky and identifying the main axes of the flock. Alternatively, it tells you where the cars are concentrated in our city. It is purely a diagnostic snapshot of the data cloud.

2. SVD (Singular Value Decomposition): The Map of the Roads While PCA looks at the data, SVD studies the matrix transformation itself. It decomposes a matrix into the main input directions it is sensitive to, the strength of those directions, and the main output directions it writes into.

The Analogy: If PCA is the shape of the traffic, SVD is the shape of the road system. It tells you how the road network channels movement. It is a snapshot of transformation geometry.

3. LoRA (Low-Rank Adaptation): The Scalpel of City Planning LoRA states that instead of fully updating a giant weight matrix during fine-tuning, we can just learn a low-rank update. It modifies behavior by updating only a small number of matrix directions.

The Analogy: If you want your city to adapt to a new industry, you don’t rebuild the entire city from scratch. You just alter a few major transit routes. If SVD is the microscope that shows us the channels of action, LoRA is the scalpel we use to intervene.

4. Attention-Head Subspaces: The Couriers in Motion Attention heads take hidden states and project them into query, key, and value spaces. This defines what the head notices, measures for compatibility, and writes back into the model.

The Analogy: A head is like a little latent courier system. It tunes into certain cues, searches for them, and transports the information back. Unlike PCA and SVD, which are static snapshots, attention heads act in context, making them geometry in motion.

3 Practical Examples of Directional Engineering

How do we use these concepts in a practical, real-world development pipeline?

Diagnosing Model Health with PCA: You can use PCA on a model’s hidden states to understand its representation landscape. For example, if you run PCA, you might discover that a model is underusing large parts of its latent space, or that its representations are heavily redundant in certain dimensions.
Compressing Networks with SVD: Because neural networks are built of matrices, you can use SVD to see if a weight matrix has only a few strong singular values. If it does, that layer is effectively operating in a lower-dimensional way than its raw size suggests, which is incredibly practically relevant for pruning, compression, and efficient training.
Targeted Fine-Tuning with LoRA: Imagine you want your model to become better at medical summarization or legal language. Instead of an expensive global update, you use LoRA to steer the model’s behavior. You nudge the transformation along a restricted set of directions without touching the vast base model.

Common Misconceptions

Before we open the floor, I want to clear up three dangerous misconceptions.

Misconception 1: In PCA, the largest variance means the most important feature.
- Reality: PCA is a map of size, not a map of intelligence. A principal component might just reflect token frequency or corpus artifacts, while a subtle, critically important reasoning feature might live in a low-variance direction.
Misconception 2: LoRA is just an efficiency trick to save compute.
- Reality: LoRA is actually a profound hypothesis about the nature of intelligence in these models. It suggests that task adaptation is fundamentally low-dimensional, and that behavioral shifts naturally cluster into steerable subspaces.
Misconception 3: Every attention head is doing entirely unique, vital work.
- Reality: Spectral analysis reveals that many heads are actually redundant. If multiple heads project into nearly the same effective subspace, they might just be echoing each other or acting as backup copies, allowing us to merge or prune them.

Q&A Section

We have a few minutes left for questions. Yes, in the front?

Student 1: Professor, if PCA and SVD do different things, are they mathematically entirely separate? Professor: Excellent question. They are mathematically related. If you apply SVD to a centered data matrix, you actually recover the PCA structure, because the eigenvectors of the covariance matrix relate to the right singular vectors. However, functionally, they are distinct: we use PCA to find dominant directions in a data cloud, and we use SVD to find dominant directions in a weight matrix.

Student 2: You mentioned attention heads are “geometry in motion.” Can you elaborate on why they aren’t static like PCA or SVD? Professor: Absolutely. PCA gives you a snapshot of a cloud, and SVD gives you a snapshot of a matrix. But an attention head’s effective role depends dynamically on the current token sequence, the current hidden states, and layer depth. They dynamically route information contextually, token by token.

Student 3: How do we make LoRA more effective using these insights? Professor: By combining it with SVD! If SVD reveals that a matrix has dominant singular directions, we can create an “eigenspace-aware” LoRA. We initialize the LoRA matrices along those top singular vectors to align the low-rank update with the model’s most meaningful subspaces, giving us precision steering rather than a generic update.

Student 4: What does it mean for an attention head to have a “clean” directional role? Professor: It means the head has “spectral cleanliness”. If a head’s internal query-key space is noisy or badly conditioned, it will attend too diffusely and confuse irrelevant tokens. If it learns sharp, meaningful directions, it separates signal from clutter and becomes highly selective.

Student 5: Ultimately, why does thinking in “eigenspaces” matter for the future of LLMs? Professor: Because the next leap in LLM effectiveness won’t come from just adding more parameters. It will come when we treat intelligence as the disciplined organization of latent directions. Eigenspaces help us figure out which directions preserve meaning, which carry memory, and which are stable across long contexts. Once we identify these clean directional roles, we can distill them perfectly into much smaller, highly effective models.

Thank you all. Class dismissed!