← Back to videos

Anthropic pays $750,000+ a year for engineers who can build LLM architectures from scratch. Stanford

Ishika Rawat

Ishika Rawat

@Ishh_021

November 18, 2026· 89:02

Anthropic pays $750,000+ a year for engineers who can build LLM architectures from scratch. Stanford taught the entire thing in 1 hour lecture & released it for free. Bookmark & watch this today before someone takes it down and read this article below. https://t.co/t9JuPgqs3h

Transcript

So today we're going to talk about architecture
0:02
which
0:02
at least to me
0:03
has always been pretty inscrutable
0:06
and so I'm going to take the approach
0:08
of just telling you kind of everything
0:10
right? I'm going to go through all of the modern papers
0:13
and we're going to just look through what
0:15
has everyone done
0:17
and so I've titled this Everything You Didn't
0:18
Want to Know About Architecture and Hyperparameters
0:21
because I think we all wished
0:23
we lived in a world where the only
0:24
things you had to know were like VC dimension or something
0:26
right? Like very simple
0:28
you know, theoretical tools
0:29
but that's not really where we are
0:31
So okay
0:32
what we are going to do
0:33
is we are going to
0:35
try to understand architecture from kind of like a survey lens
0:38
right?
0:39
The best thing to do
0:40
you know, better than listening to this lecture even
0:42
is for you to go out and like
0:43
train your own models and try different architectures
0:45
right? That's by far the best thing to
0:47
do
0:48
That's part of the philosophy of the course
0:50
But
0:50
we're not going to be able to cover
0:52
the whole design space of all the different
0:53
architectures that are out there
0:55
right? Like that's not something that we have
0:56
the compute or the time to do
0:58
So
0:59
my opinion is the second best thing that
1:01
we could do
1:02
is to try to learn from the experience of others
1:04
right? What has, what has everyone else done?
1:06
What are the choices that they are making
1:08
right?
1:08
And by looking at kind of a broader
1:10
somewhat zoomed out picture
1:11
maybe we can start to understand
1:13
oh
1:13
these are the kinds of parameters and choices
1:16
that are sort of fixed across all effective architectures
1:18
and these other ones can be sort of varied without impacting
1:22
how the model perform
1:23
right? So I'm going to talk about
1:24
you know, basically transformer variants, like what is the
1:27
you know, modern transformer starting with
1:30
you know, the Vaswani paper, and then, you know, as we go to more modern
1:34
more recent architectures
1:36
what do they have in common?
1:37
And then what are we allowed to vary
1:38
or not allowed
1:39
but what do people vary as they go through this
1:42
right?
1:43
So
1:44
I think many of you have taken an
1:45
NLP course of some kind
1:47
or at least seen a transformer
1:48
so you've probably seen
1:50
you know
1:50
the very vanilla transformer from Vaswani et al
1:54
you know, there
1:56
There are some fairly standard choices that you make
1:57
You say, oh, transformers don't have positional dependence
2:00
so we're going to add a position embedding
2:02
And what do we do?
2:03
We're going to add some sines and cosines
2:05
we're going to have information processing through
2:08
ReLU
2:09
and then we're going to have a
2:10
a post norm. I'll talk about what exactly
2:12
that is later
2:13
and when you look at your assignment
2:15
your A1
2:16
you're going to notice some differences between
2:18
the standard or the vanilla transformer
2:21
and what we've asked you to implement
2:22
Well
2:22
we're going to ask you to move the
2:23
layer norm to the front of each transformer block
2:26
or the non-residual layers
2:28
We're going to ask you to implement something
2:29
called rope
2:31
and
2:31
we're going to ask you to implement something
2:33
called swiglu and not ReLU
2:35
Right
2:36
Why do we pick these?
2:37
one reason is we've
2:38
you know, copied a lot of this over from Llama
2:41
but so did everyone else
2:42
Really, I think
2:43
if you were to train on your own language model
2:45
I think you'll quickly run into this question of
2:47
oh, there's so many choices
2:48
right? Like
2:49
what do I choose for all these things?
2:51
And so
2:52
let's now
2:53
sort of walk through all these different models
2:55
The way I kind of think about architectures
2:58
is to think about
2:59
to look at all the different things people
3:00
have done and say
3:01
what are the things that people have done?
3:03
Can we pick and choose from those?
3:06
Percy always makes fun of me for this a little bit
3:08
but, you know, I try to look at all the
3:10
the different models that come out each year
3:11
to try to make this lecture
3:14
and last year I thought
3:15
oh, there's just a couple papers
3:16
it's gonna be fine
3:17
it's gonna be fine
3:17
and then I looked through all the things
3:19
and there's a lot of papers
3:20
there's QEN2 and GEMA3
3:22
and internal M2, and then there were even more
3:25
there's like Nemotron4 and QEN2
3:27
and oh, oh my goodness
3:28
there were 19 new dense models
3:30
And so last year
3:31
I had my work cut out for me
3:33
And this year
3:34
you know, I thought, well
3:35
there can't be that many new LM releases
3:37
Like, it's gotta be slowing down
3:38
right? Like
3:39
people can't keep training 20 dense LMs per year
3:42
I mean, that's technically right
3:43
there aren't that as many dense LMs
3:44
Initially, you know, I was like, oh, there's Qwen 3
3:47
Gamma 4 just came out last Thursday
3:48
so I put that in there
3:49
and almost 3. You know
3:50
there's only a couple
3:51
and of course I have to
3:53
give a shout out to Percy's own 8B model
3:55
trained with Marin, and I was like
3:56
oh
3:56
we'll just have a few things to cover
3:58
But it turns out
3:59
if you start looking
3:59
there's a lot of different models
4:02
And so, the fact that we have so many different models
4:05
most of these actually are MOEs
4:07
mixtures of experts
4:08
and I'll be talking about that tomorrow rather
4:09
than today
4:10
Because we have such a big diversity of models
4:12
we actually get A pretty good picture
4:15
of all the different choices that we can
4:17
make
4:18
So I made this little table
4:20
We'll come back to this little table at
4:21
the end of the lecture
4:23
But basically at this point
4:25
starting with the original transformer
4:27
there's been actually quite a few
4:30
autoregressive language models
4:31
kind of trained on the same class of
4:33
things
4:34
And you can ask questions like
4:36
what are the different vocabulary sizes?
4:38
Or what kind of layer norms do we
4:40
use?
4:40
Or
4:41
what kind of position embeddings do people use?
4:44
And we see some fairly clear trends
4:45
I'll be talking about this as we go
4:47
Okay
4:49
So
4:51
the goal here
4:52
is that we're going to cover
4:53
a couple of different things
4:54
We're going to cover common architecture variations
4:57
So these are different building blocks of the
4:58
transformer
5:00
and after we've established what the standard building blocks are
5:03
like, you know, what do we use for the
5:05
the nonlinearities, or what do we use for position embeddings
5:08
then we're gonna talk about hyperparameters
5:10
We're gonna go down even lower detail and say
5:12
like, you know, what is FFDIM?
5:14
should we make that a multiple of four
5:16
or, like, multiply the
5:17
the hidden by four to get FFDIM?
5:19
how many vocab elements should I have?
5:22
and then, after that
5:23
we're gonna talk about very low-level tricks
5:26
of how to get models to train stably
5:29
And the reason why I'm gonna talk about
5:30
that in this lecture
5:31
is because these stability tricks
5:33
have a pretty close connection with the architecture variation
5:37
right?
5:38
one of the things that
5:39
you know
5:39
higher level I wanna sort of impress upon
5:41
you is that architectures are actually a
5:43
a very complex set of trade-offs
5:45
right? Like, what does a architecture have to do?
5:48
Well, it has to learn from data
5:49
so it has to generalize
5:51
it has to train efficiently on GPUs
5:53
and it has to not blow up
5:54
right? Like halfway through training
5:56
if your, you know
5:57
training
5:58
losses just go like down like this and
5:59
then suddenly blow up
6:01
that's no good at all
6:01
right?
6:02
So all these different requirements
6:04
end up getting baked straight into the architecture
6:06
and that's why these things are a little
6:08
bit messy and a little bit complex
6:10
but you should keep that in mind
6:11
and that's why, you know, things are in
6:13
in many ways not so elegant
6:15
Okay
6:16
So
6:17
we're gonna start with the core architecture piece
6:20
And as a high level view
6:22
you know, I think the
6:23
the way that I see a lot of the architecture stuff
6:25
you know, looking basically historically
6:27
is kind of in the early days of
6:29
you know, starting with the transformer until
6:31
you know, GPT-3 or so
6:33
there's a lot of experimentation that happens
6:35
People try lots of different things
6:37
there's no like gold standard that everyone has
6:39
unified on
6:40
And then, you know, Llama 2 comes out
6:42
and everyone's like, wow, Llama 2 is great
6:43
I want my own Llama 2
6:45
And so everyone starts training Llama 2-alikes with
6:48
you know, minor variation, that people have
6:52
And then finally, you know, last year
6:54
we saw really big differences or
6:56
or sort of a trend towards
6:58
architecture modifications that make training more stable
7:01
And this year, we see lots of trends towards
7:04
architecture variations that enable longer context dependence
7:08
So there are these big themes that are happening
7:10
but really I think
7:11
you know, you see this like big point
7:13
when Llama 2 comes out
7:14
and everyone's like, wow
7:15
I wanna train some with that
7:16
And then suddenly, or not suddenly, but after that
7:18
people are starting to explore once again
7:19
So it's kind of cool to see
7:21
all of these different changes
7:24
Okay
7:25
I think
7:26
people can disagree about a lot of things on architectures
7:29
but there is one thing that everyone agrees on
7:32
you know? Like, if you take the transformer paper
7:34
I think a lot of people will say
7:35
like, the transformer people got
7:37
like, most of the things right
7:39
except this
7:40
And the thing that they really did not get right
7:42
or like
7:42
I think most people agree they did not get right
7:45
is where you put the layer norm
7:47
right? So in the original
7:50
transformer paper, you know
7:52
the layer norm
7:53
goes in what you would call the residual path
7:55
right? So in the transformer
7:56
you know, you have the residual stream
7:58
this X, that kind of runs through the whole network
8:00
and then you are going to be applying a tension
8:03
which, you know, does some computation
8:05
and it will
8:05
add back sort of a delta
8:07
back into the residual stream
8:10
And then
8:10
in order to make sure that these gradients
8:12
are sort of stable across layers
8:14
you know
8:15
a layer norm is placed at the end
8:17
of each of these components
8:19
Now, instead of putting the layer norms in the residual stream
8:24
there's an alternative, I'll refer to this as pre-norm
8:28
in which
8:29
you can put the layer norm outside of the residual stream
8:32
but before each of the computations
8:34
So you can put it before the multi-head attention
8:36
you can put it before the FFN
8:38
right?
8:38
we'll call this pre-norm
8:40
the nomenclature will get a little bit confusing
8:43
you can call this
8:44
post-norm for now, but let's call this sort of residual norm
8:47
right? Because you're putting the norm in the residual layer
8:51
basically, all modern language models
8:54
Push the layer norm
8:55
outside of the residual stream
8:57
This is just like a thing that
8:58
basically everybody does
9:01
there is one funny exception
9:03
but it is OPT 350M
9:05
and if you all are familiar with sort of language models
9:09
we kind of know OPT in general was
9:11
you know, kind of a mess of a language model
9:13
right? And OPT 350M
9:15
is even more so because I don't know
9:17
why only that model
9:19
has a post-layer norm in the residual stream
9:23
Okay, so this is one of the things that
9:25
like, everyone agrees on, and so you might wonder
9:27
like, why is this, like, such a, you know
9:31
like a
9:32
unified thing across all the different models
9:37
and if you look at some of the early works studying
9:39
like, where do you place the layer norm style research
9:43
what you really see is that
9:46
you know
9:46
the early motivation for a lot of this
9:48
was When you train a transformer
9:51
you need to do a warm-up
9:52
Actually, you know
9:52
modern transformer training still does warm-ups as well
9:55
But you definitely need to do warm-up when you train
9:59
Now, wouldn't it be nice if we could remove the warm-up
10:02
right? So that was kind of the initial
10:03
motivation for a lot of this research
10:05
but people quickly realized that removing the warm-up
10:08
had very serious issues in terms of the stability and convergence
10:12
of these things, right? So if you did
10:14
post-norm plus layer norm
10:16
which is, you know, basically the original transformer thing
10:18
you got this purple dash line
10:20
oh, you just don't converge as well
10:23
compared to doing something like pre-norm
10:25
you can ignore the other
10:26
terms, you would get much nicer convergence even without warm-up
10:30
right? So this was the original motivation
10:33
But really what people kind of quickly realized is that
10:36
you know
10:37
moving the
10:38
layer norms outside the residual stream
10:40
has some
10:41
pretty important implications as you make
10:43
your network deeper
10:45
and as you start to grapple with stability issues
10:48
to me
10:49
I think the gradient attenuation issues
10:51
are kind of the most clear
10:54
when you sort of talk to people who do architecture design
10:57
I'm not really one of the
10:59
the people that deeply engages in this
11:01
but one of the things that people often
11:02
say is keep your residual stream clean
11:05
right? So in this case
11:06
you have your Xs coming in from the
11:07
bottom on the pre-norm side
11:09
and this X propagates all the way up to the top
11:12
right, all the way up to your
11:13
your final output
11:14
And that allows gradients to propagate
11:16
if you're in the backward pass
11:17
straight through this, right?
11:20
that makes gradient propagation very simple
11:23
which improves both stability and signal propagation
11:26
And that's sort of what people realized very
11:28
very quickly, that if you do something like pre-norm in blue
11:33
at initialization
11:35
sort of the gradient sizes kind of remains the same
11:37
right? Because you have this nice straight through
11:39
propagation in the backward pass
11:41
On the other hand
11:42
if you have post-layer norm
11:44
you have these kind of complicated effects that
11:46
happen
11:47
because you're layer norming
11:48
each time you're going through a transformer block
11:51
And that's going to change the norm of
11:52
your gradients as you go backwards through
11:55
So you can kind of see
11:58
you know, from the principle of keep your residual stream clean
12:01
that pre-norm makes a lot of sense
12:03
people also realized through experimentation that this also
12:06
improves stability in general
12:08
that the sizes and frequencies of gradient spikes
12:13
were improved under pre-norm compared to post-norm
12:16
And, you know, this is a figure from Salazar and Yuen
12:19
who were one of the first ones
12:20
I think, to, to study this phenomenon carefully
12:23
And I think this is the reason why it stuck around
12:25
right?
12:26
stability and the ability to go deep are both very
12:28
very important for modern large language models
12:32
And so this idea of moving your layer
12:34
norm outside of the residual stream is one that basically everyone
12:38
has adopted
12:40
so now, you know
12:42
if putting layer norms in residual streams is bad
12:46
why does layer norm have to be
12:48
at the start? Of course
12:49
we have pre-norm, which is, you know, before computation
12:52
but we could have it after computation as well
12:54
right? That's equally good
12:55
at least under that logic
12:58
and that's exactly right
13:00
many recent models, like Grok or Gemma 2 or Olmo 2
13:04
have this structure where they move the layer
13:06
norm after the computation
13:08
so it's a post-norm of a kind
13:09
but it's outside the residual stream
13:11
other models still actually just put layer norms
13:13
everywhere. They put a layer norm here
13:15
they put a layer norm after
13:17
I'll get to this later as we talk about stability
13:20
but one of the other lessons that it
13:22
seems to have held up very well
13:24
is if you have stability issues
13:25
you can kind of sprinkle in layer norms everywhere
13:28
and that will generally improve stability
13:30
It's almost very strange to be saying this
13:32
because it's so ridiculous
13:33
and yet that statement has actually been proven right
13:35
Every time, you know, people have encountered stability issues
13:38
they say, oh
13:39
but what if we just
13:40
throw a layer norm into attention?
13:42
Turns out that works too
13:43
We'll get to that later
13:44
So, okay, that's post-norm, or double norm in this case
13:46
where you have two layer norms here
13:49
Okay
13:50
the other thing that you can do
13:52
is in the original transformer
13:54
you have the layer norm
13:55
which is this operation right here
13:56
So you have your activations x
13:59
you're going to mean subtract
14:00
divide the variance, and then scale it back up
14:02
right?
14:03
And this works just fine
14:04
right? It's not like this is wrong
14:06
And many models have successfully trained on this scheme
14:09
But basically, most or all modern models
14:13
I think, use RMS norm
14:15
which doesn't subtract the mean or add a bias term
14:17
right? So it's just a scaling down and scaling back up
14:21
right? So you can see this in the
14:22
equation here
14:24
and really, layer norm is more expressive than RMS norm
14:27
so there's really representationally no reason
14:30
why you have to use RMS norm
14:32
But RMS norm is nice because in practice
14:36
there's really no expressiveness loss
14:39
RMS norm models just as well as layer norm
14:41
but more importantly
14:43
it is, you know, faster, right?
14:45
This is the part where kind of the
14:46
systems
14:47
and sort of architecture co-design starts to come in
14:50
Percy mentioned, you know, in the previous lecture
14:52
this idea of arithmetic intensity
14:54
right? We want to keep our GPUs hot by doing
14:57
you know
14:57
matrix multiplies and other very intense computations
15:00
We do not want to be wasting our
15:02
GPUs by having them move
15:03
little tiny bits of memory back and forth
15:06
right? That's a very inefficient use of our
15:08
you know, very powerful GPU
15:10
And so what we want
15:11
is to remove operations that are small and
15:14
involve memory movement
15:16
But don't give us much expressive power
15:18
right? So by that view
15:20
what we really want to be doing here is
15:22
you know
15:22
if the mean subtraction
15:24
and addition isn't really doing much for us
15:26
just get rid of it
15:27
right?
15:28
you might think, okay, why does this matter?
15:31
We're just optimizing this teeny tiny operation that accounts for
15:35
you know, in this case
15:37
something like 0.17%
15:39
of the total floating point operations of our system
15:43
But, you know, as Percy mentioned
15:46
it's not really about the flops
15:47
right? The flops are the
15:48
the floating point operations we do
15:50
that's sort of multiplying matrices
15:52
but that's not runtime
15:53
right? Runtime is a much more complicated object
15:57
and, you know, statistical normalizations, things like layer norms
16:01
even though they're only 0.17% of the flops
16:03
depending on your workload and depending on the setup
16:06
can be up to 25% of the runtime
16:08
right? That's kind of crazy
16:09
On tiny models, this can be really
16:10
really big
16:11
because you're still having to move all these
16:13
parameters back and forth from fast to slow
16:16
memory and vice versa
16:17
when you're doing these operations
16:19
So, data movement is really
16:20
really important
16:22
and RMS norm can still matter a lot because of this
16:25
right? So, you can see kind of the difference here
16:29
the arithmetic intensity is in white
16:31
and then you can kind of see
16:33
the flops involved in black
16:34
and you see that layer norm has a
16:36
very low arithmetic intensity
16:38
which is the operation we try to want
16:39
to remove as much as possible
16:40
Yeah, question over there
16:43
You know, movement for normalization is so disproportionate compared to
16:46
like, tensor contraction
16:49
So for something like tensor contraction
16:51
which is, in this case, matrix multiplies
16:54
the majority of the workload is
16:56
you know, multiplying, whereas for stat normalization
16:59
the majority of the workload is memory movement
17:01
and memory movement is quite slow
17:03
So imagine the case where
17:03
like, moving something is, like, almost all of the compute
17:07
then you're still paying quite a bit here
17:08
right? Because activations can be quite large
17:12
Yeah
17:13
I think the percent runtime in this case
17:14
is quite extreme. This is
17:15
like, in the, you know, tiny models with, like
17:18
matrices that don't really generally make sense in modern workloads
17:20
but this is, you know
17:21
giving you a sense of why this is
17:23
a free optimization
17:26
and you do see this
17:27
right? This is another paper in which people were
17:29
evaluating different architecture intervention. They're on get all
17:32
in 2020
17:33
I think this was a Google paper
17:34
and they show, you know
17:35
for teeny tiny transformer of a 200 million parameters
17:38
you get more steps per second
17:39
That's the third column over here when you
17:41
switch to RMS norm
17:42
and in fact, you actually get better performance
17:44
which I don't think is
17:45
is something that you're guaranteed
17:47
but it's a nice bonus regardless
17:48
right? So you got a free systems win
17:51
by just moving to RMS norm
17:52
And so basically everyone has
17:53
has decided to move over to this now
17:57
And in general
17:59
there's a more general version of this
18:01
There's no guarantee to any of the things I'm saying
18:03
but bias terms
18:05
in transformers and neural networks
18:07
are generally not That useful
18:09
So in the original transformer
18:10
the linear terms all have biases
18:13
but most implementations actually just drop the biases entirely
18:17
right? Once again
18:17
this is another example of something that's not very arithmetically intense
18:21
but fairly memory intensive
18:24
relatively speaking
18:25
And so you might as well just drop these
18:27
right, and get the free systems win
18:29
There's also some cases
18:31
I'll just mention this offhand
18:32
where the bias terms can also induce stability issues
18:35
so they're useful in other ways
18:37
but really I think the primary reason these
18:38
are dropped is just to simplify things
18:41
from the system's perspective
18:43
Cool
18:46
Okay
18:46
So I think layer norms
18:48
the story is pretty easy
18:50
It's easy in the sense that what people
18:52
do is fairly standardized
18:54
Our understanding, not at like a deep theoretical level
18:57
but our understanding of like
18:58
what layer norm does is fairly good
19:01
right?
19:01
Everyone moves the layer norm outside the residual stream
19:05
often pre-norm
19:06
but I think this might partially
19:07
be because LAMA2 did that
19:10
And we
19:11
roughly have a sense of how to use
19:12
layer norm to control things like
19:15
gradient spikes and keep signal propagation nice
19:20
Related to that, we also now
19:22
you know, basically always use RMS norm
19:23
and you hopefully understand the general principles here
19:26
of basically just dropping bias terms
19:29
And that allows us
19:30
to keep our system more arithmetically intense
19:34
while keeping the expressive power the same
19:37
I think the unsatisfying thing about a lot
19:39
of architectures is that
19:41
you know, you can't really reason about this beforehand
19:43
right? Like
19:43
we don't know beforehand that dropping the bias terms is okay
19:46
but from a lot of experimentation and now collectively acquired knowledge
19:50
we roughly know
19:51
Dropping the bias terms on both the linear
19:53
and RMS norm is okay for typical language modeling workloads
19:56
right?
19:57
This is the kind of statement that we
19:58
can make on the basis of
20:00
of what we do when we look at
20:01
a variety of different models
20:04
Okay, any questions, for layered stuff?
20:08
Good
20:09
Okay, so now I'm gonna talk about activations
20:12
and there's a whole zoo of activations
20:14
There's just a lot
20:15
right? There's relu, gelu, swish, lu, geglu, selu, swiglu, liglu
20:21
and what are these things?
20:24
I think at one point
20:24
one point of my
20:26
you know, more stats ML training
20:28
I thought to myself
20:29
I will never learn these things
20:30
I will make it a point of pride
20:31
to never know what a swiglu is
20:33
but now it's actually very important for us to
20:35
to actually, like
20:36
have a general sense of what these objects are
20:39
and which parts of these names actually kind
20:41
of matter for performance
20:43
right?
20:45
so
20:46
you can
20:47
build and train a language model
20:49
on just a fairly vanilla activation
20:52
even, you know
20:53
I guess Chinchilla is probably the best model
20:56
out of that group
20:56
but even if you just want a ReLU
20:58
you know
20:59
you can train a reasonably performant language model
21:01
using just that activation. There's nothing wrong with that
21:03
right?
21:04
And if we move to GELU
21:05
which is a Gaussian error unit
21:07
and really the only difference is this tiny
21:09
divot at the bottom here
21:11
which really, you know
21:12
for the most of the activation doesn't change anything
21:14
but changes the gradients right near zero
21:17
then you can train models like GPT-3
21:19
right? That's a perfectly good large language model
21:22
not, you know, modern by modern standards
21:24
but perfectly fine
21:26
but then, you know, we get to the gated linear units
21:29
like Swiglu and Gaglu
21:30
and these are really where most of the
21:32
action is. You know
21:33
this is very similar to LayerNorm
21:35
in that I think almost all
21:38
credible modern language models
21:40
use a gated linear unit of some kind
21:44
Okay, so what is a gated linear unit?
21:46
So these are gated activations
21:48
So if we want to look at something
21:50
like a feed-forward layer
21:52
we can just look at this first part
21:53
This is, you know, a very standard ReLU feed-forward
21:56
right? I have my x
21:57
I hit it with a w1
21:59
you know, I, I entry-wise threshold at zero
22:01
and then I hit it with another w2
22:03
I get my output
22:03
right? Very straightforward ReLU network
22:07
another thing, I don't say this as my personal experience
22:11
but another thing that is often said in
22:13
architecture design
22:14
is that gating is often very helpful
22:16
So if you apply that very general heuristic
22:18
what you might get is to say
22:20
okay, well, instead of just having
22:22
you know, this entry-wise ReLU
22:24
why don't we also have a gate?
22:26
And the second gate
22:27
the second term here
22:29
this is just going to multiply the output of my ReLU
22:32
entry-wise, and I have a second matrix V
22:35
okay?
22:36
Now this is just gonna modulate the output
22:38
of my ReLU
22:40
and then I'm gonna do everything else the same
22:42
right? So instead of
22:44
you know, just having X W1 W2
22:46
I have X W1
22:48
and I'm gonna gate that with X V
22:50
this is another, activation the same size as this
22:53
and then I'm going to
22:54
you know, down project it back with W2
22:57
Okay, so what is this?
22:58
Now this is a Reglu
22:59
this is, you
23:01
you make these names by adding the first activation
23:03
in this case, Relu, and GLU, right?
23:05
So the Relu gated linear unit
23:08
and gating has been a
23:09
you know, very effective other primitive in architecture design
23:13
and it turns out
23:14
that this is very effective in language modeling as well
23:18
So, if you take something like a Gelu
23:21
we've already talked about that
23:22
right? That's like the Relu
23:23
but with a little divot at the bottom here
23:25
you will get a Gelu
23:28
and if you take a squigloo
23:29
which is x times a sigmoid
23:31
then you will get a squigloo
23:33
So this is a swish times
23:35
you know, the rest of it
23:37
and this really covers a lot of the modern models
23:40
right?
23:41
generally the Google folks have used gegloo
23:44
so like the gamma models
23:45
the T5 models are those
23:47
and everything that's kind of like a llama
23:49
descendant uses a squigloo
23:51
so
23:52
palm and the llama descendants
23:54
are all kind of squigloo models
23:57
I would say that squigloo is probably the more dominant one
24:00
but honestly amongst the gated units
24:02
doesn't really matter
24:04
Now here's a side note that will
24:07
be a semi-important piece of trivia later
24:10
if you look up here
24:12
right
24:13
you will notice that there are more parameters for the gated
24:16
model
24:17
right? Cause I now have this parameter of
24:18
v
24:19
And so if you do a little bit of math
24:21
right, I now have three matrices instead of two matrices
24:25
right?
24:26
What you should do is you should maybe
24:28
use a
24:28
smaller feedforward dimension
24:30
by a factor of two thirds
24:32
in order to keep the total parameter count the same
24:34
right? So this is roughly the idea of
24:37
well
24:37
I want to keep the same number of
24:38
total parameters as my original MLP
24:41
but I now want to make it gated
24:42
so I'm going to make the feedforward dimension
24:44
which is the output dimension of this W
24:47
a little bit smaller by two thirds
24:49
right? So this is a general rule of
24:50
thumb that people have followed
24:51
but it's not really an iron rule
24:56
you know, the original Noam Shazeer paper that
24:58
you know, proposed this
25:00
had some, you know, very small deltas originally
25:05
but they're consistent deltas
25:07
and I think to his credit
25:09
I think a lot of his papers have these
25:11
like, error bar assessments of
25:12
like
25:13
training multiple replicates and checking to see if
25:15
they're better
25:17
and if you look
25:17
the GLU variants are
25:19
almost always consistently better than the non-GLU variants
25:23
And this is a parameter matched comparison because
25:26
Noam Shazeer is always doing this two thirds
25:28
adjustment
25:29
to make sure that all of the models have the total
25:31
same total number, of parameters
25:35
So this is quite nice
25:36
It's in some ways a free win
25:38
almost everyone uses a GLU
25:41
There are
25:41
there have been other
25:43
sort of more controlled systematic comparisons
25:45
This is, the same paper I was talking about before
25:47
Narang et al in 2020
25:49
Google actually in the 2020s
25:51
did quite a few nice large scale architecture comparison papers
25:56
although with a T5 architecture and not
25:59
a autoregressive, language model
26:01
and they, you know, basically comprehensively compare things like GLUs
26:04
and you see once again
26:06
if we look at the Swiglu
26:07
or the Geglu or the GLUs in general
26:10
they do significantly better at loss or the other downstream metrics
26:13
right?
26:14
Fairly compelling on pa- on these papers
26:17
Also clear
26:18
from now a lot of model training runs
26:20
that SWIGLU and GLU are good
26:23
right?
26:24
So there's a lot of variations in gating
26:26
but really the important single axis to know is that gating
26:31
for these nonlinearities is actually quite important
26:33
Gives you
26:35
nice boost without much of a computational cost
26:38
you know
26:39
that's not to say that gated linear units
26:41
are necessary. I mean
26:42
GPT-3 was that
26:44
I think the Nemotron 340B model used a squared relu
26:47
which is a kind of a crazy choice
26:48
but that works too
26:50
both of these models are perfectly performant
26:53
but it's actually quite rare
26:54
to see anything that's not trained on a gated linear unit
26:58
right? So evidence is pointing towards consistent gains
27:00
on using these gating tricks
27:03
Okay
27:04
So those are, I think
27:06
the more consensus choices
27:08
for things that we can do in architecture
27:11
Now, this one, I think, is a really fun idea
27:13
but one that I think now the test
27:15
of time has shown
27:16
maybe is not quite as good or maybe
27:18
not as popular of an idea
27:20
Normally, we do our transformer blocks serially
27:23
right? We compute our attention
27:25
then we compute the MLP
27:27
right, one after the other
27:29
If you're very systems-minded
27:30
you might say, well, this introduces a bottleneck
27:32
right? I have to wait for the computation
27:34
of one to do the other
27:36
If they were instead in parallel
27:38
I could bring to bear some new and cool systems optimizations
27:41
potentially, right?
27:42
So you might ask
27:43
could we parallelize the transformer block?
27:46
And this was originally an idea that was in GPT-J
27:50
which is the open source
27:52
attempted replication of GPT-3
27:56
And kind of Very interestingly
27:58
I think GPT-J has been surprisingly influential
28:01
in sort of propagating a lot of ideas
28:04
I mean, Palm as well
28:05
Google
28:07
is actually surprisingly bold with the architectures that
28:09
they do
28:10
but the description in
28:12
Palm, which you can see in their report
28:14
is kind of the following
28:15
Instead of nesting this
28:16
which is the sequential form at the top
28:18
you know
28:18
you're just gonna add together
28:20
the output of the MLP and the attention layer
28:22
and just add both of those back
28:24
into the residual stream
28:27
if you implement this right
28:28
you can actually share a lot of the components
28:30
like you can share the layer norms
28:32
you can fuse the matrix multiplies
28:34
This allows you to potentially get additional systems
28:37
optimizations
28:39
and I think a lot of the people
28:41
that have been influenced by Google
28:43
so Cohere, you know, was founded from one of the former
28:46
transformer authors
28:47
They do a lot of Google-inspired optimizations
28:49
They followed kind of this architecture
28:52
but not very many others
28:54
this has been a
28:54
a approach that has really fallen out of
28:56
popularity over the past
28:57
I think, two years
28:59
I think mainly because optimization of the serial
29:01
form
29:02
has gotten sufficiently good that the systems gains
29:05
from the second one just isn't worth the small hits to
29:09
representation power that you end up getting going from
29:12
parallel to serial
29:14
Effectively
29:14
you can think about it as
29:16
you've lost half of your depth
29:17
right? And that can be
29:19
a, a deleterious
29:21
thing to do to your model
29:23
Okay
29:25
So in terms of the architecture of things
29:27
actually, you know
29:28
the fact that this is so short
29:30
should kind of suggest to you how
29:31
much the original transformer
29:33
formulation has somewhat stood the test of time
29:36
right? Because the only thing I'm really talking about changing here
29:40
is, you know, where the norms go
29:42
or, you know, whether we have bias terms
29:44
or whether we gate the MLPs
29:46
but those are actually pretty minor changes
29:48
compared to all the things that you can do
29:51
Now, those of you that are
29:52
you know, sort of carefully paying attention might say
29:54
but wait, you know
29:55
there's a lot of transformer alternatives that change the attention
29:59
yes, you'll have to wait until next lecture
30:01
because today I'm just only going to cover
30:03
sort of core attention-based methods
30:06
and next lecture I'll throw in a little bit of
30:08
state-space model stuff. But as long as you're in this
30:11
like, dense attention land
30:13
actually the architecture from the original transformer paper
30:15
is pretty close to what we do
30:18
So you see, quite a bit of this
30:20
right? So, just now going back to this
30:22
blue here is RMS norm
30:23
black is layer norm
30:25
You see most of the modern
30:26
models are sort of RMS norm models
30:28
serial versus parallel layers. The blue one's parallel
30:30
the rest is serial. You see mostly serial
30:32
layers
30:33
pre-norm versus post-norm. Some of these
30:36
ones that I marked as post-norm are actually
30:38
pre and post-norm
30:40
and then these ones on the right
30:42
these are GLUs, almost always, with the exception of things like
30:46
Falcon, which is a gated linear unit
30:48
But almost all of these are really
30:50
gated linear units for modern models
30:53
So, you can see the trends quite visually
30:56
from what I'm telling you
30:58
Okay, so really the thing that is very different across implementations
31:03
and I think a place where a lot
31:05
of the architecture stuff is still in flux
31:07
is how you do
31:08
kind of position dependence and incorporate information from other positions
31:13
right? So the core attention component in some
31:15
sense
31:17
so there are lots of different ways
31:19
that you can encode position into a transformer
31:22
And just so you know
31:23
to, to remind you, right
31:24
this is very, very important because the tension is positionally independent
31:28
right? They're just inner products
31:29
so you can just shuffle them and the
31:31
tension would be the same
31:32
if you don't have a position embedding
31:34
The original transformer had sine and cosine embeddings
31:37
kind of like a Fourier transform intuition that
31:39
if you have sines and cosines
31:40
then you can kind of recover position from
31:42
that no matter what
31:44
a number of other sort of large models that
31:46
you know, followed soon after that used absolute embeddings
31:50
where each position had its own different embedding
31:54
and then
31:55
several other
31:56
sort of Google models like to use relative
31:58
embedding. So here you're not adding
32:01
embeddings into the, into the embedding, like word vector embeddings
32:05
but instead you're adding a vector to the attention computation itself
32:08
right? So if you're three positions off
32:10
sort of the attention matrix
32:12
gets a different offset added to it
32:13
And, and, you know
32:14
models like T5 and Chinchilla use kind of
32:16
this scheme
32:19
the thing that has really become pretty
32:21
dominant in terms of position embedding
32:23
is this class of embeddings called ROPE
32:26
which some of you may be familiar with
32:28
most models past 2024 use this type of
32:31
embedding. And it's kind of remarkable given that ROPE
32:34
you know
32:34
in some ways came out of nowhere
32:37
originally I think this was also a GPT-J innovation
32:41
from I think a
32:42
a sort of not very well-known sort of blog post and
32:46
paper combination, from an author in China
32:49
but really, it has some really interesting ideas for
32:53
for why you would do something like rope
32:55
So rope, you know, is a relative position embedding
32:59
and a relative position embedding
33:00
let's
33:01
make an opinionated stance
33:03
that
33:03
I should not care
33:05
about the absolute position of any words
33:08
So if, you know, A, and apple appear together
33:11
even if it appears at the start or at the end
33:14
right, in rope embeddings, they should kind of get the same
33:17
sort of result
33:19
and we do know that
33:21
you know, or
33:22
and we want to sort of represent it in this way
33:25
right? So I have an embedding F
33:27
and I have another embedding F
33:28
and these are going to take in the
33:30
identity of the words X and
33:31
Y and the positions absolute of I and J
33:34
and I want this to be equal
33:36
if I take the inner product of these
33:37
embeddings to be equal to
33:40
a
33:40
function that only depends on the relative difference
33:44
Right?
33:45
and every
33:45
existing embedding before it didn't really fulfill this
33:47
equality. Like sine
33:49
is not relative because it has these absolute
33:51
cross terms that are not relative
33:53
Absolute position embeddings, just by the, the name of it
33:56
is obviously not relative
33:58
And then relative embeddings
34:00
technically these are relative
34:01
but they're not kind of embeddings because they're
34:03
just adding to the attention matrix
34:05
right? So there's no inner product structure that
34:08
you know
34:08
you can extract out of this guy
34:10
So given this, you might ask
34:12
is there a nice way
34:14
that we can truly have this relative embedding?
34:18
And the idea is very cool
34:19
it's really just looking at kind of
34:22
properties about angles and cosines
34:25
So we want our embeddings to be invariant to absolute positions
34:29
and we know that inner products
34:30
of any kind are invariant to arbitrary rotation
34:33
right?
34:34
So the idea is to say
34:36
I'm going to take my semantic word vectors
34:38
the ones that are independent of any position
34:41
so this is my starting point
34:43
and then I'm going to rotate each of these vectors
34:45
in this case in 2D
34:48
based on the position that the words appear
34:51
So, you know, just as a, simple example, we, let's say we have the
34:57
sentence we know that
34:58
right?
34:59
We appears at position zero
35:01
so I'm not going to touch that at all
35:02
right? I'm just going to keep that where
35:03
it is
35:04
The word know is at position one
35:07
so I'm going to rotate it by some angle
35:08
right? And that's my
35:10
my one position rotation
35:12
Now what happens if I apply the same idea to
35:16
the following sequence? Of course we know
35:18
right? In this case
35:19
we and know are still adjacent
35:21
they're right next to each other
35:22
but their absolute position has shifted
35:24
right? Of course, you know
35:25
comes before we know now
35:27
In this case
35:28
I'm going to rotate the word we by two positions
35:30
because it's two index
35:31
right?
35:32
Zero, one, two
35:33
So the word we is in the second
35:35
position number two, so I rotate by two
35:38
I rotate no by three positions
35:40
because it's in position number three
35:42
And what do you know
35:42
the relative angle between these two is still separated by one
35:46
right?
35:47
So this is a very
35:48
very simple idea of just using rotations
35:51
to represent, position. And if we do that
35:55
then any time we take an inner product
35:57
those inner products are going to be invariant
35:59
of absolute position
36:01
Now you might say
36:02
well, in two dimensions, that's pretty easy
36:04
because you've only really got one choice
36:05
You've got clockwise and counterclockwise
36:08
But in high dimensions
36:09
there's an infinite
36:10
space of ways that you can rotate vectors
36:12
So what do you do in D dimensions?
36:15
Well, you do the simplest possible thing
36:16
and it works. The simplest possible thing
36:18
is to
36:19
reduce it to the 2D case repeatedly
36:21
So you have a D-dimensional vector
36:23
just cut it up into chunks of two
36:25
and each pair of two dimensions gets rotated
36:29
And the theta at which these things rotate vary
36:32
right?
36:32
Some of them are very low frequency
36:34
so they rotate very slowly
36:35
so they can capture long-range dependence
36:38
Some of them rotate very quickly
36:40
so they capture things like
36:41
are they neighbors to each other
36:42
right?
36:44
and then at the end
36:45
you know, after I've rotated every pair of vectors
36:47
I get sort of my final embedding
36:49
So this is the
36:50
you know, rope approach
36:53
the paper, if you read it
36:55
has a very complex motivation about complex numbers
36:58
but really, I think the intuitive way
37:00
at least to me
37:01
to think about it
37:02
is to just
37:03
you want to rotate by reducing to the two-dimensional case
37:05
and you're just rotating every pair of coordinates
37:10
Gamma 4 just came out on Thursday
37:12
and they have, like
37:13
another different kind of fun thing that they do
37:15
which they call, I think proportional rope or P-rope
37:19
which is a really strange way to just
37:21
say that the only thing they rotate is
37:22
the first two coordinates
37:23
but that's another valid thing
37:25
that you can do as well
37:25
So there's a lot of different things that
37:27
you can do in this space that end
37:28
up working
37:30
Okay
37:30
In practice, what you're gonna end up doing is
37:34
you know
37:34
you can
37:35
take
37:35
your vector and you can make a sparse
37:37
multiply with sines and cosines
37:39
and this is going to be giving you
37:41
some way of rotating your input vectors x's
37:44
right? So x times
37:46
w times r
37:47
this is gonna be your final embedding
37:49
that you get
37:50
Okay
37:52
and finally, you know, this is a sine and cosine
37:54
which looks a little like sine embeddings
37:56
but it's really important that I'm multiplying with
37:58
these sines and cosines rather than using them as embeddings
38:01
cause that means that there are no cross
38:02
terms
38:03
and this is purely relative
38:05
right? There's no absolute position information that you'll
38:07
get out of inner products
38:10
if we really
38:10
really wanted to get into low-level details and you ask
38:13
like, how do I actually implement this thing
38:14
you know, you're gonna have to do that
38:17
you have your usual attention stuff
38:19
and then what you do is you generate
38:20
cosine and sine angles
38:22
based on the position IDs
38:24
of sort of where your sequence is
38:25
and then
38:26
you're gonna apply those cosines and sines
38:28
onto both your queries and keys
38:30
for your sort of attention computation
38:33
and you can either apply them as a
38:34
matrix multiplier or you can go through and apply them manually
38:36
just as a rotation
38:38
right? Fairly straightforward
38:39
and you would do this
38:40
at the attention level
38:41
rather than at the very bottom
38:42
to sort of enforce position invariance
38:44
every time you're doing attention computations
38:47
Okay, so that was rope
38:49
it is a little bit confusing
38:50
but once you understand the geometry of just rotating things
38:53
it's actually fairly, straightforward
38:56
Okay, I'm gonna pause here for one moment
38:59
in case anyone has questions about the various
39:02
like, architecture bits
39:04
we're gonna then talk about even lower-level details about hyperparameters
39:08
so yes? Do you know about any papers
39:10
that do a high-dimensional rotation?
39:14
Higher dimensional rotations. That's a good question
39:18
I don't think so. By a higher dimensional rotation
39:20
like any, you know
39:21
2D rotation
39:22
in the space would just be kind of
39:23
a variant of this
39:25
You could certainly do like
39:27
any one manifold that like is a closed
39:29
loop. I have not seen that
39:31
Yes
39:33
What do you think is the best way
39:34
to instill this kind of knowledge throughout the
39:36
papers and like for reports?
39:40
It's a good question
39:43
I don't know if there's a way
39:45
beyond some combination of like looking broadly enough to get a
39:49
to get, get a pattern
39:49
which is what
39:50
the procedure I'm trying to do in this
39:51
lecture here
39:52
And then the other one is to try it yourself
39:54
even a much smaller scale to form an
39:56
intuition and like a theory for how these
39:57
things come together
39:58
I think those two are really the right
40:00
ways
40:01
I think reading any single paper in isolation is very
40:03
very difficult, especially now
40:05
because
40:06
no single paper seems to give any
40:07
full detail for a lot of language models these days
40:12
Oh, lots of questions now
40:13
Okay, good. We'll go in
40:15
so I have a question
40:16
I have a question on the parallel layers and
40:18
and the sequential layers. Yeah
40:20
I understand the
40:21
the modern models are thinking of the use of the latency
40:24
so they will use the parallel layers
40:27
And, yeah, but there's a, there's a, there's a, there's a difference between
40:30
between accuracy, for these two patterns
40:33
right? I wanna know
40:34
like, what's the, what's the difference of
40:36
of, of accuracy? Is it big enough to
40:38
is it small enough to allow the current
40:40
model trainers to ignore that
40:41
or is there any? Yeah
40:44
I think, you know, the
40:46
that's actually really mixed
40:48
So if you read the original Palm paper
40:50
I think they're, like
40:50
very confident about the use of parallel layers
40:53
like no performance drop
40:54
fifteen percent systems utilization improvement
40:56
So if you read just that
40:57
you'll kind of say
40:58
like, oh, it's just as good
41:00
but I think a lot of the
41:01
the later Google models have stopped using this
41:04
which you can take on as an implicit
41:05
signal that actually there might be some
41:07
losses
41:08
And once again, this one is a little bit hard to
41:11
to get precise numbers on
41:12
because no one's done the ablations
41:14
as far as I know
41:14
in parallel versus serial
41:16
controlled nice ablations at least
41:19
Yeah. Yeah, yeah, I mean, this
41:30
this difference is really just like which of
41:32
the coordinates you're working
41:33
Like you don't rotate most of them
41:35
because a lot of the
41:36
I mean, the argument originally, I think
41:38
is that the low frequency parts
41:41
just aren't rotating very much
41:42
And so you can drop them if you're really strapped for
41:46
you know, sort of extra space
41:48
And these, this is really a
41:49
a optimization for teeny tiny models
41:50
where like you don't have very much like hidden dimensions to
41:53
to have activations for
41:56
Yeah
41:58
For the relative embeddings
42:00
not having an inner
42:04
product, is that because it only applies to keys
42:07
basically? Like, I was trying to understand
42:07
Yeah, so they apply both to the keys and values
42:09
which is kind of why
42:11
you know, you get this, like
42:11
relative effect from where you are
42:15
you want to not have cross terms
42:17
right? So
42:17
so if you look at the sine and cosine embeddings
42:20
then you'll not only get sort of the
42:22
you know, the original vectors
42:24
you'll kind of get these weird cross terms
42:26
between the position embeddings and the word embedding themselves
42:30
and so on and so forth
42:31
And then you can kind of back out
42:32
what the absolute position is
42:34
So even sine and cosine embeddings are not
42:36
like, pure relative position embeddings
42:39
you know, you have to accept the premise that
42:41
you know, the relative embedding is what you want
42:43
but once you do
42:44
kind of
42:44
you end up at the rope solution somewhat
42:46
naturally
42:52
Yeah
42:54
Yeah
42:56
So the issue with this is that it
42:57
just can't be factorized as a product
42:59
That's more of an aesthetic problem
43:00
right? Like if, if your constraints are
43:02
I need it to be
43:04
relative
43:05
and I need it to factorize as
43:06
f of xi and f of yj
43:09
then this is not a solution in that
43:10
class
43:11
to be fair, there's a lot of
43:13
embeddings that work this way
43:15
that do work, like alibi and other kinds of
43:17
like, approaches, like do
43:18
do this kind of inject into the attention matrix
43:21
and they do reasonably well
43:24
it's not necessarily the one that's become the dominant approach
43:27
is all I can say
43:30
Cool. Okay
43:32
Great
43:34
Okay
43:35
Now I'll talk about hyperparameters
43:38
and I think hyperparameters are really something that
43:39
you start to engage with once you
43:41
like, actually have to train a model
43:43
right? When your knowledge about language models are abstract
43:46
you don't have to care about any of
43:47
these. But once you have to instantiate it
43:49
you start to ask questions like
43:50
well, how big should the feedforward size be?
43:54
how many heads should I have?
43:56
what should my vocab size be
43:58
right?
43:59
and you might also have questions of like
44:01
what should my weight decay or dropout be?
44:03
Like, do I even need to regularize?
44:04
I have a lot of tokens
44:05
right? So do I need regularization?
44:08
and do I need very deep models or
44:10
very wide models? Like
44:11
what are the, the right kinds of things to do here
44:13
right?
44:14
in all of these
44:15
if you start out with no knowledge
44:16
it's actually very daunting
44:17
because you have to search this
44:18
like, very big high-dimensional space
44:21
but the space of things that people try
44:23
is actually pretty small. And from that
44:25
maybe you can start to think about
44:26
you know, smarter search processes of
44:28
like, where you want to vary things
44:31
one of the things that's a really consensus hyperparameter
44:35
is this idea of the
44:37
ratio between the feedforward size
44:39
which is kind of the output of your
44:41
first matrix multiply in a MLP
44:43
and the model dimension
44:44
right? So this is really the
44:46
the, ratio of the two dimensions of your W1 and your
44:49
as well your W2 matrix
44:52
this seems like a thing that's very important
44:54
and controls kind of the richness
44:55
of your MLPs
44:57
So what should it be?
44:58
Well, for whatever reason
45:00
it should maybe be four times your hidden dimension
45:03
right?
45:04
and this is a rule of thumb that works remarkably well
45:08
and I will show you some data on
45:10
like
45:10
why maybe this is a fine number to
45:12
choose. There's a few exceptions
45:14
and funnily enough
45:15
the really extreme exceptions kind of backtrack on
45:18
that
45:19
Okay
45:20
exception number one
45:22
is variance of the gated linear unit
45:24
I already told you about this
45:25
so if you were thinking about it
45:27
this is probably cached in your head
45:28
right?
45:29
GLUs have more parameters
45:31
right, if you keep the same dimensions
45:33
So if you want to keep the parameter
45:34
size of your MLPs the same
45:36
well, you need to scale down by two thirds
45:38
right?
45:39
So most GLU variants
45:41
this means that you're going to end up
45:42
with something like 2.67-ish
45:45
right? So everyone that's
45:47
down here, 2.67 to 2.5, this is roughly applying this
45:52
like, two thirds correction
45:55
and then for whatever reason
45:58
the LAMA2 folks decided
46:00
well, we actually have very efficient
46:03
attention heads with, like
46:07
MQA, which I'll talk about later
46:10
and because of that
46:11
We can multiply this ratio by an arbitrary 1.33
46:14
and we'll get roughly 3.5
46:16
And so the llama people kind of like
46:17
arbitrarily chose a slightly different ratio
46:19
which essentially emphasizes the MLPs a little bit more
46:23
But really, if you actually look through all the papers
46:25
you'll find, you know, either 2.6-ish or 3.5 for GLUs
46:30
or 4 if you're doing non-GLU models
46:36
Okay, there's another exception, which I find to be very funny
46:38
but also very, very cool, which is, you know, throughout
46:43
as you read these like technical reports
46:45
you'll find that most people are just very
46:47
boring in their choice of architectures
46:49
They're like, we did llama
46:50
but we changed one thing
46:52
But You know
46:53
folks at Google are very bold sometimes
46:56
and T5 is one of my favorite ones
46:58
because they have some really bold settings
47:01
they decided that, instead of following this
47:03
like, 4x rule of thumb
47:05
they decided that they want to have a 64x
47:08
multiplier, which is, like, way bigger than 4
47:12
and they have a reasonable argument for this
47:13
as well. This is another
47:14
like, systems-based argument, right? They said, oh, well, you know
47:17
if the bigger my matrix multiplies
47:19
the more efficient I can keep my hardware
47:21
So if I make this
47:23
you know, multiplier really big
47:25
then, you know
47:26
my matrix multiplies can potentially be sort of more efficiently utilized
47:31
right?
47:33
and some others, like Gamma 2
47:35
have also tried to really
47:36
push a little bit higher on this
47:39
But really
47:39
T5 is an kind of astounding exception at
47:42
64. I don't think
47:43
any other model has really gone that high
47:45
in the feed-forward multiplier
47:49
And empirically
47:51
if you look at other sort of works
47:52
that try to do more controlled comparisons of this ratio
47:56
I've taken this one from Kaplan in 2020
47:57
This is the classic
47:59
neural scaling loss paper
48:02
where they, they do sort of various controlled
48:05
studies on language models
48:06
You'll see, you know, this wasn't the point of the study
48:09
right? It was a scaling loss study
48:10
but you'll see in one of the panels
48:12
that they actually
48:13
have a
48:13
sort of ablation or a sweep where they
48:15
change the feedforward ratio and they look at the loss
48:17
right?
48:18
for a very small model here
48:19
right?
48:20
But what they
48:21
what they find in this paper
48:23
is
48:23
there's a basin
48:24
where you start at about
48:26
one and you end at about maybe ten
48:28
where this hyperparameter is like pretty good and very
48:30
very flat. You lose very little
48:32
relative to the optimal loss down here
48:35
right?
48:36
and then if you get it really wrong
48:37
like you get, you know
48:38
above ten to a hundred or something like that
48:40
you know
48:40
then your loss starts really shooting up quadratically
48:44
and so a lot of these choices that
48:45
range between like 2.6 to 4
48:47
they're all kind of falling into this relatively nice basin
48:51
so you're fine choosing those numbers
48:53
right?
48:54
Okay, so what can we learn about this hyperparameter?
48:57
Well
48:57
the default choices have worked very well for
48:59
nearly all modern language models
49:01
so you can safely choose that
49:03
T5 was a fine model
49:05
or the version 1 T5 was a fine model
49:07
right? Like, it wasn't a bad model
49:09
and so even radical choices can technically work
49:12
but it's probably gonna be compute inefficient
49:14
And I think the funniest part of this saga
49:16
of this kind of the punchline of the
49:17
T5 saga to me
49:19
is that they have a follow-up model
49:20
T5 1 point, V, V 1.1
49:23
that's like supposed to be the improved version of T5
49:25
and they kind of go back to the standard 2.5 multiplier
49:28
you know? So there's nothing explicitly stated here
49:31
but clearly, you know, when they tried to
49:33
to update T5
49:34
they decided that they wanted to go back
49:35
to a more standard multiplier
49:36
which I find to be a little bit
49:37
funny
49:39
Okay
49:40
So that's the
49:41
you know, feed forward ratio
49:43
which now you have like a rough sense
49:45
of like what the right order of magnitude
49:46
is
49:48
Now let's talk about a different consensus hyperparameter
49:51
I always found this to be very strange
49:53
when sort of teaching
49:55
224N and, you know, just sort of teaching students about this
49:59
which is if you have a multi-head attention
50:01
where you have multiple heads for your attention in your transformer
50:05
the canonical thing to do
50:06
the thing that almost everyone does
50:08
is if you have multiple heads
50:10
you make sure that the size of those heads
50:13
the head dimension
50:14
is such that
50:15
you sort of have the same dimension as
50:17
a single head transformer
50:18
right? So you always make sure
50:20
that
50:21
you sort of divide the hidden dimension
50:23
to basically multiply with H
50:25
So in this case
50:26
right, you have H, the number of heads
50:28
and the dimension of each head is D over H
50:30
so you multiply the two and you get D
50:31
right? For some reason
50:32
this is kind of the rule of thumb
50:34
of course this doesn't have to be true
50:36
we can arbitrarily change the
50:37
the ratios between head dimensions and model dimensions
50:40
but most models do follow this guideline
50:42
and it turns out to work pretty well
50:46
you know, we can look at a variety of different models
50:47
classic and new, I, you know
50:49
have the latest and greatest quite as well
50:51
and you kind of find
50:52
yeah, the ratios are roughly around one
50:54
a model head
50:56
you know, notable exception of T5
50:59
and even lambda, which is another Google model
51:02
but really everyone sticks around one
51:04
and I think this is
51:05
kind of an interesting one
51:08
I think the thing about head dimensions
51:10
that I'll, that I'll end with here
51:12
is I think this is yet another kind
51:13
of forgiving hyperparameter
51:15
there's a couple of relations that people have done
51:17
there's
51:17
once again a pretty wide basin around one
51:20
that you can sort of get away with
51:22
Okay, but that one's maybe not the most critical
51:25
hyperparameter
51:26
I think maybe one of the most critical and interesting ones
51:29
I think conceptually, is this idea of an aspect ratio
51:32
right?
51:33
and sort of to add an extra point here
51:36
when you scale models up or down
51:38
the way you usually do that is you
51:39
fix an aspect ratio
51:40
like how wide your model is versus how deep it is
51:42
and then you make the whole model bigger
51:44
right? So the aspect ratio in some sense
51:45
controls
51:46
the entire depth to width trade-off as you make models bigger
51:50
right?
51:51
Now
51:52
you might wonder how deep should my model
51:54
be? Like if you've been following all this
51:55
stuff on like reasoning and so on
51:57
you might think
51:58
I need a really deep model or a
51:59
really shallow model if I want systems of
52:01
utilization
52:02
You might think that there's a lot of
52:03
sort of variation. And there is a lot of variation
52:06
much more so than other hyperparameters
52:08
But there's actually like a
52:09
fairly clear sweet spot that
52:12
most modern models fall into
52:14
you don't really see models go like Too
52:18
too deep, and you also don't see models go too wide
52:22
in either direction
52:23
right? You see most models have a ratio about a hundred
52:26
D model over N layers
52:29
so about hundred sort of width
52:31
for every layer that you have
52:33
I mean
52:33
this is true for like GPT-3 or LAMA
52:35
or any one of these models
52:39
and
52:40
really I think the considerations are partly a
52:42
trade-off between expressiveness and hardware
52:44
right? If you have an extremely
52:46
extremely deep model, they get very
52:49
very annoying to deal with systems-wise
52:50
The deeper your model
52:51
like what is the ways that you have for paralyzing them
52:54
well
52:55
you might have to cut up your layers
52:56
If you cut up your layers
52:58
We'll talk about this in the systems lecture
53:00
Once you start cutting up your layers depth-wise
53:03
you have very serious issues in parallelization
53:06
Pipeline parallel, which is what this is called
53:08
is something that most people really
53:10
really do not want to deal with
53:11
Whereas with is much easier to parallelize
53:14
If you have a really wide model
53:16
you can cut that up
53:17
very easily in your GPUs
53:19
Tensor parallel is what it's called
53:20
It's much, much simpler to deal with
53:24
And so in some sense
53:25
there are systems reasons to go wide
53:27
and maybe there's expressiveness reasons to go deep
53:29
and you end up at roughly 100
53:33
And
53:34
I think one of the really interesting things
53:36
about
53:37
Transformer hyperparameters
53:39
is there a lot of hyperparameters that seem quite important
53:41
but they're also fairly forgiving
53:43
and people have converged roughly on the minimum
53:46
This is yet another plot from Kaplan et al
53:48
which shows another sweep over hyperparameters for
53:51
differently sized models
53:53
and once again, you see
53:54
regardless of kind of the size of your model
53:57
roughly speaking, the optimal aspect ratio is fairly similar
54:01
and they live at about 100
54:03
maybe a little bit less
54:04
depending on how you want to do the accounting
54:06
but really, you know
54:07
anywhere near 100 is a pretty safe bet
54:09
for aspect ratios
54:12
EK and others
54:14
did a number of really interesting sort of like architecture
54:19
architecture variation experiments
54:21
in which their general conclusion on this was that
54:25
let's look at the top panel here
54:26
you have a lot of different kinds of
54:29
models that you can have in terms of
54:30
depth to width trade-offs
54:32
but as you sort of sweep the depth to width trade-offs
54:34
you find that really
54:37
the only thing that matters in some sense
54:38
is flops. As you increase the flops
54:40
the models get better
54:41
and that's really controlling the majority of the effects
54:44
not necessarily, the aspect ratio
54:46
And so
54:47
I think what has really emerged from this
54:49
is the sense that
54:50
there's a general forgiving band of hyperparameters that
54:52
people tend to choose
54:54
and then you really worry about primarily your
54:56
systems utilization rather than sort of expressiveness concerns
55:00
which are hard to reason about
55:02
Cool
55:05
okay, and then maybe the last hyperparameter thing
55:08
that I wanna mention is vocabulary sizes
55:12
and this one's kinda interesting to me
55:14
because there's a really clear difference between two
55:17
classes of models
55:19
I think in the early days of a lot of
55:21
you know, early days of open source model training
55:27
there were a lot of monolingual models whose
55:29
only goal was to be good on English
55:31
and for those models
55:32
you had these, like
55:32
much smaller vocab size in the 30,000 range
55:36
and then post-Lama
55:38
a lot of people were really interested in multilingual
55:41
or like production systems
55:42
so these include closed source models like GBD4
55:45
all these have much
55:46
much larger vocab size
55:47
and these are roughly in the 100 to 200,000
55:51
vocab range
55:53
And you see generally that
55:53
you know, Google models have a ton more vocab
55:56
Lama derivatives roughly range at about 100
55:59
Thousand tokens, and then the
56:01
the sort of monolingual models are about thirty
56:03
thousand
56:05
this is somewhat clear
56:06
the multilingual models really do need much larger
56:08
vocab to cover the whole space
56:10
Generally
56:11
the models on the right are also bigger
56:12
There have been scaling law studies showing that
56:14
the bigger your model
56:15
the larger the vocab it can handle
56:17
and so this is also partially driven by
56:20
modern scaling trends
56:21
where the models on the right are generally
56:22
bigger. No one's training small
56:24
sorry, large monolingual models, anymore
56:30
Okay, so, yeah
56:43
Sorry, the question was like
56:44
if you have, sorry, multilingual models, or sorry?
56:48
Multimodal
56:49
Yeah
56:49
so
56:49
I guess it depends on
56:51
the way that your tokens are encoded
56:53
but, you know
56:53
if you're tokenizing your images and things like that
56:55
then you need to
56:56
you know
56:56
have many more tokens to account for those
56:58
if you look at
56:59
like, various open source releases
57:00
they'll have, like, a different image tokenizer with its own vocab
57:03
which is quite large
57:05
Yeah
57:06
how valid is it to compare
57:11
bits to bytes for your image?
57:16
how valid is it to compare bits?
57:17
Oh, that is a great question
57:18
Okay, yeah
57:19
that's not a hyperparameter question
57:20
but that is a good question
57:22
so, what is the right way?
57:24
Okay, so, so let me, let me, like
57:26
step back a moment and
57:27
like, put us in the right mindset
57:29
So, if we think about language modeling
57:32
language modeling is, is a generative modeling task
57:34
right? We are modeling the probability of a sequence
57:36
Now, as long as your sequence is fixed
57:39
right, it's the same, you haven't, you know, adulterated it anyway
57:42
and you provide a probability over all strings
57:44
that's always valid to compare
57:46
right? At that level of things
57:47
it's always valid
57:48
Now, when you ask the question
57:50
is it valid to compare the bits per
57:52
byte of arbitrary token
57:54
or two arbitrary tokenizers
57:57
really, there's two things at play
57:58
The one thing is
57:59
you know, did you touch the sequence at all?
58:01
Like, if you look at some tokenizers in the past
58:03
before subword tokenizers, they would drop some tokens
58:06
or drop some words
58:07
That changes, that makes the comparisons invalid
58:10
But modern tokenizers are complete
58:11
they can model any sequence
58:13
so that's not a concern
58:15
The other thing that you have to worry about is
58:17
are we, like, length normalizing it in any way
58:19
right? But for bits per byte
58:20
you're always normalizing with the same number
58:23
which is the number of bytes
58:24
and so this is always a valid comparison
58:26
right? So that's kind of how to think about
58:27
you know, tokenizer comparisons
58:36
This, this, this, this, always leads to
58:40
to better battery performance
58:41
on battery life, perhaps, is
58:43
has the same thing been proven for
58:45
complexity of BPB are kind of dual to each other
58:49
so yes, if that's what you're asking
58:53
It's only yes, it's only no
58:54
because if you're comparing
58:58
You could, you could frame, compare complexity as comparing
59:02
but you're changing the difference of
59:04
Okay, we'll have to, we'll have to talk later
59:05
because I'm not sure I understand the question
59:09
but I think that
59:10
that's an interesting set of questions
59:12
Okay, good
59:13
Okay
59:14
All right
59:16
So, you know, we're
59:17
we're going through really the
59:19
The low level, lowest levels of details of language modeling
59:22
which I think is really
59:24
exposes a lot of interesting ideas while we
59:26
sort of talk through this
59:27
And I think dropout is one of the
59:29
and regularization
59:30
I think is another
59:31
very interesting class of ideas
59:33
Also one that I think is very counterintuitive
59:36
from your machine learning 101 intuition
59:39
So let's, go through what I think is like the
59:42
the standard argument for
59:44
you know, regularization
59:46
well, if I'm doing language modeling
59:47
I have a lot of data
59:48
right? I have more data than I can
59:50
process most of the time
59:51
right? Unless you're at
59:52
you know, Google, maybe even then
59:54
there is more internet data than there is flops
59:57
so, you know
59:58
I'm probably not even gonna see the same data twice
60:00
right?
60:02
so I'm only gonna do a single pass on a corpus
60:04
and there's very good
60:06
reasons and arguments to believe that a single
60:08
pass of SGD or other optimizers
60:10
is never really going to memorize my data very much
60:13
right?
60:14
So this means
60:15
overfitting is not really a problem
60:17
Almost ever during compute constrained language modeling
60:21
Now, you know
60:22
some people even actually
60:23
only look at training loss because they believe
60:25
so strongly that overfitting doesn't happen in single pass SG
60:29
Now, given this, you know
60:31
you can sort of sit and think about
60:32
this
60:33
Should I use dropout or weight decay in language model training
60:37
right?
60:38
Okay, you can think about it a bit
60:42
you know
60:42
one unfortunate thing is that a lot of
60:44
recent models don't talk about this stuff at
60:46
all
60:46
it's really lower level details than
60:49
like, tech reports are willing to expose
60:52
but if you look
60:54
actually, you find a lot of models
60:56
do both
60:57
Especially weight decay, actually, is a fairly popular intervention
61:01
even for modern high performance language models
61:06
this is very, very surprising, you know?
61:08
I mean, some of the dropout things
61:11
you know, maybe
61:14
Have gotten out of favor
61:15
but weight decay actually remains fairly popular
61:17
And this is very mystifying
61:19
Like, why is this?
61:21
and this is, you know
61:22
one of the reasons why I think deep learning is hard
61:24
and this architecture lecture is very strange and
61:26
hard
61:27
it's because these things interact in very strange ways
61:30
So, there have been papers that have argued
61:33
and shown nice evidence
61:35
that
61:36
weight decay is actually not a regularizer sometimes
61:39
It actually interacts with the optimizer to essentially
61:46
make optimization better
61:49
so if you look
61:49
at the training versus validation loss across different
61:53
weight decay settings on
61:54
you know, language model training for single pass SGD
61:57
you don't really see any difference
61:58
Like, weight decay isn't shifting things
62:00
so the validation loss is better
62:01
There's already no overfitting. We're on the x
62:03
equals y line here
62:04
right?
62:05
So, doesn't control overfitting
62:07
but But
62:08
if we kind of look at different levels of weight decay
62:11
and not only just different levels of weight decay
62:13
we look at weight decay combined with learning rate decay
62:18
what we find is that the stronger weight decay runs
62:22
these blue dash lines on the bottom
62:24
you know, do significantly better because they start out slow
62:28
but they essentially end up
62:31
converging to a much better minimum later
62:33
And this is, you know
62:34
generally true when we decay learning rate
62:37
not necessarily true when we're in constant learning rate
62:39
which is maybe
62:40
somewhat more of where your intuition is coming from
62:44
So, you know
62:44
this is part of why it's very difficult
62:46
to reason sort of a priori or like from scratch
62:50
you know, the behavior of all these different choices and why
62:53
you know
62:53
I think Percy and I have designed this
62:55
class so that you interact with stuff because
62:57
you know, you might come upon this thing that we're basically
63:00
weight decay is actually an optimization intervention
63:03
and not necessarily a regularization intervention
63:06
which is, you know, what you would expect here
63:08
right? So
63:08
always keep that in mind that
63:10
these kinds of unexpected effects can really start to kick in
63:14
for these kinds of
63:16
settings
63:18
Cool
63:20
All right, so to put everything together for hyperparameters
63:24
there's actually for, you know, a lot of the
63:26
the maybe more hairy looking hyperparameters
63:28
actually just fairly standard choices that have worked well for everybody
63:32
right? You know, factor of four rule of thumb
63:34
keep your head dim and your number of heads
63:37
equal to the model dimension
63:39
pick an aspect ratio roughly around 100
63:43
and, you know, if you ask about regularization
63:45
right
63:46
you want to maybe try a couple of
63:47
things because regularization actually does
63:49
interact with optimizers in ways that are quite counterintuitive
63:52
right? So this is the thing that some people
63:54
still do even though you don't need
63:56
the regularization at all
63:58
Actually, maybe I'll stop here
64:00
Okay, yeah
64:08
That I have not looked into enough
64:10
to be honest
64:11
there aren't that many people training big diffusions
64:13
is one issue
64:14
and many of the models that have been
64:16
trained are retrofitted
64:17
So I think the architectures are actually the same as the
64:19
you know, like a llama-like model
64:21
but if you're asking the question of like
64:22
what's the optimal architecture if you were to train from scratch
64:25
I don't know what that is actually off
64:27
the top of my head
64:28
Yeah
64:33
Well
64:33
I guess it's not that regularization in general
64:36
affects optimization. I don't think people do drop out anymore because
64:38
you know, it doesn't really
64:40
interact well with optimization. But for example
64:41
weight decay, you know, which is shrinkage to zero
64:44
that might allow you to use a higher
64:46
learning rate or it might allow you to
64:47
decay faster
64:49
There are lots of ways in which all
64:50
these terms are interrelated
64:53
Cool. Okay
64:56
Now, I've talked a lot about how to design
64:59
expressive models
65:01
by
65:01
sort of looking at all these other models
65:03
that have been trained
65:04
one of the things that I'll
65:05
I'll highlight now is over the last few years
65:08
a really big emphasis has not been on performance alone
65:12
it has actually been on stability
65:14
And this becomes an increasingly important concern
65:17
as your models get more and more expensive to train
65:20
right?
65:21
we've kind of seen that a lot of
65:22
these choices are forgiving
65:23
right? Everyone's kind of doing similar stuff
65:25
And so, you know, you can mess with these
65:27
but you're not going to get a big performance difference
65:29
that's fine
65:30
But if your model
65:31
you know, suddenly blows up some part into training
65:34
like you get these like horrible looking spikes
65:36
all over the place
65:38
you know, you might end up with a model that is
65:40
you know, actually not very good quality
65:42
right? Or it might be unrecoverable
65:44
You might have spent
65:44
you know, millions of dollars in training
65:46
and, you know
65:47
you get to a point where the model
65:48
is no longer able to be trained any further
65:50
right? That would be a horrible thing to
65:51
happen
65:52
if you have a lot of compute that
65:53
you want to spend
65:54
So
65:55
you don't want to train models that look
65:56
kind of like this blue curve or like
65:57
spikes everywhere and these
65:59
you know, big gradient norms happening
66:01
so what do we do to fix these
66:03
stability issues? I mean
66:04
this is really, I would say, like a core, poor issue
66:08
And, you know
66:09
if you have stability issues
66:11
in language models or in general neural networks
66:14
there's a few, you know
66:16
usual suspects that you've
66:17
got to start looking at
66:19
one of them is the softmaxes
66:21
and the softmax has two things that are
66:23
both really bad for stability
66:25
One of them's an exponential
66:26
right? You can see how that blows up
66:27
very quickly
66:29
you also divide two numbers
66:30
and that's also a potentially very dangerous operation
66:33
right?
66:34
So a softmax is one place where you
66:36
got to be extra
66:37
extra careful
66:38
And where are the softmaxes in a language model?
66:40
Well, there's two of them
66:42
There's
66:42
one on the output side when we output our probability distribution
66:46
and then in attention
66:47
when we normalize the attention
66:48
there's going to be another softmax
66:49
right?
66:50
So
66:50
we can think of both of those as
66:52
really kind of danger zones for our model
66:55
especially the attention
66:59
but okay
67:01
let's start with thinking about the output softmax
67:03
The output softmax can blow up on us
67:07
and one of the things that we can
67:08
do is we can try to control
67:11
sort of the, the normalizer problem
67:13
So, you know
67:14
let's sort of think about the softmax calculation
67:16
We want to compute a log probability to
67:18
compute the loss
67:20
Now, what is the log probability?
67:21
Well, it's, you know, the output of your model
67:23
u, and then you've got this log normalizer
67:26
right? This u is well behaved
67:28
because in some sense this is the output of your model
67:30
right? This is just the
67:31
output of your residual stream with all the
67:33
things that are added in
67:34
So if u is well behaved
67:36
then log p, the first term, is well behaved, right?
67:38
The model is being okay
67:40
Now the second term
67:41
this log z, this might not be so okay
67:44
right? If z is really big or really small
67:46
even if the output of your model is somewhat well behaved
67:49
it could blow up. And what is z?
67:50
Well, it's an exponential, right?
67:52
So it could potentially blow up very quickly
67:54
on you
67:54
Or if this is zero
67:55
it could also blow up on you
67:56
right? So both of those directions are very
67:58
very bad
67:59
Now
67:59
we would ideally like our z to be somewhere near one
68:03
right?
68:04
Or log z to be somewhere near zero
68:07
What can we do?
68:09
Well, one of the things that you notice
68:11
right
68:11
if you've sort of thought about the action of the softmax
68:14
is this whole thing is over-parameterized
68:16
right?
68:17
I could sort of push things in and out
68:19
so if I add a constant to u
68:21
I can manipulate the z's
68:23
without really affecting the output of the softmax
68:25
right? It can cancel out between the normalizer
68:28
and sort of the output of my model
68:30
So because of that Property
68:32
one thing that I could do is I
68:33
could add a regularizer
68:34
This is from Jacob Devlin's paper 2024
68:37
sorry, 2014
68:39
in which he adds sort of this squared
68:41
log Z term
68:43
And what this is doing is it's just
68:45
penalizing how far away your log Z
68:48
is from zero. And if log Z is near zero
68:51
that's nice because this whole expression
68:53
is kind of numerically stable
68:55
This is called the Z-loss trick
68:58
It's been used by a number of papers
69:00
Jacob Devlin and others sort of popular
69:03
or initially pioneered this back in 2014
69:06
and then it's become popular again through a
69:08
number of open source models
69:09
Like Bai Chuan, I think
69:10
was the first open source model to do it
69:12
but then DCLM and Olmo and others have
69:14
been using this trick
69:15
to stabilize their output softmaxes
69:18
So this is a surprisingly effective thing
69:21
Now, okay
69:22
so let's say we've handled
69:23
the instability issues on the output softmax
69:27
Now we have to sort of turn our
69:28
attention towards the other potential problem
69:30
which is attention, right? And this is a
69:33
a place where lots of degeneracies happen
69:36
Lots of techniques have been developed
69:37
to control the instability that attention
69:40
operations generate
69:43
and really the, you know
69:44
the high level thing that I'll say is
69:47
if you have instability
69:48
if you can throw a layer norm in there somehow
69:51
it might control it
69:52
And that's really, in some sense, the
69:54
the design philosophy behind this idea called the
69:57
QK norm
69:59
so what you do is remember that we have
70:01
you know, our Qs and Ks
70:03
that are going to be multiplied together
70:05
and then they're going to go into the softmax
70:07
right? So in the standard attention operation
70:09
I'm going to layer norm as a pre-layer norm
70:11
multiply with a QKV
70:13
and I'm going to get my Qs and
70:14
Ks
70:15
Those will get multiplied by a matrix multiply
70:17
I'll softmax them
70:18
and I'll multiply that with V to get the weighted average
70:21
and then I'll output whatever comes after
70:23
right? So this is our usual attention
70:25
Now what happens
70:26
if we just throw in a layer norm
70:28
before we multiply the Qs and Ks?
70:32
If we do that
70:33
then we know that the inputs to this matrix multiply
70:36
and therefore the inputs to the softmax
70:38
roughly have the same scale
70:40
They're always going to have a scale of roughly one
70:43
because we've, you know
70:44
used RMS norm to divide the size of
70:47
those Qs and Ks
70:49
Okay
70:49
If we do that
70:51
then, you know
70:51
we're kind of going to keep
70:53
this softmax operation stable. Tons of different models
70:55
do this. It's originally from
70:58
the multimodal world
71:01
you know, some folks who were doing
71:02
making multimodal models sort of initially discovered QK
71:05
norm
71:06
Edifix and Chameleon really
71:08
you know, used this and, like
71:09
proved it out. And then a number of other
71:12
open source language models
71:13
you know, realized that the same tricks are entirely applicable to
71:18
stabilizing attention for language models
71:20
And I think this is now very
71:21
very standard. Like
71:22
QK norm is actually a very standard intervention
71:25
that most of the large models now introduce
71:28
it doesn't seem to affect performance
71:30
from lots of different training runs
71:32
but it does definitely prevent the kinds of
71:35
attention degeneracies
71:37
and, you know, I, I'm really, The
71:40
the way that I've seen this is
71:41
you know, we have layer norms
71:43
initially in the pre-norm
71:44
now we add them after the
71:46
the, nonlinearities in each block
71:48
and now we're throwing them in both the
71:50
Qs and the Ks
71:51
and really I think this is
71:52
this is kind of getting at
71:54
the stabilization tricks that people apply to this world
71:57
Okay
71:58
Now
71:59
the final set of things that I'll talk
72:01
about is a stability intervention
72:03
and frankly this one
72:05
is not as popular and more of a
72:07
of a Google specific trick that I've seen
72:11
but
72:12
logit soft capping is a much harder intervention
72:14
that some people apply. So this one
72:17
you know, in QK norm
72:19
what we're doing
72:20
is we are controlling the inputs to the
72:21
softmax
72:22
and sort of hoping that the outputs are
72:24
well behaved
72:25
If we really, really want to enforce
72:28
well behaved outputs
72:29
what we can do
72:30
is we can kind of take the logits
72:32
the things that go straight into the softmax
72:34
and we can just cap them off so
72:35
they can never be too large or too small
72:37
right? This is a hard
72:38
almost a hard constraint
72:40
it's called a soft cap
72:41
of course, but a tanh, you know
72:42
is bounded at some value
72:45
and so, this is in the GEMA models
72:48
I think both GEMA
72:49
or like GEMA 2
72:51
3, and 4, all use the Lojitz soft cap trick
72:55
and what they do
72:56
is they take all of their Lojitz for the potential layers
72:59
and then they soft cap them at some
73:00
value
73:03
some NVIDIA folks have done actually quite nice
73:05
work doing systematic comparisons of these stability interventions
73:09
and what they find is
73:11
if you start with a baseline model
73:13
you can do all sorts of different interventions
73:15
and QK norm is here
73:18
and it does slightly better due to the
73:19
fact that you can crank up the learning
73:21
rate a little bit
73:23
but if you do soft capping alone
73:25
you actually end up losing performance
73:26
So there is a
73:27
a quality degradation that happens
73:29
this is a very strong intervention
73:30
you can never express very confident
73:33
signals in your soft max beyond a certain
73:34
point
73:36
so it does have some negative consequences
73:38
but this is a very safe way
73:39
of stabilizing the outputs of your attention
73:42
Or sorry, the inputs to your attention
73:44
the logents that go into the softmax
73:47
Okay
73:48
so
73:48
that's kind of the end of the stability
73:50
components
73:51
I can pause for a moment here
73:53
and I'll talk about sort of various attention things
73:56
after that
74:00
Cool
74:01
Okay
74:03
All right
74:03
so
74:04
the last thing I want to talk about
74:05
today
74:06
is various interventions that you can make your
74:08
attention
74:10
and as I was saying at the beginning of this lecture
74:13
I'm only going to talk about all the
74:14
things that you can do
74:16
to sort of dense all by all attention
74:18
today. So if you
74:18
if you were interested in hearing about state
74:20
space models or linear time attention
74:23
sadly today is not the day for you
74:26
the things that I do want to talk about
74:27
which are really commonly implemented
74:30
attention interventions today are
74:33
group query attention
74:34
which really saves inference costs by reducing the number of heads
74:38
and sparser sliding window attention
74:41
which really originally came from the GPT-3-ish family
74:45
but have now really been adopted widely
74:47
by most models that are looking to do
74:51
long context, unless they're doing exotic
74:53
SSM stuff
74:56
So I'll start with
74:57
group query attention, or GQA or MQA
75:00
this
75:01
I'm gonna
75:02
first set up the need for these kinds of things
75:04
and then you'll kind of hopefully see what the
75:07
what the trick is
75:08
and why it's fairly natural
75:10
So for the moment
75:11
we've been talking about
75:13
you know, training and modeling and all these things
75:15
but like, let's take a pause
75:16
and now let's think about deployment
75:18
right?
75:19
You've trained this very big model
75:21
and now you need to serve it to lots of users
75:23
and you're gonna pay a cost for serving
75:26
And you're gonna have to
75:27
in abstract sense, pay for two different resources
75:30
right? You're gonna have to pay for your flops
75:32
right, the computation that you perform
75:34
but you also have to pay for another
75:36
thing. You have to pay for your memory accesses
75:38
right?
75:38
Because the memory accesses are also going to impact
75:41
you know, your system's characteristics, your latency, your utilization, right?
75:45
So you want both of these things to be small
75:48
Now, let's think about what happens during training
75:50
or, alternatively, prefill, when you're looking at your prompt
75:54
where someone gives you the stuff
75:55
In this case, you know
75:57
the total arithmetic operations you have is
76:00
you know, order of magnitude
76:01
batch size, sequence length, hidden dim square, right?
76:04
That's roughly the size of things that you
76:06
get. And of course
76:07
you know, we're doing quadratic attention
76:08
so we've got d squared
76:12
we've got, total memory accesses
76:14
like what is our memory access that we
76:16
have here?
76:17
we have batch times sequence length times
76:19
hidden dim, plus, the sort of cost of the softmax
76:23
which has a n squared component
76:24
and then we've got a d squared component
76:27
for the, for the projections
76:28
So the arithmetic intensity here is pretty good
76:32
it's gonna be one over k
76:33
this is the number of heads
76:34
so you need to have
76:35
in, sorry, head dims
76:36
so your head dims need to be big
76:37
enough that you're multiplying some reasonably sized matrices
76:40
and you've got a one over bn
76:42
so your sequences need to be long enough
76:44
or your batch sizes need to be big
76:45
enough
76:46
As long as both of these are true
76:48
your GPUs are gonna be fully utilized
76:49
Great, right, you're, you're using all of your resources
76:54
Now, you know, we have done, we have finished training
76:56
and now we're serving our users
76:58
How do we serve our users?
76:59
We're gonna generate tokens and send it to them
77:01
right?
77:02
Now, if we're doing that
77:03
I can't paralyze the generation process
77:06
What I'm going to do is I'm going
77:07
to generate a token
77:08
I'm going to condition on it
77:08
I'm going to generate the next token
77:10
and I'm going to repeat this process one
77:11
by one. This is just sort of
77:13
the curse of auto-regressive language modeling
77:15
we have to do this
77:17
In order to do this
77:18
the efficient way to do it is to
77:20
maintain
77:21
all of the sort of past keys and
77:23
queries that I've had
77:25
in what's called a KV cache
77:26
So the KV cache
77:27
is going to maintain this matrix of
77:30
Q dot K over the past
77:32
and then whenever I need to compute something new
77:34
I can reuse sort of
77:36
the sub-matrices
77:37
that I've already had from the past
77:39
And I only really need to compute
77:41
sort of the new
77:43
query key
77:44
interactions that I need to fill out
77:46
the rest of this matrix
77:48
So every sub-matrix I've computed before
77:51
I can keep
77:52
I only need to compute my new one
77:53
So this saves a lot on compute
77:54
right? That's great
77:57
But
77:58
The issue here
77:59
is now our arithmetic intensity is not so good
78:02
right?
78:03
As you might sort of intuit
78:04
this KD cache approach is going to be reading and
78:07
and, reading, parameters all the time
78:10
right? Each time I have a new step
78:12
right, I'm gonna have to read in my parameters
78:14
I'm gonna have to take these dot products
78:16
and I'm gonna do this once every step
78:19
And so now what do I have?
78:20
Well, you know, my total memory act
78:22
oh, sorry, my total arithmetic operations are the same
78:24
I'm multiplying the same matrices still
78:26
right, just incrementally rather than all at once
78:29
But because I'm doing this incrementally
78:32
you know, now I have
78:34
a memory access pattern of batch
78:36
by sequence squared by hidden dim
78:38
plus, sequence by hidden dim squared
78:41
And the second term is not so pleasant
78:44
right? It used to be that it was just D squared
78:48
but now we've got N times D squared
78:51
and if we compute the arithmetic intensity
78:53
which is the ratio of these two guys
78:56
now we have N over D plus one over B
78:59
so now what we need
79:00
is large batches plus short sequence length
79:03
or we need really big model dimensions
79:05
So if we wanna serve a small model efficiently
79:08
this is not so good
79:10
right?
79:12
this is really difficult to deal with
79:13
right? This N over D term
79:15
this first term over here
79:16
which is sequence length over hidden dim
79:19
is very difficult to reduce if we're doing
79:21
this incremental computation
79:23
this is just a hard thing to deal with
79:26
So, this leads to this idea of MQA
79:29
or multi-query attention
79:31
Normally, you have multiple heads in your attention
79:34
operation, and you're gonna have different keys
79:36
different values, and different queries
79:38
That's normally how you do things
79:40
But one thing that we could do
79:42
is maybe we can keep the Ks and
79:44
the Vs the same across all the heads
79:46
and the only thing that's different across the
79:48
heads are the queries
79:50
If we do this
79:51
then this drastically removes the amount of
79:55
items that need to be moved in and out of memory
79:57
right? Because the KV cache
79:59
right, are now significantly smaller
80:01
These are all shared across all the heads
80:04
this
80:05
significantly reduces the total memory access as well
80:07
as the arithmetic intensity
80:09
and we're kind of
80:10
the key term that we were talking about here
80:12
we had the N over D term
80:14
now we have, H multiplying this, right?
80:17
And so this H term
80:19
allows us to significantly reduce the
80:23
the, sorry
80:24
increase the arithmetic intensity if we have a lot of heads
80:26
right? This is a significant gain over what we had
80:29
before
80:31
So this gets us significant efficiency improvements
80:34
but the issue with MQA is
80:38
This is on the right here
80:39
You have one value and one key
80:41
for all these queries. You do
80:43
in fact
80:44
lose significant expressive power if you do this
80:47
and so there's this trade-off between system sufficiency and expressiveness
80:51
and you might wonder
80:52
is there sort of a sweet spot
80:54
in which we can avoid trading off
80:57
you know, quite a significantly
81:00
expressive power and computation? And that's where GQA
81:03
or grouped query attention
81:04
comes in
81:05
You know, the original transformer is multi-head
81:07
We have queries and keys for each head
81:09
In multi-query, we have one key and value for each
81:13
for all the heads. In grouped query
81:15
we reduce the amount of keys and values
81:17
but we keep the number of queries the
81:19
same. So we now have this ratio that
81:20
we can play with
81:21
which is kind of the number of key heads
81:23
or the number of value heads
81:25
while keeping the total number of heads much
81:27
larger than that. So this allows us to
81:28
very simply control the
81:30
the, trade-off between expressiveness and inference efficiency
81:35
there are other sort of tricks from DeepSeek V2
81:37
multi-head latent attention, that I'll sort of mention briefly
81:41
next time
81:43
which sort of have a different kind of
81:44
factorization structure and a different set of trade-offs
81:47
But really
81:49
the nice thing about GQA
81:51
is that in practice the trade-off is quite
81:53
favorable. So if you have multi-head
81:55
your performance, this was
81:56
you know, in the, I think T5 days
81:58
if I remember right
82:00
this is your, downstream model performance
82:03
This is your time per sample
82:04
You want to reduce this as much as
82:05
possible
82:06
With multi-head attention
82:07
you have best performance but very high cost
82:10
with MQA, you know, you have lower
82:14
cost but much lower performance
82:16
Similarly
82:17
if you make your model smaller to try
82:19
to hit your performance targets
82:20
you got much worse performance
82:22
GQA really does get the best of both worlds
82:24
you know, very low inference cost
82:27
nearly the same performance as your full multi-head
82:31
and you see sort of this like
82:33
GQA group structure where if you have
82:36
a small reduction in the number of heads
82:39
you basically have most of the gains in your performance
82:42
which allows you to sort of keep most
82:44
of the expressive power
82:46
while getting significant, inference improvements
82:48
And first
82:49
you'll talk a bunch more about sort of
82:50
the inference mechanics later
82:53
but sort of this should give you a
82:54
flavor of like why models today almost all adopt this GQA
82:58
structure, because it gives you a lot of this inference cost
83:00
which is really critical
83:02
without very much of a
83:05
expressiveness hit
83:08
Cool
83:09
Any questions for, for GQA, or KV cache?
83:11
Yeah
83:14
But, GitNet has so many
83:18
like, rules of thumbs for what hyperpar fences are good
83:20
Like
83:21
to what extent are you still searching over
83:22
hyperpar fences versus
83:24
exploiting these rules of thumb that you have?
83:28
I think it's a mix of both
83:29
I think every
83:30
sort of model training run has some theses
83:32
about what can be varied
83:34
and so you see this in a lot of the reports
83:36
where I think the hyperparameters are often not
83:38
where people are touching too much
83:40
but you see, like, architecture changes, like, one at a time
83:43
in a lot of these reports
83:44
But it's very rare to
83:45
like, go and change everything up
83:47
I think Google is one of the only
83:49
orgs that seems to
83:50
like
83:50
really spice things up in a significant way
83:53
the Gamma series
83:54
has done some pretty interesting things
83:57
the most recent Gamma 4 release
83:58
for example, now has, like, individual embedding for every layer
84:02
in a way to control the trade-offs between
84:04
like, memory use and flops
84:05
It's a very interesting set of things that they've done
84:09
Oh, yeah, back there
84:10
Do experiments with, like, dynamically altering some of these
84:13
parameters during training? During training
84:18
let me think
84:20
Weight decay
84:21
yes
84:22
Weight decay people change
84:24
in concert with like learning rate
84:26
that is actually a heuristic that people do
84:28
that works very well
84:31
other than that
84:32
I don't know if there's a lot of
84:34
different hypers that people change during training
84:37
especially cause the architecture ones
84:38
just make training incompatible
84:40
So you can't really
84:41
you know, change them while you're
84:42
you're training
84:43
yeah. So I think
84:44
I think weight decay is probably the one
84:45
that I can think of
84:46
The others are usually fixed
84:49
Yeah
84:50
Yeah, and QA is a
84:53
it's not just inference time fixed
84:55
It's a pleasure time. That's right
84:57
yeah. You, you
84:57
you train with a certain number of keys
84:59
Yeah
85:02
Okay, the last thing I'll talk about is
85:05
sliding window attention, which is a really old idea
85:08
Like, you know, GPT-3 used actually this
85:11
you know, if you read the paper
85:12
they'll say we alternate between full attention
85:15
which where every position can attend to everyone in the past
85:18
in a banded matrix
85:20
style attention where you can
85:21
attend to everyone within a fixed window
85:24
and, you know
85:25
OpenAI has some early work on these kinds of
85:27
like
85:27
different kinds of attention patterns
85:29
that you can use
85:31
but actually this has become really
85:32
really popular over the past year
85:36
this idea of alternating
85:38
you know
85:38
the big full attention
85:40
and a
85:41
more local attention actually hits a sweet spot
85:44
for how to manage
85:45
like
85:45
long context performance while not paying
85:48
too much for inference
85:51
I think, you know, the more recent revival in open models
85:55
I would maybe say
85:56
Cohere Command A was the first one I saw do it
85:59
where they, you know, had this, like
86:01
structure where every four layers
86:04
they would have a full attention that attended
86:05
to everything
86:06
The three layers in between would use a
86:08
sliding window attention
86:10
that would only be able to look at
86:11
local structure. And of course
86:12
you know, as you go up
86:14
sorry, in this case down
86:15
because they ordered the
86:16
the diagram the other way
86:18
as you go down sort of these blocks
86:20
you know
86:20
you're aggregating local information into global ones
86:22
So the local attentions at the end can
86:24
of course, access more global information
86:26
but this, you know, allows you to manage the
86:30
the
86:30
the cost
86:31
of having a really long context
86:33
without having to go for something like
86:36
a state space model or more exotic intervention
86:38
and that's worked quite well
86:41
there's also some innovation where
86:43
people change the, embedding format for the long-range information
86:47
where they get rid of things like
86:49
rope
86:49
so you have no position embeddings at all
86:51
So you're really looking almost at bags
86:54
where the short-range information still gets position information
86:57
so people do all sorts of
86:59
you know, kinds of interventions involving these
87:02
these
87:02
both the embeddings
87:04
and alternating local and global structure
87:07
I'll say that this is a
87:09
you know
87:09
a tension in general how to manage the trade-off between
87:12
long context and sort of
87:14
long context cost and performance is
87:17
still an active area of investigation
87:18
It's a place where
87:19
the most architecture work and changes are still
87:22
being done
87:23
we see essentially a
87:25
a bunch of other models adopt this idea
87:27
Llama4, most recently Gemma4, Omo3
87:30
They all do
87:31
this combination of sliding window attention and full attention
87:34
in their case using full rope instead of
87:36
nope as the, embedding
87:39
So as I said
87:41
this is becoming really
87:41
really popular
87:43
Qwen 3.5, which I've put on the right
87:45
they're actually a little bit different because they
87:47
alternate a state space model
87:50
called, gated DeltaNet and a full attention
87:53
every sort of, you know, one
87:55
or one full attention every four layer
87:57
every four layers
87:59
so it's the same alternating structure
88:00
but they're using a different
88:02
sort of cheap layer. In their case
88:03
they're using a state space model
88:05
I'll explain what that is
88:06
next lecture
88:07
instead of a sliding window sort of local
88:09
attention
88:10
But you see this is like
88:11
I think, a new theme over the past year
88:13
where, you know
88:14
open models are really trying to grapple with long context performance
88:17
and the way to do that
88:18
at least so far
88:20
is to have these hybrid models that aren't just global attention
88:23
aren't just cheap attention
88:24
there's some sort of mix in between
88:26
and that's
88:26
that seems to have worked very well so
88:28
far in a lot of these models
88:32
Okay, cool
88:33
So as I was sort of trying to emphasize
88:35
when you look across all of these models
88:37
you start to see a lot of patterns
88:39
and hopefully a sense of
88:40
general understanding about what
88:42
things you can do and what things are
88:44
good defaults
88:45
we also see a lot of differences in
88:47
how we handle context and how we handle
88:49
position embeddings
88:51
even tokenization, there's some differences, right?
88:53
So there are differences across these models
88:55
but there's also commonalities that hopefully now give
88:57
you some intuition
88:58
as you go out and do your assignments and
88:59
and sort of
89:00
mess with the leaderboard and so on
89:02
thanks

Watch these videos on your TV

XTTV brings video content from X to your living room. Download now and start watching on the big screen.