← Back to videos
Anthropic pays $750,000+ a year for engineers who can build LLM architectures from scratch. Stanford

Dhruv kumar
@Dhruvkumar16797
November 18, 2026· 89:02
Anthropic pays $750,000+ a year for engineers who can build LLM architectures from scratch. Stanford taught the entire thing in 1 hour lecture & released it for free. Bookmark & watch this today before someone takes it down and read this article below. https://t.co/tApBhohRAS
Transcript
- So today we're going to talk about architecture
- 0:02
- which, at least to me
- 0:03
- has always been pretty inscrutable
- 0:06
- and so I'm going to take the approach
- 0:08
- of just telling you kind of everything
- 0:10
- right? I'm going to go through all of the modern papers
- 0:14
- and we're going to just look through what
- 0:15
- has everyone done
- 0:17
- and so I've titled this Everything You Didn't
- 0:18
- Want to Know About Architectures and Hyperparameters
- 0:21
- because I think we all wished
- 0:23
- we lived in a world where the only
- 0:24
- things you had to know were like VC dimension or something
- 0:26
- right? Like very simple
- 0:28
- you know, theoretical tools
- 0:29
- but that's not really where we are
- 0:31
- So okay
- 0:32
- what we are going to do
- 0:33
- is we are going to
- 0:35
- try to understand architecture from kind of like a survey lens
- 0:38
- right?
- 0:39
- The best thing to do
- 0:40
- you know, better than listening to this lecture even
- 0:42
- is for you to go out and like
- 0:43
- train your own models and try different architectures
- 0:46
- right? That's by far the best thing to
- 0:47
- do
- 0:48
- That's part of the philosophy of the course
- 0:50
- But
- 0:50
- we're not going to be able to cover
- 0:52
- the whole design space of all the different
- 0:54
- architectures that are out there
- 0:55
- right? Like that's not something that we have
- 0:56
- the compute or the time to do
- 0:58
- So
- 0:59
- my opinion is the second best thing that
- 1:01
- we could do
- 1:02
- is to try to learn from the experience of others
- 1:04
- right? What has, what has everyone else done?
- 1:06
- What are the choices that they are making
- 1:08
- right?
- 1:08
- And by looking at kind of a broader
- 1:10
- somewhat zoomed out picture
- 1:11
- maybe we can start to understand
- 1:13
- oh
- 1:13
- these are the kinds of parameters and choices
- 1:16
- that are sort of fixed across all effective architectures
- 1:19
- and these other ones can be sort of varied without impacting
- 1:22
- how the model perform
- 1:23
- right? So I'm going to talk about
- 1:24
- you know, basically transformer variants, like what is the
- 1:28
- you know, modern transformer starting with
- 1:30
- you know, the Vaswani paper, and then, you know, as we go to more modern
- 1:34
- more recent architectures
- 1:36
- what do they have in common?
- 1:37
- And then what are we allowed to vary
- 1:38
- or not allowed
- 1:39
- but what do people vary as they go through this
- 1:42
- right?
- 1:43
- So
- 1:44
- I think many of you have taken an
- 1:45
- NLP course of some kind
- 1:47
- or at least seen a transformer
- 1:49
- so you've probably seen
- 1:50
- you know
- 1:50
- the very vanilla transformer from Vaswani et al
- 1:54
- you know, there
- 1:56
- There are some fairly standard choices that you make
- 1:58
- You say, oh, transformers don't have positional dependence
- 2:00
- so we're going to add a position embedding
- 2:02
- And what do we do?
- 2:03
- We're going to add some sines and cosines
- 2:05
- we're going to have information processing through
- 2:08
- ReLU
- 2:09
- and then we're going to have a
- 2:10
- a post norm. I'll talk about what exactly
- 2:12
- that is later
- 2:14
- and when you look at your assignment
- 2:15
- your A1
- 2:16
- you're going to notice some differences between
- 2:18
- the standard or the vanilla transformer
- 2:21
- and what we've asked you to implement
- 2:22
- Well
- 2:22
- we're going to ask you to move the
- 2:24
- layer norm to the front of each transformer block
- 2:26
- or the non-residual layers
- 2:28
- We're going to ask you to implement something
- 2:29
- called row
- 2:31
- and
- 2:32
- we're going to ask you to implement something
- 2:33
- called swiglu and not ReLU
- 2:35
- Right
- 2:36
- Why do we pick these?
- 2:37
- one reason is we've
- 2:39
- you know, copied a lot of this over from Llama
- 2:41
- but so did everyone else
- 2:42
- Really
- 2:43
- I think
- 2:43
- if you were to train on your own language model
- 2:45
- I think you'll quickly run into this question of
- 2:47
- oh, there's so many choices
- 2:49
- right? Like
- 2:49
- what do I choose for all these things?
- 2:51
- And so let's now
- 2:53
- sort of walk through all these different models
- 2:56
- The way I kind of think about architectures
- 2:58
- is to think about
- 2:59
- to look at all the different things people
- 3:00
- have done and say
- 3:01
- what are the things that people have done?
- 3:03
- Can we pick and choose from those?
- 3:06
- Percy always makes fun of me for this a little bit
- 3:08
- but, you know, I try to look at all the
- 3:10
- the different models that come out each year
- 3:12
- to try to make this lecture
- 3:14
- and last year I thought
- 3:15
- oh, there's just a couple papers
- 3:16
- it's gonna be fine
- 3:17
- it's gonna be fine
- 3:18
- and then I looked through all the things
- 3:19
- and there's a lot of papers
- 3:20
- there's Qwen2 and Gamma3 and InternLM2
- 3:24
- and then there were even more
- 3:25
- there's like Nemotron4 and Qwen2 and oh
- 3:27
- oh my goodness
- 3:28
- there were 19 new dense models
- 3:30
- And so last year
- 3:31
- I had my work cut out for me
- 3:32
- And then this year
- 3:34
- you know, I thought, well
- 3:35
- there can't be that many new LM releases
- 3:37
- like it's gotta be slowing down
- 3:38
- right? Like people can't keep training 20 dense
- 3:41
- LMs per year
- 3:42
- and that's technically right
- 3:43
- there aren't that as many dense LMs
- 3:44
- Initially, you know, I was like, oh, there's Qwen3, Gamma4 just came out last Thursday
- 3:48
- so I put that in there
- 3:50
- and almost three, you know, there's only a couple
- 3:52
- and of course I have to
- 3:53
- Give a shout out to Percy's own AP model
- 3:55
- trained with Marin. And I was like
- 3:56
- oh
- 3:56
- we'll just have a few things to cover
- 3:58
- But it turns out if you start looking
- 3:59
- there's a lot of different models
- 4:02
- And so the fact that we have so many different models
- 4:05
- most of these actually are MOEs
- 4:07
- mixtures of experts
- 4:08
- and I'll be talking about that tomorrow rather
- 4:09
- than today
- 4:10
- Because we have such a big diversity of models
- 4:12
- we actually get a pretty good picture
- 4:15
- of all the different choices that we can
- 4:17
- make
- 4:18
- So I made this little table
- 4:20
- We'll come back to this little table at
- 4:21
- the end of the lecture
- 4:24
- But basically at this point
- 4:25
- starting with the original transformer
- 4:28
- there's been actually quite a few
- 4:30
- auto-regressive language models
- 4:31
- kind of trained on the same class of
- 4:33
- things
- 4:35
- And you can ask questions like
- 4:36
- what are the different vocabulary sizes?
- 4:38
- Or what kind of layer norms do we
- 4:40
- use?
- 4:41
- Or
- 4:41
- what kind of position embeddings do people use?
- 4:44
- And we see some fairly clear trends
- 4:45
- I'll be talking about this as we go
- 4:47
- Okay
- 4:49
- So
- 4:51
- The goal here
- 4:52
- is that we're going to cover
- 4:53
- a couple of different things
- 4:54
- We're going to cover common architecture variations
- 4:57
- so these are different building blocks of the
- 4:58
- transformer
- 5:00
- And after we've established what the standard building blocks are
- 5:03
- like, you know
- 5:03
- what do we use for
- 5:05
- the nonlinearities or what do we use for position embeddings
- 5:08
- then we're going to talk about hyperparameters
- 5:10
- We're going to go down even lower detail and say
- 5:12
- like, you know, what is FFDIM?
- 5:14
- Should we make that a multiple of four or
- 5:16
- like
- 5:16
- multiply the hidden by four to get FFDIM?
- 5:19
- How many vocab elements should I have?
- 5:22
- And then after that
- 5:23
- we're going to talk about
- 5:25
- very low-level tricks
- 5:26
- of how to get models to train stably
- 5:29
- And the reason why I'm going to talk
- 5:30
- about that in this lecture
- 5:31
- is because these stability tricks
- 5:33
- have a pretty close connection with the architecture variation
- 5:37
- right?
- 5:38
- One of the things that
- 5:39
- you know, higher level
- 5:40
- I want to sort of impress upon you
- 5:41
- is that architectures are actually a very complex set of trade-offs
- 5:45
- right? Like, what does an architecture have to do?
- 5:48
- Well, it has to learn from data
- 5:49
- so it has to generalize
- 5:51
- it has to train efficiently on GPUs
- 5:53
- and it has to not blow up
- 5:55
- right? Like halfway through training
- 5:56
- if your, you know
- 5:57
- training
- 5:58
- losses just go like down like this and
- 5:59
- then suddenly blow up
- 6:01
- that's no good at all
- 6:01
- right?
- 6:02
- So all these different requirements
- 6:04
- end up getting baked straight into the architecture
- 6:06
- and that's why these things are a little
- 6:08
- bit messy and a little bit complex
- 6:10
- but you should keep that in mind
- 6:12
- and that's why, you know, things are in
- 6:13
- in many ways not so elegant
- 6:15
- Okay
- 6:16
- So
- 6:17
- we're gonna start with the core architecture piece
- 6:20
- And as a high level view
- 6:22
- you know, I think the
- 6:23
- the way that I see a lot of the architecture stuff
- 6:25
- you know, looking basically historically
- 6:27
- is kind of in the early days of
- 6:29
- you know, starting with the transformer until
- 6:31
- you know, GPT-3 or so
- 6:33
- there's a lot of experimentation that happens
- 6:35
- People try lots of different things
- 6:37
- there's no like gold standard that everyone has
- 6:39
- unified on
- 6:40
- And then, you know, Llama 2 comes out
- 6:42
- and everyone's like, wow, Llama 2 is great
- 6:43
- I want my own Llama 2
- 6:45
- And so everyone starts training Llama 2-alikes with
- 6:48
- you know, minor variation, that people have
- 6:52
- And then finally, you know, last year
- 6:54
- we saw really big differences or
- 6:56
- or sort of a trend towards
- 6:58
- architecture modifications that make training more stable
- 7:01
- And this year, we see lots of trends towards
- 7:04
- architecture variations that enable longer context dependence
- 7:08
- So there are these big themes that are happening
- 7:10
- but really, I think, you know
- 7:11
- you see this like big point
- 7:13
- Lam, when Llama 2 comes out
- 7:14
- and everyone's like, wow
- 7:15
- I wanna train some with that
- 7:16
- And then suddenly, or not suddenly, but after that
- 7:18
- people are starting to explore once again
- 7:20
- So it's kind of cool to see
- 7:21
- all of these different changes
- 7:24
- Okay
- 7:25
- I think
- 7:26
- people can disagree about a lot of things on architectures
- 7:29
- but there is one thing that everyone agrees on
- 7:32
- you know? Like, if you take the transformer paper
- 7:34
- I think a lot of people will say
- 7:36
- like, the transformer people got
- 7:37
- like, most of the things right
- 7:39
- except this
- 7:40
- And the thing that they really did not get right
- 7:42
- or like
- 7:42
- I think most people agree they did not get right
- 7:45
- is where you put the layer norm
- 7:47
- right? So in the original
- 7:50
- transformer paper, you know
- 7:52
- the layer norm
- 7:53
- goes in what you would call the residual path
- 7:55
- right? So in the transformer
- 7:56
- you know, you have the residual stream
- 7:58
- this X, that kind of runs through the whole network
- 8:00
- and then you are going to be applying a tension
- 8:03
- which, you know, does some computation
- 8:05
- and it will
- 8:05
- add back sort of a delta
- 8:07
- back into the residual stream
- 8:10
- And then
- 8:10
- in order to make sure that these gradients
- 8:12
- are sort of stable across layers
- 8:15
- you know
- 8:15
- a layer norm is placed at the end
- 8:17
- of each of these components
- 8:19
- Now, instead of putting the layer norms in the residual stream
- 8:24
- there's an alternative, I'll refer to this as pre-norm
- 8:28
- in which
- 8:29
- you can put the layer norm outside of the residual stream
- 8:32
- but before each of the computations
- 8:34
- So you can put it before the multi-head attention
- 8:36
- you can put it before the FFN
- 8:38
- right?
- 8:38
- we'll call this pre-norm
- 8:40
- the nomenclature will get a little bit confusing
- 8:43
- you can call this
- 8:44
- post-norm for now, but let's call this sort of residual norm
- 8:47
- right? Because you're putting the norm in the residual layer
- 8:51
- basically, all modern language models
- 8:54
- push the layer norm
- 8:56
- outside of the residual stream
- 8:57
- This is just like a thing that
- 8:59
- basically everybody does
- 9:01
- there is one funny exception
- 9:03
- but it is OPT 350M
- 9:06
- and if you all are familiar with sort of language models
- 9:09
- we kind of know OPT in general was
- 9:11
- you know, kind of a mess of a language model
- 9:13
- right? And OPT 350M
- 9:16
- is even more so because I don't know
- 9:17
- why only that model
- 9:19
- has a post-layer norm in the residual stream
- 9:23
- Okay, so this is one of the things that
- 9:25
- like, everyone agrees on, and so you might wonder
- 9:28
- like, why is this, like, such a, you know
- 9:31
- un- like a uni- unified thing
- 9:33
- across all the different models?
- 9:37
- and if you look at some of the early works studying
- 9:39
- like, where do you place the layer norm style research
- 9:43
- what you really see is that
- 9:46
- you know
- 9:46
- the early motivation for a lot of this
- 9:48
- was when you train a transformer
- 9:51
- you need to do a warm-up
- 9:52
- Actually, you know, modern transformer training still does warm-ups as well
- 9:55
- but you definitely need to do warm-up when you train
- 9:59
- Now, wouldn't it be nice if we could remove the warm-up
- 10:02
- right? So that was kind of the initial
- 10:03
- motivation for a lot of this research
- 10:05
- but people quickly realized that removing the warm-up
- 10:08
- had very serious issues in terms of the stability and convergence
- 10:12
- of these things, right? So if you did
- 10:14
- post-norm plus layer norm
- 10:16
- which is, you know, basically the original transformer thing
- 10:19
- you got this purple dashed line
- 10:20
- Oh, you just don't converge as well
- 10:23
- compared to doing something like pre-norm
- 10:25
- you can ignore the other
- 10:26
- terms, you would get much nicer convergence even without warm-up
- 10:30
- right? So this was the original motivation
- 10:33
- But really what people kind of quickly realized is that
- 10:36
- you know
- 10:37
- moving the
- 10:38
- layer norms outside the residual stream
- 10:40
- has some
- 10:41
- pretty important implications as you make
- 10:43
- your network deeper
- 10:45
- and as you start to grapple with stability issues
- 10:48
- to me
- 10:49
- I think the gradient attenuation issues
- 10:51
- are kind of the most clear
- 10:54
- when you sort of talk to people who do architecture design
- 10:57
- I'm not really one of the
- 10:59
- the people that deeply engages in this
- 11:01
- but one of the things that people often
- 11:02
- say is keep your residual stream clean
- 11:05
- right? So in this case
- 11:06
- you have your Xs coming in from the
- 11:07
- bottom on the pre-norm side
- 11:09
- and this X propagates all the way up to the top
- 11:12
- right, all the way up to your
- 11:13
- your final output
- 11:14
- And that allows gradients to propagate
- 11:16
- if in the backward pass
- 11:17
- straight through this, right?
- 11:20
- that makes gradient propagation very simple
- 11:23
- which improves both stability and signal propagation
- 11:26
- And that's sort of what people realized very
- 11:28
- very quickly, that if you do something like pre-norm in blue
- 11:33
- at initialization
- 11:35
- sort of the gradient sizes kind of remains the same
- 11:37
- right? Because you have this nice straight through
- 11:39
- propagation in the backward path
- 11:41
- On the other hand
- 11:42
- if you have post-layer norm
- 11:44
- you have these kind of complicated effects that
- 11:46
- happen
- 11:47
- because you're layer norming
- 11:48
- each time you're going through a transformer block
- 11:51
- and that's going to change the norm of
- 11:52
- your gradients as you go backwards through
- 11:55
- So you can kind of see You know
- 11:58
- from the principle of keep your residual stream clean
- 12:01
- that pre-norm makes a lot of sense
- 12:04
- people also realized through experimentation that this also
- 12:06
- improves stability in general
- 12:08
- that the sizes and frequencies of gradient spikes
- 12:13
- were improved under pre-norm compared to post-norm
- 12:16
- And, you know, this is a figure from Salazar and Yuen
- 12:19
- who were one of the first ones
- 12:20
- I think, to, to study this phenomenon carefully
- 12:23
- And I think this is the reason why it stuck around
- 12:25
- right?
- 12:26
- stability and the ability to go deep are both very
- 12:28
- very important for modern large language models
- 12:32
- and so this idea of moving your layer
- 12:34
- norm outside of the residual stream is one that basically everyone
- 12:38
- has adopted
- 12:41
- so now, you know
- 12:42
- if putting layer norms in residual streams is bad
- 12:46
- why does layer norm have to be
- 12:48
- at the start? Of course
- 12:49
- we have pre-norm, which is, you know, before our computation
- 12:52
- but we could have it after our computation as well
- 12:54
- right? That's equally good
- 12:55
- at least under that logic
- 12:58
- and that's exactly right
- 13:00
- many recent models, like Grok or Gemma 2 or Olmo 2
- 13:04
- have this structure where they move the layer
- 13:06
- norm after the computation
- 13:08
- so it's a post-norm of a kind
- 13:09
- but it's outside the residual stream
- 13:12
- other models still actually just put layer norms
- 13:14
- everywhere. They put a layer norm here
- 13:15
- they put a layer norm after
- 13:17
- I'll get to this later as we talk about stability
- 13:20
- but one of the
- 13:21
- other lessons that it seems to have held
- 13:23
- up very well is if you have stability issues
- 13:25
- you can kind of sprinkle in layer norms everywhere
- 13:28
- and that will generally improve stability
- 13:30
- it's almost very strange to be saying this
- 13:32
- because it's so ridiculous
- 13:33
- and yet that statement has actually been proven right
- 13:36
- Every time, you know, people have encountered stability issues
- 13:39
- they say, oh
- 13:39
- but what if we just
- 13:40
- throw a layer norm into attention?
- 13:42
- Turns out that works too
- 13:43
- We'll get to that later
- 13:44
- So, okay, that's post-norm, or double norm in this case
- 13:47
- where you have two layer norms here
- 13:49
- Okay
- 13:50
- The other thing that you can do
- 13:52
- is in the original transformer
- 13:54
- you have the layer norm
- 13:55
- which is this operation right here
- 13:57
- So you have your activations x
- 13:59
- you're going to mean subtract
- 14:00
- divide the variance, and then scale it back up
- 14:02
- right?
- 14:03
- And this works just fine
- 14:04
- right? It's not like this is wrong
- 14:06
- And many models have successfully trained on this scheme
- 14:09
- But basically, most or all modern models
- 14:13
- I think, use RMS norm
- 14:15
- which doesn't subtract the mean or add a bias term
- 14:18
- right? So it's just a scaling down and scaling back up
- 14:21
- right? So you can see this in the
- 14:22
- equation here
- 14:24
- And really, layer norm is more expressive than RMS norm
- 14:27
- so there's really representationally no reason
- 14:30
- why you have to use RMS norm
- 14:33
- But RMS norm is nice because in practice
- 14:36
- there's really no expressiveness loss
- 14:39
- RMS norm models just as well as layer norm
- 14:41
- but more importantly
- 14:43
- it is, you know, faster, right?
- 14:45
- This is the part where kind of the
- 14:46
- systems
- 14:47
- and sort of architecture co-design starts to come in
- 14:50
- Percy mentioned, you know, in the previous lecture
- 14:53
- this idea of arithmetic intensity
- 14:54
- right? We want to keep our GPUs hot by doing
- 14:57
- you know
- 14:57
- matrix multiplies and other very intense computations
- 15:00
- We do not want to be wasting our
- 15:02
- GPUs by having them move
- 15:03
- little tiny bits of memory back and forth
- 15:06
- right? That's a very inefficient use of our
- 15:08
- you know, very powerful GPU
- 15:10
- And so what we want
- 15:12
- is to remove operations that are small and involve memory movement
- 15:16
- but don't give us much expressive power
- 15:18
- right? So by that view
- 15:20
- what we really want to be doing here is
- 15:22
- you know
- 15:22
- if the mean subtraction
- 15:24
- and addition isn't really doing much for us
- 15:26
- just get rid of it
- 15:27
- right?
- 15:29
- you might think, okay, why does this matter?
- 15:31
- We're just optimizing this teeny tiny operation that Accounts for
- 15:36
- you know, in this case
- 15:37
- something like 0.17%
- 15:39
- of the total floating point operations of our system
- 15:43
- But, you know, as Percy mentioned
- 15:46
- it's not really about the flops
- 15:47
- right? The flops are the
- 15:48
- the floating point operations we do
- 15:50
- that sort of multiplying matrices
- 15:52
- but that's not runtime
- 15:53
- right? Runtime is a much more complicated object
- 15:57
- and, you know, statistical normalization, things like layer norms
- 16:01
- even though they're only 0.17% of the flops
- 16:03
- depending on your workload and depending on the setup
- 16:06
- can be up to 25% of the runtime
- 16:08
- right? That's kind of crazy
- 16:09
- On tiny models, this can be really
- 16:10
- really big
- 16:12
- because you're still having to move all these
- 16:13
- parameters back and forth from fast to slow
- 16:16
- memory and vice versa
- 16:17
- when you're doing these operations
- 16:19
- So data movement is really
- 16:20
- really important
- 16:22
- and RMS norm can still matter a lot because of this
- 16:25
- right? So
- 16:27
- You can see kind of the difference here
- 16:29
- the arithmetic intensity is in white
- 16:31
- and then you can kind of see
- 16:33
- the flops involved in black
- 16:34
- And you see that layer norm has a
- 16:36
- very low arithmetic intensity
- 16:38
- which is the operation we try to want
- 16:39
- to remove as much as possible
- 16:41
- Yeah, question over there
- 16:49
- So for something like tensor contraction
- 16:51
- which is in this case matrix multiplies
- 16:54
- the majority of the workload is
- 16:56
- you know, multiplying
- 16:57
- Whereas for stat normalization
- 16:59
- the majority of the workload is memory movement
- 17:01
- And memory movement is quite slow
- 17:03
- So imagine the case where
- 17:04
- like, moving something is, like, almost all the compute
- 17:07
- then you're still paying quite a bit here
- 17:09
- right? Because activations can be quite large
- 17:13
- Yeah
- 17:13
- I think the percent runtime in this case
- 17:14
- is quite extreme. This is
- 17:15
- like, in the, you know, tiny models with, like
- 17:18
- matrices that don't really generally make sense in modern workloads
- 17:20
- but this is, you know
- 17:21
- giving you a sense of why this is
- 17:23
- a free optimization
- 17:26
- and you do see this
- 17:27
- right? This is another paper in which people were
- 17:29
- evaluating different architecture intervention
- 17:31
- the wrong at all in 2020
- 17:33
- I think this was a Google paper
- 17:34
- and they show, you know
- 17:35
- for teeny tiny transform of a 200 million parameters
- 17:38
- you get more steps per second
- 17:39
- that's the third column over here
- 17:40
- when you switch to RMS norm
- 17:42
- and in fact, you actually get better performance
- 17:44
- which I don't think is
- 17:45
- is something that you're guaranteed
- 17:47
- but it's a nice bonus regardless
- 17:48
- right? So you got a free systems win
- 17:51
- by just moving to RMS norm
- 17:52
- And so basically everyone has
- 17:53
- has decided to move over to this now
- 17:57
- And in general
- 17:59
- there's a more general version of this
- 18:01
- There's no guarantee to any of the things I'm saying
- 18:03
- but bias terms
- 18:05
- in transformers and neural networks
- 18:07
- are generally not that useful
- 18:09
- So in the original transformer
- 18:10
- the linear terms all have biases
- 18:13
- but most implementations actually just drop the biases entirely
- 18:17
- right? Once again
- 18:18
- this is another example of something that's not very arithmetically intense
- 18:21
- but fairly memory, intensive, relatively speaking
- 18:25
- And so you might as well just drop these
- 18:27
- right, and get the free systems win
- 18:30
- there's also some cases
- 18:31
- I'll just mention this offhand
- 18:32
- where the bias terms can also induce stability issues
- 18:35
- so they're useful in other ways
- 18:37
- but really I think the primary reason these
- 18:39
- are dropped is just to simplify things
- 18:41
- from a systems perspective
- 18:43
- Cool
- 18:46
- Okay
- 18:47
- So I think layer norms
- 18:48
- the story is pretty easy
- 18:50
- it's easy in the sense that what people
- 18:52
- do is fairly standardized. Our
- 18:54
- our understanding, not at like a deep theoretical level
- 18:57
- but our understanding of like
- 18:58
- what layer norm does is fairly good
- 19:01
- right? Everyone moves the layer norm outside the residual stream
- 19:05
- often pre-norm, but I think this might partially be
- 19:08
- be because Lama 2 did that
- 19:10
- and we
- 19:11
- roughly have a sense of how to use
- 19:12
- layer norm to control things like
- 19:15
- gradient spikes, and keep signal propagation nice
- 19:20
- Related to that, we also now
- 19:22
- you know, basically always use RMS norm
- 19:23
- and you hopefully understand the general principles here
- 19:26
- of basically just dropping bias terms
- 19:29
- and that allows us to
- 19:30
- to keep our system more arithmetically intense
- 19:34
- while keeping the expressive power the same
- 19:37
- I think the unsatisfying thing about a lot
- 19:39
- of architectures is that
- 19:41
- you know, you can't really reason about this beforehand
- 19:43
- right? Like
- 19:43
- we don't know beforehand that dropping the bias terms is okay
- 19:47
- but from a lot of experimentation and now collectively acquired knowledge
- 19:50
- we roughly know
- 19:51
- that dropping the bias terms on both the
- 19:53
- linear and RMS norm is okay for typical language modeling workloads
- 19:57
- right?
- 19:57
- this is the kind of statement that we
- 19:58
- can make on the basis of
- 20:00
- of what we do when we look at
- 20:01
- a variety of different models
- 20:04
- Okay, any questions, for Laird on this topic?
- 20:08
- Good
- 20:09
- Okay, so now I'm gonna talk about activations
- 20:12
- and there's a whole zoo of activations
- 20:14
- There's just a lot
- 20:15
- right? There's relu, gelu, swish, lu, geglu, selu, swiglu, liglu
- 20:21
- and what are these things?
- 20:24
- I think at one point
- 20:25
- one point of my
- 20:26
- you know, more stats ML training
- 20:28
- I thought to myself
- 20:29
- I will never learn these things
- 20:30
- I will make it a point of pride
- 20:31
- to never know what a squigloo is
- 20:33
- but now it's actually very important for us to
- 20:35
- to actually, like
- 20:36
- have a general sense of what these objects are
- 20:39
- and which parts of these names actually kind
- 20:41
- of matter for performance
- 20:43
- right?
- 20:45
- so
- 20:46
- you can
- 20:47
- build and train a language model
- 20:49
- on just a fairly vanilla activation
- 20:53
- even, you know
- 20:53
- I guess Chinchilla is probably the best model
- 20:56
- out of that group
- 20:56
- but even if you just want a ReLU
- 20:58
- you know
- 20:59
- you can train a reasonably performant language model
- 21:01
- using just that activation. There's nothing wrong with that
- 21:04
- right?
- 21:04
- And if we move to Gelu
- 21:05
- which is a Gaussian error unit
- 21:07
- and really the only difference is this tiny
- 21:09
- divot at the bottom here
- 21:11
- which really, you know
- 21:12
- for the most of the activation doesn't change anything
- 21:14
- but changes the gradients right near zero
- 21:17
- then you can train models like GPT-3
- 21:19
- right? That's a perfectly good large language model
- 21:22
- not, you know, modern by modern standards
- 21:24
- but perfectly fine
- 21:26
- but then, you know, we get to the gated linear units
- 21:29
- like Swiglu and Gaglu
- 21:30
- and these are really where most of the
- 21:32
- action is. You know
- 21:33
- this is very similar to LayerNorm
- 21:36
- in that I think almost all
- 21:38
- credible modern language models
- 21:40
- use a gated linear unit of some kind
- 21:44
- Okay, so what is a gated linear unit?
- 21:47
- So these are gated activations
- 21:48
- So if we want to look at something
- 21:50
- like a feed-forward layer
- 21:52
- we can just look at this first part
- 21:53
- This is, you know, a very standard ReLU feed-forward
- 21:56
- right? I have my x
- 21:57
- I hit it with a w1
- 21:59
- you know, I, I entry y threshold at zero
- 22:01
- and then I hit it with another w2
- 22:03
- I get my output
- 22:03
- right? Very straightforward ReLU network
- 22:08
- another thing, I don't say this as my personal experience
- 22:11
- but another thing that is often said in
- 22:13
- architecture design
- 22:14
- is that gating is often very helpful
- 22:16
- So if you apply that very general heuristic
- 22:18
- what you might get is to say
- 22:20
- okay, well, instead of just having
- 22:22
- you know, this entry y's ReLU
- 22:24
- why don't we also have a gate?
- 22:26
- And the second gate
- 22:27
- the second term here
- 22:29
- this is just going to multiply the output of my ReLU
- 22:32
- entry y's, and I have a second matrix v
- 22:35
- okay?
- 22:36
- Now this is just gonna modulate the output
- 22:38
- of my ReLU
- 22:40
- and then I'm gonna do everything else the same
- 22:43
- right? So instead of
- 22:44
- you know, just having x w1 w2
- 22:47
- I have x w1
- 22:48
- and I'm gonna gate that with xv
- 22:50
- this is another, activation the same size as this
- 22:53
- and then I'm going to
- 22:54
- you know, down-project it back with w2
- 22:57
- Okay, so what is this?
- 22:58
- Now this is a reglu
- 23:00
- this is, you
- 23:01
- you make these names by adding the first activation
- 23:03
- in this case relu
- 23:04
- and glu
- 23:05
- right? So the relu gated linear unit
- 23:08
- and gating has been a
- 23:09
- you know, very effective other primitive in architecture design
- 23:13
- and it turns out
- 23:14
- that this is very effective in language modeling as well
- 23:18
- So, if you take something like a glu
- 23:21
- we've already talked about that
- 23:22
- right? That's like the relu
- 23:23
- but with a little divot at the bottom here
- 23:26
- you will get a geglu
- 23:28
- and if you take a squigloo
- 23:29
- which is x times a sigmoid
- 23:31
- then you will get a squigloo
- 23:33
- So this is a swish times
- 23:35
- you know, the rest of it
- 23:37
- and this really covers a lot of the modern models
- 23:40
- right?
- 23:41
- generally the Google folks have used gegloo
- 23:44
- so like the Gemma models
- 23:45
- the T5 models are those
- 23:48
- and everything that's kind of like a Llama
- 23:49
- descendant uses a squigloo
- 23:51
- so
- 23:52
- Palm and the Llama descendants
- 23:54
- are all kind of squigloo models
- 23:57
- I would say that squigloo is probably the more dominant one
- 24:00
- but honestly amongst the gated units
- 24:03
- doesn't really matter
- 24:04
- Now here's a side note that will
- 24:07
- be a semi-important piece of trivia later
- 24:10
- if you look up here
- 24:12
- right
- 24:13
- you will notice that there are more parameters for the gated
- 24:16
- model, right
- 24:17
- because I now have this parameter of v
- 24:19
- And so if you do a little bit of math
- 24:21
- right, I now have three matrices instead of two matrices
- 24:26
- right
- 24:26
- What you should do is you should maybe
- 24:28
- use a
- 24:28
- smaller feedforward dimension
- 24:30
- by a factor of two thirds
- 24:32
- in order to keep the total parameter count the same
- 24:34
- right? So this is roughly the idea of
- 24:37
- well
- 24:37
- I want to keep the same number of
- 24:38
- total parameters as my original MLP
- 24:41
- but I now want to make it gated
- 24:42
- so I'm going to make the feedforward dimension
- 24:44
- which is the output dimension of this W
- 24:47
- a little bit smaller by two thirds
- 24:49
- right? So this is a general rule of
- 24:50
- thumb that people have followed
- 24:52
- but it's not really an iron rule
- 24:56
- you know, the original Noam Shazeer paper that
- 24:58
- you know, proposed this
- 25:01
- had some, you know, very small deltas originally
- 25:05
- but they're consistent deltas
- 25:07
- and I think to his credit
- 25:10
- I think a lot of his papers have
- 25:11
- these like error bar assessments of like training
- 25:13
- multiple replicates and checking to see if they're
- 25:15
- better
- 25:17
- and if you look
- 25:17
- the GLU variants are
- 25:20
- almost always consistently better than the non-GLU variants
- 25:23
- And this is a parameter matched comparison because
- 25:26
- Noam Shazeer is always doing this two thirds
- 25:28
- adjustment
- 25:29
- to make sure that all of the models have the total
- 25:31
- same total number, of parameters
- 25:35
- So this is quite nice
- 25:36
- It's in some ways a free win
- 25:38
- almost everyone uses a GLU
- 25:41
- There are
- 25:41
- there have been other
- 25:43
- sort of more controlled systematic comparisons
- 25:45
- This is, the same paper I was talking about before
- 25:47
- Narang et al. in 2020
- 25:49
- Google actually in the 2020s
- 25:52
- did quite a few nice large-scale architecture comparison papers
- 25:56
- although with a T5 architecture and not
- 25:59
- a autoregressive, language model
- 26:01
- and they, you know, basically comprehensively compare things like GLUs
- 26:04
- and you see once again
- 26:06
- if we look at the Swiglu
- 26:07
- or the Geglu or the GLUs in general
- 26:10
- they do significantly better at loss or the other downstream metrics
- 26:13
- right?
- 26:14
- Fairly compelling on pa- on these papers
- 26:17
- Also clear
- 26:18
- from now a lot of model training runs
- 26:20
- that Swiglu and GLU are good
- 26:23
- right?
- 26:24
- So there's a lot of variations in gating
- 26:26
- but really the important single axis to know is that gating
- 26:31
- for these nonlinearities is actually quite important
- 26:33
- gives you
- 26:34
- a nice boost without much of a computational cost
- 26:39
- you know
- 26:39
- that's not to say that gated linear units
- 26:41
- are necessary. I mean
- 26:42
- GPT-3 was that
- 26:44
- I think the Nemotron 340B model used a squared ReLU
- 26:47
- which is a kind of a crazy choice
- 26:49
- but that works too
- 26:50
- both of these models are perfectly performant
- 26:53
- but it's actually quite rare
- 26:54
- to see anything that's not trained on a gated linear unit
- 26:58
- right? So evidence is pointing towards consistent gains
- 27:00
- on using these gating tricks
- 27:04
- Okay
- 27:04
- So those are, I think, the
- 27:06
- the more consensus choices
- 27:08
- for things that we can do in architecture
- 27:11
- now this one I think is a really fun idea
- 27:13
- but one that I think now the test
- 27:15
- of time has shown
- 27:16
- maybe is not quite as good or maybe
- 27:18
- not as popular of an idea
- 27:20
- normally, we do our transformer blocks serially
- 27:23
- right? We compute our attention
- 27:25
- then we compute the MLP
- 27:27
- right? One after the other
- 27:29
- if you're very systems-minded
- 27:30
- you might say, well, this introduces a bottleneck
- 27:33
- right? I have to wait for the computation
- 27:34
- of one to do the other
- 27:36
- If they were instead in parallel
- 27:38
- I could bring to bear some new and cool systems optimizations
- 27:41
- potentially, right?
- 27:42
- So you might ask
- 27:43
- could we parallelize the transformer block?
- 27:46
- And, this was originally an idea that was in GPT-J
- 27:51
- which is an open source
- 27:52
- attempted replication of GPT-3
- 27:56
- and kind of very interestingly
- 27:58
- I think GPT-J has been surprisingly influential
- 28:02
- in sort of propagating a lot of ideas
- 28:04
- I mean, Palm as well, Google
- 28:07
- is actually surprisingly bold with the architectures that
- 28:09
- they do
- 28:10
- but the description in
- 28:12
- Palm, which you can see in their report
- 28:14
- is kind of the following
- 28:15
- Instead of nesting this
- 28:16
- which is the sequential form at the top
- 28:18
- you know
- 28:18
- you're just gonna add together
- 28:20
- the output of the MLP and the attention layer
- 28:22
- and just add both of those back
- 28:24
- into the residual stream
- 28:27
- if you implement this right
- 28:28
- you can actually share a lot of the components
- 28:30
- Like, you can share the layer norms
- 28:32
- you can fuse the matrix multiplies
- 28:34
- This allows you to potentially get additional systems
- 28:37
- optimizations
- 28:39
- and I think a lot of the people
- 28:41
- that have been influenced by Google
- 28:43
- so Cohere, you know, was founded from one of the former
- 28:46
- transformer authors
- 28:47
- they do a lot of Google-inspired optimizations
- 28:50
- They followed kind of this architecture
- 28:52
- but not very many others
- 28:54
- this has been a
- 28:55
- a approach that has really fallen out of
- 28:56
- popularity over the past
- 28:57
- I think, two years
- 28:59
- I think mainly because optimization of the serial
- 29:01
- form
- 29:02
- has gotten sufficiently good that the systems gains
- 29:05
- from the second one just isn't worth the small hits to
- 29:09
- representation power that you end up getting going from
- 29:12
- parallel to serial
- 29:14
- Effectively
- 29:15
- you can think about it as
- 29:16
- you've lost half of your depth
- 29:17
- right? And that can be
- 29:19
- a, a deleterious
- 29:21
- thing to do to your model
- 29:23
- Okay
- 29:25
- So in terms of the architecture of things
- 29:27
- actually, you know
- 29:28
- the fact that this is so short
- 29:30
- should kind of suggest to you how
- 29:32
- much the original transformer fo- formulation has
- 29:35
- somewhat stood the test of time
- 29:36
- right? Because the only thing I'm really talking about changing here
- 29:40
- is, you know, where the norms go
- 29:42
- or, you know, whether we have bias terms
- 29:44
- or whether we gate the MLPs
- 29:46
- but those are actually pretty minor changes
- 29:48
- compared to all the things that you can do
- 29:51
- Now, those of you that are
- 29:52
- you know, sort of carefully paying attention might say
- 29:54
- but wait, you know
- 29:55
- there's a lot of transformer alternatives that change the attention
- 29:59
- yes, you'll have to wait until next lecture
- 30:01
- because today I'm just only going to cover
- 30:03
- sort of core attention-based methods
- 30:06
- and next lecture I'll throw in a little bit of
- 30:08
- state-space model stuff. But as long as you're in this
- 30:11
- like, dense attention land
- 30:13
- actually the architecture from the original transformer paper
- 30:15
- is pretty close to what we do
- 30:18
- So you see, quite a bit of this
- 30:20
- right? So, just now going back to this
- 30:22
- blue here is RMS norm
- 30:23
- black is layer norm
- 30:25
- You see most of the modern
- 30:26
- models are sort of RMS norm models
- 30:28
- serial versus parallel layers. The blue one's parallel
- 30:31
- the rest is serial. You see mostly serial
- 30:33
- layers
- 30:34
- pre-norm versus post-norm. Some of these
- 30:36
- ones that I marked as post-norm are actually
- 30:38
- pre and post-norm
- 30:40
- and then these ones on the right
- 30:42
- these are GLUs, almost always, with the exception of things like
- 30:46
- Falcon, which is a gated linear unit
- 30:48
- But almost all of these are really
- 30:50
- gated linear units for modern models
- 30:53
- So, you can see the trends quite visually
- 30:56
- from what I'm telling you
- 30:58
- Okay, so really the thing that is very different across implementations
- 31:04
- and I think a place where a lot
- 31:05
- of the architecture stuff is still in flux
- 31:07
- is how you do
- 31:09
- kind of position dependence and incorporate information from other positions
- 31:13
- right? So the core attention component in some
- 31:15
- sense
- 31:17
- so there are lots of different ways
- 31:19
- that you can encode position into a transformer
- 31:22
- And just to, you know, to, to remind you, right
- 31:24
- this is very, very important because attention is positionally independent
- 31:28
- right? They're just inner products
- 31:30
- so you can just shuffle them and attention
- 31:31
- would be the same
- 31:32
- if you don't have a position embedding
- 31:34
- The original transformer had sine and cosine embeddings
- 31:37
- kind of like a Fourier transform intuition that
- 31:39
- if you have sines and cosines
- 31:40
- then you can kind of recover position from
- 31:42
- that no matter what
- 31:44
- a number of other sort of large models that
- 31:47
- you know, followed soon after that used absolute embeddings
- 31:50
- where each position had its own different embedding
- 31:54
- and then
- 31:55
- several other
- 31:56
- sort of Google models like to use relative
- 31:58
- embeddings. And here you're not adding
- 32:01
- embeddings into the, into the embedding, like word vector embeddings
- 32:05
- but instead you're adding a vector to the attention computation itself
- 32:08
- right? So if you're three positions off
- 32:10
- sort of the attention matrix
- 32:12
- gets a different offset added to it
- 32:14
- And, and, you know
- 32:14
- models like T5 and Chinchilla use kind of
- 32:16
- this scheme
- 32:19
- the thing that has really become pretty
- 32:21
- dominant in terms of position embedding
- 32:24
- is this class of embeddings called ROPE
- 32:26
- which some of you may be familiar with
- 32:29
- most models past 2024 use this type of
- 32:31
- embedding. And it's kind of remarkable given that ROPE You know
- 32:34
- in some ways came out of nowhere
- 32:37
- originally, I think this was also a GPT-J innovation
- 32:41
- from, I think, a
- 32:42
- a sort of not very well-known sort of blog post and
- 32:46
- paper combination, from an author in China
- 32:50
- but really, it has some really interesting ideas for
- 32:53
- for why you would do something like rope
- 32:55
- So rope, you know, is a relative position embedding
- 32:59
- and a relative position embedding
- 33:00
- let's
- 33:01
- make an opinionated stance
- 33:03
- that
- 33:04
- I should not care
- 33:05
- about the absolute position of any words
- 33:08
- So if, you know, A, and apple appear together
- 33:11
- even if it appears at the start or at the end
- 33:14
- right, in rope embeddings, they should kind of get the same
- 33:17
- sort of result
- 33:19
- and we do know that
- 33:21
- you know, or
- 33:23
- and we want to sort of represent it in this way
- 33:25
- right? So I have an embedding F
- 33:27
- and I have another embedding F
- 33:28
- and these are going to take in the
- 33:30
- identity of the words X and
- 33:31
- Y and the positions absolute of I and J
- 33:34
- and I want this to be equal
- 33:36
- if I take the inner product of these embeddings
- 33:38
- to be equal to
- 33:40
- a function that only depends on the relative difference
- 33:44
- right?
- 33:45
- and every existing embedding before it didn't really fulfill this equality
- 33:48
- like sine
- 33:49
- is not relative because it has these absolute
- 33:51
- cross terms that are not relative
- 33:53
- Absolute position embeddings, just by the, the name of it
- 33:56
- is obviously not relative
- 33:58
- And then relative embeddings
- 34:00
- technically these are relative
- 34:01
- but they're not kind of embeddings because they're
- 34:03
- just adding to the attention matrix
- 34:05
- right? So there's no inner product structure that
- 34:08
- you know
- 34:08
- you can extract out of this guy
- 34:10
- So given this, you might ask
- 34:13
- is there a nice way
- 34:14
- that we can truly have this relative embedding?
- 34:18
- And the idea is very cool
- 34:19
- it's really just looking at kind of
- 34:22
- properties about angles and cosines
- 34:25
- So, we want our embeddings to be invariant to absolute positions
- 34:29
- and we know that inner products
- 34:30
- of any kind are invariant to arbitrary rotation
- 34:33
- right?
- 34:34
- So, the idea is to say
- 34:36
- I'm going to take my semantic word vectors
- 34:38
- the ones that are independent of any position
- 34:41
- so this is my starting point
- 34:43
- and then I'm going to rotate each of these vectors
- 34:45
- in this case in 2D
- 34:48
- based on the position that the words appear
- 34:51
- So, you know, just as a, simple example, we, let's say we have the
- 34:57
- sentence, we know that, right?
- 34:59
- We appears at position zero
- 35:01
- so I'm not going to touch that at all
- 35:02
- right? I'm just going to keep that where
- 35:03
- it is
- 35:04
- The word know is at position one
- 35:07
- so I'm going to rotate it by some angle
- 35:09
- right? And that's my
- 35:10
- my one position rotation
- 35:12
- Now what happens if I apply the same idea to
- 35:16
- the following sequence? Of course we know
- 35:18
- right? In this case
- 35:20
- we and no are still adjacent
- 35:21
- they're right next to each other
- 35:22
- but their absolute position has shifted
- 35:24
- right? Of course, you know
- 35:25
- comes before we know now
- 35:27
- In this case
- 35:28
- I'm gonna rotate the word we by two positions
- 35:30
- because it's two index
- 35:31
- right?
- 35:32
- Zero, one, two
- 35:34
- So the word we is in the second
- 35:35
- position number two, so I rotate by two
- 35:38
- I rotate no by three positions
- 35:40
- cause it's in position number three
- 35:42
- and what do you know
- 35:42
- the relative angle between these two is still separated by one
- 35:46
- right?
- 35:47
- So this is a very
- 35:48
- very simple idea of just using rotations
- 35:51
- to represent, position. And if we do that
- 35:55
- then anytime we take an inner product
- 35:57
- those inner products are gonna be invariant
- 35:59
- of absolute positions
- 36:01
- Now you might say
- 36:02
- well, in two dimensions that's pretty easy
- 36:04
- because you've only really got one choice
- 36:06
- you've got clockwise and counterclockwise
- 36:08
- but in high dimensions
- 36:09
- there's an infinite
- 36:10
- space of ways that you can rotate vectors
- 36:12
- So what do you do in D dimensions?
- 36:15
- well, you do the simplest possible thing
- 36:16
- and it works. The simplest possible thing
- 36:18
- is to
- 36:19
- reduce it to the 2D case repeatedly
- 36:21
- So you have a D-dimensional vector
- 36:23
- just cut it up into chunks of two
- 36:25
- and each pair of two dimensions gets rotated
- 36:29
- And the theta at which these things rotate vary
- 36:32
- right?
- 36:33
- Some of them are very low frequency
- 36:34
- so they rotate very slowly
- 36:35
- so they, they can capture long-range dependence
- 36:38
- Some of them rotate very quickly
- 36:40
- so they capture things like
- 36:41
- are they neighbors to each other
- 36:42
- right?
- 36:44
- and then at the end
- 36:45
- you know, after I've rotated every pair of vectors
- 36:47
- I get sort of my final embedding
- 36:49
- So this is the
- 36:50
- you know, rope approach
- 36:53
- the paper, if you read it
- 36:55
- has a very complex motivation about complex numbers
- 36:58
- but really, I think the intuitive way
- 37:00
- at least to me
- 37:01
- to think about it
- 37:02
- is to just
- 37:03
- you want to rotate by reducing to the two-dimensional case
- 37:06
- and you're just rotating every pair of coordinates
- 37:10
- Gamma 4 just came out on Thursday
- 37:12
- and they have, like
- 37:13
- another different kind of fun thing that they do
- 37:15
- which they call, I think proportional rope or P rope
- 37:19
- which is a really strange way to just
- 37:21
- say that the only thing they rotate is
- 37:22
- the first two coordinates
- 37:23
- but that's another valid thing
- 37:25
- that you can do as well
- 37:26
- So there's a lot of different things that
- 37:27
- you can do in this space that end
- 37:28
- up working
- 37:30
- Okay
- 37:31
- In practice, what you're gonna end up doing is
- 37:34
- you know
- 37:34
- you can
- 37:35
- take
- 37:35
- your vector and you can make a sparse
- 37:37
- multiply with sines and cosines
- 37:39
- and this is going to be giving you
- 37:41
- some way of rotating your input vectors xs
- 37:44
- right? So x times
- 37:46
- w times r
- 37:47
- this is gonna be your final embedding
- 37:49
- that you get
- 37:50
- Okay
- 37:52
- and finally, you know, this is a sine and cosine
- 37:54
- which looks a little like sine embeddings
- 37:56
- but it's really important that I'm multiplying with
- 37:58
- these sines and cosines rather than using them as embeddings
- 38:01
- cause that means that there are no cross
- 38:03
- terms
- 38:03
- and this is purely relative
- 38:05
- right? There's no absolute position information that you'll
- 38:07
- get out of inner products
- 38:10
- if we really
- 38:10
- really wanted to get into low-level details and you ask
- 38:13
- like, how do I actually implement this thing
- 38:14
- you know, you're gonna have to do that
- 38:17
- you have your usual attention stuff
- 38:19
- and then what you do is you generate
- 38:20
- cosine and sine angles
- 38:22
- based on the position IDs
- 38:24
- of sort of where your sequence is
- 38:26
- and then
- 38:26
- you're gonna apply those cosines and sines
- 38:28
- onto both your queries and keys
- 38:30
- for your sort of attention computation
- 38:33
- and you can either apply them as a matrix multiplier
- 38:35
- or you can go through and apply them manually
- 38:37
- just as a rotation
- 38:38
- right? Fairly straightforward
- 38:39
- and you would do this at the attention level
- 38:41
- rather than at the very bottom
- 38:42
- to sort of enforce position invariance
- 38:45
- every time you're doing attention computations
- 38:47
- Okay, so that was rope
- 38:49
- it is a little bit confusing
- 38:51
- but once you understand the geometry of just rotating things
- 38:53
- it's actually fairly, straightforward
- 38:56
- Okay, I'm gonna pause here for one moment
- 39:00
- in case anyone has questions about the various
- 39:02
- like, architecture bits
- 39:04
- we're gonna then talk about even lower-level details about hyperparameters
- 39:08
- So, yes?
- 39:10
- Do you
- 39:11
- know of any papers that do a high-dimensional
- 39:12
- representation?
- 39:14
- Higher dimensional rotation
- 39:16
- That's a good question
- 39:18
- I don't think so. By higher dimensional rotation
- 39:20
- like any, you know
- 39:21
- 2D rotation
- 39:22
- in the space would just be kind of
- 39:24
- a variant of this
- 39:25
- You could certainly do like
- 39:27
- any
- 39:27
- one manifold that like is a closed loop
- 39:29
- I have not seen that
- 39:31
- Yes. What do
- 39:33
- you think is the best way to do
- 39:35
- this kind of knowledge about papers and technical
- 39:38
- reports?
- 39:40
- It's a good question
- 39:43
- I don't know if there's a way
- 39:45
- beyond some combination of like looking broadly enough to get a
- 39:49
- to get, get a pattern, which is what I
- 39:50
- the procedure I'm trying to do in this
- 39:51
- lecture here
- 39:52
- And then the other one is to try it yourself
- 39:54
- even a much smaller scale
- 39:55
- to form an intuition and like a theory
- 39:57
- for how these things come together
- 39:59
- I think those two are really the right
- 40:00
- ways
- 40:01
- I think reading any single paper in isolation is very
- 40:03
- very difficult, especially now
- 40:05
- because
- 40:06
- no single paper seems to give any
- 40:07
- full detail for a lot of language models these days
- 40:12
- Oh, lots of questions now
- 40:13
- Okay, good. We'll go in
- 40:15
- so how come you
- 40:16
- how come you're on the parallel layers and
- 40:18
- the serial layers? Yeah
- 40:20
- I understand the, the modern models are thinking of
- 40:23
- resource dependency, so they will use the parallel layers
- 40:26
- Yeah, but, yeah, but that's a, that's a different, that's a
- 40:29
- that's a big point between
- 40:30
- you know, accuracy, for these two patterns
- 40:33
- right? I want to know
- 40:34
- like, what's the, what's the difference of
- 40:36
- of, of accuracy? Is it big enough to
- 40:39
- is this small enough to allow the current
- 40:40
- model trainers to ignore that
- 40:41
- or is there any sort of
- 40:44
- Yeah, I think, you know, the
- 40:46
- that's actually really mixed
- 40:48
- So if you read the original Palm paper
- 40:50
- I think they're, like
- 40:50
- very confident about the use of parallel layers
- 40:53
- like no performance drop
- 40:54
- fifteen percent systems utilization improvement
- 40:56
- So if you read just that
- 40:57
- you'll kind of say
- 40:58
- like, oh, it's just as good
- 41:00
- but I think a lot of the
- 41:01
- the later Google models have stopped using this
- 41:04
- which you can take on as an implicit
- 41:05
- signal that actually there might be some
- 41:07
- losses
- 41:08
- And once again, this one is a little bit hard to
- 41:11
- to get precise numbers on
- 41:13
- because no one's done the ablations
- 41:14
- as far as I know
- 41:15
- in parallel versus serial
- 41:16
- controlled nice ablations at least
- 41:18
- Yeah. Yeah, yeah, I
- 41:27
- mean, this
- 41:30
- this difference is really just like which of
- 41:32
- the coordinates you rotate
- 41:33
- Like you don't rotate most of them
- 41:35
- because a lot of the
- 41:36
- I mean, the argument originally, I think
- 41:38
- is that the low frequency parts
- 41:41
- just aren't rotating very much
- 41:42
- And so you can drop them if you're really strapped for
- 41:46
- you know, sort of extra space
- 41:48
- And these, this is really a
- 41:49
- a optimization for teeny tiny models
- 41:51
- where like you don't have very much like hidden dimensions to
- 41:54
- to have activations for
- 41:56
- Yeah
- 41:58
- For the relative embeddings
- 42:00
- not having an inner product
- 42:06
- is that because it only applies to keys
- 42:07
- basically? Like I was trying to understand
- 42:08
- Yeah, so they apply both the
- 42:08
- the keys and values
- 42:09
- which is kind of why
- 42:11
- you know
- 42:11
- you get this like relative
- 42:12
- effect from where you are
- 42:15
- you want to not have cross terms
- 42:17
- right? So
- 42:17
- so if you look at the sine and cosine embeddings
- 42:20
- then you'll not only get sort of the
- 42:22
- you know, the original vectors
- 42:24
- you'll kind of get these weird cross terms
- 42:26
- between the position embeddings and the word embedding themselves
- 42:30
- and so on and so forth
- 42:31
- And then you can kind of back out
- 42:32
- what the absolute position is
- 42:34
- So even sine and cosine embeddings are not
- 42:36
- like pure relative position embeddings
- 42:39
- you know, you have to accept the premise that
- 42:41
- you know, the relative embedding is what you want
- 42:43
- but once you do
- 42:44
- kind of you end up at the rope
- 42:46
- solution somewhat naturally
- 42:52
- Yeah
- 42:54
- Yeah
- 42:56
- So the issue with this is that it
- 42:57
- just can't be factorized as a product
- 42:59
- That's more of an aesthetic problem
- 43:01
- right? Like if, if your constraints are
- 43:03
- I need it to be
- 43:04
- relative
- 43:05
- and I need it to factorize as
- 43:06
- f of xi and f of yj
- 43:09
- then this is not a solution in that
- 43:10
- class
- 43:11
- to be fair, there's a lot of
- 43:13
- embeddings that work this way
- 43:15
- that do work, like alibi and other kinds of
- 43:17
- like, approaches, like do
- 43:18
- do this kind of inject into the attention matrix
- 43:21
- and they do reasonably well
- 43:24
- it's not necessarily the one that's become the dominant approach
- 43:27
- is all I can say
- 43:30
- Cool. Okay
- 43:32
- Great
- 43:35
- Okay
- 43:36
- Now I'll talk about hyperparameters
- 43:38
- and I think hyperparameters are really something that
- 43:39
- you start to engage with once you
- 43:41
- like, actually have to train a model
- 43:43
- right? When your knowledge about language models are abstract
- 43:46
- you don't have to care about any of
- 43:47
- these. But once you have to instantiate it
- 43:49
- you start to ask questions like
- 43:51
- well, how big should the feedforward size be?
- 43:54
- how many heads should I have?
- 43:56
- what should my vocab size be
- 43:58
- right?
- 43:59
- and you might also have questions of like
- 44:01
- what should my weight decay or dropout be?
- 44:03
- Like, do I even need to regularize?
- 44:04
- I have a lot of tokens
- 44:05
- right? So do I need regularization?
- 44:08
- and do I need very deep models or
- 44:10
- very wide models? Like
- 44:11
- what are the, the right kinds of things to do here
- 44:13
- right?
- 44:14
- and all of these
- 44:15
- if you start out with no knowledge
- 44:16
- it's actually very daunting
- 44:17
- because you have to search this
- 44:18
- like, very big high-dimensional space
- 44:21
- but the space of things that people try
- 44:23
- is actually pretty small. And from that
- 44:25
- maybe you can start to think about
- 44:26
- you know, smarter search processes of
- 44:28
- like, where you want to vary things
- 44:31
- one of the things that's a really consensus hyperparameter
- 44:35
- is this idea of the
- 44:37
- Ratio between the feedforward size
- 44:39
- which is kind of the output of your
- 44:41
- first matrix multiply in a MLP
- 44:43
- and the model dimension
- 44:44
- right? So this is really the
- 44:46
- the, ratio of the two dimensions of your W1 and your
- 44:49
- as well your W2 matrix
- 44:52
- this seems like a thing that's very important
- 44:54
- and controls kind of the richness
- 44:56
- of your MLPs
- 44:57
- So what should it be?
- 44:58
- Well, for whatever reason
- 45:00
- it should maybe be four times your hidden dimension
- 45:03
- right?
- 45:05
- and this is a rule of thumb that works remarkably well
- 45:08
- and I will show you some data on
- 45:10
- like
- 45:10
- why maybe this is a fine number to
- 45:12
- choose. There's a few exceptions
- 45:14
- and funnily enough
- 45:15
- the really extreme exceptions kind of backtrack on
- 45:18
- that
- 45:19
- Okay
- 45:21
- exception number one
- 45:22
- is variance of the gated linear unit
- 45:24
- I already told you about this
- 45:26
- so if you were thinking about it
- 45:27
- this is probably cached in your head
- 45:28
- right?
- 45:29
- GLUs have more parameters
- 45:31
- right, if you keep the same dimensions
- 45:33
- So if you want to keep the parameter
- 45:34
- size of your MLPs the same
- 45:36
- well, you need to scale down by two thirds
- 45:38
- right?
- 45:39
- So most GLU variants
- 45:41
- this means that you're going to end up
- 45:42
- with something like 2.67-ish
- 45:45
- right? So everyone that's
- 45:48
- down here, 2.67 to 2.5, this is roughly applying this
- 45:52
- like, two thirds correction
- 45:55
- and then for whatever reason
- 45:58
- the LAMA2 folks decided
- 46:00
- well, we actually have very efficient
- 46:03
- attention heads with, like
- 46:07
- MQA, which I'll talk about later
- 46:10
- and because of that
- 46:11
- We can multiply this ratio by an arbitrary 1.33
- 46:14
- and we'll get roughly 3.5
- 46:16
- And so the Lama people kind of like
- 46:17
- arbitrarily chose a slightly different ratio
- 46:19
- which essentially emphasizes the MLPs a little bit more
- 46:23
- but really, if you actually look through all of the
- 46:25
- the papers, you'll find, you know
- 46:26
- either 2.6-ish or 3.5 for GLUs
- 46:30
- or 4 if you're doing
- 46:32
- non-GLU models
- 46:36
- Okay, there's another exception, which I find to be very funny
- 46:39
- but also very, very cool, which is, you know, throughout, as you read these
- 46:43
- like, technical reports
- 46:45
- you'll find that most people are just very
- 46:47
- boring in their choice of architectures
- 46:49
- They're like, we did Lama, but we changed one thing
- 46:52
- but, you know
- 46:53
- folks at Google are very bold sometimes
- 46:56
- and T5 is one of my favorite ones
- 46:58
- because they have some really bold settings
- 47:01
- they decided that, instead of following this
- 47:03
- like, 4x rule of thumb
- 47:05
- they decided that they want to have a 64x
- 47:08
- multiplier, which is, like, way bigger than 4
- 47:12
- and they have a reasonable argument for this
- 47:13
- as well. This is another
- 47:14
- like, systems-based argument, right? They said, oh, well, you know
- 47:17
- if the bigger my matrix multiplies
- 47:19
- the more efficient I can keep my hardware
- 47:22
- So if I make this
- 47:23
- you know, multiplier really big
- 47:25
- then, you know
- 47:26
- my matrix multiplies can potentially be sort of more efficiently utilized
- 47:31
- right?
- 47:33
- and some others, like Gamma2
- 47:35
- have also tried to really
- 47:36
- push a little bit higher on this
- 47:39
- but really
- 47:39
- T5 is an kind of astounding exception at
- 47:42
- 64. I don't think
- 47:43
- any other model has really gone that high
- 47:45
- in the feedforward multiplier
- 47:49
- And empirically
- 47:51
- if you look at other sort of works
- 47:52
- that try to do more controlled comparisons of this ratio
- 47:56
- I've taken this one from Kaplan in 2020
- 47:58
- this is the classic
- 47:59
- neural scaling loss paper
- 48:02
- where they, they do sort of various controlled
- 48:05
- studies on language models
- 48:06
- You'll see, you know, this wasn't the point of the study
- 48:09
- right? It was a scaling loss study
- 48:10
- but you'll see in one of the panels
- 48:12
- that they actually
- 48:13
- have a
- 48:13
- sort of ablation or a sweep where they
- 48:15
- change the feedforward ratio and they look at the loss
- 48:17
- right?
- 48:18
- for a very small model here
- 48:19
- right?
- 48:21
- But what they
- 48:21
- what they find in this paper
- 48:23
- is
- 48:24
- there's a basin
- 48:25
- where you start at about
- 48:26
- one and you end about maybe ten
- 48:28
- where this hyperparameter is like pretty good and very
- 48:30
- very flat. You lose very little
- 48:32
- relative to the optimal loss down here
- 48:35
- right?
- 48:36
- and then if you get it really wrong
- 48:37
- like you get, you know
- 48:38
- above ten to a hundred or something like that
- 48:40
- you know
- 48:41
- then your loss starts really shooting up quadratically
- 48:44
- and so a lot of these choices that
- 48:45
- range between like 2.6 to 4
- 48:48
- they're all kind of falling into this relatively nice basin
- 48:51
- so you're fine choosing those numbers
- 48:53
- right?
- 48:54
- Okay, so what can we learn about this hyperparameter?
- 48:57
- Well
- 48:57
- the default choices have worked very well for
- 49:00
- nearly all modern language models
- 49:01
- so you can safely choose that
- 49:03
- T5 was a fine model
- 49:05
- or the version 1 T5 was a fine model
- 49:07
- right? Like, it wasn't a bad model
- 49:09
- and so even radical choices can technically work
- 49:12
- but it's probably gonna be compute inefficient
- 49:14
- And I think the funniest part of this saga
- 49:16
- of this kind of the punchline of the
- 49:17
- T5 saga to me
- 49:19
- is that they have a follow-up model
- 49:20
- T5 1. V, V 1.1
- 49:23
- that's like supposed to be the improved version of T5
- 49:25
- and they kind of go back to the standard 2.5 multiplier
- 49:29
- you know? So there's nothing explicitly stated here
- 49:31
- but clearly, you know, when they tried to
- 49:33
- to update T5
- 49:34
- they decided that they wanted to go back
- 49:35
- to a more standard multiplier
- 49:37
- which I find to be a little bit
- 49:38
- funny
- 49:39
- Okay
- 49:40
- So that's the, you know, feed-forward ratio
- 49:43
- which now you have like a rough sense
- 49:45
- of like what the right order of magnitude
- 49:47
- is
- 49:48
- Now let's talk about a different consensus hyperparameter
- 49:51
- I always found this to be very strange
- 49:53
- when sort of teaching
- 49:55
- 224n and, you know, just sort of teaching students about this
- 49:59
- which is if you have a multi-head attention
- 50:01
- where you have multiple heads for your attention in your transformer
- 50:05
- the canonical thing to do
- 50:06
- the thing that almost everyone does
- 50:08
- is if you have multiple heads
- 50:10
- you make sure that the size of those heads
- 50:13
- the head dimension
- 50:14
- is such that
- 50:15
- you sort of have the same dimension as
- 50:17
- a single head transformer
- 50:18
- right? So you always make sure
- 50:20
- that
- 50:21
- you sort of divide the hidden dimension
- 50:24
- to basically multiply with h
- 50:25
- So in this case
- 50:26
- right, you have h, the number of heads
- 50:28
- and the dimension of each head is d over h
- 50:30
- so you multiply the two and you get d
- 50:31
- right? For some reason
- 50:32
- this is kind of the rule of thumb
- 50:34
- of course this doesn't have to be true
- 50:36
- We can arbitrarily change the
- 50:37
- the ratios between head dimensions and model dimensions
- 50:40
- but most models do follow this guideline
- 50:43
- and it turns out to work pretty well
- 50:46
- you know, we can look at a variety of different models
- 50:47
- classic and new, I, you know
- 50:49
- have the latest and greatest quite as well
- 50:51
- and you kind of find
- 50:52
- yeah, the ratios are roughly around one
- 50:54
- a model head
- 50:56
- you know, notable exception of T5
- 50:59
- and even Lambda, which is another Google model
- 51:02
- but really everyone sticks around one
- 51:04
- And I think this is
- 51:05
- kind of an interesting one
- 51:08
- I think the thing about head dimensions
- 51:10
- that I'll, that I'll end with here
- 51:12
- is I think this is yet another kind
- 51:13
- of forgiving hyperparameter
- 51:15
- there's a couple of relations that people have done
- 51:17
- there's
- 51:17
- once again a pretty wide basin around one
- 51:20
- that you can sort of get away with
- 51:22
- Okay, but that one's maybe not the most critical
- 51:25
- hyperparameter
- 51:26
- I think maybe one of the most critical and interesting ones
- 51:29
- I think conceptually, is this idea of an aspect ratio
- 51:32
- right?
- 51:33
- and sort of to add an extra point here
- 51:36
- when you scale models up or down
- 51:38
- the way you usually do that is you
- 51:39
- fix an aspect ratio
- 51:40
- like how wide your model is versus how deep it is
- 51:43
- and then you make the whole model bigger
- 51:44
- right? So the aspect ratio in some sense
- 51:45
- controls the entire depth with trade-off as you make models bigger
- 51:50
- right?
- 51:51
- Now, you might wonder, how deep should my model be?
- 51:54
- Like, if you've been following all this stuff on
- 51:56
- like, reasoning and so on
- 51:57
- you might think
- 51:58
- I need a really deep model or a
- 51:59
- really shallow model if I want systems of
- 52:01
- utilization
- 52:02
- You might think that there's a lot of
- 52:03
- sort of variation. And there is a lot of variation
- 52:06
- much more so than other hyperparameters
- 52:08
- but there's actually, like
- 52:09
- a fairly clear sweet spot that
- 52:12
- most modern models fall into
- 52:14
- you don't really see models go
- 52:16
- like, Too, too deep
- 52:19
- and you also don't see models go too wide
- 52:22
- in either direction
- 52:23
- right? You see most models have a ratio about a hundred
- 52:26
- D model over N layers
- 52:29
- so about hundred sort of width
- 52:31
- for every layer that you have
- 52:33
- I mean
- 52:33
- this is true for like GPT-3 or LAMA
- 52:35
- or any one of these models
- 52:39
- And really
- 52:40
- I think the considerations are partly a trade-off
- 52:42
- between expressiveness and hardware
- 52:45
- right? If you have an extremely
- 52:46
- extremely deep model, they get very
- 52:49
- very annoying to deal with systems-wise
- 52:50
- The deeper your model
- 52:52
- like
- 52:52
- what is the ways that you have for paralyzing them?
- 52:54
- Well
- 52:55
- you might have to cut up your layers
- 52:56
- If you cut up your layers
- 52:58
- we'll talk about this in the systems lecture
- 53:00
- once you start cutting up your layers
- 53:02
- depth-wise, you have very serious issues in paralyzation
- 53:06
- Pipeline parallel, which is what this is called
- 53:08
- is something that, like, most people really
- 53:10
- really do not want to deal with
- 53:12
- Whereas with is much easier to paralyze
- 53:14
- If you have a really wide model
- 53:16
- you know
- 53:16
- you can cut that up
- 53:17
- very easily in your GPUs
- 53:19
- tensor parallel is what it's called
- 53:20
- is much, much simpler to deal with
- 53:24
- and so, in some sense, you know
- 53:25
- there are systems reasons to go wide
- 53:27
- and maybe there's expressiveness reasons to go deep
- 53:29
- and you end up at roughly a hundred
- 53:33
- and I think one of the really interesting things about
- 53:37
- Transformer hyperparameters
- 53:39
- is there a lot of hyperparameters that seem quite important
- 53:42
- but they're also fairly forgiving
- 53:43
- and people have converged roughly on the minimum
- 53:46
- This is yet another plot from Kaplan et al
- 53:48
- which shows another sweep over hyperparameters for
- 53:51
- differently sized models
- 53:53
- and once again, you see
- 53:54
- regardless of kind of the size of your model
- 53:57
- roughly speaking, the optimum aspect ratio is fairly similar
- 54:01
- and they live at about a hundred
- 54:03
- maybe a little bit less
- 54:04
- depending on how you want to do the accounting
- 54:06
- but really, you know
- 54:07
- anywhere near a hundred is a pretty safe
- 54:09
- bet for aspect ratios
- 54:12
- EK and others
- 54:14
- did a number of really interesting sort of like architecture
- 54:19
- architecture variation experiments
- 54:21
- in which their general conclusion on this was that
- 54:25
- let's look at the top panel here
- 54:26
- you have a lot of different kinds of
- 54:29
- models that you can have in terms of
- 54:30
- depth-to-width trade-offs
- 54:32
- but as you sort of sweep the depth-to-width trade-offs
- 54:35
- you find that really
- 54:37
- the only thing that matters in some sense
- 54:38
- is flops. As you increase the flops
- 54:40
- the models get better
- 54:41
- and that's really controlling the majority of the effects
- 54:44
- not necessarily, the aspect ratio
- 54:46
- And so
- 54:47
- I think what has really emerged from this
- 54:49
- is the sense that
- 54:50
- there's a general forgiving band of hyperparameters that
- 54:53
- people tend to choose
- 54:54
- and then you really worry about primarily your
- 54:56
- system's utilization rather than sort of expressiveness concerns
- 55:00
- which are hard to reason about
- 55:03
- Cool
- 55:05
- okay
- 55:06
- And then maybe the last hyperparameter thing
- 55:08
- that I wanna mention is vocabulary sizes
- 55:12
- and this one's kinda interesting to me
- 55:14
- because there's a really clear difference between two
- 55:17
- classes of models
- 55:19
- I think in the early days of a lot of
- 55:21
- you know, early days of open source model training
- 55:27
- there were a lot of monolingual models whose
- 55:29
- only goal was to be good on English
- 55:31
- And for those models
- 55:32
- you had these, like
- 55:32
- much smaller vocab size in the 30,000 range
- 55:37
- and then post-Lama
- 55:38
- a lot of people were really interested in multilingual or
- 55:41
- like, production systems
- 55:42
- so these include closed source models like GBD4
- 55:45
- all these have much
- 55:46
- much larger vocab size
- 55:47
- and these are roughly in the 100 to 200,000
- 55:51
- vocab range
- 55:53
- And you see generally that
- 55:54
- you know, Google models have a ton more vocab
- 55:56
- Lama derivatives roughly range at about 100
- 55:59
- thousand tokens, and then the
- 56:01
- the sort of monolingual models are about 30,000
- 56:05
- this is
- 56:05
- somewhat clear. The multilingual models really do need
- 56:08
- much larger vocab to cover the whole space
- 56:10
- Generally
- 56:11
- the models on the right are also bigger
- 56:12
- There have been scaling law studies showing that
- 56:14
- the bigger your model
- 56:15
- the larger the vocab it can handle
- 56:18
- And so this is also partially driven by
- 56:20
- modern scaling trends
- 56:21
- where the models on the right are generally
- 56:23
- bigger. No one's training small mo- sorry
- 56:25
- large monolingual models, anymore
- 56:30
- Okay
- 56:34
- So, yeah
- 56:43
- Sorry, the question was like
- 56:44
- if you have, sorry, multi-lingual models or, sorry, multimodal
- 56:49
- Yeah
- 56:49
- so
- 56:50
- I guess it depends on
- 56:51
- the way that your tokens are encoded
- 56:53
- but, you know
- 56:53
- if you're tokenizing your images and things like that
- 56:55
- then you need to
- 56:56
- you know
- 56:56
- have many more tokens to account for those
- 56:58
- if you look at like various open source releases
- 57:00
- they'll have like a different image tokenizer with its own vocab
- 57:03
- which is quite large
- 57:05
- Yeah
- 57:16
- how valid is it to compare bits
- 57:17
- oh, that is a great question
- 57:18
- Okay, yeah
- 57:19
- that's not a hyper-primary question
- 57:20
- but that is a good question
- 57:22
- so, what is the right way
- 57:24
- okay, so, so let me, let me, like
- 57:26
- step back a moment and
- 57:27
- like, put us in the right mindset
- 57:29
- So, if we think about language modeling
- 57:32
- language modeling is, is a generative modeling task
- 57:34
- right? We are modeling the probability of a sequence
- 57:36
- Now, as long as your sequence is fixed
- 57:39
- right, it's the same, you haven't, you know, adulterated it anyway
- 57:42
- and you provide a probability over all strings
- 57:44
- that's always valid to compare
- 57:46
- right? At that level of things
- 57:47
- it's always valid
- 57:48
- Now, when you ask the question
- 57:50
- is it valid to compare the bits per byte of
- 57:53
- arbitrary token, or, or two arbitrary tokenizers
- 57:57
- really, there's two things at play
- 57:58
- The one thing is
- 57:59
- you know, did you touch the sequence at all?
- 58:01
- Like, if you look at some tokenizers in the past
- 58:04
- before subword tokenizers, they would drop some tokens
- 58:06
- or drop some words
- 58:08
- That changes, that makes the comparisons invalid
- 58:10
- But modern tokenizers are complete
- 58:12
- they can model any sequence
- 58:13
- so that's not a concern
- 58:15
- the other thing that you have to worry about is
- 58:17
- are we, like, length normalizing in any way
- 58:19
- right? But for bits per byte
- 58:21
- you're always normalizing with the same number
- 58:23
- which is the number of bytes
- 58:24
- and so this is always about comparison
- 58:26
- right? So that's kind of how to think about
- 58:27
- you know, tokenizer comparisons
- 58:40
- complexity and BPD are kind of dual to each other
- 58:42
- so yes, if that's what you're asking
- 58:53
- It's only yes, it's only no, because if you're comparing
- 58:59
- you could, you could train compare complexity with
- 59:02
- if you're changing a different thing
- 59:04
- Okay, we'll have to, we'll have to talk later
- 59:08
- because I'm not sure I understand the question
- 59:09
- But I think that
- 59:10
- that's an interesting set of questions
- 59:11
- Okay, good
- 59:13
- Okay
- 59:14
- All right
- 59:16
- So, you know, we're, we're going through really the
- 59:18
- the, the low level, lowest levels of details of language modeling
- 59:23
- which I think is really
- 59:24
- exposes a lot of interesting ideas while we
- 59:26
- sort of talk through this
- 59:27
- And I think dropout is one of the
- 59:29
- and regularization
- 59:30
- I think is another
- 59:31
- very interesting class of ideas
- 59:34
- Also one that I think is very counterintuitive
- 59:36
- for your machine learning 101 intuition
- 59:39
- So let's, go through what I think is like the
- 59:42
- the standard argument for
- 59:44
- you know, regularization
- 59:46
- well, if I'm doing language modeling
- 59:48
- I have a lot of data
- 59:48
- right? I have more data than I can
- 59:50
- process most of the time
- 59:51
- right? Unless you're at
- 59:53
- you know, Google, maybe even then
- 59:54
- there is more internet data than there is FLOPS
- 59:57
- So, you know
- 59:58
- I'm probably not even going to see the same data twice
- 60:00
- right?
- 60:02
- so I'm only gonna do a single pass on a corpus
- 60:04
- and there's very good
- 60:06
- reasons and arguments to believe that a single
- 60:08
- pass of SGD or other optimizers
- 60:10
- is never really going to memorize my data very much
- 60:13
- right?
- 60:14
- So this means overfitting is not really a problem
- 60:17
- almost ever during compute-constrained language modeling
- 60:21
- Now, you know
- 60:22
- some people even actually
- 60:23
- only look at training loss because they believe
- 60:25
- so strongly that overfitting doesn't happen in single pass SGD
- 60:29
- Now, given this, you know
- 60:31
- you can sort of sit and think about this
- 60:33
- should I use dropout or weight decay in language model training
- 60:37
- right?
- 60:38
- Okay, you can think about it a bit
- 60:42
- you know
- 60:42
- one unfortunate thing is that a lot of
- 60:44
- recent models don't talk about this stuff at
- 60:46
- all
- 60:47
- it's really lower-level details than
- 60:49
- like, tech reports are willing to expose
- 60:53
- but if you look
- 60:54
- Actually, you find a lot of models
- 60:56
- do both
- 60:57
- Especially weight decay, actually, is a fairly popular intervention
- 61:01
- even for modern high-performance language models
- 61:06
- this is very, very surprising, you know?
- 61:08
- I mean, some of the dropout things
- 61:11
- you know, maybe, have gone out of favor
- 61:15
- but weight decay actually remains fairly popular
- 61:17
- and this is very mystifying
- 61:19
- Like, why is this?
- 61:21
- and this is, you know
- 61:22
- one of the reasons why I think deep learning is hard
- 61:24
- and this architecture lecture is very strange and
- 61:26
- hard
- 61:27
- it's because these things interact in very strange ways
- 61:30
- So, there have been papers that have argued
- 61:33
- and shown nice evidence
- 61:35
- that
- 61:36
- weight decay is actually not a regularizer sometimes
- 61:39
- It actually interacts with the optimizer to essentially
- 61:46
- make optimization better
- 61:49
- So if you look at the training versus
- 61:51
- validation loss across different weight decay settings on
- 61:54
- you know, language model training for single pass SGD
- 61:57
- you don't really see any difference
- 61:58
- Like weight decay isn't shifting things
- 62:00
- so the validation loss is better
- 62:01
- There's already no overfitting. We're on the x
- 62:03
- equals y line here
- 62:04
- right?
- 62:05
- So doesn't control overfitting
- 62:07
- but
- 62:08
- if we kind of look at different levels of weight decay
- 62:11
- and not only just different levels of weight decay
- 62:14
- we look at weight decay combined with learning rate decay
- 62:19
- what we find is that the stronger weight decay runs
- 62:22
- these blue dash lines on the bottom
- 62:24
- you know, do significantly better because they start out slow
- 62:28
- but they essentially end up
- 62:31
- converging to a much better minimum later
- 62:33
- And this is, you know
- 62:34
- generally true when we decay learning rate
- 62:37
- not necessarily true when we're in constant learning rate
- 62:39
- which is maybe
- 62:40
- somewhat more of where your intuition is coming from
- 62:44
- So, you know
- 62:44
- this is part of why it's very difficult
- 62:46
- to reason sort of a priori or like from scratch
- 62:50
- you know, the behavior of all these different choices and why
- 62:53
- you know
- 62:53
- I think Percy and I have designed this
- 62:55
- class so that you interact with stuff
- 62:57
- because, you know
- 62:58
- you might
- 62:58
- come upon this thing that where basically
- 63:00
- weight decay is actually an optimization intervention and not necessarily a
- 63:05
- regularization intervention, which is, you know, what you would expect here
- 63:08
- right? So
- 63:08
- always keep that in mind that
- 63:10
- these kinds of unexpected effects can really start to kick in
- 63:14
- for these kinds of
- 63:16
- settings
- 63:18
- Cool
- 63:20
- All right
- 63:21
- So to put everything together for hyperparameters
- 63:24
- there's actually for, you know, a lot of the
- 63:26
- the maybe more hairy looking hyperparameters
- 63:28
- actually just fairly standard choices that have worked well for everybody
- 63:32
- right? You know, factor of four rule of thumb
- 63:35
- keep your head dim and your number of heads
- 63:37
- equal to the model dimension
- 63:39
- take an aspect ratio roughly around 100
- 63:43
- and, you know, if you ask about regularization
- 63:45
- right
- 63:46
- you want to maybe try a couple of
- 63:47
- things because regularization actually does
- 63:49
- interact with optimizers in ways that are quite counterintuitive
- 63:52
- right? So this is a thing that some people
- 63:54
- still do, even though you
- 63:56
- you don't need the regularization at all
- 63:58
- Actually, maybe I'll stop here
- 64:00
- I guess, yeah
- 64:10
- I have not looked into enough
- 64:10
- to be honest
- 64:11
- there aren't that many people training big diffusions
- 64:13
- is one issue
- 64:15
- and many of the models that have been
- 64:16
- trained are retrofitted
- 64:17
- So I think the architectures are actually the same as the
- 64:19
- you know, like a llama-like model
- 64:21
- but if you're asking the question of like
- 64:22
- what's the optimal architecture if you were to train from scratch
- 64:25
- I don't know what that is actually off
- 64:27
- the top of my head
- 64:28
- Yeah
- 64:33
- Well
- 64:33
- I guess it's not that regularization in general
- 64:36
- affects optimization. I don't think people do dropout anymore because
- 64:39
- you know, it doesn't really
- 64:40
- interact well with optimization. But for example
- 64:42
- weight decay, you know, which is shrinkage to zero
- 64:45
- that might allow you to use a
- 64:46
- higher learning rate or it might allow you
- 64:47
- to decay faster
- 64:49
- There are lots of ways in which all
- 64:50
- these terms are interrelated
- 64:53
- Cool. Okay
- 64:56
- Now, I've talked a lot about how to design
- 64:59
- expressive models
- 65:01
- by
- 65:01
- sort of looking at all these other models
- 65:03
- that have been trained
- 65:04
- one of the things that I'll
- 65:05
- I'll highlight now is over the last few years
- 65:09
- a really big emphasis has not been on performance alone
- 65:12
- it has actually been on stability
- 65:14
- And this becomes an increasingly important concern
- 65:17
- as your models get more and more expensive to train
- 65:20
- right?
- 65:21
- we've kind of seen that a lot of
- 65:22
- these choices are forgiving
- 65:23
- right? Everyone's kind of doing similar stuff
- 65:26
- And so, you know, you can mess with these
- 65:27
- but you're not going to get a big performance difference
- 65:29
- that's fine
- 65:31
- But if your model
- 65:32
- you know, suddenly blows up some part into training
- 65:34
- like you get these like horrible looking spikes
- 65:36
- all over the place
- 65:38
- you know, you might end up with a model that is
- 65:40
- you know, actually not very good quality
- 65:42
- right? Or it might be unrecoverable
- 65:44
- You might have spent
- 65:44
- you know, millions of dollars in training
- 65:46
- and, you know
- 65:47
- you get to a point where the model
- 65:48
- is no longer able to be trained any further
- 65:50
- right? That would be a horrible thing to
- 65:51
- happen
- 65:52
- if you have a lot of compute that
- 65:53
- you want to spend
- 65:54
- So
- 65:55
- you don't want to train models that look
- 65:56
- kind of like this blue curve or like
- 65:58
- spikes everywhere and these
- 65:59
- you know, big gradient norms happening
- 66:02
- so what do we do to fix these
- 66:03
- stability issues? I mean
- 66:04
- this is really, I would say
- 66:06
- like a core poor issue
- 66:08
- And, you know
- 66:09
- if you have stability issues in
- 66:11
- language models or in general neural networks
- 66:14
- there's a few, you know
- 66:16
- usual suspects that you've
- 66:17
- got to start looking at
- 66:19
- one of them is the softmaxes
- 66:21
- and the softmax has two things that are
- 66:23
- both really bad for stability
- 66:25
- One of them's an exponential
- 66:26
- right? We can see how that blows up
- 66:27
- very quickly
- 66:29
- you also divide two numbers
- 66:30
- and that's also a potentially very dangerous operation
- 66:33
- right?
- 66:34
- So a softmax is
- 66:35
- one place where you got to be extra
- 66:37
- extra careful
- 66:38
- And where are the softmaxes in a language model?
- 66:40
- Well, there's two of them
- 66:42
- There's
- 66:43
- one on the output side when we output our probability distribution
- 66:46
- and then in attention
- 66:47
- when we normalize the attention
- 66:48
- there's going to be another softmax
- 66:49
- right?
- 66:50
- So
- 66:50
- we can think of both of those as
- 66:52
- really kind of danger zones for our model
- 66:55
- especially the attention
- 66:59
- but okay
- 67:01
- Let's start with thinking about the output softmax
- 67:03
- The output softmax can blow up on us
- 67:07
- and one of the things that we can
- 67:08
- do is we can try to control
- 67:11
- sort of the, the normalizer problem
- 67:13
- So, you know
- 67:14
- let's sort of think about the softmax calculation
- 67:16
- We want to compute a log probability to
- 67:18
- compute the loss
- 67:20
- Now, what is the log probability?
- 67:21
- Well, it's, you know, the output of your model
- 67:23
- U, and then you've got this log normalizer
- 67:26
- right? This U is well behaved
- 67:28
- because in some sense this is the output of your model
- 67:30
- right? This is just the
- 67:31
- output of your residual stream with all the
- 67:33
- things that are added in
- 67:34
- So if U is well behaved
- 67:36
- then log P, the first term, is well behaved, right?
- 67:38
- The model is being okay
- 67:40
- Now, the second term, this log Z
- 67:42
- this might not be so okay
- 67:44
- right? If Z is really big or really small
- 67:47
- even if the output of your model is somewhat well behaved
- 67:49
- it could blow up. And what is Z?
- 67:50
- Well, it's an exponential, right?
- 67:52
- So it could potentially blow up very quickly
- 67:54
- on you
- 67:54
- Or if this is zero
- 67:55
- it could also blow up on you
- 67:56
- right? So both of those directions are very
- 67:58
- very bad
- 67:59
- Now
- 67:59
- we would ideally like our z to be somewhere near one
- 68:03
- right?
- 68:05
- or log z to be somewhere near zero
- 68:08
- what can we do?
- 68:09
- Well, one of the things that you notice
- 68:11
- right
- 68:11
- if you've sort of thought about the action of the softmax
- 68:14
- is this whole thing is over-parameterized
- 68:16
- right?
- 68:17
- I could sort of push things in and out
- 68:19
- so if I add a constant to u
- 68:21
- I can manipulate the z's
- 68:23
- without really affecting the output of the softmax
- 68:25
- right? It can cancel out between the normalizer
- 68:28
- and sort of the output of my model
- 68:30
- So because of that property
- 68:32
- one thing that I could do is I
- 68:33
- could add a regularizer
- 68:34
- this is from, from Jacob Devlin's paper
- 68:36
- 2024, sorry, 2014
- 68:39
- in which he adds sort of this squared log z term
- 68:43
- and what this is doing is it's just
- 68:45
- penalizing how far away your log z
- 68:48
- is from zero. And if log z is near zero
- 68:51
- that's nice because this whole expression
- 68:53
- is kind of numerically stable
- 68:56
- This is called the Z-loss trick
- 68:58
- it's been used by a number of papers
- 69:00
- Jacob Devlin and others
- 69:02
- sort of popular, or initially pioneered this back in 2014
- 69:06
- and then it's become popular again through a
- 69:08
- number of open source models
- 69:09
- Like, Bai Chuan, I think
- 69:10
- was the first open source model to do it
- 69:12
- but then DCLM and Olmo and others have
- 69:14
- been using this trick
- 69:15
- to stabilize their output softmaxes
- 69:18
- So this is, this is a surprisingly effective thing
- 69:21
- Now, okay
- 69:22
- so let's say we've handled the
- 69:24
- instability issues on the output softmax
- 69:27
- Now we have to sort of turn our
- 69:28
- attention towards the other potential problem
- 69:30
- which is attention, right? And this is a
- 69:33
- a place where lots of degeneracies happen
- 69:36
- Lots of techniques have been developed
- 69:38
- to control the instability that attention
- 69:40
- operations generate
- 69:43
- and really the, you know
- 69:44
- the high-level thing that I'll say is
- 69:47
- If you have instability
- 69:49
- if you can throw a layer norm in there somehow
- 69:51
- it might control it
- 69:52
- And that's really, in some sense, the
- 69:54
- the design philosophy behind this idea called the
- 69:57
- QK norm
- 69:59
- so what you do is remember that we have
- 70:01
- you know, our Qs and Ks
- 70:03
- that are going to be multiplied together
- 70:05
- and then they're going to go into the softmax
- 70:07
- right? So in the standard attention operation
- 70:09
- I'm going to layer norm as a pre-layer norm
- 70:11
- multiply with a QKV
- 70:13
- and I'm going to get my Qs and
- 70:14
- Ks
- 70:15
- Those will get multiplied by a matrix multiply
- 70:17
- I'll softmax them
- 70:18
- and I'll multiply that with V to get the weighted average
- 70:21
- and then I'll output whatever comes after
- 70:23
- right? So this is our usual attention
- 70:25
- Now what happens
- 70:26
- if we just throw in a layer norm
- 70:29
- before we multiply the Qs and Ks?
- 70:32
- If we do that
- 70:33
- then we know that the inputs to this matrix multiply
- 70:36
- and therefore the inputs to the softmax
- 70:38
- roughly have the same scale
- 70:40
- They're always going to have a scale of roughly one
- 70:43
- because we've, you know
- 70:44
- used RMS norm to divide the size of
- 70:47
- those Qs and Ks
- 70:49
- Okay
- 70:49
- If we do that
- 70:51
- then, you know
- 70:51
- we're kind of going to keep
- 70:53
- this softmax operation stable. Tons of different models
- 70:56
- do this. It's originally from
- 70:58
- the multimodal world
- 71:01
- you know, some folks who were doing
- 71:02
- making multimodal models sort of initially discovered QK-norm
- 71:06
- Edifix and Chameleon really
- 71:08
- you know, used this and, like
- 71:09
- proved it out. And then a number of other
- 71:12
- open source language models
- 71:13
- you know, realized that the same tricks are entirely applicable to
- 71:18
- stabilizing attention for language models
- 71:20
- And I think this is now very
- 71:21
- very standard. Like
- 71:22
- QK-norm is actually a very standard intervention
- 71:25
- that most of the large models now introduce
- 71:28
- it doesn't seem to affect performance
- 71:30
- from lots of different training runs
- 71:32
- but it does definitely prevent the kinds of
- 71:35
- attention degeneracies
- 71:37
- and, you know, I, I'm really
- 71:39
- The, the way that I've seen this is
- 71:41
- you know, we have layer norms initially in the pre-norm
- 71:44
- now we add them after the
- 71:46
- the, nonlinearities in each block
- 71:48
- and now we're throwing them in both the
- 71:50
- Qs and the Ks. And really
- 71:51
- I think this is
- 71:52
- this is kind of getting at
- 71:54
- the stabilization tricks that people apply to this world
- 71:57
- Okay
- 71:58
- Now
- 71:59
- the final set of things that I'll talk
- 72:01
- about is a stability intervention
- 72:03
- and frankly
- 72:04
- this one is not as popular and more of a
- 72:07
- of a Google-specific trick that I've seen
- 72:11
- but
- 72:12
- logit soft capping is a much harder intervention
- 72:14
- that some people apply. So this one
- 72:17
- you know, in QK norm
- 72:19
- what we're doing
- 72:20
- is we are controlling the inputs to the
- 72:21
- softmax
- 72:22
- and sort of hoping that the outputs are
- 72:24
- well-behaved
- 72:25
- If we really, really want to enforce
- 72:28
- well-behaved outputs
- 72:29
- what we can do
- 72:30
- is we can kind of take the logits
- 72:32
- the things that go straight into the softmax
- 72:34
- and we can just cap them off so
- 72:35
- they can never be too large or too small
- 72:38
- right? This is a hard
- 72:38
- almost a hard constraint
- 72:40
- it's called a soft cap
- 72:41
- of course, but a tanh, you know
- 72:42
- is bounded at some value
- 72:45
- and so, this is in the GEMA models
- 72:48
- I think both GEMA or like GEMAs two
- 72:51
- three
- 72:51
- and four
- 72:52
- all use the logit softmax cap trick
- 72:55
- and what they do
- 72:56
- is they take all of their logits for the pension layers
- 72:59
- and then they soft cap them at some
- 73:01
- value
- 73:03
- some NVIDIA folks have done actually quite nice
- 73:05
- work doing systematic comparisons of these stability interventions
- 73:09
- and what they find is
- 73:11
- if you start with a baseline model
- 73:14
- you can do all sorts of different interventions
- 73:15
- and QK norm is here
- 73:18
- and it does slightly better due to the
- 73:19
- fact that you can crank up the learning
- 73:21
- rate a little bit
- 73:23
- but if you do soft capping alone
- 73:25
- you actually end up losing performance
- 73:27
- So there is a
- 73:27
- a quality degradation that happens
- 73:29
- This is a very strong intervention
- 73:30
- You can never express very confident
- 73:33
- signals in your softmax beyond a certain point
- 73:36
- so it does have some negative consequences
- 73:38
- but this is a very safe way
- 73:39
- of stabilizing the outputs of your attention
- 73:42
- or sorry, the inputs to your attention
- 73:44
- the logits that go into the softmax
- 73:47
- Okay
- 73:48
- so
- 73:48
- that's kind of the end of the stability
- 73:50
- components
- 73:51
- I can pause for a moment here
- 73:53
- and I'll talk about sort of various attention things
- 73:56
- after that
- 74:01
- Cool
- 74:01
- Okay
- 74:03
- All right
- 74:03
- so
- 74:04
- the last thing I want to talk about
- 74:06
- today
- 74:06
- is various interventions that you can make to
- 74:08
- your attention
- 74:10
- and as I was saying at the beginning of this lecture
- 74:13
- I'm only going to talk about all the
- 74:14
- things that you can do
- 74:16
- to sort of dense all-by-all attention today
- 74:18
- So if you
- 74:18
- if you were interested in hearing about state
- 74:20
- space models or linear time attention
- 74:23
- sadly today is not the day for you
- 74:26
- the things that I do want to talk about
- 74:27
- which are really commonly implemented
- 74:30
- attention interventions today are
- 74:33
- group query attention
- 74:34
- which really saves inference costs by reducing the number of heads
- 74:39
- and sparser sliding window attention
- 74:41
- which really originally came from the GPT-3-ish family
- 74:45
- but have now really been adopted widely
- 74:48
- by most models that are looking to do
- 74:51
- long context, unless they're doing exotic
- 74:53
- SSM stuff
- 74:56
- So I'll start with
- 74:57
- group query attention, or GQA or MQA
- 75:01
- this
- 75:01
- I'm gonna
- 75:02
- first set up the need for these kinds of things
- 75:05
- and then you'll kind of hopefully see what the
- 75:07
- what the trick is
- 75:08
- and why it's fairly natural
- 75:10
- So for the moment
- 75:12
- we've been talking about
- 75:13
- you know, training and modeling and all these things
- 75:15
- but like, let's take a pause
- 75:16
- and now let's think about deployment
- 75:18
- right?
- 75:19
- You've trained this very big model
- 75:21
- and now you need to serve it to lots of users
- 75:23
- and you're gonna pay a cost for serving
- 75:26
- And you're gonna have to
- 75:27
- in abstract sense, pay for two different resources
- 75:30
- right? You're gonna have to pay for your flops
- 75:32
- right, the computation that you're performing
- 75:34
- but you also have to pay for another
- 75:36
- thing. You have to pay for your memory accesses
- 75:38
- right?
- 75:39
- Because the memory accesses are also going to impact
- 75:41
- you know, your system's characteristics, your latency, your utilization, right?
- 75:45
- So you want both of these things to be small
- 75:48
- Now, let's think about what happens during training
- 75:50
- or, alternatively, prefill, when you're looking at your prompt
- 75:54
- where someone gives you the stuff
- 75:55
- In this case, you know
- 75:57
- the total arithmetic operations you have is
- 76:00
- you know, order of magnitude
- 76:01
- batch size, sequence length, hidden dim squared, right?
- 76:04
- That's roughly the size of things that you
- 76:06
- get. And of course
- 76:07
- you know, we're doing quadratic attention
- 76:08
- so we've got d squared
- 76:12
- we've got, total memory accesses
- 76:14
- like what is our memory access that we
- 76:16
- have here?
- 76:17
- we have batch times sequence length times
- 76:19
- hidden dim, plus, the sort of cost of the softmax
- 76:23
- which has an n squared component
- 76:24
- and then we've got a d squared component
- 76:27
- for the, for the projections
- 76:28
- So the arithmetic intensity here is pretty good
- 76:32
- it's gonna be one over k
- 76:33
- this is the number of heads
- 76:34
- so you need to have
- 76:35
- in, sorry, head dims
- 76:36
- so your head dims need to be big
- 76:37
- enough that you're multiplying some reasonably sized matrices
- 76:40
- and you've got a one over bn
- 76:42
- so your sequences need to be long enough
- 76:44
- or your batch sizes need to be big
- 76:45
- enough
- 76:46
- As long as both of these are true
- 76:48
- your GPUs are gonna be fully utilized
- 76:50
- Great, right, you're, you're using all of your resources
- 76:54
- Now, you know, we have done, we have finished training
- 76:57
- and now we're serving our users
- 76:58
- How do we serve our users?
- 76:59
- We're gonna generate tokens and send it to them
- 77:01
- right?
- 77:02
- Now, if we're doing that
- 77:03
- I can't paralyze the generation process
- 77:06
- What I'm going to do is I'm going
- 77:07
- to generate a token
- 77:08
- I'm going to condition on it
- 77:09
- I'm going to generate the next token
- 77:10
- and I'm going to repeat this process one
- 77:11
- by one. This is just sort of
- 77:13
- the curse of auto-regressive language modeling
- 77:15
- we have to do this
- 77:17
- In order to do this
- 77:18
- the efficient way to do it is to
- 77:20
- maintain
- 77:21
- all of the sort of past keys and
- 77:23
- queries that I've had
- 77:25
- in what's called a KV cache
- 77:26
- So the KV cache is going to maintain this matrix of
- 77:30
- you know, Q dot K over the past
- 77:32
- and then whenever I need to compute something new
- 77:34
- I can reuse sort of
- 77:36
- the sub-matrices
- 77:37
- that I've already had from the past
- 77:39
- And I only really need to compute
- 77:41
- sort of the new
- 77:43
- query key
- 77:44
- interactions that I need to fill out
- 77:46
- the rest of this matrix
- 77:48
- right? So every sub-matrix I've computed before
- 77:51
- I can keep
- 77:52
- I only need to compute my new one
- 77:53
- So this saves a lot on compute
- 77:55
- right? That's great
- 77:57
- But
- 77:58
- The issue here
- 77:59
- is now our arithmetic intensity is not so good
- 78:02
- right?
- 78:03
- As you might sort of intuit
- 78:04
- this KB cache approach is going to be reading and
- 78:08
- and, reading, parameters all the time
- 78:10
- right? Each time I have a new step
- 78:12
- right, I'm gonna have to read in my parameters
- 78:14
- I'm gonna have to take these dot products
- 78:16
- and I'm gonna do this once every step
- 78:19
- And so now what do I have?
- 78:20
- Well, you know, my total memory ac
- 78:22
- oh, sorry, my total arithmetic operations are the same
- 78:25
- I'm multiplying the same matrices still
- 78:26
- right, just incrementally rather than all at once
- 78:29
- But because I'm doing this incrementally
- 78:32
- you know, now I have
- 78:34
- a memory access pattern of batch
- 78:36
- by sequence squared by hidden dim plus
- 78:40
- sequence by hidden dim squared
- 78:41
- And the second term is not so pleasant
- 78:44
- right? It used to be that it was just d squared
- 78:48
- but now we've got n times d squared
- 78:51
- And if we compute the arithmetic intensity
- 78:53
- which is the ratio of these two guys
- 78:56
- now we have
- 78:57
- n over d plus one over b
- 78:59
- So now what we need
- 79:00
- is large batches plus short sequence length
- 79:03
- or we need really big model dimensions
- 79:05
- So if we want to serve a small model efficiently
- 79:08
- this is not so good
- 79:11
- right?
- 79:12
- this is really difficult to deal with
- 79:13
- right? This n over d term
- 79:15
- this first term over here
- 79:16
- which is sequence length over hidden dim
- 79:19
- is very difficult to reduce if we're doing
- 79:21
- this incremental computation
- 79:23
- this is just a hard thing to deal
- 79:24
- with
- 79:27
- So this leads to this idea of MQA
- 79:29
- or multi-query attention
- 79:31
- Normally you have multiple heads in your attention
- 79:34
- operation, and you're gonna have different keys
- 79:36
- different values, and different queries
- 79:38
- That's normally how you do things
- 79:40
- But one thing that we could do
- 79:42
- is maybe we can keep the Ks and
- 79:44
- the Vs the same across all the heads
- 79:47
- and the only thing that's different across the
- 79:48
- heads are the queries
- 79:50
- If we do this
- 79:51
- then this drastically removes the amount of
- 79:55
- items that need to be moved in and out of memory
- 79:57
- right? Because the KV cache
- 79:59
- right, are now significantly smaller
- 80:01
- These are all shared across all the heads
- 80:04
- this
- 80:05
- significantly reduces the total memory access as well
- 80:08
- as the arithmetic intensity
- 80:09
- and we're kind of
- 80:10
- the key term that we were talking about here
- 80:12
- we had the N over D term
- 80:14
- now we have, H multiplying this, right?
- 80:18
- So this H term
- 80:19
- allows us to significantly reduce the
- 80:23
- the, sorry
- 80:24
- increase the arithmetic intensity if we have a lot of heads
- 80:26
- right? This is a significant gain over what we had
- 80:29
- before
- 80:31
- So this gets us significant efficiency improvements
- 80:34
- but the issue with MQA is
- 80:38
- This is on the right here
- 80:39
- You have one value and one key
- 80:41
- for all these queries. You do
- 80:43
- in fact
- 80:44
- lose significant expressive power if you do this
- 80:47
- and so there's this trade-off between systems efficiency and expressiveness
- 80:51
- and you might wonder
- 80:52
- is there sort of a sweet spot
- 80:54
- in which we can avoid trading off
- 80:57
- you know, quite a significantly
- 81:00
- expressive power and computation? And that's where GQA
- 81:03
- or grouped query attention
- 81:04
- comes in
- 81:05
- You know, the original transformer is multi-head
- 81:07
- We have queries and keys for each head
- 81:09
- In multi-query, we have one key and value for each
- 81:13
- for all the heads. In grouped query
- 81:15
- we reduce the amount of keys and values
- 81:17
- but we keep the number of queries the
- 81:19
- same. So we now have this ratio that
- 81:20
- we can play with
- 81:21
- which is kind of the number of key heads
- 81:23
- or the number of value heads
- 81:25
- while keeping the total number of heads much
- 81:27
- larger than that. So this allows us to
- 81:29
- very simply control the
- 81:30
- the, trade-offs between expressiveness and inference efficiency
- 81:35
- there are other sort of tricks from DeepSeek v2
- 81:37
- multi-head latent attention, that I'll sort of mention briefly
- 81:41
- next time
- 81:43
- which sort of have a different kind of
- 81:44
- factorization structure and a different set of trade-offs
- 81:47
- But really, the nice thing about GQA is that in practice
- 81:52
- the trade-off is quite favorable
- 81:53
- So if you have multi-head
- 81:55
- your performance, this was
- 81:56
- you know, in the, I think T5 days
- 81:59
- if I remember right
- 82:00
- this is your, downstream model performance
- 82:03
- This is your time per sample
- 82:04
- You want to reduce this as much as
- 82:05
- possible
- 82:06
- With multi-head attention, you have best performance
- 82:08
- but very high cost
- 82:10
- with MQA, you know, you have
- 82:13
- lower, cost, but much lower performance
- 82:16
- Similarly
- 82:17
- if you make your model smaller to try
- 82:19
- to hit your performance targets
- 82:20
- you get much worse performance
- 82:22
- GQA really does get the best of both worlds
- 82:24
- you know, very low inference cost
- 82:27
- nearly the same performance as your full multi-head
- 82:31
- and you see sort of this like
- 82:33
- GQA group structure where if you have
- 82:37
- a small reduction in the number of heads
- 82:39
- you basically have most of the gains in your performance
- 82:43
- which allows you to sort of keep most
- 82:44
- of the expressive power
- 82:46
- while getting significant, inference improvements
- 82:48
- And first
- 82:49
- you'll talk a bunch more about sort of
- 82:50
- the inference mechanics later
- 82:53
- but sort of this should give you a
- 82:54
- flavor of like why models today almost all adopt this GQA
- 82:58
- structure, because it gives you a lot of this inference cost
- 83:01
- which is really critical
- 83:02
- without very much of a
- 83:05
- expressiveness hit
- 83:08
- Cool
- 83:09
- Any questions for, for GQA, or KDKAT?
- 83:11
- Yeah. But, didn't
- 83:14
- they have so many
- 83:18
- Bruce Dunn's for what hyperframes are good?
- 83:20
- Like
- 83:20
- do you
- 83:21
- think they still switching over hyperframes
- 83:23
- versus exploiting these?
- 83:28
- I think it's a mix of both
- 83:29
- I think every
- 83:30
- sort of model training run has some theses
- 83:32
- about what can be varied
- 83:34
- and so you see this in a lot of the reports
- 83:36
- where I think the hyperparameters are often not
- 83:38
- where people are touching too much
- 83:40
- but you see, like, architecture changes, like, one at a time
- 83:43
- in a lot of these reports
- 83:44
- But it's very rare to
- 83:45
- like, go and change everything up
- 83:47
- I think Google is one of the only
- 83:49
- orgs that seems to
- 83:50
- like
- 83:50
- really spice things up in a significant way
- 83:53
- the Gamma series
- 83:54
- has done some pretty interesting things
- 83:57
- the most recent Gamma 4 release
- 83:58
- for example, now has, like, individual embedding for every layer
- 84:02
- in a way to control the trade-offs between
- 84:04
- like, memory use and flops
- 84:05
- It's a very interesting set of things that they've done
- 84:09
- Oh, yeah, back there. Do you think frameworks will
- 84:11
- like, dynamically alter the training
- 84:12
- parameters during training? During training?
- 84:18
- let me think
- 84:20
- Weight decay
- 84:21
- yes. Weight decay people change
- 84:24
- in concert with like learning rate
- 84:26
- that is actually a heuristic that people do
- 84:28
- that works very well
- 84:31
- other than that
- 84:32
- I don't know if there's a lot of
- 84:34
- different hypers that people change during training
- 84:37
- especially cause the architecture ones
- 84:38
- just make training incompatible
- 84:40
- So you can't really
- 84:41
- you know, change them while you're
- 84:42
- you're training
- 84:44
- yeah. So I think
- 84:44
- I think weight decay is probably the one
- 84:46
- that I can think of
- 84:46
- The others are usually fixed
- 84:49
- Yeah
- 84:50
- Thank you. It's
- 84:53
- it's not just inference time fixed
- 84:55
- It's applied during training. That's right
- 84:57
- yeah. You, you
- 84:57
- you train with a certain number of keys
- 85:00
- Yeah
- 85:02
- Okay, the last thing I'll talk about is
- 85:05
- sliding window attention, which is a really old idea
- 85:08
- Like, you know, GPT-3 used actually this
- 85:11
- you know, if you read the paper
- 85:12
- they'll say we alternate between full attention
- 85:15
- which where every position can attend to everyone in the past
- 85:18
- in a banded matrix
- 85:20
- style attention where you can
- 85:21
- attend to everyone within a fixed window
- 85:24
- and, you know
- 85:25
- OpenAI has some early work on these kinds of
- 85:27
- like
- 85:27
- different kinds of attention patterns
- 85:30
- that you can use
- 85:31
- but actually this has become really
- 85:32
- really popular over the past year
- 85:36
- this idea of alternating
- 85:38
- you know
- 85:38
- the big full attention
- 85:40
- and a
- 85:41
- more local attention actually hits a sweet spot
- 85:44
- for how to manage
- 85:45
- like
- 85:45
- long context performance while not paying
- 85:48
- too much for inference
- 85:51
- I think, you know, the more recent revival in open models
- 85:55
- I would maybe say
- 85:57
- Cohere Command A was the first one I saw do it
- 85:59
- where they, you know, had this, like
- 86:01
- structure where
- 86:03
- every
- 86:03
- four layers they would have a full attention
- 86:05
- that attended to everything
- 86:07
- The three layers in between would use a
- 86:08
- sliding window attention
- 86:10
- that would only be able to look at
- 86:11
- local structure. And of course
- 86:12
- you know, as you go up
- 86:14
- sorry, in this case down
- 86:15
- because they ordered the
- 86:16
- the diagram the other way
- 86:18
- as you go down sort of these blocks
- 86:20
- you know
- 86:20
- you're aggregating local information into global ones
- 86:23
- So the local attentions at the end can
- 86:24
- of course, access more global information
- 86:27
- but this, you know, allows you to manage the
- 86:30
- the
- 86:30
- the cost
- 86:32
- of having a really long context
- 86:33
- without having to go for something like
- 86:36
- a state space model or a more exotic intervention
- 86:38
- and that's worked quite well
- 86:41
- there's also some innovation where
- 86:43
- people change the, embedding format for the long-range information
- 86:47
- where they get rid of things like
- 86:49
- rope
- 86:49
- so you have no position embeddings at all
- 86:51
- So you're really looking almost at bags
- 86:54
- where the short-range information still gets position information
- 86:57
- so people do all sorts of
- 86:59
- you know, kinds of interventions involving these
- 87:02
- these
- 87:03
- both the embeddings
- 87:04
- and alternating local and global structure
- 87:07
- I'll say that this is a
- 87:09
- you know, attention in general, how to manage the trade-off between
- 87:12
- long context and sort of
- 87:14
- long context cost and performance is
- 87:17
- still an active area of investigation
- 87:18
- It's a place where
- 87:19
- the most architecture work and changes are still
- 87:22
- being done
- 87:23
- we see essentially a
- 87:25
- a bunch of other models adopt this idea
- 87:27
- Llama4, most recently Gamma4, OMO3
- 87:30
- They all do
- 87:31
- this combination of sliding window attention and full attention
- 87:34
- in their case using full rope instead of
- 87:36
- nope as the, embedding
- 87:39
- So as I said
- 87:41
- this is becoming really
- 87:41
- really popular
- 87:43
- Qwen 3.5, which I put on the right
- 87:45
- they're actually a little bit different because they
- 87:47
- alternate a state space model
- 87:50
- called, gated delta net and a full attention
- 87:53
- every sort of, you know
- 87:55
- one or one full attention every four layer
- 87:57
- every four layers
- 87:59
- so it's the same alternating structure
- 88:01
- but they're using a different
- 88:02
- sort of cheap layer. In their case
- 88:03
- they're using a state space model
- 88:05
- I'll explain what that is
- 88:06
- next lecture
- 88:07
- instead of a sliding window sort of local
- 88:09
- attention
- 88:10
- But you see this is like
- 88:11
- I think, a new theme over the past year where
- 88:14
- you know
- 88:14
- open models are really trying to grapple with long context performance
- 88:17
- and the way to do that
- 88:19
- at least so far
- 88:20
- is to have these hybrid models that aren't just global attention
- 88:23
- aren't just cheap attention
- 88:24
- there's some sort of mix in between
- 88:26
- and that's
- 88:26
- that seems to have worked very well so
- 88:28
- far in a lot of these models
- 88:32
- Okay, cool
- 88:33
- So as I was sort of trying to emphasize
- 88:35
- when you look across all of these models
- 88:37
- you start to see a lot of patterns
- 88:39
- and hopefully a sense of
- 88:40
- general understanding about what
- 88:42
- things you can do and what things are
- 88:44
- good defaults
- 88:45
- we also see a lot of differences in
- 88:47
- how we handle context and how we handle
- 88:49
- position embeddings
- 88:51
- even tokenization, there's some differences, right?
- 88:53
- So there are differences across these models
- 88:55
- but there's also commonalities that hopefully now give
- 88:57
- you some intuition
- 88:58
- as you go out and do your assignments and
- 88:59
- and sort of
- 89:00
- mess with the leaderboard and so on
- 89:02
- thanks
Watch these videos on your TV
XTTV brings video content from X to your living room. Download now and start watching on the big screen.