Instead of spending 2 hours watching Netflix tonight, invest that time in this Stanford lecture. It

Rahul Kumar
@RahulTechX
Instead of spending 2 hours watching Netflix tonight, invest that time in this Stanford lecture. It's one of the clearest end-to-end breakdowns I've seen of how ChatGPT and Claude are built. It covers everything from tokenization and BPE to transformer architecture, the training pipeline, and next-token prediction. Whether you're completely new to AI or already building AI agents every day, this 2-hour-and-34-minute lecture will help you understand what's happening behind the scenes. Bookmark this, set aside some time this weekend, and give it your full attention. Because understanding how AI actually works could be one of the most valuable skills you develop this year.
Transcript
- So large language models
- 0:02
- I guess you always
- 0:02
- probably all of you have heard of this word
- 0:06
- large language models. This is
- 0:09
- precisely, I think in some sense
- 0:10
- this is all starting from GPT-3
- 0:12
- when OpenAI released GPT-3
- 0:15
- it has a, it used this new paradigm which
- 0:18
- used autoregressive large language models
- 0:22
- so, so basically we're going to focus on that
- 0:24
- you know, autoregressive large language models
- 0:26
- We're going to skip some of the early versions of Transformer
- 0:28
- which are, not autoregressive
- 0:32
- so
- 0:34
- and
- 0:35
- and another regressive basically means that you generate
- 0:37
- tokens one by one
- 0:38
- and each of the new tokens depends on the previous tokens
- 0:40
- and you do this
- 0:42
- and you model probability in this kind of like
- 0:44
- autoregressive way
- 0:45
- I'm going to talk more about details
- 0:47
- In some sense, you model your language distribution by using a
- 0:51
- a sequentially, sequential model one by one in some sense
- 0:55
- anyway, so, let's, I think your point
- 0:59
- our goal is to talk about some details
- 1:01
- so I'll
- 1:01
- I'll fill in more high-level stuff as we
- 1:03
- talk about the details
- 1:04
- So
- 1:05
- the first thing we're going to talk about
- 1:06
- is tokenization
- 1:10
- This is, you know, just, to prepare us for
- 1:13
- to talk about the real meat in some
- 1:15
- sense
- 1:15
- So what does tokenization mean?
- 1:17
- It means that you need to You know
- 1:19
- these transformers, they are numerical, you know, architecture models, right?
- 1:23
- So they take in
- 1:24
- in some sense, numerical numbers, right?
- 1:26
- So, but, you have tax
- 1:28
- and how do you turn tax into some
- 1:30
- kind of things that
- 1:32
- the models can understand?
- 1:33
- So, the first step is to tokenize
- 1:36
- Tokenize means that you need to decide
- 1:38
- you know, what is the smallest unit
- 1:40
- of the input that you give to the
- 1:42
- large language model
- 1:43
- And often it's not a
- 1:44
- a, a word
- 1:46
- right?
- 1:46
- The most natural thing you can think of
- 1:47
- is
- 1:48
- maybe a smallest unit is a character or
- 1:49
- a word
- 1:50
- If you use character level tokenization
- 1:53
- then the issue is that often you have so many characters
- 1:55
- right? You, you need so many tokens
- 1:57
- so many steps
- 1:59
- and, if you use word-level
- 2:01
- tokenization, then often the issue is that you don't
- 2:04
- how do you deal with these long words
- 2:06
- which are very rare? For example
- 2:07
- you have a word called
- 2:08
- say, interna-intern, nationalization, right? So, yeah
- 2:12
- if you treat this word as a single word
- 2:14
- then maybe it's too rare
- 2:16
- and you don't leverage your understanding about other words
- 2:19
- right? So if you treat internationalization as one word
- 2:21
- then it is different from internationalized
- 2:24
- It, it's different from, like, a na-nation, right?
- 2:27
- And, and they are all completely different kind of things
- 2:30
- and you cannot leverage
- 2:32
- that, you know, for example, if you understand interna-nationalized
- 2:35
- then you should probably understand internationalization to some
- 2:37
- degree without even seeing more data
- 2:40
- And, so, so that's why people don't always use
- 2:44
- subword tokenization, or, or don't use word tokenization
- 2:47
- Maybe another, okay, maybe let me give you another example
- 2:49
- right? For example, if you have used this
- 2:51
- I guess in my notes
- 2:52
- there is this word LM
- 2:54
- magnification
- 2:55
- Like, actually, I don't know whether this is a real word
- 2:57
- you know, like
- 2:57
- I think it's a new word that is
- 2:59
- complete new word. Nobody have used it before 2020
- 3:03
- 2020, I guess, right? So, even, even right now
- 3:05
- it's not that kind of like common
- 3:08
- And if you have to treat this word as a unit
- 3:12
- then you
- 3:13
- you have to see this word enough times
- 3:15
- to understand what it means
- 3:17
- right? However, if you treat it as subwords
- 3:19
- then maybe you have a
- 3:20
- a token called ILM. You have a token
- 3:22
- which is, you have a two unit
- 3:24
- you know, this is a token
- 3:25
- this is a token
- 3:26
- So ILM is a token
- 3:27
- invocation is a token
- 3:28
- Then in some sense
- 3:30
- you have seen ILMs many
- 3:31
- many times, you ought to understand it
- 3:33
- And then invocation has a special meaning in English
- 3:37
- and then you combine these two understanding
- 3:39
- you, you understand the word
- 3:41
- So that's much more efficient
- 3:42
- understanding of the word
- 3:44
- than just, treating this whole thing as a single unit
- 3:48
- So, and then, and this happens more often in
- 3:52
- for example, biology, when you have this very long word
- 3:54
- or maybe in German you have very long word
- 3:56
- but actually there are fundamentally many combinations of many subwords
- 4:00
- And, and
- 4:02
- and using subword tokenization allows you to understand
- 4:04
- each of the subwords
- 4:05
- and, and then you can understand the combination of the
- 4:07
- those subwords, as a, as a, as a whole, very efficiently
- 4:12
- So, yeah, so basically tokenization, you know
- 4:14
- just means that you have a predefined vocabulary
- 4:18
- vocabulary, and this vocabulary is a list of like
- 4:22
- subwords, right? So for example
- 4:23
- you can have, you know, many, many of this, like maybe A is a subword
- 4:26
- you know, is a one of the token
- 4:28
- you know
- 4:29
- I think maybe like a
- 4:30
- dog is a token
- 4:32
- Pretty much
- 4:33
- all common words these days is a token
- 4:35
- And actually, if you look at the
- 4:36
- the, the
- 4:37
- actually you can play with the tokenization online
- 4:39
- if you search OpenAI tokenizer
- 4:41
- OpenAI give you a
- 4:42
- playground where you can see how the texts are tokenized
- 4:46
- And, and you'll
- 4:47
- you'll find that actually blank dog is also a token
- 4:51
- So
- 4:52
- so the blank is actually merged with dog
- 4:54
- as a token
- 4:55
- And, you know, and you have so many of this
- 4:57
- I, typically this kind of like a tokenization right now
- 5:01
- I think for the open source models
- 5:02
- we know the number of tokens
- 5:03
- they have, I think it's, you know
- 5:05
- often on the order of like ten to the five
- 5:07
- like a hundred K at least
- 5:09
- I think QN, QN 3.5
- 5:12
- this is the newest version of the open source models
- 5:13
- which are pretty good
- 5:14
- I think they have like two hundred fifty
- 5:16
- tokens
- 5:18
- I think two hundred fifty probably is mostly because
- 5:20
- you know, I think two hundred fifty is
- 5:21
- is somehow you can represent this token with like a
- 5:24
- Maybe it's eight, eight bytes in some sense
- 5:27
- like, like, it, it
- 5:28
- it also depends on how many bits
- 5:30
- you can present the tokens
- 5:32
- so, anyway, so you have this vocabulary
- 5:34
- and then you just
- 5:36
- given a long
- 5:37
- string of text, you just
- 5:40
- take all the subwords that are tokens
- 5:41
- and you turn it into a sequence of
- 5:43
- like
- 5:44
- segments, right? For example, if you say, if I have a
- 5:47
- a dog runs, suppose this is the text you see
- 5:49
- and then you'll, you'll say, this is a token
- 5:52
- so this is one part
- 5:53
- and then I think blank dog is a token
- 5:57
- and blank runs is also a token
- 6:00
- and, and comma, I think, is a token by itself
- 6:05
- So, and every token has an ID
- 6:06
- right? So let's say A is probably the ID number one
- 6:09
- dog is maybe the ID number like 356
- 6:12
- whatever, and, and run is, some ID
- 6:15
- maybe, I don't know, like, maybe 250
- 6:18
- I'm just writing some random numbers
- 6:19
- Every, everything has an ID
- 6:21
- so basically you turn every tax into a sequence of IDs
- 6:24
- you know, each ID corresponds to one token
- 6:27
- and, and I think, so in this case
- 6:30
- you know
- 6:30
- most of these words are actually one token
- 6:32
- I think, I just checked recently
- 6:34
- if you have the word happiness
- 6:37
- I think the tokenization will be unhappiness
- 6:40
- let's say
- 6:41
- I think this part is a token
- 6:43
- H is a token
- 6:44
- and A P I N E S S is a token
- 6:48
- for some reason
- 6:50
- and maybe another example is this
- 6:52
- oh, I, I already read it
- 6:53
- that I am identification
- 6:55
- and this is another example
- 6:58
- I come, we come up with
- 7:00
- you know, just, I don't know actually what these words mean
- 7:02
- if you have a blast
- 7:04
- Oh, Ma, I think this part is a token
- 7:05
- this part is a token
- 7:06
- and this part is token
- 7:07
- I think this is just because for this biological
- 7:09
- biology words, you know, biological words, you know
- 7:12
- actually every part of the word has some special meaning
- 7:15
- and, so you tokenize them into
- 7:18
- tokens
- 7:19
- And, and what is the exact vocabulary?
- 7:21
- It depends on the model
- 7:23
- so basically the, the algorithm that turns the text into these
- 7:27
- tokens is called a tokenizer
- 7:29
- and, and every model has a different tokenizer
- 7:32
- Not, not everyone has a different one
- 7:34
- Some, some of them shared the same tokenizers
- 7:36
- but generally it's specific to the tokenizer
- 7:38
- to, to the model
- 7:40
- and different companies have different
- 7:42
- proprietary tokenization approach
- 7:45
- you probably heard of that
- 7:46
- recently, I think Cloud Code
- 7:48
- Cloud, they update their tokenizers so that
- 7:51
- I think they make the tokenizers more granular
- 7:53
- to some degree
- 7:54
- so that
- 7:55
- the same amount of tax will be
- 7:56
- tokenized into more tokens
- 7:58
- So before it was like one thousand tokens
- 8:00
- now it's one point five thousand tokens
- 8:02
- you know, I didn't verify it myself
- 8:03
- but I read some news about this
- 8:05
- Assuming they are true
- 8:06
- that means that you
- 8:07
- you spend more tokens
- 8:08
- you know, you, you pay more to Cloud Code
- 8:12
- with the same amount of tax
- 8:14
- anyway, so, cool
- 8:17
- And there are some details about how to
- 8:20
- come up with the vocabulary and how to kind of like
- 8:24
- run this algorithm to tokenize
- 8:26
- the, the key word is called VPE
- 8:30
- VPE
- 8:31
- tokenizer, byte, pair, encoding
- 8:35
- So I'm not going through the details because
- 8:37
- this is just a very small
- 8:38
- quick, setup
- 8:40
- it's not that important
- 8:42
- but roughly speaking, the BP
- 8:44
- tokenization idea is that you try to
- 8:46
- in some sense
- 8:46
- you just try to find frequent subwords that are most frequent
- 8:51
- right? And
- 8:51
- and you use them in the vocabulary
- 8:53
- But how do you decide which one is more frequent and
- 8:55
- and not? I think there
- 8:56
- there's a sequential algorithm. You first go through maybe like the
- 8:59
- the, the, the smallest kind of like
- 9:04
- combinations of like, you go through all the characters
- 9:06
- you know, and you, you try to find
- 9:07
- you know, I think it's a greedy algorithm
- 9:09
- First, you try to find the most frequent
- 9:11
- you know, one character thing
- 9:12
- and you find in the most frequent combination of two characters
- 9:15
- and you remove, you know, it's, it's a little complicated
- 9:17
- which I, I mean, I'll try to
- 9:19
- give you a precise version
- 9:20
- but it's kind of like a
- 9:21
- a greedy, algorithm that decides what one
- 9:24
- which one is more frequent
- 9:26
- so and so forth
- 9:28
- and if you are interested
- 9:28
- you can look up the
- 9:29
- the exact algorithm
- 9:32
- okay, so that's the tokenization
- 9:34
- So that means that from now on
- 9:36
- you know, we just assume we have a tokenizer
- 9:38
- so every tax
- 9:39
- every piece of tax will be turned into
- 9:41
- a sequence of tokens
- 9:42
- So from now on
- 9:43
- like a, a tax, it just means a sequence of tokens
- 9:47
- and each of this XI
- 9:49
- is in the vocabulary. Let's call the vocabulary V
- 9:53
- and the size of the vocabulary could be
- 9:55
- like two hundred and fifty
- 9:57
- thousands
- 9:58
- and, you know
- 10:00
- each of this is kind of like an index
- 10:02
- in the vocabulary, like, you know, this is the fifth token
- 10:04
- this is the, a hundred fifth token
- 10:06
- so on and so forth
- 10:07
- Okay?
- 10:08
- So, and now the questions
- 10:10
- you know, now we go into the next question
- 10:12
- which is how do we model this distribution
- 10:14
- of sequence of tokens?
- 10:16
- How do we create a probabilistic model to
- 10:18
- model it?
- 10:19
- If you think about the support
- 10:22
- of, this, this distribution, like how many possible combinations
- 10:26
- that there are possible here
- 10:28
- so the number of possible combinations is You have V choice
- 10:33
- this is a cardinality
- 10:34
- you have V choice
- 10:35
- cardinality of V choice for each of the
- 10:39
- the tokens, and you have a sequence of last T
- 10:41
- so you take the power to the T
- 10:43
- right?
- 10:43
- So everyone has V choice
- 10:46
- and the
- 10:47
- V to the power T
- 10:48
- is the total number of choices
- 10:49
- So you cannot assign one probability for each of this
- 10:52
- sequence, like, like, like a, like a
- 10:57
- like a discrete categorical tokens
- 10:58
- right? Categorical distributions
- 11:00
- right? If you treat this as a categorical distributions
- 11:02
- then every choice here will have one probability
- 11:06
- and that's just way too many parameters to
- 11:08
- describe it
- 11:09
- So that's why
- 11:10
- what people do is that people
- 11:11
- decompose this distribution using the chain rule
- 11:14
- to write it this way
- 11:15
- So you say
- 11:16
- suppose I have a probability distribution
- 11:20
- over the joint, of the sequence
- 11:22
- then I can write it as the following
- 11:24
- so I can say it's equals to
- 11:26
- P of X one times P of
- 11:28
- X two given X one
- 11:30
- P of X three given X two and X one
- 11:34
- so on and so forth
- 11:35
- and P of X T
- 11:36
- given X one
- 11:37
- up to X T minus one
- 11:40
- And this is the
- 11:41
- the so-called, you know, conditional, probability decomposition
- 11:45
- You decompose the joint distribution
- 11:47
- into a product of conditional probabilities
- 11:51
- And you model each of these conditional probabilities
- 11:53
- The benefit here is that each of these
- 11:55
- conditional probabilities is a distribution
- 11:58
- is a conditional distribution
- 11:59
- but the choice of X two
- 12:02
- for example, here, is only V
- 12:03
- You have V choices here
- 12:05
- You only have a
- 12:06
- you only have to have a
- 12:07
- kind of like a softmax or logis over
- 12:09
- V choices for each of this
- 12:11
- right? So each of this will be eventually you'll see is
- 12:14
- some kind of like a
- 12:17
- you have some logis
- 12:17
- then you apply some softmax
- 12:18
- you get a proper distribution over
- 12:20
- v choice
- 12:22
- So that's what we're gonna do
- 12:24
- So basically, each of this
- 12:26
- will be parameterized by a new artwork
- 12:30
- the input will be
- 12:31
- what's after the conditioning
- 12:33
- right? So input here is the x minus x two
- 12:34
- and output is the distribution of x three
- 12:37
- And, and this input-output relationship is modeled by
- 12:41
- a, a new artwork, which is a transformer
- 12:46
- Okay, so that's, what we are, going towards
- 12:49
- How do we model each of this conditional distribution?
- 12:53
- So
- 12:56
- so basically we are gonna describe a
- 12:57
- conditional distribution P of
- 12:59
- P sub theta X T
- 13:01
- given X one up to X T minus one
- 13:04
- So
- 13:09
- Before going to details
- 13:10
- let's treat this as a kind of like
- 13:11
- a black box first
- 13:12
- And let's first kind of like discuss a little bit about
- 13:16
- you know, the, the boundary, right, in some sense, right?
- 13:19
- So it's kind of like a
- 13:19
- there are multiple layers here
- 13:21
- So the typical way to
- 13:24
- parameterize this is the following
- 13:26
- So, you use a transformer
- 13:28
- So basically what happens is that you have
- 13:30
- a sequence X1 up to XT
- 13:32
- And, so first of all
- 13:35
- these are discrete IDs
- 13:37
- right? So each of these means
- 13:39
- a ID for the token
- 13:41
- You first have to turn them into
- 13:42
- some numerical numbers
- 13:44
- So the, the, the first step is that you say
- 13:47
- I'm going to turn this into
- 13:49
- an embedding E of X1
- 13:51
- And so basically I turn each of the token X into
- 13:56
- XT into embedding of XT
- 13:59
- And what is the embedding?
- 14:00
- Embedding is, As a D-dimensional vector
- 14:04
- in, the Euclidean space
- 14:06
- And this embedding is
- 14:07
- is trimmed. So basically
- 14:09
- you don't know what the parameters are
- 14:11
- you will just, parameterize it, you know, as unknown numbers
- 14:14
- and you trim the model to
- 14:16
- find the right numbers
- 14:18
- for, for these embeddings
- 14:20
- So every, every, token in the vocabulary will be
- 14:25
- will have a corresponding embeddings
- 14:27
- So basically, you're gonna have a
- 14:32
- a sequence of embeddings
- 14:33
- E1, E2, this is the embedding for the first token
- 14:35
- embedding for the second token
- 14:37
- up to the embedding for the last token
- 14:39
- and in this lecture
- 14:40
- I think we're gonna use row vectors
- 14:42
- because I think when you get into the implementations
- 14:45
- you know, every vector is actually row vectors in the
- 14:48
- in the Python code
- 14:49
- even though in mathematics people often use column vectors
- 14:52
- So, so I, I thought about this
- 14:54
- and I, I thought just for this
- 14:56
- for large language model sections
- 14:57
- we'll just always use row vectors
- 14:59
- which is more consistent with the practice
- 15:01
- It looks a bit weird to me
- 15:03
- actually, from a mathematical perspective
- 15:05
- and if you read the papers
- 15:06
- many of the papers do use column vectors
- 15:08
- but I think we'll just use row vectors
- 15:10
- to be consistent with the code
- 15:11
- So basically every embedding is a row vector
- 15:15
- and then you have this
- 15:16
- embedding matrix
- 15:18
- so this is, you have
- 15:22
- V of them, and each of them is of dimension D
- 15:25
- so you have this matrix
- 15:26
- And, and to read off which row you are
- 15:28
- you just have to multiply
- 15:30
- either you just read off the ID
- 15:31
- right, so if you have the fifth token
- 15:33
- you just read the fifth row
- 15:34
- or you can multiply this matrix with
- 15:37
- an indicator vector to gather that row
- 15:39
- which is the same thing
- 15:41
- Anyway, so we'll just read off from this embed
- 15:43
- embedding matrix the corresponding embeddings for each of the token
- 15:46
- so you get E of X2
- 15:48
- up to E of Xt
- 15:51
- and, and I think in
- 15:54
- in many cases, often people append
- 15:57
- prepend, a, beginning of sentence token
- 16:01
- which is just a fixed token
- 16:02
- in the very beginning
- 16:03
- This is a kind of like not always
- 16:05
- necessary. In some cases
- 16:06
- you just don't do it
- 16:07
- In some cases, you do it
- 16:08
- But let's, probably just introduce that
- 16:10
- You have a beginning of sentence token
- 16:12
- which is one of the special token
- 16:13
- in this vocabulary. You probably
- 16:15
- often actually what happens is that
- 16:17
- your vocabulary is bigger than what is needed
- 16:19
- You always reserve a few tokens
- 16:21
- you know, actually maybe a thousand
- 16:23
- you know
- 16:23
- a few thousand tokens for special tokens that you can introduce
- 16:27
- in, in, for special purposes
- 16:30
- And you can introduce this beginning of sentence token
- 16:32
- as X zero
- 16:33
- so which is always the same
- 16:35
- and then you turn them into embeddings
- 16:38
- And then what you do is that you
- 16:40
- go through this transformer
- 16:41
- which I'll discuss in the later half of
- 16:43
- the lecture
- 16:44
- So this is a transformer
- 16:48
- For now, you just think of this as a new network
- 16:50
- which does a lot of computation
- 16:52
- machine application, you know, tensions, which I'm going to discuss
- 16:55
- a lot of operations
- 16:56
- and then you output
- 16:58
- some logits of, like, you output some kind of logits for
- 17:03
- at each of the locations
- 17:08
- So
- 17:08
- each of these logits is supposed to describe the conditional distribution
- 17:12
- before the, the softmax. So basically
- 17:21
- what you do is you say
- 17:23
- the probability of, so let's just
- 17:26
- you say each of this U1 is
- 17:30
- let's call this U1 U theta X zero
- 17:34
- and then P of X one is
- 17:39
- modeled as, you know, just the softmax
- 17:44
- Its distribution P of XI is the
- 17:47
- XY is modeled as the softmax of
- 17:52
- F theta X zero
- 17:55
- So the first, distribution, which is not conditional
- 17:58
- you just, get the output from the X zero
- 18:01
- you get some logit
- 18:02
- So this U, UI is in dimension V
- 18:06
- and you take a softmax
- 18:07
- you get a probability vector
- 18:09
- So this is a probability vector
- 18:15
- right, in this, simplex of dimension V
- 18:18
- right? So the sum
- 18:19
- this, this, after taking a softmax
- 18:21
- the sum of the entries is one
- 18:23
- and all the
- 18:23
- all of the entries are non-active
- 18:25
- So then this is your probability for X
- 18:27
- one. This is the distribution
- 18:28
- Technically, I think the distribution of this is equal to softmax
- 18:31
- okay?
- 18:32
- and then the same thing
- 18:34
- you know, when you have X two given X one
- 18:37
- you say this is equals to softmax
- 18:41
- of, U2, where U2 is the output at the second position
- 18:48
- and, and U2, you know
- 18:49
- we'll just call U2 the same as
- 18:50
- you know, so we call U2
- 18:55
- the output, you know, of, this model given X
- 19:03
- zero and X one
- 19:06
- So, so each of these UIs
- 19:08
- you know, is, basically the output
- 19:15
- for, the sequence before it
- 19:17
- right? So UT plus one is
- 19:19
- F theta X zero up to XT
- 19:23
- So, so, so basically generically, you know
- 19:26
- P of XT
- 19:27
- given X one up XT minus one is the softmax
- 19:34
- of, UT
- 19:39
- is it UT or UT plus one?
- 19:41
- UT plus UT, yes, UT
- 19:44
- which is the softmax
- 19:47
- of F theta X zero to X
- 19:52
- T minus
- 20:00
- one
- 20:02
- Okay. Any questions so far?
- 20:07
- So I, I, I particularly chose it to index at one
- 20:09
- yes
- 20:13
- Yes
- 20:18
- U is just a vector
- 20:19
- you know, I think, you know, I can just call U
- 20:21
- I think, it's kind of like
- 20:23
- I can just call U F theta X zero
- 20:24
- it's the same thing
- 20:25
- I'm not, did I answer the question?
- 20:28
- One second
- 20:43
- So, oh, I think maybe, maybe that's the
- 20:47
- So
- 20:48
- so
- 20:49
- I'm just using f theta x zero
- 20:51
- x one as a
- 20:52
- more explicit way to
- 20:54
- represent UI because I'm trying to express the dependency
- 20:58
- So, so basically I'm saying like
- 21:01
- basically I, I think they are the same thing
- 21:04
- Like, it's just there
- 21:05
- there are no difference at all in some sense
- 21:06
- just how you view it
- 21:07
- right? Like in one way you just say
- 21:08
- oh, I have, I need a, a
- 21:10
- a notation for the vector
- 21:11
- and the other way you say I have
- 21:12
- I
- 21:12
- I need to interpret this vector as a
- 21:14
- function of the inputs
- 21:18
- Oh, okay, okay
- 21:19
- okay. So maybe I didn't get the question
- 22:00
- Yes, yes, yes
- 22:01
- So
- 22:01
- I think maybe I'm trying to figure out
- 22:04
- what's the question here. So
- 22:07
- I think maybe what they're asking is the following
- 22:09
- So, so maybe let's take this one
- 22:12
- right? So this is a
- 22:13
- so this is a vector
- 22:15
- This is a vector in dimension V
- 22:19
- So in, in a vocabulary size
- 22:21
- And you have to take a softmax
- 22:22
- it's a, it's a sequence of numbers which are
- 22:27
- all non-negative and they sum to one
- 22:29
- So, and what this really means is that
- 22:33
- P theta of xt is equals to one
- 22:37
- given x one up to xt minus one
- 22:39
- The chance to see the first token under this probabilistic model
- 22:42
- parametric theta, is equals to, this is the vector
- 22:47
- the first entry of this vector
- 22:49
- right, is equals to softmax
- 22:52
- f theta x zero up to xt minus one
- 22:57
- indexed by one. This is the first entry
- 22:59
- of this vector
- 23:00
- And you have many
- 23:00
- many other entries, right? Each entry is
- 23:02
- you have another entry softmax
- 23:06
- f theta, sorry
- 23:14
- two
- 23:14
- and that
- 23:15
- is equals to P theta of xt is equal to two
- 23:19
- the chance of seeing the
- 23:21
- second token given x one
- 23:23
- up to xt minus one
- 23:25
- And you have so on and so forth
- 23:28
- Does that answer the question?
- 23:31
- V, the, yeah
- 23:32
- this V absolute value is the
- 23:34
- size of the vocabulary. Yep
- 23:45
- Yep. So this is the causal model so far
- 23:47
- but I haven't defined the transformer yet
- 23:49
- So, but I'm insisting that this is only
- 23:52
- so guys
- 23:52
- this is one of the reasons why I'm
- 23:54
- using this notation. So I'm insisting that UT
- 23:57
- is only a function of X zero and
- 23:58
- X T minus one
- 23:59
- but not a function of the
- 24:01
- the, the later, inputs
- 24:04
- So, but that will be
- 24:06
- that will be guaranteed when we introduce the
- 24:08
- exact inverse working of this transformer
- 24:27
- Okay, good question. So the first
- 24:28
- the second question is easier to answer
- 24:30
- So the embeddings is only a property of the
- 24:33
- this X, right? You don't have to see any other things
- 24:35
- right? It's just actually
- 24:36
- it's literally, if this is the first token
- 24:38
- then you just read the first row
- 24:39
- of this matrix. If this is the tenth token
- 24:42
- you raise the tenth row
- 24:43
- and that's your embedding. No any other dependencies
- 24:45
- And while we are writing this as
- 24:47
- as functions of the X
- 24:48
- but not the embeddings
- 24:49
- that's just, it's just
- 24:53
- is sometimes the embeddings is the inner working
- 24:55
- to some degree
- 24:56
- I'm just trying to give you more
- 24:57
- I'm trying to unpack it a bit
- 24:58
- right? So, so, so
- 25:00
- so if you don't have to unpack it
- 25:01
- you just work
- 25:02
- you can work with the text
- 25:03
- Yeah. That's just the representation
- 25:05
- Yeah. And it's one-to-one
- 25:07
- so in some sense
- 25:07
- there's no difference
- 25:13
- Okay, so this is a
- 25:14
- a full description of the probabilistic model
- 25:16
- Once you have this probabilistic model
- 25:17
- you can compute the maximum likelihood
- 25:18
- given any data. We can say
- 25:20
- I have a data distribution
- 25:21
- I have a data point
- 25:22
- a sequence of data
- 25:23
- I can see under this probabilistic model
- 25:25
- P theta, what is the chance of seeing this
- 25:29
- this data
- 25:30
- So that's, that's what you need for computing the
- 25:33
- do the training, right? You, you
- 25:34
- you need to compute the likelihood
- 25:35
- and then you maximize the likelihood as a function of theta
- 25:39
- right?
- 25:39
- So, and, let's briefly discuss, you know, for generation, right?
- 25:44
- So suppose you already have the generational decoding
- 25:50
- suppose you already have the model
- 25:52
- then a, a model theta
- 25:54
- then what do you do?
- 25:55
- So the, as you can see
- 25:56
- the most, natural thing is you just keep generating X one
- 26:00
- X two, X three, so on and so forth
- 26:01
- right?
- 26:02
- You just say, I'm gonna generate X one first from this
- 26:05
- you know, P, say, softmax
- 26:12
- Softmax, you know
- 26:13
- f theta
- 26:14
- x one up to x zero up to x t
- 26:17
- x, I guess I just write
- 26:20
- generically, so you generate x t minus one from
- 26:24
- f theta x zero up to x t minus one
- 26:27
- right? And you do this sequentially
- 26:28
- Sometimes people draw this in this way
- 26:29
- so you say, oh, you have x zero
- 26:31
- you first go through the transformer
- 26:35
- and you get x one
- 26:36
- and you feed x one back to this transformer
- 26:40
- right? So, and
- 26:43
- and then you generate x two by sampling from the softmax
- 26:48
- and then you feed it back
- 26:49
- So that's kind of why it's called autoregressive
- 26:51
- because you You have to
- 26:53
- like when you are generating
- 26:54
- you don't have the sequence
- 26:55
- right? When you are computing log likelihood
- 26:57
- you see the whole
- 26:58
- you already see the whole sequence
- 26:59
- You are just trying to figure out
- 27:00
- what's the chance of seeing this sequence
- 27:02
- But when you are generating
- 27:03
- you start with nothing
- 27:04
- and then you have to generate your sequence
- 27:06
- yourself
- 27:06
- And that's why you start with the beginning of sentences token
- 27:10
- and then you generate first token
- 27:12
- and then you
- 27:12
- give the first token back to transformer and compute this softmax
- 27:16
- and then you generate second token
- 27:19
- you do a sampling to generate second token
- 27:20
- and you give it back
- 27:21
- and you keep doing this
- 27:23
- So that's the autoregressive generation
- 27:26
- the
- 27:28
- and sometimes, you know, like
- 27:30
- this generation only start from certain part
- 27:33
- because in many cases
- 27:34
- you are not generating from nothing
- 27:37
- So you are given maybe a few tokens
- 27:39
- maybe you are given
- 27:40
- so the, sometimes it's called prompt
- 27:43
- So prompt is like something
- 27:44
- the models are given. It's maybe like a question
- 27:46
- you know, or instructions
- 27:47
- so on and so forth
- 27:48
- You say you are given
- 27:50
- X amount to X K
- 27:51
- And then the model is supposed to complete the generation
- 27:55
- so it start with X T plus one
- 27:56
- So X T plus one
- 27:58
- so this is given
- 28:00
- And then you start with X T plus one
- 28:02
- you first generate X T plus one from
- 28:03
- this softmax
- 28:05
- of the F theta X one up to X K
- 28:10
- and then you generate X T K plus two
- 28:14
- and then you do the same thing
- 28:15
- So it doesn't have to be always from
- 28:17
- the beginning
- 28:18
- And then there's another small thing
- 28:20
- which is that when you generate
- 28:22
- you don't necessarily have to always follow the same
- 28:26
- model. And there's one thing that you can deviate from it
- 28:28
- which is that you can
- 28:30
- add additional temperature where
- 28:35
- you say, as opposed to generating From the original
- 28:40
- probabilistic model I'm generate
- 28:42
- I'm gonna divide this by a temperature T
- 28:45
- tau, or T sometimes
- 28:48
- so, and, what this ha- what
- 28:50
- what does this mean is that you change the
- 28:53
- you don't change the orders between these
- 28:55
- choices, right? So this is a
- 28:58
- this, this F theta is a vector
- 28:59
- right? It's a
- 29:00
- a V-dimensional vector
- 29:01
- that describes the logis for each of the
- 29:03
- tokens in the vocabulary
- 29:05
- And when you divide by tau
- 29:07
- you don't change their relative orders
- 29:08
- The bigger ones are still bigger
- 29:09
- the smaller ones are small
- 29:11
- still smaller, but you change the
- 29:13
- the scale
- 29:14
- So, so for example, before you could have like a
- 29:19
- so suppose, you know
- 29:22
- F theta X zero up to X K is
- 29:26
- is equals to, let's say, maybe like a two
- 29:30
- one, minus one, something like this
- 29:34
- And if you divide by tau
- 29:35
- let's say suppose you divide by tau
- 29:37
- which is very, very small
- 29:38
- Suppose tau is almost zero
- 29:40
- Then after dividing by tau
- 29:41
- like if you divide by tau
- 29:44
- then what happens is that maybe this becomes two hundred
- 29:47
- this becomes one hundred. I'm
- 29:48
- I'm giving you a concrete example
- 29:50
- which is, a little bit unrealistic
- 29:51
- but, to demonstrate a point
- 29:53
- And what's the difference between the softmax of
- 29:55
- this vector versus the softmax of the readjusted
- 29:58
- vector?
- 29:59
- The difference is that
- 30:00
- the softmax of this is much sharper
- 30:03
- So maybe let me just give you an example here
- 30:05
- So, so if you just have
- 30:08
- let's say, maybe I just have three dimensions
- 30:10
- so that, it's easier to demonstrate a point
- 30:16
- So if you do softmax of this vector
- 30:22
- of two, one, minus one, what happens is that you get
- 30:28
- in some sense you get exponential two
- 30:30
- over exponential two
- 30:33
- Plus exponential one plus exponential minus one
- 30:37
- That's the, after softmax was the probability of the first
- 30:41
- token, right? And the second token will be exponential one
- 30:46
- exponential two plus exponential one plus exponential minus one
- 30:52
- right? And the third one will be
- 30:54
- the, the same denominator, the different numerator, right?
- 30:57
- So that's, that's the softmax of this
- 31:00
- But if you take the softmax of the two hundred
- 31:02
- one hundred minus one hundred
- 31:03
- suppose you do that
- 31:11
- then this will give you
- 31:13
- exponential two hundred
- 31:16
- versus exponential two hundred plus exponential
- 31:21
- one hundred plus exponential minus one hundred
- 31:25
- blah, blah, blah
- 31:27
- So what's the difference here?
- 31:28
- The difference is that in terms of the
- 31:30
- the three numbers, in terms of the order
- 31:31
- it's always like this is bigger than this
- 31:33
- this is bigger than this
- 31:34
- right?
- 31:34
- Because 200 is big
- 31:35
- 2 is bigger than 1
- 31:36
- 1 is bigger than minus 1
- 31:37
- the order is always maintained
- 31:39
- But the, the, the relativeness
- 31:44
- scale is different
- 31:45
- because here you can see exponential 200 is way
- 31:48
- way bigger than exponential 100
- 31:50
- The difference is like so astronomical
- 31:53
- like it's too, the
- 31:54
- the difference is too
- 31:55
- big so that
- 31:56
- these two numbers are just so small compared
- 31:57
- to exponential 200
- 31:59
- So they are even negligible
- 32:01
- So basically, in this calculation
- 32:03
- these two numbers don't even have to show up here
- 32:05
- like it, it don't even matter
- 32:06
- So you just get what?
- 32:07
- So basically, the first token will just have probably exactly
- 32:10
- almost exactly what, maybe 0.9999 something
- 32:12
- And then if you look at the second entry
- 32:16
- which is exponential 100
- 32:18
- divided by exponential 200 plus exponential 100 plus exponential minus 100
- 32:27
- Again, everything is negligible compared to exponential 200
- 32:31
- So this is not very useful
- 32:32
- this is not very useful
- 32:33
- this is kind of like close to zero
- 32:35
- So basically this entry will be zero
- 32:37
- So that's why after you
- 32:38
- you, you divide by tau to make this bigger
- 32:41
- then this is just approximately one
- 32:44
- zero, and zero
- 32:45
- So that's
- 32:45
- that's what I mean by it's much sharper
- 32:47
- So you, you, you, you sharpen the
- 32:49
- the distribution to make it
- 32:51
- most of the distribution to be
- 32:53
- the, the probability mass to be on the
- 32:54
- the most likely token
- 32:56
- And however here
- 32:58
- exponential two and exponential one are not that
- 33:00
- different
- 33:00
- If you compute this
- 33:01
- you'll get something like maybe
- 33:03
- the first one will be like maybe point seven
- 33:05
- the second one will be point three
- 33:06
- I, I don't know
- 33:07
- like we can use a calculator to see
- 33:09
- But, you know
- 33:09
- it will still be like this one is bigger than this
- 33:12
- and the second one is bigger than the third one
- 33:13
- but it won't be like
- 33:14
- all the mass is on the first token
- 33:16
- So, so
- 33:17
- so basically what I'm trying to say is
- 33:18
- that if tau is a small number
- 33:20
- so that you You
- 33:22
- you amplify the logis
- 33:24
- then you are sharpening the
- 33:25
- resulting distribution to make it more like a
- 33:27
- deterministic distribution
- 33:28
- And then as a result
- 33:29
- you are just generating
- 33:32
- you are just sampling from the
- 33:34
- the most likely token. You are taking the
- 33:35
- max as opposed to sampling from it
- 33:38
- So, and, and, and that's actually what people actually
- 33:44
- do, like they say, basically, like I think actually
- 33:48
- in API call, you have the choice to choose the temperature
- 33:51
- and if you choose a small temperature
- 33:52
- it means that you want more deterministic generation
- 33:55
- if you choose a temperature to be zero
- 33:56
- it means that your generation will just be
- 33:58
- always deterministic every time you take the largest on
- 34:02
- on, on most likely token
- 34:04
- and then keep going on
- 34:05
- right? And of course
- 34:06
- you can also change the temperature to be smaller than
- 34:09
- bigger than one, which makes the, the distribution more softer
- 34:12
- so that there's a
- 34:13
- you, you focus on more the long tail
- 34:14
- like you can generate more
- 34:17
- tokens from the tail
- 34:18
- so you have more stochasticity and
- 34:20
- and certainty in your generation
- 34:22
- Sometimes that's useful because
- 34:23
- you want to encourage some diversity in your generation
- 34:26
- and you, you do that
- 34:27
- especially in the RL
- 34:28
- sometimes you have to do that
- 34:31
- cool
- 34:32
- Okay, sounds good
- 34:34
- And sometimes there are also other ways of sampling
- 34:36
- which is, probably I shouldn't go into details
- 34:39
- there are some notes
- 34:40
- in the lecture notes
- 34:42
- So another way to do it is that you
- 34:45
- this vector right now I have three dimensions
- 34:47
- but actually it's like a
- 34:48
- the dimensionality is V
- 34:49
- right? Like, which is like 250 or something
- 34:52
- You can take top K
- 34:54
- and then drop all of the other
- 34:55
- dimensions. So you only sample from the
- 34:57
- the most likely K tokens
- 34:59
- and you re-normalize the distribution
- 35:01
- So basically the, the first K tokens may not cover the
- 35:04
- all the probabilistic, probabilistic noise
- 35:07
- So
- 35:08
- but still that's probably easier to generate because
- 35:10
- you don't want to generate too rare tokens
- 35:12
- so you just drop everything
- 35:14
- below top K and then generate
- 35:18
- okay, sounds good. So I think I'm probably
- 35:27
- And then, okay, so once you have all
- 35:29
- so this is the generation
- 35:30
- and then for the training
- 35:32
- I think I already kind of like describe it
- 35:35
- a little bit
- 35:36
- so when you do the training
- 35:40
- you just, computer log likelihood, the negative
- 35:46
- log likelihood
- 35:52
- The negative log likelihood
- 35:55
- and you say this is a log
- 35:57
- so if you have example x1 up to xt
- 36:01
- so the minus log p theta x1 up to xt
- 36:06
- capital T, is just a minus log
- 36:11
- the product of all of this
- 36:13
- right, p theta x1, p theta x2 given x1
- 36:19
- so on and so forth
- 36:27
- And this is the
- 36:29
- minus the sum of the log of the
- 36:31
- p of xt
- 36:33
- given x1 up to xt minus one
- 36:37
- And then you change this to
- 36:40
- log of the softmax
- 36:44
- of p1 of this f theta
- 36:53
- And here, this XT is a particular choice of XT
- 36:56
- right, which is seen in the data
- 36:59
- So that means you are looking at the
- 37:00
- XT's entry of this softmax
- 37:02
- This softmax will give you the probability for every possible tokens
- 37:06
- and then you are only interested in the
- 37:08
- probability to see that particular X token XT
- 37:10
- so that's why you're looking at XT's index of that vector
- 37:15
- and that's a single scalar
- 37:16
- You take the log
- 37:17
- and you take the sum over T
- 37:19
- and that's your loss function
- 37:22
- And then you can optimize this loss function by
- 37:24
- for example, some optimizers, like, SGD or IDAM
- 37:29
- we didn't discuss IDAM in this lecture
- 37:30
- which is an extension of the X
- 37:32
- SGD optimizer
- 37:33
- but mostly
- 37:34
- you can just call the
- 37:35
- optimizer as a black box to optimize the
- 37:37
- loss function
- 37:54
- It's actually not the last one
- 37:55
- right? Because this is sum from t
- 37:57
- to one to capital T
- 38:00
- The little t and larger
- 38:10
- Okay, sounds good
- 38:13
- Okay, so then in the next
- 38:15
- thirty minutes, I'm going to tell you what this
- 38:18
- this function f is
- 38:19
- right?
- 38:20
- Or maybe what's hidden in this transformer
- 38:22
- What's the operation we have to do to compute
- 38:25
- the, the, the logic here?
- 38:29
- and it's pretty complicated
- 38:30
- so, it probably takes more than thirty minutes
- 38:33
- but I'll see how much I can do
- 38:36
- So
- 38:41
- any questions before that?
- 38:43
- Okay
- 38:44
- So at a high level
- 38:46
- this transformer has multiple layers
- 38:49
- And, so you start with like EX1 up to EXT
- 38:54
- and you first apply a so-called attention layer
- 38:58
- which I'll define, and then you apply the so-called MLP
- 39:02
- So, first of all, the attention layer
- 39:05
- the attention always takes in
- 39:08
- a sequence of vectors and output
- 39:09
- a sequence of vectors
- 39:11
- So you output a bunch of vectors
- 39:13
- at each position you output one vector
- 39:15
- and then
- 39:17
- each of these vector will go through a
- 39:18
- MLP
- 39:19
- MLP means like a multilayer perception
- 39:21
- which is just a two or three layers
- 39:23
- two or three layer network
- 39:25
- And, each of these vector will go through this MLP
- 39:31
- and then you apply another attention
- 39:33
- This attention will takes in many
- 39:36
- many vectors, and, and then output a sequence of vectors
- 39:40
- and then you do the MLP
- 39:44
- So that's the, the high-level, structure of this, attention
- 39:49
- And eventually, you have to output
- 39:51
- remember, eventually
- 39:53
- you have to output a sequence of vectors
- 39:54
- So basically, you do enough of these alternations between these two
- 39:58
- and eventually, you have this, MLP
- 40:01
- and you output
- 40:03
- some vectors u1 up to ut
- 40:04
- And you alternate between attention and MLP
- 40:09
- That's the high-level idea
- 40:12
- And, and I think we'll focus most of the
- 40:15
- our energy on defining attention
- 40:17
- which is kind of like the most
- 40:19
- new thing, right? Because MLP is just a two-layer network
- 40:22
- with some special kind of gating
- 40:24
- or kind of like
- 40:25
- some kind of like
- 40:26
- special activation functions
- 40:28
- and another small thing is that
- 40:29
- the MLP are sharing the parameters
- 40:32
- and they are applying independently
- 40:34
- So each of these vectors is
- 40:37
- Processed by the MLP in an independent fashion
- 40:39
- there's no
- 40:40
- kind of dependencies between
- 40:41
- positions when you apply the MLP
- 40:43
- They are just very separate operations
- 40:45
- So the only way that you are using
- 40:48
- you are kind of like
- 40:49
- you fuse the input from different positions or
- 40:52
- different time steps is the attention
- 40:54
- So the attention will look at all those vectors and
- 40:57
- and process a little bit and then output
- 40:58
- a sequence of vectors
- 41:00
- And the MLP is
- 41:00
- is very parallelized and parallelizable and independent
- 41:05
- Okay, so now let's, let's define attention
- 41:15
- This entire thing will be treated as a single system
- 41:17
- So, so, so basically once you define the architecture
- 41:19
- you apply the auto-differentiation
- 41:22
- you, you, like, you treat this whole thing as a
- 41:25
- as a gigantic network
- 41:26
- yeah, end-to-end
- 41:37
- Yeah, but auto-differentiation still works
- 41:39
- So, so the auto-differentiation, I think as we discussed
- 41:43
- if you remember, like
- 41:44
- it's kind of like fundamental is claiming that
- 41:45
- if you have a way to compute your forward pass
- 41:48
- as long as there are compositions of many
- 41:50
- many different components, you know
- 41:52
- like, basic components
- 41:53
- you can compute the backward pass the same fast
- 41:55
- the same way
- 41:56
- So here is, is indeed a composition of many
- 41:59
- many small arithmetic operations
- 42:01
- because as you see
- 42:02
- the attention layer has many small matrix modifications
- 42:06
- and the MLP has many matrix modifications
- 42:08
- and you can automatically differentiate
- 42:14
- okay, so
- 42:17
- okay, so now let's, let's talk about attention
- 42:30
- So for attention
- 42:32
- one thing to remember
- 42:33
- first is that the type signature is that your
- 42:36
- your input, a sequence of vectors to the attention
- 42:39
- and they output a sequence of vectors
- 42:41
- And, and
- 42:43
- we're gonna start with the so-called single header
- 42:45
- tension
- 42:51
- So the single header tension is like this
- 42:53
- So you first have
- 42:56
- a input as a
- 42:58
- a sequence of vectors
- 43:00
- let's say H one
- 43:03
- say, in up to H T in
- 43:07
- And, and you want to output
- 43:09
- you know, H, I'm trying to define the
- 43:11
- the type signature
- 43:13
- So you output
- 43:16
- H one out up to H T out
- 43:20
- And what you do is the following
- 43:21
- So
- 43:23
- the first thing is that you turn
- 43:25
- There's input vectors into a set of query key
- 43:29
- and, and value vectors
- 43:32
- So
- 43:34
- so you define, so basically you say H
- 43:38
- one in, you multiply it by a matrix W Q
- 43:42
- this is a weight matrix
- 43:43
- this is a parameter
- 43:46
- And, by the way
- 43:48
- let's treat all of this as row vectors
- 43:49
- and then all the multiplication is on the right-hand side
- 43:51
- as opposed to on the left-hand side
- 43:53
- So every vector is a row vector
- 43:55
- and you multiply on the right-hand side
- 43:56
- you get another row vector
- 43:58
- So, so this H in
- 44:00
- let's say this is like a dimension D
- 44:02
- R D, let's say, say R one by D
- 44:05
- and this W Q is in dimension
- 44:09
- so this is R one by D
- 44:11
- and this W Q is in R D by
- 44:17
- D H
- 44:18
- dh is a parameter
- 44:20
- it's a dimension you choose
- 44:22
- and, and then the resulting thing is called q1
- 44:28
- which is in dimension dh
- 44:31
- maybe one by dh
- 44:32
- depending on how you think about it
- 44:35
- and you can do this for every index
- 44:37
- every time step. Here I'm only doing it
- 44:38
- for time step one
- 44:39
- you can do it for time step t
- 44:41
- so you have qt
- 44:43
- So basically this give you a factor
- 44:44
- a sequence of vectors q1 up to q capital T
- 44:47
- each of them is in dimension
- 44:50
- one by dh
- 44:52
- And this called, these are called queries
- 44:56
- And
- 44:57
- you do the same thing for keys and
- 44:59
- values. So keys Let me define them first
- 45:02
- So it's the same thing you say the
- 45:05
- hT in multiplied by your so-called key projection matrix
- 45:10
- W K, and that's your K T
- 45:12
- And the dimensionality, I think, is the same
- 45:14
- so this is also in the same dimension
- 45:17
- and the K T is in the same
- 45:18
- dimension. And it's the same thing for value func- value vectors
- 45:21
- not value functions, hT in times W V
- 45:28
- So in some sense
- 45:30
- maybe
- 45:32
- maybe this is the right time to
- 45:34
- talk about some of the intuitions
- 45:35
- you know, for the tensions
- 45:36
- which is not that obvious because eventually it just works
- 45:39
- you know
- 45:39
- like nobody really know exactly how it works
- 45:42
- but the, the
- 45:43
- the rough intuition is that you have to
- 45:44
- have some interactions between these inputs to make it meaningful
- 45:48
- right? Because if everything is independent
- 45:49
- then you're just
- 45:50
- dealing with all the time steps independently
- 45:52
- So that's why you have to have to inter
- 45:55
- interactions. And interactions sometimes is about
- 45:57
- how do you kind of like
- 46:00
- retrieve or kind of pay attention to
- 46:02
- each other
- 46:03
- So that's actually how the word attention come from
- 46:05
- So basically, when you are processing the
- 46:07
- maybe this vector
- 46:09
- you probably should attend to
- 46:11
- or pay attention to some other vectors which are
- 46:13
- have relevant information
- 46:15
- And which one you
- 46:16
- you pay attention to
- 46:17
- is what we're going to decide here using
- 46:19
- with this query and key and vectors
- 46:21
- So in some sense
- 46:22
- intuitively, the attention will, what attention will do
- 46:24
- which I didn't show you yet
- 46:26
- is that you try to
- 46:29
- for processing each of the vectors
- 46:31
- you try to figure out
- 46:32
- which are the relevant vectors that has relevant information
- 46:36
- and, and then you pay actual attention to those vectors
- 46:40
- And, and then you compute the output depends on
- 46:42
- depending on those special vectors to some degree
- 46:46
- So that's what we are trying to do
- 46:48
- and
- 46:49
- and to kind of like decide which one
- 46:50
- you pay attention to
- 46:52
- you have to prepare these queries and key and value vectors
- 46:56
- and how to use them
- 46:57
- So use the query and keys as a
- 46:59
- way to decide which one you pay attention to
- 47:02
- So basically, let's say, suppose you have, the T's position, right?
- 47:08
- I want to decide
- 47:10
- what the T's position should pay attention to
- 47:13
- and, and
- 47:14
- and the way to do it is you
- 47:15
- say you compute the query Q T
- 47:19
- this is, the query at T's position
- 47:22
- and you multiply the query
- 47:24
- with the keys at all the other positions
- 47:28
- So maybe I, I will just draw it
- 47:30
- So you have queries Q one up to Q T
- 47:34
- and one of them is Q little t
- 47:37
- and you have Q
- 47:38
- K one up to
- 47:41
- K T
- 47:43
- And you, you multiply Qs with all the Ks
- 47:49
- so you have Q1 times
- 47:51
- you know, K1 transpose, Qt times K2 transpose
- 47:57
- and Qt times K capital T transpose
- 48:02
- and there is a
- 48:04
- sometimes there's a scaling factor here
- 48:06
- to kind of like normalize this in a way that
- 48:09
- makes it a little bit more kind of like well-behaved
- 48:11
- but I think for the moment you probably
- 48:13
- don't have to pay too much attention to the scaling factor
- 48:15
- so
- 48:16
- the important things that you are taking the
- 48:17
- inner product between Qs and
- 48:19
- each of the other Ks
- 48:20
- And this Qt transpose K
- 48:23
- Q, K transpose is actually inner product
- 48:26
- because Q is a
- 48:28
- is a row vector
- 48:29
- and K transpose is a column vector
- 48:34
- And you are taking the inner product between
- 48:35
- these row vectors and the column vectors
- 48:39
- and you get one scalar out of it
- 48:41
- So So this is
- 48:44
- all of these numbers are
- 48:45
- are real numbers, right? So this is in R T
- 48:47
- so you have T real numbers
- 48:51
- and then you take a softmax
- 48:52
- over these T real numbers to turn it
- 48:54
- into a probability vector
- 48:55
- So then you say
- 48:56
- I'm gonna do a softmax
- 49:01
- Say softmax
- 49:04
- of this whole thing
- 49:14
- And now you get what?
- 49:15
- You get a, this is a vector
- 49:17
- the dimensionality here is R T
- 49:19
- and each of this
- 49:21
- this is, the
- 49:22
- the outcome of the softmax is a sequence of numbers
- 49:24
- right?
- 49:27
- And, they are T numbers
- 49:28
- and, and all of these numbers are non-active
- 49:30
- and the sum of them is one
- 49:32
- So it's a probability vector over T positions
- 49:35
- And if you see a large number
- 49:36
- it means that Let's suppose this number is big
- 49:41
- it means that
- 49:42
- this number is somewhat big
- 49:43
- compared to other ones
- 49:44
- It means that the Qs and Qs have
- 49:46
- like more higher inner product
- 49:48
- right?
- 49:49
- And, it means that you should pay more attention to K
- 49:53
- K1, right? So
- 49:54
- so basically if either of these entries is big
- 49:56
- it means that
- 49:58
- intuitively Qt needs to pay more attention to K1
- 50:03
- and
- 50:03
- and that will be reflected by the probability vectors because the
- 50:06
- the corresponding probability will also be high
- 50:09
- And, so I think let's call this
- 50:11
- I guess I have a notation here
- 50:13
- so let's call this
- 50:14
- let's denote this to be Pt1
- 50:18
- Pt2 up to Pt capital T
- 50:23
- So the
- 50:23
- so each of the PtI is
- 50:26
- Each of the pT i is non-negative
- 50:29
- and the sum of the pT i
- 50:31
- over i from one to t
- 50:33
- is equal to one. So this is a probability vector
- 50:36
- and, and if this, if, and if this entries is big
- 50:38
- it means that we should pay more attention
- 50:40
- to that particular position when processing the
- 50:43
- little t's query
- 50:45
- And then how do you turn this
- 50:46
- how does this relate to the final
- 50:49
- outcome?
- 50:50
- So finally, once you have this
- 50:52
- then, you say the, the output at t's position
- 50:58
- which is a vector
- 50:59
- is using these probability vectors to weight
- 51:03
- the values
- 51:05
- The values is, v's, which are we are defining here
- 51:10
- right, v t, sorry, v t
- 51:11
- So you say it's pT
- 51:13
- one times v one plus
- 51:20
- pT
- 51:20
- two times v two plus pT t times
- 51:22
- v t. So basically
- 51:23
- if each of
- 51:24
- any of this
- 51:25
- Probability is
- 51:27
- much higher than you are multiplying this high probability with the
- 51:31
- the corresponding value at that position
- 51:34
- So basically you are using the V
- 51:36
- right, if, for example, if this number is very big
- 51:38
- then you are using V2
- 51:40
- a lot in the output at time T
- 51:48
- So you determine them by multiplying the matrices
- 51:51
- and the matrices will be trained
- 51:55
- like, like, yeah. How
- 51:56
- how they attend to each other is also trained
- 51:58
- Like, so
- 51:59
- so you're only giving a template in some sense
- 52:01
- Like, how, how, you know, like, it may be that
- 52:03
- you know, like
- 52:05
- it could be that all of these are zero
- 52:07
- you know, then you are not doing anything with it
- 52:09
- and then your neural network
- 52:10
- the loss function will be not good
- 52:12
- but it's still a valid neural network
- 52:14
- it's just the loss is not good
- 52:15
- So eventually you are going to train the
- 52:17
- model to behave well
- 52:20
- so that
- 52:20
- so basically
- 52:22
- The training algorithms will choose the right
- 52:24
- W so that this mechanism is
- 52:27
- doing the right thing
- 52:43
- So, let's try to articulate
- 52:45
- you know, I think it's a good question
- 52:46
- but let's try to articulate so that we
- 52:48
- can answer it
- 52:49
- So what's the
- 52:54
- your
- 53:00
- Yes. Why ain't, particular education, like I think this year
- 53:08
- that would be available for the interview when you train it
- 53:12
- and it's like not five minutes
- 53:13
- so we need to
- 53:14
- like, continue this
- 53:17
- So I think you are asking why you need such architecture
- 53:19
- but not just only using MLP
- 53:21
- for example
- 53:21
- right? That's one way to rephrase the question
- 53:25
- I think
- 53:27
- so if you are only using MLP in this way
- 53:30
- suppose you remove all the tensions
- 53:32
- you are only using MLP in this way
- 53:34
- then there's no interactions between different positions whatsoever
- 53:38
- So there's no chance that it's gonna work
- 53:39
- right? You are just applying
- 53:41
- separate networks for each of the tokens
- 53:44
- there's no chance you can understand the
- 53:46
- the, the, the, the dependencies, you know, the, the meanings, right?
- 53:49
- Like, there's no chance I
- 53:50
- I can just give you
- 53:51
- individual words and you can understand a sentence
- 53:53
- There's no chance, right?
- 53:54
- So, so you do have to
- 53:57
- have a model that can read
- 54:00
- all the
- 54:01
- tokens, you know, and, and use
- 54:03
- and study their kind of correlations and combinations
- 54:05
- so on and so forth
- 54:06
- right?
- 54:06
- But, there are other possible
- 54:09
- possible ways to deal with that
- 54:11
- For example, what if you use
- 54:12
- use a gigantic MLP
- 54:14
- that takes in all of this
- 54:15
- maybe you concatenate all of them
- 54:17
- and you take in one MLP
- 54:19
- right? So, so, for example
- 54:21
- you just concatenate all of these tokens
- 54:23
- vectors into one long vector
- 54:25
- and you go through a network
- 54:26
- So there are several issues with that
- 54:28
- One is that, you, will have way too many parameters
- 54:32
- because your sequence length
- 54:34
- Maybe sometimes even millions
- 54:37
- Then if you have a million by million
- 54:38
- matrices in your MLP
- 54:40
- it's going to be way too big
- 54:41
- Right now, the MLP is actually very small
- 54:44
- It only takes in a vector
- 54:45
- for one position. It doesn't even depend on
- 54:47
- T
- 54:48
- And another thing is that you're not only the
- 54:50
- the size of the MLP is big
- 54:52
- but also the size of the MLP is also changes over
- 54:55
- as you change the sequence length
- 54:57
- which is also very undesirable
- 54:58
- Because, for example, if you train with sequence length a thousand
- 55:01
- and then you test with sequence length a thousand and one
- 55:04
- then you have to change your parameter count
- 55:06
- which is not good
- 55:08
- And, and here, if you really look into details
- 55:10
- you'll find out that
- 55:11
- the number of parameters doesn't depend on the sequence
- 55:15
- length
- 55:33
- Exactly, right? So, so, so you, you know that, you know
- 55:36
- if you have sufficient compute
- 55:38
- you know, probably a gigantic MLP will work
- 55:40
- If you have sufficient data so you don't overfit
- 55:43
- and you have sufficient compute
- 55:44
- then there is some MLP that can actually solve this problem
- 55:47
- but that's too expensive
- 55:49
- So you need to prescribe some
- 55:52
- structure so that you can reduce the complexity
- 55:55
- right? And, and this is the
- 55:57
- the, the prescription we have
- 55:58
- but you still have to leave enough flexibilities
- 56:01
- You cannot just say
- 56:02
- oh, for example
- 56:03
- one way to do it is you just
- 56:04
- say let's average
- 56:05
- Then there's no parameters
- 56:06
- you just, prescribe too much, right?
- 56:07
- You, you, you dictate too much
- 56:09
- then that's not good. So it's kind of like an
- 56:11
- you need
- 56:11
- you need to strike a balance between
- 56:13
- how much structure you introduce and how
- 56:15
- much flexibilities you give the models to figure
- 56:17
- out
- 56:19
- Yeah
- 56:22
- yeah. Any other questions?
- 56:26
- By the way, there is actually, in the history, there was
- 56:28
- actually other ways to deal with this sequence
- 56:30
- For example, you can use a convolutional network on this
- 56:33
- which is kind of like MLP
- 56:34
- So you have use a
- 56:35
- use a MLP, but with a, a specific receptive field
- 56:39
- so that each time you only see a
- 56:41
- kind of like a window
- 56:43
- I think there are still papers that show
- 56:46
- says that that's not a terrible idea
- 56:49
- Actually, if you use a convolutional network
- 56:51
- it ac-actually can work to some degree
- 56:52
- It's just not as good as this one
- 56:55
- you know
- 56:56
- and, and, and also you can see
- 56:58
- like, there are also other
- 57:00
- probably we'll talk about that later
- 57:01
- so there are other variants of tensions to
- 57:03
- that does similar stuff
- 57:05
- but have better efficiency
- 57:07
- anyway, so let's continue. We ha-we are not done yet
- 57:11
- actually. This is only a sin-so-called single-head attention
- 57:14
- you know
- 57:15
- actually
- 57:16
- the real thing is a little more complicated
- 57:19
- but Let's see. Before I go into that
- 57:22
- let's see
- 57:27
- So I think
- 57:29
- you can see the notation skills is very messy
- 57:31
- so
- 57:32
- let's try to
- 57:33
- simplify it a little bit
- 57:34
- so that we don't have to always work
- 57:36
- with this messy notation
- 57:38
- So
- 57:39
- one typical way that people simplify this is
- 57:41
- that you
- 57:41
- try to organize all the q-queries
- 57:44
- into a matrix
- 57:47
- We call that each of the queries a row vector
- 57:49
- and if you put them all in your matrix
- 57:51
- then this is a
- 57:52
- a matrix of dimension T by dH
- 57:56
- and you can put all the keys into a matrix
- 58:02
- and you can put all the values in
- 58:03
- in your matrix as well
- 58:09
- right, so, and then you can write this H out
- 58:22
- The C- all two-seating factors in either kind of like
- 58:28
- also as a matrix is
- 58:29
- in some sense
- 58:30
- you do a softmax
- 58:32
- of the row of
- 58:34
- this Q K transpose
- 58:36
- divided by C matrix times V
- 58:39
- So here the, basically this Q K matrix
- 58:41
- Q K transpose matrix is the
- 58:44
- the formula of Q
- 58:46
- I times K J transpose divided by C
- 58:50
- So every entry is of this form
- 58:53
- which is kind of like this guys
- 58:56
- And then if you do a row softmax
- 58:58
- that corresponds to this softmax
- 59:00
- And then if you multiply by V
- 59:01
- it means, it means this linear combination
- 59:04
- I think you, we probably, we don't have time
- 59:06
- enough time to go into the details
- 59:09
- like, but you can double check
- 59:11
- you know
- 59:11
- it's just a very basic linear algebra
- 59:13
- It's just real, real and ranging, like, it's
- 59:14
- it's mo-mostly just notations
- 59:16
- Right
- 59:16
- so this Q case
- 59:17
- divided by C matrix is the inner product
- 59:19
- between the queries and keys
- 59:21
- and then you take a row
- 59:22
- row by softmax to get the probabilities
- 59:24
- So after the softmax
- 59:25
- this matrix becomes, every row is a probability vector
- 59:29
- and then you multiply that with the V to
- 59:31
- take the linear combination of Vs.
- 59:35
- and
- 59:36
- there's one
- 59:37
- thing that we kind of cheat a little bit here
- 59:39
- which is that if you look at this architecture
- 59:43
- then
- 59:44
- it's not true that the T's output
- 59:49
- only depends on everything before it
- 59:52
- There's no this kind of
- 59:53
- like, auto-recursive kind of, like, properties yet, right?
- 59:56
- Because the output at time T Depends on everything
- 60:03
- right? It depends on V capital T
- 60:05
- even the last input
- 60:07
- V capital T
- 60:07
- V capital T is a function of the last input
- 60:10
- And, so
- 60:12
- and the output at time T depends on the input
- 60:16
- depends on V T and it depends on H T
- 60:18
- in
- 60:19
- So
- 60:20
- so basically there's no kind of like this
- 60:21
- dependency. So then what you do is you say
- 60:25
- you introduce the concept of masking to
- 60:28
- make sure that there's no
- 60:31
- this kind of like
- 60:32
- wrong dependency you want. So basically what you
- 60:34
- really want to do is that you just
- 60:36
- in some sense, for H little t
- 60:39
- you want to delete everything after
- 60:43
- time T. So basically
- 60:44
- let's say, before you have this
- 60:48
- you know, P T V T plus
- 60:55
- P T T V T
- 60:57
- P T capital T V capital T
- 61:00
- So basically you say these dependencies are invalid for me
- 61:03
- so I just, don't have them
- 61:04
- So basically you just delete all of this
- 61:07
- And
- 61:08
- and then you renormalize these Ps
- 61:10
- so that the sum of these Ps
- 61:13
- got normalized to one. So it's still a
- 61:14
- linear combination or convex combination of the V
- 61:17
- one up to V T
- 61:18
- So that's pretty much what you do
- 61:19
- So you, you just say if the dependency is
- 61:22
- not good, you just
- 61:23
- in some sense mask them out
- 61:25
- And mathematically what you do
- 61:27
- actually also implementation-wise, what you do is through this masking
- 61:31
- which means that you
- 61:39
- which means that you add
- 61:43
- a masking here
- 61:45
- Maybe let's use a different color to
- 61:47
- show this
- 61:49
- So you add a masking here
- 61:51
- I think they're different by their color
- 61:59
- Okay
- 62:00
- So you add a mask here so that
- 62:04
- you can mask out the
- 62:07
- Let me see. You can mask out half of the
- 62:09
- this matrix
- 62:11
- So, so you have like a Q1
- 62:14
- K1 transpose, Q1, K2 transpose
- 62:17
- so on and so forth
- 62:19
- And Q1, K2 transpose is not valid in some sense
- 62:22
- because you are looking
- 62:23
- you are tending to something you shouldn't attend
- 62:25
- right? Q1 shouldn't be able to see K2
- 62:28
- So basically you add this masking matrix
- 62:30
- what you add is that you add minus infinity
- 62:35
- you add minus infinity here
- 62:39
- And for Q1, K3 transpose, you also add minus infinity
- 62:45
- so on and so forth
- 62:47
- And then in the next row
- 62:48
- you have Q2, K1 transpose, which is good
- 62:52
- You have Q2, K2 transpose, which is good
- 62:54
- And you have Q2
- 62:55
- K3 transpose, which is not good
- 62:58
- So you add minus infinity here
- 63:00
- so on and so forth
- 63:01
- So basically everything here
- 63:03
- will be adding additional minus infinity
- 63:06
- So basically this upper triangle
- 63:09
- part of this matrix will be all minus infinity
- 63:11
- You know, I mean, we'll
- 63:13
- By adding minus infinity
- 63:14
- it just means like you
- 63:15
- you turn it into minus infinity
- 63:16
- right? Any number add
- 63:18
- added by minus infinity is the minus infinity
- 63:20
- And then what happens is that
- 63:22
- or in other words
- 63:23
- it's kind of like you are just
- 63:25
- trying to
- 63:27
- add all of
- 63:28
- minus infinity to all of this part
- 63:30
- And then when you add
- 63:31
- when you turn all of this into minus infinity
- 63:33
- when you do the softmax
- 63:35
- this part just doesn't help
- 63:37
- doesn't change, doesn't
- 63:39
- like, doesn't, by this part, I mean, everything after T, right?
- 63:44
- So everything after little t
- 63:45
- you, you turn them into minus infinity
- 63:46
- and after you do a softmax
- 63:48
- they become zero
- 63:50
- So because exponential to the minus infinity becomes like
- 63:54
- so, so, so, become basically zero
- 63:56
- So that's why in this P
- 63:59
- so everything after PTT
- 64:04
- everything after Q will become zero
- 64:06
- and the
- 64:07
- the sum of the rest will be one
- 64:09
- So that's exactly what you
- 64:10
- how you achieve this
- 64:11
- because
- 64:12
- basically the leading combination of the rest of
- 64:13
- the terms doesn't matter because their coefficients are zero
- 64:23
- Yeah, we are masking this because you shouldn't
- 64:26
- in this autoregressive model
- 64:28
- you don't allow
- 64:30
- the output
- 64:31
- at time t
- 64:32
- to depend on anything that is after t
- 64:35
- So you have to respect this kind of
- 64:36
- like a sequential dependency
- 64:38
- So, you know, in generation, you
- 64:41
- you are not able to see the future
- 64:43
- tokens to generate a current token
- 64:45
- So
- 64:46
- so every
- 64:47
- token can only depends on everything before it
- 64:50
- So that's why we have to mask it out
- 64:57
- Okay, we have, yeah, we are almost done there
- 65:00
- Okay, so
- 65:06
- okay
- 65:06
- so this is the so-called single head attention
- 65:10
- and we have to make it
- 65:11
- more complicated to even finish the attention part
- 65:14
- So then what we do next is that
- 65:18
- maybe let's erase here
- 65:30
- So in some sense
- 65:31
- what we are doing here is like we
- 65:33
- have H1 in
- 65:35
- And hT in
- 65:37
- and we go through this single header tension
- 65:41
- I guess, in the note, I call it
- 65:43
- a tension one h
- 65:45
- and you get this h out
- 65:50
- h one out, up to hT out
- 65:53
- That's what we define so far
- 65:55
- And, in practice
- 65:57
- you have multiple copies of this single header tension with different
- 66:02
- weight matrices
- 66:03
- So you get another copy
- 66:05
- of the single header tension
- 66:08
- Maybe let's call this single header tension one
- 66:11
- this is called single tension
- 66:12
- header tension two, and they have the same
- 66:17
- input, and then you drop
- 66:19
- you get the, another copy of the output
- 66:25
- But here, by copy, I don't really mean literally copy
- 66:27
- because then it's not useful
- 66:28
- So what's, the difference between this and this is that the
- 66:31
- the weight are, the weight matrices are different
- 66:36
- So basically this WQ
- 66:38
- WKWV for these two copies of the tension are different
- 66:43
- So, so, so that's why they're not really literary copies
- 66:46
- they are just, like you have another
- 66:49
- another version of it
- 66:51
- And
- 66:58
- yeah, so
- 66:59
- so in some sense you should index this
- 67:00
- by something else because the output will be different
- 67:02
- right? So maybe you should index this by
- 67:07
- let's say this is
- 67:09
- because this is the first attention
- 67:10
- you say H11 and HT1
- 67:12
- and this is H12
- 67:14
- HT2
- 67:15
- And you also have another one
- 67:17
- and you get H13 up to HT3
- 67:23
- So, so, so when you have so many
- 67:25
- copies, you have, several sequences of vectors
- 67:28
- but eventually you only want one sequence of vectors
- 67:30
- so you have to combine them in some
- 67:32
- way
- 67:33
- And the way to combine them is that you
- 67:37
- you concatenate these three things
- 67:43
- So you concatenate
- 67:46
- H one one out
- 67:47
- H one two out
- 67:53
- one, I think how many
- 67:56
- each of this is called one hat
- 67:58
- So you have, let's say, N H
- 68:00
- N H hat
- 68:02
- So this is a tension N sub H
- 68:06
- So you concatenate all of these vectors
- 68:09
- and you get, and then you multiply that with
- 68:11
- another projection matrix, and then this will give you the final
- 68:17
- H, the output at one
- 68:21
- So that's the final output at position one
- 68:24
- And you do the same thing for position two
- 68:25
- where you concatenate
- 68:29
- maybe these guys, these guys, And
- 68:33
- and then you do a projection
- 68:34
- you get the output at position two
- 68:37
- So in some sense
- 68:37
- this is just give you more power
- 68:39
- to compute different kind of like
- 68:44
- kind of attention. In some sense
- 68:45
- you know, I think actually in practice
- 68:47
- if you
- 68:48
- I think the intuition is that
- 68:50
- maybe each of these attention is doing
- 68:52
- single head attention is trying to pay special type of like
- 68:57
- is doing some special type of job
- 68:58
- right? For example
- 68:59
- maybe here it's trying to find out
- 69:01
- which entity in the path passage matters to me
- 69:05
- and here this is trying to Find out
- 69:08
- you know, which
- 69:11
- what are kind of like the sentiment
- 69:13
- of the past passages
- 69:15
- and, and, and which part of the sentiment matters to me
- 69:17
- so and so forth
- 69:18
- right? So each of these attention probably pay
- 69:19
- attention to different type of information
- 69:22
- and then eventually you aggregate the outcome of
- 69:24
- all of this information into one
- 69:27
- big vector, and you project it into
- 69:30
- a single output
- 69:39
- Cool
- 69:42
- No, no, they are, they are all parallel
- 69:43
- yes
- 69:45
- They are, they are parallel, and then you aggregate
- 69:46
- So
- 69:47
- and all of this is in one of this
- 69:50
- And then
- 69:51
- so basically in one of this you have multiple parallel heads
- 69:54
- and then next round you have another
- 69:56
- number of parallel heads
- 70:03
- I think the number of heads is
- 70:04
- maybe on the order of like maybe And most a hundred
- 70:07
- I guess?
- 70:10
- let me think about this
- 70:11
- I think something on that order
- 70:14
- I think
- 70:15
- Yeah. It also depends on the model size
- 70:18
- of course, right? I think the
- 70:19
- the biggest one, I don't know
- 70:20
- I don't know whether a hundred is
- 70:22
- is for which model size
- 70:23
- I think maybe for like a
- 70:25
- a hundred billion parameter models we have
- 70:27
- have a hundred heads. Yeah
- 70:28
- Yeah
- 70:52
- Yeah. I think, I, I don't know exactly what's your proposal
- 70:55
- but, maybe it, it sounds like a simple
- 70:58
- simplified answer, but like a, like a
- 71:00
- like many of these proposals has been considered in
- 71:03
- in various ways, and, and some of them are For
- 71:06
- on most of our efficiency perspectives
- 71:08
- because eventually you can just have a
- 71:10
- like as we discussed
- 71:11
- you know, like if you really don't care about efficiency
- 71:13
- you can just have a generic
- 71:15
- very gigantic network space. So
- 71:17
- all of these kind of like
- 71:19
- special structures to try to
- 71:21
- use your parameters in the right way so
- 71:23
- that you can use it more efficiently in some sense
- 71:31
- okay, so
- 71:35
- and then, okay, so I guess, let's try to
- 71:38
- let's just give me two or three minutes to talk about
- 71:41
- to talk about some very quick things
- 71:44
- So one thing is that actually when I draw this
- 71:49
- it's actually a bit more complicated than
- 71:51
- Just doing this
- 71:52
- So because there's also a residual part in the
- 71:56
- in this kind of like how to
- 71:57
- combine the MLP with attention
- 71:59
- So, so actually what
- 72:01
- actually you really do is that you say you apply
- 72:07
- so after you do the
- 72:09
- attention, you
- 72:14
- you do, some kind of like normalization here
- 72:18
- So there's a normalization layer
- 72:20
- Often this is using the R
- 72:21
- RSM norm that we
- 72:22
- briefly discussed involved in the network section
- 72:25
- And then you apply MLP
- 72:26
- and then you add the residuals back to this
- 72:30
- You, you
- 72:31
- you kind of like in the residual network
- 72:32
- that we discussed
- 72:33
- And then you go to the next layer
- 72:35
- And there are different ways to kind of
- 72:36
- like add residuals. You can add residuals after you do the
- 72:40
- attention or, or before you do the attention
- 72:42
- So there's something called Pre-norm and the
- 72:45
- and post-norm, thing, so I guess the details you
- 72:48
- you, you can check out the lecture notes
- 72:50
- so there's some kind of like orders on how you
- 72:52
- when you apply your normalization
- 72:53
- residues, and attention, so on and so forth
- 72:56
- And, and finally, I think one important thing
- 72:59
- especially for the next lecture about the system ML
- 73:03
- perspective, is that the, we should try to quickly
- 73:08
- think about the computational efficiency here and the memory foot
- 73:12
- footprint
- 73:13
- So how many operations we have to do
- 73:15
- especially as a function of the
- 73:17
- of T
- 73:19
- as a function of other things
- 73:20
- it's actually relatively easier because as a function
- 73:22
- of the number of parameters
- 73:23
- you know, like all of those is kind of like a
- 73:25
- it's all multiple kind of and linear in
- 73:27
- some sense
- 73:27
- The main important thing is what is the
- 73:29
- the dependency on T
- 73:31
- And you can see that for each of the T's position
- 73:34
- you have to compute all of this inner
- 73:36
- products
- 73:37
- So for each of the position
- 73:38
- you have to compute capital T inner product
- 73:41
- So, and
- 73:42
- and you have capital T of these positions
- 73:45
- So that means that you have to
- 73:47
- basically do a T squared dependency
- 73:49
- right? So, so, or in other words
- 73:51
- you know, even this, just this matrix itself is a T
- 73:56
- T by T matrix
- 73:57
- capital T by capital T matrix
- 73:58
- and each of these entry requires some inner product
- 74:01
- So, so I think the dimension
- 74:03
- if you still remember the QIs
- 74:05
- or the QTs are in
- 74:07
- are dH
- 74:08
- and you need to compute T squared of this inner product
- 74:11
- so basically the
- 74:13
- the number of operations is T squared times
- 74:15
- dH
- 74:16
- The dependence on dH is less relevant
- 74:18
- the, the, you know, less, less important to some degree
- 74:20
- the dependence on T squared is very important
- 74:22
- because this means that if you have
- 74:24
- capital T being a million
- 74:26
- it's gonna take a million times square
- 74:28
- square, operations, which is prohibitive
- 74:31
- So, and this is one of the issues with
- 74:34
- the long context issues
- 74:35
- probably have heard of
- 74:36
- like, like I think we are using ChatGPT or Cloud Code
- 74:39
- you see the context
- 74:40
- and, and after some point they say
- 74:42
- let me compact my context
- 74:43
- right?
- 74:43
- And the reason is that you have to shrink your context
- 74:45
- otherwise your computational efficiency is too bad
- 74:49
- but of course, you know, in the, in the, in the real production system
- 74:52
- it's not like all of this capital T is really
- 74:54
- like there are other ways to
- 74:55
- reduce the dependencies
- 74:57
- I think in practice
- 74:58
- I don't think necessarily is T squared is
- 74:59
- probably more closer to linear in T
- 75:02
- because there are other
- 75:03
- ways
- 75:04
- variants of transformers that can reduce the dependency
- 75:06
- on T
- 75:07
- I think after, in the next Wednesday
- 75:09
- we're going to discuss a few
- 75:11
- things that can
- 75:12
- reduce the dependency on T a little bit
- 75:15
- exactly how we do it
- 75:16
- we don't really know in some sense
- 75:18
- because the there are a lot of open source papers that
- 75:22
- propose different approach, on which
- 75:25
- at least I don't know exactly which one is used
- 75:28
- in the, in OpenAI topic
- 75:31
- and another thing is that what is the
- 75:33
- memory usage here?
- 75:35
- So the memory usage
- 75:36
- you know, as, as, at, at the first side
- 75:38
- you have to spend T squared memory
- 75:41
- because this matrix itself requires T squared memory
- 75:44
- but that actually can be
- 75:45
- reduced because you don't have to actually
- 75:48
- compute the whole matrix
- 75:50
- one by one and then do the multiplication
- 75:52
- with V
- 75:53
- So in some sense
- 75:53
- to some degree
- 75:54
- you can
- 75:55
- do some part of this matrix and multiply
- 75:56
- with part of the V first
- 75:58
- and then you don't have to save that
- 75:59
- part of the matrix
- 76:00
- you can compute the second part
- 76:02
- So, and, and the, the idea is called flash
- 76:04
- flash attention, probably you have heard of
- 76:06
- which is a way to reduce the
- 76:07
- the, the memory
- 76:10
- footprint, in these, systems
- 76:14
- So
- 76:15
- yeah, I think that's, but, but, but, you know, like, some
- 76:18
- some of these kind of
- 76:18
- like, ideas do have a trade-off because when you reduce the
- 76:24
- for example
- 76:24
- if you change your architecture to make the
- 76:25
- dependency on T squared better
- 76:27
- for example, we'll discuss some of this
- 76:30
- on Wednesday. So if you change the architecture
- 76:32
- the dependency on T is better
- 76:33
- but your architecture is less expressive
- 76:35
- and then you may lose some
- 76:37
- performance, to some degree
- 76:39
- Okay, sounds good
- 76:41
- Yeah, thanks
- 76:47
- So I guess, you know, I think you have
- 76:50
- you know, Chris has talked about a linear model
- 76:52
- which is kind of like you have a
- 76:53
- model which is parameterized by theta
- 76:56
- and the input is X
- 76:57
- Let's just recall the notation so that we're
- 76:59
- on the same page about notation
- 77:01
- I think Chris is using Almost exactly the same notation
- 77:04
- you know
- 77:04
- it cannot be exactly the same because sometimes
- 77:06
- it's not a linear model
- 77:07
- right? So, but for, for linear model, recall that, you know
- 77:10
- if you have a linear model
- 77:12
- then the model is kind of like this
- 77:13
- right? You have theta h theta x
- 77:15
- which is equal to theta one
- 77:17
- x one plus theta two x two
- 77:20
- up to theta d x d
- 77:23
- where x is the input
- 77:25
- Which is a d-dimensional vector
- 77:27
- and theta is the parameters
- 77:28
- which are also a d-dimensional vector
- 77:31
- and once you have nonlinear models
- 77:33
- what does that mean?
- 77:34
- You know
- 77:34
- generally nonlinear just means that as long as
- 77:36
- this form is nonlinear
- 77:38
- then it's nonlinear. And
- 77:40
- and here we are mostly talking about nonlinear in the parameters
- 77:43
- because this nonlinear in the
- 77:45
- data is not nonlinear enough in some sense
- 77:47
- So I suppose you have something like
- 77:49
- theta one x one squared plus theta two x two squared
- 77:54
- so on and so forth
- 77:56
- So that dx d squared
- 77:58
- I think we didn't talk about this kind of models
- 77:59
- you know, it, you know
- 78:00
- bef- because we skipped them
- 78:03
- in some sense in the new version
- 78:04
- but this is
- 78:05
- not sufficiently nonlinear to be interesting because you can just
- 78:09
- define new data
- 78:09
- which is the square of the old data
- 78:11
- So suppose you say
- 78:12
- I'm going to replace this by z one
- 78:14
- and replace this by z two
- 78:16
- and then it still becomes a linear model
- 78:18
- And anyway, you are not learning the
- 78:20
- you are not in
- 78:20
- you are not trying to find out what are the
- 78:22
- you know, you are trying to learn the theta
- 78:24
- theta i's
- 78:24
- right? You're going to figure out the
- 78:26
- You don't really have to figure out
- 78:28
- there's no learning on ZI
- 78:29
- right? So whether Z is ZI or XI squared
- 78:32
- it doesn't really matter
- 78:33
- right? So that's why this is not exactly
- 78:35
- what we're talking about
- 78:35
- even though some people will say this is
- 78:36
- still a nonlinear model
- 78:37
- but
- 78:38
- this is not exactly what we are talking
- 78:40
- about. We are talking about cases like you
- 78:41
- have theta one squared
- 78:42
- for example, X one, plus theta two, maybe
- 78:46
- You know, I'm just giving an example
- 78:47
- something like this, right? Whatever nonlinear model you can write
- 78:49
- as long as it's nonlinear
- 78:50
- it's, it's called nonlinear model
- 78:52
- Okay, so, of course, you know, the
- 78:54
- the real model we are talking about is
- 78:56
- not that like this
- 78:56
- this is just one example
- 78:58
- so the model we are
- 79:00
- talk about is neural networks
- 79:01
- which is, more structured and more
- 79:03
- you know, you know, you know, you know, you know, specialized way
- 79:06
- Anyway, so, but before talking about neural networks
- 79:09
- let's talk about how do you learn nonlinear models
- 79:12
- and what's the kind of the framework
- 79:15
- this is in some sense
- 79:16
- mostly just recalling what Chris was talking about
- 79:19
- with a little bit twist about
- 79:20
- you know, the model
- 79:21
- So basically the model can be a general function
- 79:24
- as opposed to, a linear function
- 79:26
- So, let's just quickly recall some of this
- 79:29
- So let's say you have a data set
- 79:31
- I think Chris probably mentions the concept of data set
- 79:34
- which is, a finite data point with
- 79:37
- pairs of inputs and outputs
- 79:40
- So here, let's talk about supervised learning
- 79:42
- by the way
- 79:43
- you know, in other cases, you know
- 79:45
- we have data sets of different format
- 79:46
- but here, if you have a supervised learning data set
- 79:48
- you have a sequence of input and output pairs
- 79:51
- and let's say we have N of this
- 79:54
- and let's say we assume xi
- 79:57
- is in D
- 79:58
- And just recall that in this lecture
- 80:01
- in this whole course
- 80:02
- we are trying to use this notation where
- 80:04
- the superscript is for the indices of the examples
- 80:08
- and the subscript is for the
- 80:09
- the dimensions, the coordinates, right?
- 80:11
- So when I write x sub i
- 80:13
- this is the first coordinate of the vector xi
- 80:16
- and we have superscript is
- 80:17
- is index, is indexing one particular example
- 80:20
- Go
- 80:23
- ahead. Yeah, yeah, that's great
- 80:37
- So if you are only nonlinear in
- 80:41
- x by non-linear theta
- 80:42
- then you c-
- 80:43
- it means that you can replace the x by something else
- 80:45
- right? You can replace xi
- 80:47
- for example, in this example
- 80:48
- you can replace x1 squared by z1
- 80:51
- and then you just say z is my inputs now
- 80:53
- and then I have a linear model
- 80:55
- in both z and theta
- 80:56
- So fundamentally
- 80:57
- it's the same as linear model from a
- 80:58
- learning perspective
- 81:00
- but here you cannot do this
- 81:01
- right? You, you
- 81:02
- there is no way you can
- 81:03
- easily transform it into a linear model
- 81:05
- Actually, there's no way just fundamentally
- 81:07
- So that's why it's
- 81:08
- it's becomes, it requires different kind of like
- 81:12
- slightly different kind of techniques
- 81:13
- right? So some of the linear models techniques
- 81:15
- cannot work anymore
- 81:17
- Yeah
- 81:20
- yeah, you can write, you know, yeah, yeah
- 81:22
- This is just the one
- 81:24
- you know, you can even add more things
- 81:25
- you know, you can take more square roots
- 81:26
- you know, whatever, you can take squares
- 81:28
- you know, yeah. You
- 81:29
- you'll see some real examples
- 81:31
- Yeah
- 81:40
- In which cases, in this case?
- 81:45
- Okay
- 81:52
- Sorry. So you add one of these
- 82:00
- two?
- 82:03
- But then you change the model
- 82:04
- right? Suppose my model is really
- 82:06
- you know
- 82:06
- suppose my model assumption is just literally this
- 82:10
- what my argument was that you can replace
- 82:12
- X Y square by Z one
- 82:13
- so that you maintain the model's functionality
- 82:16
- but, but, but you can reapply the same techniques that
- 82:21
- that is useful for linear models
- 82:26
- Okay, sounds good. So, anyway, so, okay, so how do we
- 82:31
- so now let's just assume we have a general
- 82:33
- you know, okay, so let's talk about the framework
- 82:35
- So, and, And you have a
- 82:39
- let's say we have a regression problem
- 82:40
- I guess, you know
- 82:41
- Chris's lecture talked about regression versus classification
- 82:44
- And for regression problem
- 82:48
- let's say
- 82:50
- one of the
- 82:51
- the distinction between regression classification is that we
- 82:53
- have a regression problem
- 82:54
- the YI, the labels are
- 82:57
- in real number, right? These are real numbers
- 82:59
- And then your model
- 83:00
- let's say, is the H parameterized by theta
- 83:03
- which is a function that maps the input
- 83:05
- which is the, you know, RD to a real number R
- 83:10
- right? And
- 83:11
- you can have a cost function or maybe
- 83:13
- loss function
- 83:15
- I think these days people use the word loss function
- 83:18
- more often
- 83:19
- In the lecture notes
- 83:20
- we still use cost function
- 83:21
- you know, but most researchers understand
- 83:24
- what you are talking about
- 83:26
- so, so this is the most likely Of course
- 83:31
- there are also many different type of mean square
- 83:33
- average functions or loss functions you can
- 83:36
- you can define
- 83:37
- By the way, average loss and
- 83:38
- and cost is the same
- 83:40
- let's just use loss for all the cases
- 83:44
- and the loss function could be something like
- 83:45
- this
- 83:47
- With J for the ith example
- 83:49
- the loss function is a function of the
- 83:51
- parameter because the data is not something you optimize for
- 83:53
- right? It's not vari- It's not a variable
- 83:55
- it's given
- 83:56
- So
- 83:57
- so loss function
- 83:58
- for the j's example is yi
- 84:03
- minus h theta xi. You apply the data
- 84:06
- you apply the model to the data
- 84:07
- you get some prediction
- 84:09
- and you compare that with the real label
- 84:10
- and you take the square
- 84:13
- Right
- 84:14
- Something like this
- 84:15
- And then, you can also have like the
- 84:18
- the average loss
- 84:19
- which is something like one over n
- 84:22
- times the sum of
- 84:25
- all the losses
- 84:27
- So
- 84:28
- and then you try to minimize this function
- 84:30
- over the parameter theta
- 84:32
- So that's for regression. And
- 84:34
- let me give another example
- 84:35
- you know, I think you are
- 84:36
- you know, this, this is mostly just a
- 84:37
- re- like
- 84:39
- repeating some of the stuff we have discussed
- 84:41
- The only thing is here is just that
- 84:43
- this is a nonlinear model as opposed to a linear model
- 84:45
- right? So like, in crisp lecture
- 84:46
- I think this is like just a theta
- 84:48
- one X one plus X two
- 84:50
- so on and so forth
- 84:51
- And, I guess for multi-class classification
- 84:57
- we can also talk about single class classification
- 84:58
- but, which is actually in the lecture notes
- 85:01
- I think I just
- 85:03
- thought to give you an instance of time
- 85:04
- let's just only talk about multi-class classification
- 85:06
- Basically
- 85:07
- you just have to define the loss function
- 85:08
- the same way as we have done
- 85:10
- for, for linear models
- 85:12
- except that you replace the models by nonlinear
- 85:14
- functions. By the way
- 85:15
- I think in most cases
- 85:16
- you don't have to think about the superscript I
- 85:18
- because You know, this is just indexing examples
- 85:20
- right? So if you remove
- 85:21
- if you say, I only have one example
- 85:23
- X and one, and X and Y
- 85:25
- then everything without a superscript still works
- 85:28
- maybe I should just write in that notation
- 85:30
- to simplify it a little bit
- 85:32
- you know, so let's say you have
- 85:35
- so in this case
- 85:36
- the Y, you know
- 85:37
- I can write Y superscript I for the I's label
- 85:39
- or I can just
- 85:40
- use Y to indicate one label
- 85:42
- Let's say, suppose you have one label
- 85:44
- in Mathematica's application, you have, it's a discrete choice of labels
- 85:48
- right? One, two, K
- 85:51
- and here, I guess, you know
- 85:54
- you can say you have a model
- 85:56
- let's use H bar
- 85:58
- which is mapping the input X
- 86:01
- So the X is still in RD
- 86:03
- Think of this as an image
- 86:04
- think of this as some kind of labels
- 86:06
- for the image
- 86:07
- And then the
- 86:11
- which is mapping the input from to K numbers
- 86:14
- and these K numbers are often called logits
- 86:21
- This is the same largest in the
- 86:23
- in the linear case
- 86:25
- So, and, and then you say the
- 86:29
- which is largest, the predicted probability of each class
- 86:34
- of each class, for this example X
- 86:38
- it, for each class, let's say class J
- 86:41
- is the, this kind of like exponential form
- 86:44
- right? You look at the J's
- 86:47
- this is the J's output
- 86:49
- right?
- 86:50
- Index J means like this is the
- 86:51
- the J's output of this K-dimensional vector
- 86:55
- and then divided by
- 86:58
- the sum of the exponential of all the
- 87:02
- this is the so-called softmax
- 87:03
- So this is the so-called softmax
- 87:15
- sorry
- 87:18
- So this is the so-so-called softmax function of this
- 87:21
- h
- 87:22
- and then, so basically you say
- 87:27
- yeah, so basically your model is like this
- 87:29
- right? So this, this is your model
- 87:30
- so p of
- 87:32
- y is equal to j given x
- 87:35
- is this
- 87:36
- This is my parametric model for the prediction
- 87:41
- parameters by theta
- 87:43
- And then I need to compute a log likelihood
- 87:45
- so I'm going to use negative
- 87:48
- log likelihood
- 87:50
- Which is also what we have discussed before
- 87:52
- By the way, here
- 87:53
- this is also log likelihood model if you
- 87:55
- assume there's some Gaussian probability distribution
- 87:57
- I didn't really define
- 87:58
- go through that calculation
- 87:59
- So if you think of that your model
- 88:02
- your probabilistic model is something you add a
- 88:03
- Gaussian to the h theta
- 88:05
- and then you compute the log likelihood under that Gaussian
- 88:07
- then it's the same mean squared error
- 88:09
- Here, I'm just going through more details
- 88:11
- So, I define my probability distribution
- 88:14
- I compute log likelihood on the data
- 88:16
- so this is the minus log p of y given x
- 88:21
- and then my parameterized model
- 88:23
- then this is a minus log of this thing
- 88:32
- So, oh, sorry
- 88:33
- I think one subtle thing here is that this is the
- 88:37
- so this model
- 88:39
- Specifies what's the probability for every choice of J
- 88:43
- but now I have a particular choice of J
- 88:45
- which is equals to my label Y
- 88:47
- so that's why I need to plug in my Y
- 88:50
- so this is, exponential H theta bar X
- 88:54
- This is for the particular label I'm given
- 88:57
- the width index, the, the width corner of that
- 89:02
- logic vector divided by the
- 89:05
- the, the denominator is the same
- 89:07
- just the sum of the expansions
- 89:10
- Oh, I think I have some typo here
- 89:24
- and this is just called
- 89:26
- the entropy loss
- 89:28
- I think if you look at the multi-class classification case
- 89:30
- this is the so-called
- 89:32
- cross entropy loss
- 89:36
- You have many different form of it
- 89:38
- but this is the
- 89:39
- the cross entropy loss
- 89:41
- where the
- 89:45
- you can just write this
- 89:46
- which is theta bar
- 89:51
- So this is the logic
- 89:52
- this is the label
- 89:52
- and you can plug in the cross entropy
- 89:54
- loss. If you use PyTorch to implement it
- 89:55
- I think you can find a module
- 89:57
- which is just called cross entropy loss
- 89:59
- and you give it to the inputs
- 90:00
- and you can compute the loss
- 90:02
- but of, of course, the underlying math is this
- 90:06
- okay, this all sounds somewhat familiar to some degree
- 90:09
- Okay, good
- 90:10
- Any questions?
- 90:18
- is cross entropy
- 90:21
- oh, is there, so, so
- 90:24
- so what's the second part of the question?
- 90:29
- Yeah, I think this is the cross entropy between the
- 90:34
- The, the probability distribution defined by H theta X
- 90:41
- Okay, so I think here's the thing
- 90:42
- So if you
- 90:43
- so the question is why this is called
- 90:45
- cross-entropy loss
- 90:46
- I think if you
- 90:53
- how do I
- 90:53
- how do I write it in some sense?
- 90:55
- Like, I think if you have a
- 90:59
- let's say this distribution that's quite
- 91:02
- maybe P theta
- 91:04
- Okay
- 91:05
- So, and
- 91:07
- I think this cross-entropy loss is actually equals
- 91:10
- to the cross-entropy
- 91:15
- of between the true dis
- 91:18
- I guess you have to take
- 91:22
- if you take the expectation
- 91:25
- of this L cross-entropy loss
- 91:31
- Of this, H bar and Y
- 91:34
- and expecting, so, a Y from the true distribution
- 91:37
- from the true label distribution
- 91:39
- whatever you call it
- 91:40
- and then this is equals to a cross entropy
- 91:45
- between
- 91:47
- this distribution P of theta and a true distribution P star
- 91:52
- which is something like you
- 91:54
- I guess the cross entropy just
- 91:55
- is some
- 91:56
- is actually the probability of one distribution times
- 91:58
- the log of the probability of the other distribution
- 92:03
- yeah, I think the derivation is probably a bit long
- 92:04
- especially, if you're reading this language
- 92:06
- it's a bit cumbersome
- 92:07
- but, but this is a fundamental
- 92:09
- I think actually probably I have it in the lecture notes
- 92:11
- I, I forgot exactly
- 92:13
- yeah. Any other questions?
- 92:19
- Why, what is the one hot vector?
- 92:22
- You know, see, in
- 92:23
- in my case, you know, in my notation
- 92:25
- Y is just a discrete variable
- 92:28
- So that's why I
- 92:28
- you can see I have H theta bar
- 92:30
- theta X indexed by Y
- 92:32
- I
- 92:32
- Y is like two or three or four or K
- 92:35
- So, but if
- 92:37
- you can also encode Y as a one-hot vector
- 92:39
- which is the same thing
- 92:39
- right? So like
- 92:40
- then your math needs to be a little different
- 92:42
- right? Then this is gonna be the inner
- 92:43
- product between this vector times the one-hot vector
- 92:46
- Yeah. But, but here I'm just using Y as really a
- 92:49
- a discrete choice between one to K
- 92:59
- So, can you repeat the question?
- 93:01
- Sorry. So, can you repeat the question?
- 93:02
- Sorry
- 93:09
- Okay, so the question is why for the regression case
- 93:11
- it also is not like
- 93:12
- not, not, not likelihood, right?
- 93:15
- So I think the
- 93:17
- Let me try to see what I can
- 93:17
- do it on the fly
- 93:18
- I think it should be easy
- 93:20
- So I think the
- 93:21
- the modeling
- 93:22
- so if you make the modeling assumption that if you assume
- 93:26
- suppose you assume that the
- 93:27
- your modeling assumption is that Y is generated from
- 93:32
- say, H theta bar, theta X plus some Gaussian noise
- 93:36
- with epsilon is from N zero
- 93:39
- maybe sigma squared, something like this
- 93:43
- then
- 93:46
- the likelihood of Y given X is
- 93:50
- you know, I think one over
- 93:52
- I guess this is challenging my memory of this
- 93:55
- something like this exponential
- 93:58
- Y minus H theta X squared
- 94:03
- over two
- 94:06
- sigma squared, I guess, something like this
- 94:09
- and then you take the negative log likelihood
- 94:13
- So then this is a log of
- 94:15
- P Y given X is equals to
- 94:18
- there's some constant, you take the log
- 94:19
- you get a constant
- 94:21
- which doesn't really matter
- 94:22
- and you got, the log of this
- 94:24
- which is one over two sigma squared Y
- 94:27
- minus H theta X squared
- 94:30
- So then it's, it's kind of like this
- 94:32
- right?
- 94:32
- the scaling doesn't really matter because this
- 94:34
- like, it doesn't change the optimization function fundamentally
- 94:38
- Sounds good
- 94:44
- Any other questions?
- 94:47
- Okay, cool. So
- 94:53
- and, and in this case
- 94:54
- you know, here I only have like one example
- 94:57
- right? Eventually, I'm going to average over many examples
- 94:59
- right? So, I guess maybe just
- 95:02
- let me write here
- 95:02
- so
- 95:03
- If you continue here
- 95:05
- so eventually your loss
- 95:06
- you know, your real loss is something like the
- 95:08
- the final loss
- 95:10
- is the average over all the examples
- 95:14
- You take the cross entropy loss
- 95:19
- of each of the ex- examples X
- 95:22
- i and Y i
- 95:25
- So, and
- 95:26
- and each of these functions is a function of theta
- 95:28
- so basically your total loss is also a
- 95:30
- function of theta
- 95:32
- And now
- 95:33
- the next task is to optimize to find
- 95:35
- theta that makes the loss smaller
- 95:36
- right?
- 95:37
- The same
- 95:38
- I think we also introduced a little bit about gradient descent
- 95:41
- you know, like
- 95:44
- Newton method
- 95:45
- right? So these are all ways to optimize
- 95:46
- things
- 95:47
- So here you can't use
- 95:48
- Newton method because the function is now convex
- 95:50
- so that's why Newton's method is not guaranteed to work
- 95:54
- actually, no methods can guarantee to work for all non-convex functions
- 95:57
- but
- 95:59
- stochastic gradient descent seems to be a good
- 96:00
- idea. So just to quickly recall
- 96:02
- you know, you can use an optimizer
- 96:07
- so, so optimization
- 96:09
- I think one of the options is you
- 96:10
- take gradient descent
- 96:16
- So, so let's say, suppose you have a loss function
- 96:18
- let's just, summarize all the cases as this
- 96:21
- Let's be a little abstract
- 96:22
- So let's say each of the examples have one loss
- 96:24
- have the loss J I of theta
- 96:27
- and you have an average of these loss
- 96:29
- functions. Within descent means that you just say
- 96:32
- I'm gonna ignore the structure here
- 96:35
- I just treat this as a single loss function
- 96:36
- I, I take the update theta is updated to
- 96:40
- I think we use this notation to denote update
- 96:43
- something like eta times gradient of J
- 96:45
- of theta
- 96:47
- So this is gradient descent
- 96:49
- and how do you compute this gradient?
- 96:51
- You, you compute the gradient of each of the loss
- 96:53
- the gradient, sorry
- 96:54
- you compute the gradient of each of the
- 96:55
- loss with respect to parameters
- 96:56
- and you add them up
- 96:58
- Right, so that's gradient descent
- 96:59
- And for stochastic gradient descent
- 97:08
- so what you do is you say
- 97:10
- your first sample, say B
- 97:16
- let's see. So you sample
- 97:18
- I guess they are
- 97:18
- you know, maybe let's say
- 97:20
- the most vanilla version of it is you sample one item
- 97:23
- you sample
- 97:26
- J from one to N uniformly
- 97:34
- and then you run this
- 97:36
- you know
- 97:38
- so of, of, of course there's a for loop
- 97:41
- and you sample
- 97:44
- and then you say theta is updated to theta minus eta
- 97:48
- blah, blah
- 97:50
- The cost of the example J
- 97:52
- the cost of loss of example J
- 97:54
- theta
- 97:56
- And
- 97:57
- the kind of the justification here is that
- 97:59
- if you think about what this is
- 98:01
- so the difference here is that here you
- 98:02
- compute a gradient of the average loss
- 98:04
- and here you only compute a gradient of one single example
- 98:07
- the loss on one single example
- 98:09
- And the justification here is that you say
- 98:13
- Even though this is one example
- 98:14
- it's not actually the loss
- 98:16
- loss of the whole gradient
- 98:18
- but actually if you look at the expectation of the
- 98:24
- the gradient, the per example gradient
- 98:27
- and your randomness come from J
- 98:29
- right, which is sampled from uniform rate from one to N
- 98:32
- this is equals to the gradient of the
- 98:36
- the average gradient, the gradient of the loss function
- 98:40
- So, so in some sense
- 98:42
- this quantity is approximation of this quantity
- 98:45
- and the approximation is in the sense that
- 98:47
- this quantity in average is equal to this
- 98:49
- It's just that sometimes it's biased towards
- 98:51
- it's, you know, it's, it's never approx, you know, let's say
- 98:54
- suppose this quantity is like in this direction
- 98:56
- and this quantity is just a
- 98:58
- it can be any directions
- 98:59
- but if you average out all the possibilities
- 99:01
- you get the
- 99:02
- the original direction on the goal
- 99:04
- So in some sense
- 99:05
- you have like a much larger noise
- 99:07
- right? So like if you do gradient descent
- 99:09
- you, it's kind of like
- 99:10
- I guess maybe let me use two different
- 99:11
- colors
- 99:17
- So suppose you was grading this end
- 99:19
- this is the grading of the
- 99:21
- the average loss, and if you stochastic grading this end
- 99:24
- it's stochastic, you may go here
- 99:25
- you may go here
- 99:27
- or you may go here
- 99:28
- maybe you may
- 99:28
- sometimes you may even go in the reverse direction
- 99:30
- but in average
- 99:31
- all of these
- 99:32
- blue ones will average out in
- 99:34
- the same direction as the black one
- 99:36
- And, and so, and then if you have a trajectory
- 99:39
- right, so suppose the black one goes like this
- 99:44
- the blue one, because it's stochastic, you have noise, right
- 99:47
- so it will go look like this
- 99:48
- right?
- 99:49
- Maybe something like this
- 99:51
- Maybe like even, but, you know, but in average
- 99:53
- it's kind of roughly tracking the grading descent
- 99:55
- Of course, how much this tracks
- 99:57
- you know, how much these two things tracks over time is
- 100:00
- is a very complex kind of like a matter
- 100:02
- Like, it depends on how large the
- 100:04
- the step size is
- 100:05
- you know, how
- 100:06
- You know, kind of like
- 100:07
- depends on many kind of like subtle things
- 100:08
- but con-conceptually you
- 100:10
- you should expect that the blue one sum will track the
- 100:12
- the, the, the black one
- 100:14
- You know
- 100:14
- they are, the, the blue one
- 100:15
- your average is exactly equal to the black
- 100:17
- one in one step
- 100:18
- But the error sometimes can be amplified
- 100:20
- sometimes the error can be shrinked
- 100:22
- So it depends on the situation
- 100:24
- Ideally
- 100:24
- you want them to track to some degree
- 100:33
- I think you can use gradient descent as well
- 100:35
- So, okay, but I think, I think I had a question
- 100:36
- you know, it is a quick
- 100:38
- I think, it sounds like the
- 100:39
- the, the
- 100:41
- the question is really about why you use
- 100:43
- stochastic gradient descent. In what cases you want
- 100:44
- to use stochastic gradient? That is a great
- 100:46
- question. So
- 100:46
- why you even consider stochastic gradient descent?
- 100:49
- The reason is that computing this gradient is
- 100:51
- very costly each time
- 100:52
- because you have so many examples
- 100:54
- N could be, you know, in 2015
- 100:56
- it's kind of one million examples
- 100:58
- In 2026 now, I think it's kind of like
- 101:01
- you know, like one trillion tokens
- 101:03
- right? There's no chance you can compute All the gradients
- 101:07
- even completing one gradient is very costly
- 101:09
- let alone completing all the gradients
- 101:11
- So that's why you have to find an efficient way
- 101:12
- so, so that's why, you want to
- 101:15
- have a sampler which is less costly
- 101:17
- And, and, maybe just one, so, and also another reason you
- 101:22
- you can
- 101:23
- do this is that completing the full gradient sometimes is actually
- 101:29
- wasteful, because maybe it's
- 101:30
- it's the case that all of these individual gradients are actually
- 101:33
- they are correlated, they are highly correlated
- 101:36
- Right? So
- 101:37
- there's no reason to compute each of them to get
- 101:40
- understand which direction to go
- 101:42
- right? So you just only need a subset
- 101:43
- It's kind of like every
- 101:44
- example tells you to go that way
- 101:47
- why you have to compute all of them
- 101:49
- right? You just need a subset
- 101:50
- subset of them
- 101:51
- And another thing is that even this is not entirely true
- 101:54
- in many cases, like
- 101:56
- you don't really have to be very precise in this
- 101:58
- in this dynamics
- 101:59
- It's kind of like
- 102:02
- even you have some mistakes
- 102:03
- you know, it will be corrected later
- 102:05
- So in some, maybe another thing
- 102:06
- way to think about it is that for some cases
- 102:08
- like, for example, if you have a convex function
- 102:10
- right? So in some sense
- 102:12
- like, one way to do it is that you do this
- 102:15
- right?
- 102:16
- But sometimes if you kind of like
- 102:17
- overshoot, you know, or maybe you
- 102:19
- you go in the wrong direction
- 102:20
- still eventually you're going to go down
- 102:22
- It's, how do I say this
- 102:23
- like, you know, like basically, like even you have some perturbations
- 102:28
- and this is not exactly the right example
- 102:30
- but like even you have some perturbations in your dynamics
- 102:32
- then over time, because you are, every time you, you have a shared target
- 102:37
- eventually those kind of perturbations got kind of like washed out
- 102:40
- by the, by the
- 102:41
- by the shared force in some sense
- 102:42
- So I just wanted to ask on a
- 102:42
- non-convex function like a stochastic
- 102:43
- and I believe that this is very stochastic
- 102:46
- are we still evolving through the intersect or
- 102:47
- would it not be? Yeah
- 102:56
- I think that's a
- 102:57
- that's a great question. So
- 102:58
- one of the theory for why stochastic
- 103:00
- is good is that it can
- 103:02
- work better on, on non-convex functions, and maybe let me
- 103:06
- let me first explain what you said
- 103:07
- but, but actually it's not entirely right
- 103:09
- So, so I think, so basically
- 103:13
- suppose you have a non-convex function in one D
- 103:15
- right, let's say, right? So like this
- 103:17
- So if you start here
- 103:19
- you, you keep going down
- 103:22
- This green incentive means you go downhill in
- 103:23
- some sense. So you go to the directions
- 103:25
- which is the going downhill in the
- 103:27
- sometimes it's called steepest descent
- 103:29
- It's basically you are finding the
- 103:30
- the direction which is going downhill the fastest in some sense
- 103:33
- right? So you just go here and you
- 103:34
- get stuck here because you can never leave this
- 103:36
- this place, right? Here the green is zero
- 103:38
- right? The, the, the curvature is zero
- 103:41
- and, and, and
- 103:45
- I think what you are trying to say
- 103:46
- is that if you do stochastic winning sense
- 103:48
- because there's some noise
- 103:49
- maybe even though you want to go this way
- 103:51
- maybe the first step you just
- 103:53
- because of the noise
- 103:54
- you go the other way and you jump
- 103:55
- to another local minimum
- 103:57
- I think this, is plausible, but, generally speaking, you know
- 104:01
- of course the theory here is kind of very limited
- 104:03
- but I think
- 104:05
- the community don't believe that
- 104:07
- this is the case. This is why stochastic
- 104:09
- winning sense is helping
- 104:10
- I don't think people believe that stochastic winning
- 104:12
- sense
- 104:12
- can help you to jump out of a local minimum
- 104:15
- however, on the other hand
- 104:17
- we don't even believe that there are
- 104:19
- Many bad local minimum
- 104:20
- So, so in one
- 104:21
- it's actually very easy to have local minimum
- 104:23
- but in high dimensional cases
- 104:25
- it's not that trivial to have that many local minimum
- 104:28
- So, so, so
- 104:29
- I think at least one in the actu- actually was
- 104:31
- I was involved in developing some of the theory
- 104:33
- Of course, nobody really knows exactly whether they are right
- 104:36
- but, the, the common, let's say hypothesis or
- 104:40
- or hypothesized consensus in some sense is that
- 104:43
- there are very limited number of bad local minimums in this
- 104:47
- in this high dimensional function you're optim-optimizing
- 104:49
- Maybe there's even no bad local minimum
- 104:51
- All the local minimum are actually global minimum
- 104:53
- It's kind of like if you think about
- 104:54
- you are skiing in a mountain
- 104:57
- like
- 104:59
- I think if you just randomly try to go down scales
- 105:02
- skiing downhills in 2D
- 105:03
- right?
- 105:04
- It's probably
- 105:04
- you're probably going to go back to one
- 105:05
- of the parking lots
- 105:06
- which is one of the global minimum
- 105:08
- right? So like, you know, of, but in 2D, I
- 105:10
- I think it's a bit easier to have a
- 105:12
- a local minimum. You probably go back country
- 105:13
- and you got stuck somewhere
- 105:14
- That's possible, but, you know
- 105:16
- but you always find some river going down
- 105:18
- you always find some valley going down eventually
- 105:20
- I think, you know
- 105:21
- Not, not, you know
- 105:21
- I'm just going to give an analogy
- 105:22
- You know, it's not exactly true in 2D
- 105:24
- but in, in, in high dimension, it's
- 105:25
- it's more likely to fi
- 105:26
- always find a way to
- 105:28
- to go down, downhill in some sense
- 105:33
- Yeah, I think, I, I don't
- 105:47
- think there's a
- 105:48
- a very simple way to say it
- 105:50
- I think
- 105:53
- It's not, it's not universally true
- 105:55
- right? So like, so in some senses
- 105:56
- I think maybe the
- 105:57
- you know, if I need
- 105:58
- I can only use one or two sentences
- 106:00
- I think it's kind of like just in high dimension
- 106:03
- it's always, there's always some ways to
- 106:05
- like the local minimum is your Hessian at that point
- 106:07
- it's just all, in all directions positive semi-definite
- 106:11
- That's a very hard thing to satisfy
- 106:13
- So like, so, so, so you find the local
- 106:15
- you, you find, a station, stationary point where the
- 106:18
- the gradient is zero
- 106:19
- and you look at the Hessian
- 106:20
- the Hessian is kind of like
- 106:23
- always have some direction which is negative
- 106:26
- So, and I think, I
- 106:27
- I was thinking about how to formalize this
- 106:29
- you know, like, so for example, if you think about it
- 106:31
- the, the gradient zero is already a pretty strong constraint
- 106:33
- right? You suppose you have
- 106:34
- you already have like
- 106:35
- suppose you have a d-dimensional function
- 106:36
- and you
- 106:37
- add d constraints that
- 106:39
- each of the dimension of the gradient is zero
- 106:41
- that's already restrict yourself a lot
- 106:43
- And if you add another thing
- 106:45
- which is the Hessian is positive semi-definite
- 106:48
- I think that just makes the
- 106:49
- the number of points much smaller
- 106:51
- Yeah, and, and in some cases you can compute
- 106:53
- In, in one of my papers
- 106:54
- I think we try to compute the number
- 106:55
- of local minimums by using this
- 106:57
- I think in, in differential geometry
- 106:59
- there's something called Catswright's formula
- 107:01
- And actually not only my papers
- 107:02
- but many other people's papers
- 107:04
- So in some, certain cases you can compute
- 107:05
- or you can count
- 107:06
- and you find out that
- 107:07
- oh, all the local minimum are actually just those
- 107:09
- number of local minimum is the same as
- 107:10
- number of global minimum
- 107:12
- But, but generally it's very hard to count
- 107:15
- Any other questions?
- 107:21
- Okay, sounds great. So, okay, so going back to here
- 107:24
- so, so that's the algorithm people do
- 107:26
- of course, practically speaking
- 107:27
- you don't also just only sample one example each time
- 107:31
- because sample, sampling one example, you know, Suddenly all, you know
- 107:34
- using all the examples is too inefficient because
- 107:36
- you have to complete all the gradings
- 107:38
- doing one example, the, the downside is that it's too noisy
- 107:42
- right? So sometimes you want to have like a
- 107:43
- a, a trade-off, right? So you
- 107:45
- you want to
- 107:46
- have a few examples and so that you can
- 107:49
- you can be a little bit less noisy
- 107:50
- And another thing is that
- 107:52
- which probably we won't go
- 107:53
- go in detail here
- 107:54
- but another thing is that
- 107:57
- when you have, one example, your GPU allocation is too
- 108:00
- too low. You know
- 108:02
- not, it's not always the case
- 108:03
- but in some cases
- 108:04
- using
- 108:05
- too few of examples make your GPU allocation
- 108:06
- too low. For example
- 108:07
- if you have like ten thousand GPUs
- 108:09
- and you just only complete one example of
- 108:11
- one grading each time
- 108:12
- then most of the GPUs are actually idle
- 108:14
- So
- 108:15
- so that's why you need a so-called batch grading descent
- 108:17
- Oh, sorry. like I think it's called
- 108:19
- it's also still called stochastic gradient descent with
- 108:21
- some kind of like small batches
- 108:24
- it's
- 108:24
- I think HDD just means like
- 108:27
- both the version with batch size one or
- 108:30
- whatever bigger batch. So when you have a bigger batch is
- 108:32
- you know, you can imagine what it is
- 108:34
- Basically, you just say, I'm gonna
- 108:37
- with batches
- 108:40
- I guess let's call it HDD
- 108:42
- So I think you just
- 108:43
- change the algorithm to be sample
- 108:46
- say, B, examples from one to N
- 108:53
- and then, then you say
- 108:57
- theta is equal to theta
- 109:00
- minus eta gradient
- 109:15
- Okay
- 109:16
- Sounds good. So, and, okay, so this is just
- 109:19
- a general introduction to the framework for the
- 109:22
- supervised learning
- 109:25
- and actually in some of the unsupervised learning
- 109:26
- it's the same thing. You define loss function for each examples
- 109:28
- and you average them. The only question is
- 109:30
- when you have unsupervised learning later
- 109:33
- that we'll cover, you don't have the y's
- 109:35
- So
- 109:35
- but still the kind of the framework is
- 109:36
- the same. You define the loss functions
- 109:38
- but the loss functions then involve y
- 109:40
- and then you have average loss functions
- 109:41
- and you run optimizers
- 109:43
- you can choose different type of optimizers
- 109:47
- so, and then pretty much there are two questions
- 109:51
- So, we're going to address that
- 109:53
- So, so the two remaining questions
- 109:56
- question one is that what is this function h theta
- 109:58
- right?
- 109:59
- So we haven't, right, so this whole framework
- 110:01
- you know, works for every function h theta
- 110:03
- and, or h theta bar
- 110:05
- I think I use bar just because
- 110:08
- it's a little bit different
- 110:08
- no
- 110:10
- With, with all bar register
- 110:11
- I just want to distinguish it from the regression case
- 110:13
- but they mean the same thing
- 110:15
- And so, and the question
- 110:17
- so question one is how do we define H theta
- 110:19
- and question two is how do we compute
- 110:22
- how to
- 110:25
- compute the gradient
- 110:27
- the gradient J on each of the example
- 110:31
- And once you know how to compute the
- 110:32
- gradient for each of the example
- 110:33
- you can compute the gradient for all the
- 110:34
- examples or batch examples
- 110:37
- by averaging the gradient for every examples
- 110:39
- So, and that's called
- 110:41
- so this is called back propagation
- 110:43
- The algorithm for this is called back propagation or auto-differentiation
- 110:46
- which is the, the topic for next lecture
- 110:48
- And in the rest of this lecture
- 110:50
- I'm going to define a bunch of
- 110:51
- H theta that people are using
- 110:53
- mostly for like, more classical cases, like vision cases, like superserving
- 110:59
- and, of course, for language case
- 111:02
- you can have even more complicated
- 111:04
- nonlinear functions like transformers
- 111:07
- Okay
- 111:08
- So, okay. So that's the plans
- 111:12
- I'm going to start with
- 111:13
- you know, the very basics, like new networks
- 111:16
- So
- 111:22
- so
- 111:22
- we are still using these kind of housing
- 111:23
- examples
- 111:25
- I know that this example is probably too old
- 111:27
- like it's 20 years old
- 111:28
- So, so let's say, suppose you have like
- 111:30
- to motivate, the, the, the, the introduction of like nonlinear models
- 111:34
- suppose you have like size and
- 111:36
- and, like this is the price
- 111:39
- right?
- 111:42
- That's housing price prediction
- 111:47
- So, you know, if you look at the real data
- 111:49
- most likely something like this
- 111:52
- Maybe some, I don't know exactly what it would look like
- 111:54
- but at the beginning
- 111:55
- it doesn't increase that much as the size goes up
- 111:57
- and then eventually becomes linear
- 111:59
- Let's
- 111:59
- let's say this is the case we will
- 112:00
- see in the data
- 112:01
- right?
- 112:02
- So, you know, if you use a linear model
- 112:03
- and you're going to fit something here
- 112:05
- then you're not be accurate enough
- 112:06
- So that's why, you probably have to have a nonlinear model
- 112:09
- which is something like this
- 112:11
- So you want your model to be able
- 112:12
- to output a curve like this
- 112:14
- Of course
- 112:15
- you don't really know which line is this
- 112:16
- It could be like a
- 112:17
- you know, a probably without seeing the data
- 112:19
- you can have multiple kind of like curves like this
- 112:21
- but at least you should allow
- 112:23
- the model to generate a curve like this
- 112:24
- as opposed to only generating a linear function
- 112:28
- So, and, and what this function is
- 112:30
- right? So this function
- 112:33
- so is actually of this form
- 112:37
- So this blue function is actually of the form
- 112:41
- it's a max of say
- 112:46
- wx plus b, which is a linear function
- 112:49
- and zero
- 112:51
- I guess, you know, especially if you
- 112:54
- Suppose this is zero
- 112:55
- then you just
- 112:56
- maybe I say this is max
- 112:58
- wxp and zero plus c
- 113:02
- So
- 113:03
- max of
- 113:03
- wx plus b
- 113:04
- is kind of like a function like this
- 113:08
- And then if you
- 113:09
- With the... Sorry, this one is like this
- 113:11
- And then if you plus c
- 113:13
- you can shift it
- 113:14
- above the, above or, or below a little bit
- 113:19
- So basically, you just cut off everything that is below zero
- 113:21
- which makes sense because
- 113:23
- you know, for housing price, you know, it cannot be negative
- 113:25
- you know, it has to be positive
- 113:28
- so
- 113:32
- Yeah, okay
- 113:33
- So, and, and, so this actually
- 113:37
- this introduced a primitive that is used very
- 113:39
- often in deep learning these days
- 113:40
- which is this function
- 113:42
- which is max
- 113:46
- of t and zero
- 113:48
- So, I think in, in mathematics, people call it this
- 113:51
- like, people write this, something like
- 113:55
- like this, but, but for us, we just call this value
- 113:58
- I think this is the
- 113:59
- the standard in machine learning
- 114:02
- So, this function is called rectified
- 114:03
- rectified linear units
- 114:06
- which is literally just this function where you
- 114:08
- have linear functions
- 114:10
- on the positive plane and then zero
- 114:12
- on the other side
- 114:13
- So, zero and then linear
- 114:15
- and linear with slope one
- 114:17
- So, this is a very important primitive
- 114:19
- and then you can write this function as
- 114:21
- you know
- 114:22
- this function can be written as
- 114:24
- value of t maybe plus c
- 114:26
- something like this
- 114:28
- Oh, sorry, value of
- 114:32
- Guys
- 114:32
- this one will be written as value
- 114:34
- of W X plus B plus C
- 114:37
- So basically you introduce
- 114:39
- so, you know, it's, see, if you don't have the value
- 114:41
- then it's basically a linear function
- 114:42
- But if you have the value
- 114:43
- then you somehow change the shape a little bit
- 114:46
- And
- 114:47
- and also this value is often called activation
- 114:49
- function
- 114:52
- I think the reason for the word activation is because in
- 114:55
- there's some kind of like
- 114:56
- connections to, biological neural networks
- 114:59
- I, I think in biology
- 115:01
- you know, like, in your brain, in our brain, the neurons kind of like
- 115:06
- how the neurons are fired up is a
- 115:07
- nonlinear function of the stimulus it receives from other neurons
- 115:11
- So, and the, the, it's not exactly this
- 115:14
- but it's something like this
- 115:15
- right? So like, when it's passing from threshold
- 115:17
- you got, the neurons got fired up
- 115:18
- and
- 115:21
- and, and below that threshold
- 115:22
- the neurons just doesn't do anything
- 115:24
- So, so that's why it's called
- 115:26
- activation, right? If it's fired up
- 115:28
- So, but, but I think in biology
- 115:30
- you know, I'm not a expert there
- 115:31
- but, you know, based on my
- 115:35
- knowledge of it, like the, the, the threshold, the function is, is
- 115:37
- is never exactly like this
- 115:38
- Sometimes it's like a sigmoid
- 115:40
- and that's why sigmoid was used at the
- 115:42
- very beginning of deep learning
- 115:45
- and, but
- 115:47
- but then later on people realized that this is actually
- 115:50
- this value is actually a better idea
- 115:52
- for, for neural networks for various computational reasons
- 115:56
- which we'll probably discuss
- 115:58
- in a bit
- 115:59
- Anyway, so for now, I think that's kind of like
- 116:02
- just to remember this concept
- 116:04
- So, so actually we have already defined
- 116:08
- a one-dimensional neural network. So basically this is
- 116:09
- a neural network with one in- one-dimensional input
- 116:12
- and a few parameters
- 116:14
- So that's already a neural network
- 116:15
- So then the rest of the
- 116:17
- the, the, for, for like ten, ten, fifteen minutes
- 116:20
- we are just going to generalize this to multiple
- 116:24
- higher dimensions, you know, like, and
- 116:25
- and you apply this multiple times
- 116:27
- and that will make it a
- 116:28
- a more complex neural networks
- 116:30
- And then we're going to even build more
- 116:31
- on top of that
- 116:33
- So
- 116:35
- okay, so next, what do we do?
- 116:36
- So, so this is only one-dimensional input
- 116:39
- right?
- 116:40
- And
- 116:40
- so now let's try to do a
- 116:42
- higher dimensional case. Let's say
- 116:43
- suppose you
- 116:44
- you have a data with dimension D
- 116:48
- And then how do you kind of like
- 116:49
- imitate this and make it nonlinear?
- 116:51
- So the way to do it is that
- 116:52
- you say I have X in D
- 116:54
- and then I have a parameter W
- 116:57
- which is also in the dimension D
- 116:59
- And often this is called weight
- 117:03
- vector, and actually eventually it will be a weight matrices
- 117:06
- so that's why weight and matrix
- 117:07
- you know
- 117:08
- associate together. Now weight vector
- 117:10
- is a rare cases
- 117:11
- And then you also introduce another number B
- 117:14
- this is called bias
- 117:17
- And then you define your model like this
- 117:22
- Say ReLU of W transpose X plus B
- 117:26
- This is the inner product between two vectors
- 117:28
- and you get the scalar out of it
- 117:30
- And then you add another bias
- 117:31
- which is still a scalar
- 117:32
- so the output of this is a scalar
- 117:36
- And so then basically we have like a
- 117:40
- high-dimensional inputs and low-dimensional outputs
- 117:42
- This is
- 117:43
- this is actually called a one-layer network
- 117:44
- So basically you apply this value
- 117:47
- once
- 117:48
- So, and, and now, let's try to kind of like
- 117:53
- stack multiple, so this is one neuron
- 117:56
- actually, because you, you apply one activation in some sense
- 117:59
- it's one neuron, and then you can have multiple neurons
- 118:02
- So, I mean
- 118:03
- one of the typical examples I used to give
- 118:04
- you know
- 118:04
- maybe this is not exactly the best example
- 118:06
- these days
- 118:07
- For example, let's say, suppose you think about the housing price
- 118:10
- right?
- 118:10
- So it probably depends on a lot of
- 118:12
- intermediate features
- 118:13
- Suppose housing price depends on
- 118:15
- for example, maybe, okay
- 118:17
- maybe let me give you the full example
- 118:18
- just to, to slow down a little bit
- 118:20
- Let's say
- 118:20
- suppose you have this cases where X1 is the size
- 118:25
- X2 is the, the number of bedroom
- 118:31
- and X3 is, let's say, the zip code
- 118:35
- and maybe X4 is
- 118:37
- let's see what's Maybe the
- 118:44
- I don't know what else for
- 118:46
- as far as my nose means
- 118:47
- But anyways, this is your raw inputs
- 118:48
- right?
- 118:49
- So, so, maybe you can think of like
- 118:52
- there are some intermediate things that
- 118:53
- that actually matters more than this raw inputs
- 118:57
- Let's say you have intermediate
- 119:01
- quantities that you
- 119:03
- quantities, like for example, you say
- 119:09
- walkability
- 119:12
- right?
- 119:13
- How, how easy to walk around the house
- 119:16
- right? Maybe there's another one
- 119:17
- which is like the school quality
- 119:21
- So these are, in some sense
- 119:23
- variables that are not present in your original data
- 119:25
- but they matter a lot to the
- 119:27
- the price of the house
- 119:29
- right?
- 119:30
- And, and each of this
- 119:31
- you know, maybe let's call this A1
- 119:32
- this is A2
- 119:33
- each of this is actually a function of
- 119:35
- X1 up to X4
- 119:36
- which can be computed as a nonlinear function
- 119:39
- So
- 119:39
- so you can imagine that A1
- 119:41
- is probably some kind of like a nonlinear function of
- 119:45
- you know
- 119:47
- maybe value of some of these linear combinations
- 119:49
- of this thing
- 119:54
- And you can imagine maybe A2 is also
- 119:56
- a some nonlinear function of this
- 119:59
- of, maybe I say, say the
- 120:02
- I'm using notations in a pretty
- 120:05
- sloppy way just to give you example
- 120:06
- you know
- 120:07
- finally we're going to define things more formally
- 120:09
- And then you can say
- 120:10
- oh
- 120:10
- maybe my final
- 120:11
- Y should be a linear combination of these
- 120:13
- guys. Maybe I have another parameter
- 120:15
- say the nine, A1 plus say the ten
- 120:18
- A2, something like this
- 120:20
- So then this way you
- 120:21
- you kind of like stack
- 120:22
- two layers on top. You have
- 120:23
- two neurons and you stack another linear functions
- 120:25
- on top of it
- 120:26
- So in some sense you are applying this recursively
- 120:30
- So
- 120:30
- so that's kind of like the fundamental idea
- 120:32
- on how do you kind of compose
- 120:34
- So basically you apply this nonlinear functions
- 120:37
- you know, you apply a linear function
- 120:38
- you apply some nonlinear functions
- 120:40
- and then you get some articles
- 120:41
- and then you apply another linear version
- 120:43
- and you can probably apply another nonlinear function here
- 120:46
- and you keep compose
- 120:47
- So that's the, that's the rough idea
- 120:51
- but let's try to make it a little more formal
- 120:54
- so that it's probably more obvious
- 120:59
- Let's see
- 120:59
- what's the right way for me to
- 121:02
- erase this stuff?
- 121:05
- I think we can probably start from the
- 121:06
- left
- 121:22
- So one of the ways that people
- 121:23
- at least used to draw this kind of things
- 121:25
- you know, at least, ten years ago was that they draw
- 121:29
- this kind of like inputs like this
- 121:31
- So suppose you have X one up to X D
- 121:34
- and then you say
- 121:35
- have some intermediate variables A one up to A D
- 121:39
- maybe A M
- 121:41
- and then each of the intermediate variable depend
- 121:42
- on all of this
- 121:44
- and they have some dependencies here
- 121:46
- So let's say A
- 121:46
- M depends on all of the input variables
- 121:48
- and you just draw this kind of edges
- 121:50
- and then you can say you have another
- 121:51
- layer of intermediate variables
- 121:54
- and they connect to each other
- 121:57
- So these drawings are
- 121:57
- you know, once you know what they are talking about
- 121:59
- the drawings are very useful
- 122:01
- but, when you don't really know what they are talking about
- 122:04
- it's kind of like confusing
- 122:05
- So what I really mean here is that
- 122:07
- you know, the way you do it is that you say
- 122:09
- first, let's say, let's do only one layer
- 122:12
- maybe just
- 122:13
- formally. So we have X one up to X D
- 122:16
- right? So then Let's have A1
- 122:19
- The first inconvenient variable
- 122:22
- is defined as maybe value of W
- 122:28
- say, one X one plus B one
- 122:31
- where W one is a vector
- 122:33
- even though I have an in-index one
- 122:34
- but it's the, the first, first vector
- 122:37
- W one transpose
- 122:39
- And X, so X, X is here
- 122:41
- X is a D-dimensional vector
- 122:43
- W one is a vector
- 122:44
- so the inner part together
- 122:45
- you get a scalar
- 122:45
- and you add a scalar B one here
- 122:48
- That's how you get to A one
- 122:50
- which is kind of like what I wrote there
- 122:52
- like A one
- 122:53
- So I use W just because I think
- 122:56
- people use W and B to distinguish the
- 122:58
- the weight matrices and the biases
- 123:01
- And then you have A two
- 123:03
- which is a value of
- 123:05
- W two transpose X plus B two
- 123:09
- where W two is
- 123:12
- R D and B two is in R
- 123:14
- And they have a bunch of this
- 123:15
- let's say you have like
- 123:18
- Up to a n
- 123:20
- Let me see where the n is my
- 123:21
- notation here
- 123:22
- It's very easy to mess up the notation
- 123:24
- in this kind of cases
- 123:25
- Yes
- 123:26
- a n is value of w n
- 123:29
- transpose x plus b n
- 123:32
- And that's how we define a
- 123:33
- a one up to a n
- 123:35
- And each of the a i depends on
- 123:37
- all of the x i's in this particular
- 123:38
- way
- 123:39
- And then you can say
- 123:40
- I'm going to apply another
- 123:42
- my output will just be a linear combination
- 123:44
- of all of these a i's
- 123:46
- Let's say the output is
- 123:48
- some linear combination of the a's plus another
- 123:54
- plus another biases
- 123:55
- So here is the thing
- 123:56
- How do I write a linear combination of
- 123:58
- the a i's? You know
- 123:59
- I can write something like
- 124:00
- say, the one a one
- 124:02
- so on and so forth
- 124:02
- But if I always kind of expand this
- 124:04
- it's too complicated
- 124:05
- So what I'm going to do is I'm
- 124:05
- going to write a vector
- 124:07
- let's say w transpose a plus another b
- 124:11
- But he realized that
- 124:12
- you know, when I write this
- 124:13
- you know
- 124:13
- this W needs to be distinguished from other Ws
- 124:17
- So
- 124:18
- and that's why
- 124:19
- I think I'm going to use this
- 124:22
- two to
- 124:23
- say that this is the
- 124:24
- W on the second layer
- 124:26
- In some sense, these are the Ws on the first
- 124:27
- so there is another
- 124:30
- So all of these AIs are combined to H theta X
- 124:32
- and that's why people write this is the
- 124:34
- W on the second layer
- 124:36
- and these are the Ws on the first layer
- 124:38
- So, so I'm going to write a
- 124:40
- another bracket W A one
- 124:44
- something like this, to distinguish the
- 124:49
- the first layer and the second layer
- 124:52
- So, and this W two
- 124:55
- Super square bracket two is a vector
- 124:58
- of the same dimension as A
- 124:59
- not of the same dimension as X anymore
- 125:01
- because you have to compile
- 125:02
- you have to inner product with A
- 125:04
- So this is a in dimension RM
- 125:06
- because F, M entries here
- 125:09
- and B is always a scalar
- 125:15
- So that's how you define a two-layer network
- 125:17
- So you have a lot of image
- 125:20
- and, and you can see this notation is kind of messy
- 125:22
- right? So like, you know, you probably shouldn't, you know
- 125:24
- when you have like too many super square up
- 125:27
- subscripts, at some point it
- 125:28
- it's kind of like too messy
- 125:30
- So that's why for most of the cases
- 125:32
- you know, some cases you have to live
- 125:33
- live with these types of notations
- 125:36
- so
- 125:36
- but in most cases you can simplify even further
- 125:39
- You say, actually if I write everything as
- 125:42
- as, matrices and vectors
- 125:43
- I don't have to really think about each of these
- 125:46
- the details. So basically I just write
- 125:49
- A as a vector in
- 125:51
- this is a vector in RM
- 125:53
- You can write as value
- 125:54
- of a matrix W times X plus B
- 125:59
- and here W and B are a little bit different
- 126:02
- okay, so W
- 126:03
- I guess I need to write W
- 126:06
- one and B one
- 126:10
- So this is a matrix W one
- 126:13
- If you want to look
- 126:14
- understand the, the correspondence between this notation and this notation
- 126:18
- so W one is this matrix of dimension M by D
- 126:25
- and each row is W one
- 126:29
- one transpose up to W M
- 126:35
- One transpose. Right, so you put all of this row vectors
- 126:38
- because once you transpo... By the way
- 126:40
- all the vectors are supposed to be column vectors
- 126:42
- if you don't do any transpose
- 126:43
- You know, if you really don't understand this
- 126:45
- you, you should try to understand the
- 126:47
- Like, at least the one
- 126:48
- you know, depending on the situations
- 126:50
- right? When, like, but in many cases
- 126:52
- it's good to keep in mind
- 126:53
- like
- 126:54
- whether this vector is a row vector and
- 126:55
- column vector
- 126:56
- So all of the
- 126:57
- these are column vectors
- 126:58
- and then you can transpose this
- 127:00
- into row vectors, so this is a row vector
- 127:02
- and you put all of them
- 127:03
- stack them into a matrix
- 127:06
- and
- 127:06
- and then this B1
- 127:09
- is just the vector of other scalars
- 127:16
- But in many cases
- 127:17
- you actually don't have to
- 127:18
- necessarily remember this correspondence. You just remember this is fine
- 127:21
- right?
- 127:22
- You have a vector in your multiplication matrix
- 127:23
- you get a vector
- 127:24
- and then you add a vector bias
- 127:25
- and you take value entrywise
- 127:27
- Okay, that's probably important. So value applied to a vector
- 127:30
- T, is the same as you applied entrywise
- 127:40
- So, so in many cases
- 127:41
- you just say, oh, one matrix application
- 127:43
- one, layer is just a multiplication matrix
- 127:45
- add a bias vector
- 127:47
- and apply value
- 127:48
- and you get a vector out of it
- 127:50
- So
- 127:50
- you don't have to necessarily understand all the details
- 127:52
- You know, sometimes you do, but
- 127:53
- but most cases these days you don't have to
- 127:55
- So
- 127:58
- and
- 128:01
- And if you count how many parameters are there
- 128:03
- right, like this is M by D
- 128:04
- this, this is the dimension
- 128:06
- So M by D
- 128:07
- right, because the in, your input dimension is D
- 128:09
- the output dimension is M
- 128:10
- and in the matrix that turns the D
- 128:11
- dimensional input to M dimensional outputs
- 128:14
- that's why you need M times D
- 128:16
- parameters
- 128:17
- And then this guy has M parameters
- 128:19
- So, so the total parameters
- 128:24
- parameter count is, M D plus, M
- 128:30
- So D is the input dimension
- 128:31
- M is the output dimension
- 128:32
- and D, this is out, output dimension
- 128:34
- This is B one
- 128:42
- yes
- 128:45
- And this B one is this B one
- 128:48
- So
- 128:56
- any, any questions?
- 128:58
- And I think, you know
- 128:59
- eventually, actually, some
- 129:01
- many of these details don't even matter if
- 129:02
- you don't care. So
- 129:04
- for example, you can even just write this
- 129:07
- like
- 129:15
- for example, you can just say
- 129:16
- oh, I, I even simplify notation here
- 129:19
- you can say
- 129:23
- I think you can say
- 129:23
- let's just, say this is a matrix multiplication
- 129:32
- So let's say you have a
- 129:33
- let's say you have a matrix multiplication
- 129:35
- with W and B
- 129:37
- So W is the weight matrices
- 129:39
- which is this W
- 129:40
- and B is the
- 129:43
- the, the bias, and you apply Z
- 129:45
- and, and this is just equals to WZ plus B
- 129:49
- And, and, and then you can say I'm going to apply
- 129:54
- a sigma
- 129:59
- Sorry, sig-sig, let's say sigmaism, I guess maybe I didn't
- 130:02
- I, probably I go a little too fast
- 130:04
- I see
- 130:07
- So maybe just, before I talk about that
- 130:11
- let's, I think I already allude
- 130:13
- allude to it, so
- 130:15
- this value here can be used also replaced to other
- 130:19
- nonlinear activations
- 130:25
- So you can replace
- 130:34
- so the value was
- 130:36
- you know, some other nonlinear activations
- 130:38
- and all the other activations are functions
- 130:41
- that maps R to R
- 130:43
- so it's always a scalar function
- 130:45
- and, but, you know
- 130:47
- you can apply this to vectors by entrywise applying it
- 130:51
- applying entrywise
- 130:52
- and other activations could be like sigmoid
- 130:58
- sigma looks like this
- 131:01
- you know, let me
- 131:02
- let me just save some time to now define the
- 131:04
- the equations, you know
- 131:05
- you can read the electrons to see the
- 131:07
- the exact equations. There are some
- 131:09
- there are also tan
- 131:10
- tan h, which looks like
- 131:14
- going through zero and like this
- 131:16
- So all of these activations are generally
- 131:19
- increasing functions, but not always the case
- 131:22
- I think there's also Leaky value
- 131:23
- you know
- 131:24
- there are a bunch of different activations functions
- 131:26
- For example, Leaky value looks like
- 131:29
- none of this is actually used that much
- 131:31
- if you want to see what's the
- 131:33
- use the, in the latest network
- 131:35
- I think you have to check the electrons
- 131:37
- I just added a few
- 131:38
- which I don't have it on my notes
- 131:40
- Like, but there's also functions like this
- 131:42
- you know
- 131:44
- anyway, generally it's kind of like some almost increasing functions
- 131:47
- and often on the
- 131:49
- on the left-hand side of the zero is
- 131:51
- kind of close to zero
- 131:52
- But not exactly, right? So for example
- 131:54
- the one that I just draw
- 131:55
- this is called, Jellu
- 131:58
- let me not, write, I think the
- 132:00
- the definition of it is
- 132:01
- the Jellu's definition is this
- 132:03
- just in case you
- 132:06
- you care about the exact equation
- 132:08
- So it's not exactly nonlinear
- 132:10
- it just drop, it
- 132:11
- it dips a little bit
- 132:13
- I think it doesn't go above zero
- 132:14
- I think. Exactly, strictly speaking
- 132:18
- So, it dips a little bit and then goes up
- 132:20
- So
- 132:21
- exactly why you use any of this is
- 132:23
- kind of like a
- 132:24
- a little kind of like a magic
- 132:25
- Like, I think you don't
- 132:28
- I think people seem to
- 132:30
- find out that using some kind of somewhat smooth functions is
- 132:33
- is actually useful. So value is actually not
- 132:35
- necessarily great because it's not smooth enough
- 132:38
- But other than that
- 132:38
- I don't really know
- 132:40
- exactly, there's like a, sometimes it's just a ability thing
- 132:44
- you know, some, sometimes just a trial and error
- 132:47
- Yeah
- 132:49
- so, okay, so that's the activations
- 132:51
- and then, so now I'm trying to
- 132:55
- compact it, you know, I'm trying to kind of have
- 132:57
- have more and more abstract versions
- 132:59
- so you don't have to deal with all these
- 133:00
- notations
- 133:02
- so then, once you have the activations
- 133:04
- you can define this
- 133:05
- right?
- 133:07
- You can
- 133:07
- this is just the sigma of
- 133:09
- WZ plus
- 133:10
- V. This is the abstract version of that
- 133:12
- And then you can compose many times
- 133:13
- You can say, I'm going to apply another one
- 133:16
- so my model can be like
- 133:17
- say, apply a matrix multiplication on top of this sigma
- 133:21
- and then, right, so WBZ, and then apply another thing
- 133:29
- apply another thing, right, so on and so forth
- 133:33
- And, each of this is one layer
- 133:38
- And then if you apply another matrix multiplication
- 133:40
- sigma is not a layer
- 133:41
- and if you do another one
- 133:42
- it's another layer
- 133:43
- The number of layers is
- 133:44
- defined as
- 133:45
- pretty much as the number of matrix multiplication
- 133:48
- Yeah. But here
- 133:49
- one thing I'm going to describe to you
- 133:50
- about is that
- 133:51
- I didn't really tell you what's the parameters
- 133:52
- here in this matrix multiplication
- 133:54
- So technically, if you want to write this formula
- 133:56
- you have to say
- 133:57
- oh, this is W and B
- 133:58
- and this is what?
- 133:59
- Is it still using W and B or
- 134:01
- some other W and some other B?
- 134:03
- So oftentimes you use a different
- 134:04
- W and different B. It's a
- 134:06
- The, the, the, the shape is similar
- 134:08
- but the, the variables are different variables
- 134:11
- So then you should
- 134:12
- then you have to introduce the
- 134:13
- the bracket again, so let's say this is the bracket one
- 134:16
- B one
- 134:17
- and then here the parameters are bracket W bracket two
- 134:21
- B two, so on and so forth
- 134:22
- And then you index each of them
- 134:24
- to fully specify the network
- 134:34
- Which matrix, sorry?
- 134:38
- So, so these matrix?
- 134:40
- Yeah, so, okay
- 134:41
- Yeah, so what these matrix are
- 134:43
- so the only requirement is that you need
- 134:44
- to make sure that dimensionality com- is compatible
- 134:48
- right? So
- 134:54
- so your input, wait, did I write Z?
- 134:57
- Oh, yeah. Okay. So let's say
- 134:58
- suppose your input Z is
- 135:00
- say, dimension D, then you can only apply a matrix
- 135:05
- you know, with, so to make WZ make sense
- 135:08
- you have to make sure
- 135:09
- W is of dimension something by D
- 135:13
- right?
- 135:14
- and then something is what you can choose
- 135:15
- you know, almost freely. Of course
- 135:17
- you know, not all the options are the same
- 135:19
- eventually in terms of the results
- 135:20
- but for the compatibility issue perspective
- 135:23
- you can choose them arbitrarily
- 135:25
- And let's say, suppose you, the first layer you choose
- 135:28
- let's say, suppose you choose, let's say you choose matrix multiplication
- 135:33
- say
- 135:37
- W one, B one, Z plus C
- 135:43
- And suppose this W one is
- 135:48
- M one by D
- 135:50
- So that means that you map
- 135:51
- this D dimensional vector to M
- 135:53
- one dimensional vector. So let's say
- 135:54
- so this output is going to be
- 135:56
- M one dimensional vector
- 135:58
- And then next time
- 136:00
- and if you apply a sigma
- 136:01
- it's still M one dimensional vector
- 136:03
- And next time you apply the matrix application
- 136:05
- you have to make sure that the
- 136:07
- the input dimension of that matrix to be
- 136:09
- compatible with M one
- 136:10
- So basically, if you do another one
- 136:13
- say W two, B two
- 136:16
- then it's required that W two
- 136:19
- is to be something times M one dimension
- 136:24
- and, and something
- 136:25
- the alpha dimension is still what you can choose freely
- 136:28
- but the, the input dimension is not
- 136:30
- And the next, and you have to do this
- 136:32
- you know, again, again. So once you have multiple dimensions
- 136:34
- then, so basically the, the, like, I
- 136:37
- maybe the easy
- 136:38
- way to think about it is that you have
- 136:40
- a matrix, right, times
- 136:41
- say, Z plus B, right?
- 136:43
- So this dimension is what I call the input dimension
- 136:45
- because this dimension got multiplied with the dimension of Z
- 136:48
- and this dimension is the output dimension
- 136:50
- But the output dimension becomes the input dimension
- 136:52
- for the next round
- 136:53
- so that's why you need to make sure all the
- 136:55
- dimensionalities to be compatible
- 136:58
- That make sense?
- 136:59
- And in many cases
- 137:00
- I think, you know, the, the, the
- 137:04
- the original input dimension is
- 137:06
- is something you cannot change
- 137:07
- right? So like that's fixed
- 137:08
- that's based on our data
- 137:10
- And then afterwards
- 137:11
- you don't change dimensions that often
- 137:13
- You typically use the same dimensions for quite a while
- 137:15
- or maybe you change it once or twice
- 137:17
- something like that
- 137:22
- Okay
- 137:27
- And, you know, like, if you
- 137:29
- just in case you read some of the old papers
- 137:31
- you know, which
- 137:31
- and I think even
- 137:32
- these days still the architecture papers sometimes draw like this
- 137:35
- like they, they just draw, you know
- 137:38
- people go more and more abstract
- 137:39
- right? They say, I have an input X
- 137:41
- and then apply a matrix magnification
- 137:45
- and then apply another
- 137:46
- maybe sigma, and apply another matrix magnification
- 137:50
- so on and so forth
- 137:50
- That's how, how people actually draw these networks
- 137:53
- you know, without specifying all details
- 137:55
- And
- 137:56
- and you can even
- 137:57
- merge these two together to call it MLP
- 137:59
- So sometimes people just draw this
- 138:01
- you have X, and if MLP is called multilayer perception
- 138:05
- often it, encapsulates
- 138:10
- one or two layers of this
- 138:12
- a few of these components
- 138:13
- When people call, say, MLP
- 138:15
- I think sometimes it's one matrix multiplication
- 138:17
- one sigma, sometimes it's two matrix multiplication
- 138:20
- so on and so forth
- 138:35
- I think in most cases you get smaller
- 138:37
- dimensions
- 138:38
- the intermediate dimensions become smaller
- 138:40
- and that's often because the input dimension is
- 138:41
- often very big. For example
- 138:43
- for language, it's kind of like your initial
- 138:47
- the, the vocab is kind of like
- 138:49
- the token, you know
- 138:50
- I guess I'm jumping ahead a little bit
- 138:51
- but like your
- 138:52
- your initial dimension could be like 250K or something like that
- 138:54
- and then you, you
- 138:56
- you shrink it to probably 2K and then
- 138:57
- keep using 2K. Yeah
- 138:59
- something like that. And for vision
- 139:00
- I think the same thing
- 139:01
- because initially you have a RGB
- 139:03
- encoded kind of like image
- 139:05
- You have three channels
- 139:06
- three RGB, three numbers
- 139:07
- and each is like 256 by 256 or maybe
- 139:10
- depending on the resolution
- 139:11
- and you have to shrink it a little bit
- 139:14
- yeah
- 139:17
- okay, what else?
- 139:18
- I think one thing is called ResNet
- 139:20
- I think that's still useful
- 139:20
- so probably I should tell you about it
- 139:22
- So, let me raise something here
- 139:28
- So, but eventually, you know
- 139:30
- like you can define many components like this
- 139:31
- Most of these components are machine applications
- 139:34
- activations
- 139:34
- and I'll tell you a few other components
- 139:36
- that are Still largely being used in
- 139:39
- in, in different cases
- 139:41
- and, and, and
- 139:43
- and some of the very more than cases in
- 139:47
- used in like more
- 139:49
- more than components like our attention used in
- 139:53
- language. I'm going to talk about those in a
- 139:54
- in a, in a large language model lectures
- 139:57
- So, so these are some
- 140:00
- so one of the
- 140:01
- the components called residual network
- 140:05
- or, or just a re-residual connections
- 140:07
- if you think about the
- 140:09
- the
- 140:09
- components. So the residual connection is like this
- 140:12
- So instead of define
- 140:14
- instead of using, keep using this, this kind of things
- 140:18
- the residual connection is kind of like
- 140:20
- let's call residual Z
- 140:22
- is this interesting thing where you have
- 140:27
- two applications of this
- 140:29
- alternating between Matrix multiplication and
- 140:34
- and activations
- 140:35
- and then you add
- 140:36
- your input again back to it
- 140:40
- So one thing that is actu-at least
- 140:42
- by definition, it means that you don't change dimension here
- 140:46
- right? You, because to make this compatible
- 140:49
- your, your input dimension, all dimension are exactly the same
- 140:54
- and, of course
- 140:55
- these two matrix applications are not using the
- 140:56
- same weight matrices. I'm not just
- 140:58
- right exactly what they are
- 140:59
- but, they are not using the same weight matrices
- 141:03
- and, and this is called a residual block
- 141:10
- And, the, this was introduced like in twen-twenty
- 141:13
- 2015 in this, very famous residual network paper
- 141:17
- I think the kind of the
- 141:19
- the, the rationale was that if
- 141:23
- let's say, suppose By the way
- 141:25
- this Z
- 141:26
- doesn't have to be original raw input
- 141:27
- This could be one intermediate layer
- 141:29
- and then you apply this
- 141:31
- Z, you apply this residual block
- 141:32
- and then you, you can actually apply many
- 141:34
- many residual blocks. So eventually
- 141:37
- the, this is how residual block
- 141:38
- and eventually
- 141:41
- the residual network, the residual net looks like
- 141:45
- you know, with some approximation
- 141:46
- it's kind of like you
- 141:47
- apply a lot of residual net many
- 141:50
- many times
- 141:53
- And at the beginning
- 141:54
- you need to kind of do something maybe
- 141:55
- to change the dimension
- 141:57
- because you cannot work with the original dimension
- 142:00
- So you first change the dimension to some
- 142:01
- dimension of the intermediate dimensions
- 142:03
- and then you apply many
- 142:04
- many residual network
- 142:05
- And maybe you have to apply
- 142:06
- one more at the end
- 142:07
- to change the dimension again
- 142:09
- But largely speaking, it's application of many
- 142:11
- many residual blocks, and that's called residual network
- 142:14
- and, the original motivation was that
- 142:17
- why you apply this?
- 142:18
- That's because suppose you believe that at some
- 142:20
- point you are already get
- 142:22
- getting close to your labels
- 142:23
- right? So suppose
- 142:26
- at some point Z is already approximately equal to Y
- 142:29
- let's say. Y is your target
- 142:30
- and Z is your intermediate variables in some intermediate
- 142:35
- like, layers
- 142:37
- Then
- 142:40
- You probably should model the differences explicitly
- 142:42
- because you are looking up close
- 142:43
- you just probably shouldn't use too much works to model the
- 142:46
- the, the only the differences
- 142:48
- So basically you should probably just model Z
- 142:49
- minus Y as some residual
- 142:52
- I mean Y minus Z
- 142:53
- sorry
- 142:54
- say Y minus Z as some residual of Z
- 142:57
- And, and if you believe in that
- 143:00
- like, sorry, not the residual of C
- 143:03
- Y minus Z to be some kind of
- 143:05
- like a right work
- 143:06
- say something like this
- 143:09
- Plus, right, like this
- 143:11
- Right, so if, if this is your
- 143:13
- your belief, then if you move Z to the other side
- 143:15
- then you have
- 143:15
- Y is equals to sigma M M
- 143:19
- Z plus Z
- 143:20
- And you can see that this is kind
- 143:21
- of like almost the same as here
- 143:23
- The only difference is that
- 143:24
- they found out actually you probably shouldn't use
- 143:25
- a one layer to model the difference
- 143:27
- you should use two layers to model the difference
- 143:29
- So, so they say, oh
- 143:30
- let's say you do two of this
- 143:33
- And then the same thing you just
- 143:34
- have like this
- 143:38
- So this is just a story
- 143:39
- It's not like, any, there's any formal proof there
- 143:43
- and, and actually people have tried to
- 143:45
- you know, ten years ago people have tried
- 143:47
- maybe seven years ago people have tried
- 143:50
- to see how many blocks here you should do
- 143:53
- you know, can do two layers
- 143:53
- three layers, L layers
- 143:55
- and eventually I think two or three layers are the right
- 143:57
- are the best
- 143:59
- also for reasons we don't fundamentally understand necessarily
- 144:03
- yeah, so that's residual network
- 144:05
- I thought
- 144:05
- yeah. I thought like that the term on the right
- 144:08
- that's our hypothesis
- 144:15
- So, so, so, so, so suppose this is the modeling assumption
- 144:19
- you model only the differences
- 144:20
- then, you know, so this implies this
- 144:23
- and they are, they are the same
- 144:23
- right? So like
- 144:24
- which one is the hypothesis depends on what you
- 144:26
- what you believe in
- 144:27
- So, but, but somehow if you
- 144:28
- if you somehow believe that
- 144:30
- it's more beneficial to model directly the residual
- 144:34
- Then you, you, you do this, right?
- 144:36
- And then you change
- 144:37
- and you rearrange together
- 144:40
- That's, that's the only thing I'm saying
- 144:41
- Nothing profound
- 144:47
- And there are other kind of like
- 144:49
- explanations for why this works
- 144:50
- you know. For example
- 144:51
- some of the
- 144:52
- theorists that they try to analyze the optimization
- 144:54
- perspective of different modeling process
- 144:56
- right? So basically
- 144:57
- if you don't add Z here versus you add Z
- 145:00
- you know, in some cases you can argue that
- 145:04
- adding Z will make the optimization easier
- 145:07
- make your optimization more well-conditioned
- 145:09
- including some of my papers
- 145:12
- you know, I think
- 145:13
- I think there are some
- 145:14
- degree of truth there as well
- 145:15
- It's just that it's very hard to fundamentally
- 145:17
- you know, it's like you have many explanations for
- 145:21
- for it. It's
- 145:22
- it's very hard to know exactly which one
- 145:23
- is the right explanation
- 145:24
- Yeah
- 145:28
- maybe, let's see, I have seven minutes
- 145:30
- Any, any question?
- 145:37
- Why the differences in model binding?
- 145:41
- There's no, you know, I
- 145:42
- I got a question or what's the intuition
- 145:45
- but like, maybe I see, so the intuition could be that
- 145:47
- oh, maybe this is already small
- 145:48
- then maybe I should directly model that
- 145:50
- right? As, as opposed to model the
- 145:51
- the original label. So it's a simpler task
- 145:54
- The original label, you know, predicting original label is
- 145:56
- is very hard, but if you predict the difference
- 145:58
- probably it's easier
- 145:59
- So that's why you model easier task with a two-layer networks
- 146:02
- and that's pretty much it
- 146:05
- Like, there's no, nothing more formal than that
- 146:08
- Yeah
- 146:11
- let's see, I have seven minutes
- 146:13
- I think
- 146:13
- Let's see, should I attempt to talk about two things?
- 146:18
- Okay, so maybe let me quickly talk about
- 146:22
- layer norm
- 146:25
- and I can see whether I have time
- 146:26
- to talk about the convolutional network
- 146:29
- So Layer norm is also something that people still use
- 146:34
- in large language models
- 146:36
- So, of course, it's like a different form these days
- 146:40
- So, the original layer norm is the following
- 146:43
- So basically, still, it's, it's a component, right?
- 146:46
- So like, it's a, it's a block, you know
- 146:47
- so you have some input that's called Z
- 146:48
- and
- 146:50
- and this component is trying to normalize Z
- 146:52
- in some way
- 146:53
- And the way to normalize it is that
- 146:54
- you say you first turn Z into
- 146:57
- some normalized version of Z
- 146:58
- which is Z minus
- 146:59
- mu one over sigma one
- 147:02
- comma Z minus mu M sigma M
- 147:05
- I say Z is M dimensional
- 147:06
- Z, sorry, Z one
- 147:08
- Not my bad
- 147:09
- I should, I should read the note
- 147:10
- I should follow the notes more closely
- 147:11
- I think this is
- 147:13
- of the same
- 147:19
- Yeah
- 147:22
- Maybe I should even use
- 147:31
- Yeah
- 147:31
- So let's say, okay, I only remember why I use hat
- 147:35
- but let's use hat
- 147:37
- So this is how you normalize
- 147:39
- where mu hat is the empirical mean
- 147:43
- which is the average of all the ZIs
- 147:47
- And sigma hat is the empirical
- 147:52
- sigma hat squared is the empirical average of the
- 147:57
- the standard deviation
- 148:00
- So this will empirically normalize it to norm one
- 148:04
- right? So you can see it's
- 148:05
- it's pretty sta, and
- 148:09
- and then it's not done yet
- 148:12
- And then you turn this into
- 148:14
- sometimes you find, people found out, oh
- 148:15
- actually mean zero and norm one
- 148:17
- is not that great for whatever reason
- 148:20
- So they say, oh, let's be flexible
- 148:21
- Let's also add
- 148:27
- Of flexible tunable parameters
- 148:29
- beta and gamma. So let's turn
- 148:30
- the next step is turn this into
- 148:34
- beta plus gamma z one minus mu hat
- 148:39
- sigma hat
- 148:48
- Where beta is a
- 148:49
- is a tunable param
- 148:50
- beta is a variable that you can tune
- 148:51
- and gamma is a variable you can tune
- 148:54
- and, so basically this make a mean beta and
- 148:58
- and a standard deviation gamma
- 149:01
- So that's the original layer norm
- 149:03
- And then later people realized
- 149:05
- actually, you probably don't have to do this
- 149:06
- You just, so this is the layer norm
- 149:09
- And now I think there's something called RM-SAM norm
- 149:12
- which is, even simpler
- 149:14
- You don't, subtract the mean anymore
- 149:18
- You just divide by sigma
- 149:19
- So you
- 149:21
- so you just say you do z
- 149:22
- is turned into
- 149:25
- Z one minus sigma one
- 149:27
- sigma hat, say Z M minus sigma hat
- 149:31
- And the sigma hat is also different
- 149:33
- Sigma hat is only the
- 149:36
- the average
- 149:39
- is only kind of like the
- 149:41
- you are only averaging over like the
- 149:43
- you don't have the
- 149:44
- you don't subtract here. So you just try to make the
- 149:47
- so sigma hat is something like one over M
- 149:54
- Z, Z I, sorry, Z I squared
- 150:04
- And, and then you still do the beta thing
- 150:06
- right? And then you turn this into
- 150:08
- beta plus
- 150:12
- Z one over sigma hat
- 150:17
- You know, why
- 150:18
- why this is kind of like more better or not
- 150:20
- you know?
- 150:22
- I think it's mostly empirical
- 150:24
- So basically here you don't really want to make it
- 150:27
- let me think about this
- 150:28
- Wait, sorry, I think you still need gamma
- 150:30
- sorry
- 150:31
- Did I keep gamma or keep the beta?
- 150:32
- I keep the gamma
- 150:33
- sorry, my bad. You do this
- 150:36
- So, and gamma is channel parameter
- 150:37
- So basically this is saying that I don't
- 150:39
- have to care about the
- 150:39
- the shifting mean. I don't really make it mean zero
- 150:41
- but I
- 150:42
- I still
- 150:42
- care about normalizing it in some way
- 150:44
- But I still leave us some degree of flexibility gamma
- 150:47
- which I, you know, it's not really normalizing
- 150:48
- I still allow it to
- 150:50
- to be normalized to something that I
- 150:51
- I will decide later in my training
- 150:54
- Right, so, so this is kind of like the
- 150:55
- the magic here. It's not like you
- 150:57
- you try to force it to not normalize
- 150:58
- to one standard deviation. Why you actually say
- 151:01
- I first normalize to one
- 151:02
- and then I leave a gamma
- 151:03
- parameter there so that
- 151:05
- in the training I can normalize to anything
- 151:08
- and
- 151:11
- I think, let me see
- 151:13
- Yeah, I don't have one then
- 151:14
- So
- 151:15
- so one benefit of all of this is
- 151:17
- that you make it kind of like a
- 151:20
- Scaling environment
- 151:21
- in term
- 151:21
- for both the parameters and the data
- 151:23
- So what does that mean is that if you multiply
- 151:27
- so maybe let's say you call this whole thing called
- 151:30
- let's say Z
- 151:32
- let's just, say Z, right?
- 151:35
- So like, let's say
- 151:36
- suppose this whole operation is called a layer
- 151:38
- norm
- 151:39
- So you apply Z and you'll
- 151:40
- you get a layer norm of Z
- 151:42
- And you can real
- 151:43
- you can find out that there's a scaling environment
- 151:44
- which means that if you apply
- 151:47
- a scalar, let's say you, you multiply a scalar
- 151:49
- say alpha to Z
- 151:51
- and you get L N alpha Z
- 151:54
- but L N alpha Z is actually equals
- 151:56
- to L N Z
- 152:00
- So basically, no matter how you multiply the input
- 152:02
- you don't really change it
- 152:04
- and so that's why no matter how you multiply your
- 152:08
- your matrices or, or you multiply your input data
- 152:12
- you never change the output
- 152:14
- unless you change the
- 152:15
- the, some of the parameters like beta and gamma
- 152:17
- which is, you know, in a test time
- 152:18
- you're not supposed to change them
- 152:20
- So, so basically this makes the
- 152:22
- the model more scaling environment
- 152:25
- and in some cases
- 152:25
- it, it helps, stabilize the model, for example
- 152:29
- At least intuitively, you know, with some difficulty of approximation
- 152:33
- you know, they are, they are, at least in the
- 152:34
- in the past when people tuned all of these models
- 152:36
- it happens that sometimes you apply the
- 152:39
- the, the, these layers many times
- 152:41
- and then the, the output becomes bigger and bigger
- 152:43
- and, and it becomes like a really big
- 152:46
- it's like ten to the twenty or something like that
- 152:48
- and then, and then everything just blows up
- 152:51
- So when you normalize every time
- 152:52
- then you, you never got into this issue
- 152:54
- And also it makes the initialization a little
- 152:55
- more kind of like
- 152:57
- easier so that you don't have to
- 152:58
- you know, before, like if you don't do something right
- 153:00
- then suddenly everything just blows up
- 153:02
- and now like the scaling
- 153:04
- doesn't really matter that much
- 153:05
- but there's also subtlety here
- 153:07
- because when you have
- 153:08
- okay, this might be a too advanced comment
- 153:10
- like one side effect of this is that when the
- 153:13
- when the forward pass is scaling environment
- 153:15
- but actually the, the
- 153:15
- the gradient is not scaling environment still
- 153:18
- So, so, so
- 153:19
- so some of these issues got pushed into the
- 153:21
- the optimization, so, which is, a much more complex matter
- 153:26
- anyway, so I guess, let's see
- 153:28
- So I think I'm done
- 153:31
- The only thing I didn't cover
- 153:32
- is the convolution network
- 153:34
- I think it's in the lecture notes
- 153:35
- So, if I spend one minute on it
- 153:38
- I think it just means that
- 153:40
- instead of multiplying a full matrix
- 153:42
- you actually multiply a
- 153:44
- a topless matrix. I'm not sure you've heard of this word
- 153:46
- like a
- 153:46
- So basically it's a matrix with some structures where like
- 153:50
- So, so when it's, yeah, instead of having this matrix
- 153:54
- convolution network just basically means you have a
- 153:55
- matrix that's kind of like a
- 153:57
- like, look like this
- 153:58
- The non-zeros are only like this
- 154:01
- And, and it's, it's actually the same set of non-zeros
- 154:03
- like you, you, you shift this vector here
- 154:04
- and you put it here
- 154:05
- and you put it here
- 154:06
- and you put it here
- 154:07
- It, it's actually called like a
- 154:08
- a, a, a filter in some sense
- 154:10
- like, so, so this matrix
- 154:12
- all the other parts of the matrix is zero
- 154:14
- and only this, what I drew here is kind of non-zero
- 154:17
- and also they share the parameters
- 154:18
- so basically this vector is the same as
- 154:20
- this vector
- 154:21
- So you apply kind of like a
- 154:22
- a shifting environment, kind of like filter
- 154:25
- to each, to, to the data
- 154:27
- and it's called convolution. Of course
- 154:29
- the real thing is a little bit more complicated
- 154:30
- because when you have 2D and the so and so forth
- 154:32
- it becomes, complicated. But in the lecture notes
- 154:35
- we have that
- 154:36
- But convolution network is not that
- 154:38
- used that much
- 154:39
- except for very hardcore vision cases
- 154:41
- Like even for vision cases
- 154:42
- people use, language models, like, transformers, a lot
- 154:47
- So, so that's why I probably find to keep
- 154:49
- skip them
- 154:50
- Okay, sounds good
Watch these videos on your TV
XTTV brings video content from X to your living room. Download now and start watching on the big screen.