← Back to videos

Instead of spending 2 hours watching Netflix tonight, invest that time in this Stanford lecture. It

Rahul Kumar

Rahul Kumar

@RahulTechX

November 19, 2026· 154:51

Instead of spending 2 hours watching Netflix tonight, invest that time in this Stanford lecture. It's one of the clearest end-to-end breakdowns I've seen of how ChatGPT and Claude are built. It covers everything from tokenization and BPE to transformer architecture, the training pipeline, and next-token prediction. Whether you're completely new to AI or already building AI agents every day, this 2-hour-and-34-minute lecture will help you understand what's happening behind the scenes. Bookmark this, set aside some time this weekend, and give it your full attention. Because understanding how AI actually works could be one of the most valuable skills you develop this year.

Transcript

So large language models
0:02
I guess you always
0:02
probably all of you have heard of this word
0:06
large language models. This is
0:09
precisely, I think in some sense
0:10
this is all starting from GPT-3
0:12
when OpenAI released GPT-3
0:15
it has a, it used this new paradigm which
0:18
used autoregressive large language models
0:22
so, so basically we're going to focus on that
0:24
you know, autoregressive large language models
0:26
We're going to skip some of the early versions of Transformer
0:28
which are, not autoregressive
0:32
so
0:34
and
0:35
and another regressive basically means that you generate
0:37
tokens one by one
0:38
and each of the new tokens depends on the previous tokens
0:40
and you do this
0:42
and you model probability in this kind of like
0:44
autoregressive way
0:45
I'm going to talk more about details
0:47
In some sense, you model your language distribution by using a
0:51
a sequentially, sequential model one by one in some sense
0:55
anyway, so, let's, I think your point
0:59
our goal is to talk about some details
1:01
so I'll
1:01
I'll fill in more high-level stuff as we
1:03
talk about the details
1:04
So
1:05
the first thing we're going to talk about
1:06
is tokenization
1:10
This is, you know, just, to prepare us for
1:13
to talk about the real meat in some
1:15
sense
1:15
So what does tokenization mean?
1:17
It means that you need to You know
1:19
these transformers, they are numerical, you know, architecture models, right?
1:23
So they take in
1:24
in some sense, numerical numbers, right?
1:26
So, but, you have tax
1:28
and how do you turn tax into some
1:30
kind of things that
1:32
the models can understand?
1:33
So, the first step is to tokenize
1:36
Tokenize means that you need to decide
1:38
you know, what is the smallest unit
1:40
of the input that you give to the
1:42
large language model
1:43
And often it's not a
1:44
a, a word
1:46
right?
1:46
The most natural thing you can think of
1:47
is
1:48
maybe a smallest unit is a character or
1:49
a word
1:50
If you use character level tokenization
1:53
then the issue is that often you have so many characters
1:55
right? You, you need so many tokens
1:57
so many steps
1:59
and, if you use word-level
2:01
tokenization, then often the issue is that you don't
2:04
how do you deal with these long words
2:06
which are very rare? For example
2:07
you have a word called
2:08
say, interna-intern, nationalization, right? So, yeah
2:12
if you treat this word as a single word
2:14
then maybe it's too rare
2:16
and you don't leverage your understanding about other words
2:19
right? So if you treat internationalization as one word
2:21
then it is different from internationalized
2:24
It, it's different from, like, a na-nation, right?
2:27
And, and they are all completely different kind of things
2:30
and you cannot leverage
2:32
that, you know, for example, if you understand interna-nationalized
2:35
then you should probably understand internationalization to some
2:37
degree without even seeing more data
2:40
And, so, so that's why people don't always use
2:44
subword tokenization, or, or don't use word tokenization
2:47
Maybe another, okay, maybe let me give you another example
2:49
right? For example, if you have used this
2:51
I guess in my notes
2:52
there is this word LM
2:54
magnification
2:55
Like, actually, I don't know whether this is a real word
2:57
you know, like
2:57
I think it's a new word that is
2:59
complete new word. Nobody have used it before 2020
3:03
2020, I guess, right? So, even, even right now
3:05
it's not that kind of like common
3:08
And if you have to treat this word as a unit
3:12
then you
3:13
you have to see this word enough times
3:15
to understand what it means
3:17
right? However, if you treat it as subwords
3:19
then maybe you have a
3:20
a token called ILM. You have a token
3:22
which is, you have a two unit
3:24
you know, this is a token
3:25
this is a token
3:26
So ILM is a token
3:27
invocation is a token
3:28
Then in some sense
3:30
you have seen ILMs many
3:31
many times, you ought to understand it
3:33
And then invocation has a special meaning in English
3:37
and then you combine these two understanding
3:39
you, you understand the word
3:41
So that's much more efficient
3:42
understanding of the word
3:44
than just, treating this whole thing as a single unit
3:48
So, and then, and this happens more often in
3:52
for example, biology, when you have this very long word
3:54
or maybe in German you have very long word
3:56
but actually there are fundamentally many combinations of many subwords
4:00
And, and
4:02
and using subword tokenization allows you to understand
4:04
each of the subwords
4:05
and, and then you can understand the combination of the
4:07
those subwords, as a, as a, as a whole, very efficiently
4:12
So, yeah, so basically tokenization, you know
4:14
just means that you have a predefined vocabulary
4:18
vocabulary, and this vocabulary is a list of like
4:22
subwords, right? So for example
4:23
you can have, you know, many, many of this, like maybe A is a subword
4:26
you know, is a one of the token
4:28
you know
4:29
I think maybe like a
4:30
dog is a token
4:32
Pretty much
4:33
all common words these days is a token
4:35
And actually, if you look at the
4:36
the, the
4:37
actually you can play with the tokenization online
4:39
if you search OpenAI tokenizer
4:41
OpenAI give you a
4:42
playground where you can see how the texts are tokenized
4:46
And, and you'll
4:47
you'll find that actually blank dog is also a token
4:51
So
4:52
so the blank is actually merged with dog
4:54
as a token
4:55
And, you know, and you have so many of this
4:57
I, typically this kind of like a tokenization right now
5:01
I think for the open source models
5:02
we know the number of tokens
5:03
they have, I think it's, you know
5:05
often on the order of like ten to the five
5:07
like a hundred K at least
5:09
I think QN, QN 3.5
5:12
this is the newest version of the open source models
5:13
which are pretty good
5:14
I think they have like two hundred fifty
5:16
tokens
5:18
I think two hundred fifty probably is mostly because
5:20
you know, I think two hundred fifty is
5:21
is somehow you can represent this token with like a
5:24
Maybe it's eight, eight bytes in some sense
5:27
like, like, it, it
5:28
it also depends on how many bits
5:30
you can present the tokens
5:32
so, anyway, so you have this vocabulary
5:34
and then you just
5:36
given a long
5:37
string of text, you just
5:40
take all the subwords that are tokens
5:41
and you turn it into a sequence of
5:43
like
5:44
segments, right? For example, if you say, if I have a
5:47
a dog runs, suppose this is the text you see
5:49
and then you'll, you'll say, this is a token
5:52
so this is one part
5:53
and then I think blank dog is a token
5:57
and blank runs is also a token
6:00
and, and comma, I think, is a token by itself
6:05
So, and every token has an ID
6:06
right? So let's say A is probably the ID number one
6:09
dog is maybe the ID number like 356
6:12
whatever, and, and run is, some ID
6:15
maybe, I don't know, like, maybe 250
6:18
I'm just writing some random numbers
6:19
Every, everything has an ID
6:21
so basically you turn every tax into a sequence of IDs
6:24
you know, each ID corresponds to one token
6:27
and, and I think, so in this case
6:30
you know
6:30
most of these words are actually one token
6:32
I think, I just checked recently
6:34
if you have the word happiness
6:37
I think the tokenization will be unhappiness
6:40
let's say
6:41
I think this part is a token
6:43
H is a token
6:44
and A P I N E S S is a token
6:48
for some reason
6:50
and maybe another example is this
6:52
oh, I, I already read it
6:53
that I am identification
6:55
and this is another example
6:58
I come, we come up with
7:00
you know, just, I don't know actually what these words mean
7:02
if you have a blast
7:04
Oh, Ma, I think this part is a token
7:05
this part is a token
7:06
and this part is token
7:07
I think this is just because for this biological
7:09
biology words, you know, biological words, you know
7:12
actually every part of the word has some special meaning
7:15
and, so you tokenize them into
7:18
tokens
7:19
And, and what is the exact vocabulary?
7:21
It depends on the model
7:23
so basically the, the algorithm that turns the text into these
7:27
tokens is called a tokenizer
7:29
and, and every model has a different tokenizer
7:32
Not, not everyone has a different one
7:34
Some, some of them shared the same tokenizers
7:36
but generally it's specific to the tokenizer
7:38
to, to the model
7:40
and different companies have different
7:42
proprietary tokenization approach
7:45
you probably heard of that
7:46
recently, I think Cloud Code
7:48
Cloud, they update their tokenizers so that
7:51
I think they make the tokenizers more granular
7:53
to some degree
7:54
so that
7:55
the same amount of tax will be
7:56
tokenized into more tokens
7:58
So before it was like one thousand tokens
8:00
now it's one point five thousand tokens
8:02
you know, I didn't verify it myself
8:03
but I read some news about this
8:05
Assuming they are true
8:06
that means that you
8:07
you spend more tokens
8:08
you know, you, you pay more to Cloud Code
8:12
with the same amount of tax
8:14
anyway, so, cool
8:17
And there are some details about how to
8:20
come up with the vocabulary and how to kind of like
8:24
run this algorithm to tokenize
8:26
the, the key word is called VPE
8:30
VPE
8:31
tokenizer, byte, pair, encoding
8:35
So I'm not going through the details because
8:37
this is just a very small
8:38
quick, setup
8:40
it's not that important
8:42
but roughly speaking, the BP
8:44
tokenization idea is that you try to
8:46
in some sense
8:46
you just try to find frequent subwords that are most frequent
8:51
right? And
8:51
and you use them in the vocabulary
8:53
But how do you decide which one is more frequent and
8:55
and not? I think there
8:56
there's a sequential algorithm. You first go through maybe like the
8:59
the, the, the smallest kind of like
9:04
combinations of like, you go through all the characters
9:06
you know, and you, you try to find
9:07
you know, I think it's a greedy algorithm
9:09
First, you try to find the most frequent
9:11
you know, one character thing
9:12
and you find in the most frequent combination of two characters
9:15
and you remove, you know, it's, it's a little complicated
9:17
which I, I mean, I'll try to
9:19
give you a precise version
9:20
but it's kind of like a
9:21
a greedy, algorithm that decides what one
9:24
which one is more frequent
9:26
so and so forth
9:28
and if you are interested
9:28
you can look up the
9:29
the exact algorithm
9:32
okay, so that's the tokenization
9:34
So that means that from now on
9:36
you know, we just assume we have a tokenizer
9:38
so every tax
9:39
every piece of tax will be turned into
9:41
a sequence of tokens
9:42
So from now on
9:43
like a, a tax, it just means a sequence of tokens
9:47
and each of this XI
9:49
is in the vocabulary. Let's call the vocabulary V
9:53
and the size of the vocabulary could be
9:55
like two hundred and fifty
9:57
thousands
9:58
and, you know
10:00
each of this is kind of like an index
10:02
in the vocabulary, like, you know, this is the fifth token
10:04
this is the, a hundred fifth token
10:06
so on and so forth
10:07
Okay?
10:08
So, and now the questions
10:10
you know, now we go into the next question
10:12
which is how do we model this distribution
10:14
of sequence of tokens?
10:16
How do we create a probabilistic model to
10:18
model it?
10:19
If you think about the support
10:22
of, this, this distribution, like how many possible combinations
10:26
that there are possible here
10:28
so the number of possible combinations is You have V choice
10:33
this is a cardinality
10:34
you have V choice
10:35
cardinality of V choice for each of the
10:39
the tokens, and you have a sequence of last T
10:41
so you take the power to the T
10:43
right?
10:43
So everyone has V choice
10:46
and the
10:47
V to the power T
10:48
is the total number of choices
10:49
So you cannot assign one probability for each of this
10:52
sequence, like, like, like a, like a
10:57
like a discrete categorical tokens
10:58
right? Categorical distributions
11:00
right? If you treat this as a categorical distributions
11:02
then every choice here will have one probability
11:06
and that's just way too many parameters to
11:08
describe it
11:09
So that's why
11:10
what people do is that people
11:11
decompose this distribution using the chain rule
11:14
to write it this way
11:15
So you say
11:16
suppose I have a probability distribution
11:20
over the joint, of the sequence
11:22
then I can write it as the following
11:24
so I can say it's equals to
11:26
P of X one times P of
11:28
X two given X one
11:30
P of X three given X two and X one
11:34
so on and so forth
11:35
and P of X T
11:36
given X one
11:37
up to X T minus one
11:40
And this is the
11:41
the so-called, you know, conditional, probability decomposition
11:45
You decompose the joint distribution
11:47
into a product of conditional probabilities
11:51
And you model each of these conditional probabilities
11:53
The benefit here is that each of these
11:55
conditional probabilities is a distribution
11:58
is a conditional distribution
11:59
but the choice of X two
12:02
for example, here, is only V
12:03
You have V choices here
12:05
You only have a
12:06
you only have to have a
12:07
kind of like a softmax or logis over
12:09
V choices for each of this
12:11
right? So each of this will be eventually you'll see is
12:14
some kind of like a
12:17
you have some logis
12:17
then you apply some softmax
12:18
you get a proper distribution over
12:20
v choice
12:22
So that's what we're gonna do
12:24
So basically, each of this
12:26
will be parameterized by a new artwork
12:30
the input will be
12:31
what's after the conditioning
12:33
right? So input here is the x minus x two
12:34
and output is the distribution of x three
12:37
And, and this input-output relationship is modeled by
12:41
a, a new artwork, which is a transformer
12:46
Okay, so that's, what we are, going towards
12:49
How do we model each of this conditional distribution?
12:53
So
12:56
so basically we are gonna describe a
12:57
conditional distribution P of
12:59
P sub theta X T
13:01
given X one up to X T minus one
13:04
So
13:09
Before going to details
13:10
let's treat this as a kind of like
13:11
a black box first
13:12
And let's first kind of like discuss a little bit about
13:16
you know, the, the boundary, right, in some sense, right?
13:19
So it's kind of like a
13:19
there are multiple layers here
13:21
So the typical way to
13:24
parameterize this is the following
13:26
So, you use a transformer
13:28
So basically what happens is that you have
13:30
a sequence X1 up to XT
13:32
And, so first of all
13:35
these are discrete IDs
13:37
right? So each of these means
13:39
a ID for the token
13:41
You first have to turn them into
13:42
some numerical numbers
13:44
So the, the, the first step is that you say
13:47
I'm going to turn this into
13:49
an embedding E of X1
13:51
And so basically I turn each of the token X into
13:56
XT into embedding of XT
13:59
And what is the embedding?
14:00
Embedding is, As a D-dimensional vector
14:04
in, the Euclidean space
14:06
And this embedding is
14:07
is trimmed. So basically
14:09
you don't know what the parameters are
14:11
you will just, parameterize it, you know, as unknown numbers
14:14
and you trim the model to
14:16
find the right numbers
14:18
for, for these embeddings
14:20
So every, every, token in the vocabulary will be
14:25
will have a corresponding embeddings
14:27
So basically, you're gonna have a
14:32
a sequence of embeddings
14:33
E1, E2, this is the embedding for the first token
14:35
embedding for the second token
14:37
up to the embedding for the last token
14:39
and in this lecture
14:40
I think we're gonna use row vectors
14:42
because I think when you get into the implementations
14:45
you know, every vector is actually row vectors in the
14:48
in the Python code
14:49
even though in mathematics people often use column vectors
14:52
So, so I, I thought about this
14:54
and I, I thought just for this
14:56
for large language model sections
14:57
we'll just always use row vectors
14:59
which is more consistent with the practice
15:01
It looks a bit weird to me
15:03
actually, from a mathematical perspective
15:05
and if you read the papers
15:06
many of the papers do use column vectors
15:08
but I think we'll just use row vectors
15:10
to be consistent with the code
15:11
So basically every embedding is a row vector
15:15
and then you have this
15:16
embedding matrix
15:18
so this is, you have
15:22
V of them, and each of them is of dimension D
15:25
so you have this matrix
15:26
And, and to read off which row you are
15:28
you just have to multiply
15:30
either you just read off the ID
15:31
right, so if you have the fifth token
15:33
you just read the fifth row
15:34
or you can multiply this matrix with
15:37
an indicator vector to gather that row
15:39
which is the same thing
15:41
Anyway, so we'll just read off from this embed
15:43
embedding matrix the corresponding embeddings for each of the token
15:46
so you get E of X2
15:48
up to E of Xt
15:51
and, and I think in
15:54
in many cases, often people append
15:57
prepend, a, beginning of sentence token
16:01
which is just a fixed token
16:02
in the very beginning
16:03
This is a kind of like not always
16:05
necessary. In some cases
16:06
you just don't do it
16:07
In some cases, you do it
16:08
But let's, probably just introduce that
16:10
You have a beginning of sentence token
16:12
which is one of the special token
16:13
in this vocabulary. You probably
16:15
often actually what happens is that
16:17
your vocabulary is bigger than what is needed
16:19
You always reserve a few tokens
16:21
you know, actually maybe a thousand
16:23
you know
16:23
a few thousand tokens for special tokens that you can introduce
16:27
in, in, for special purposes
16:30
And you can introduce this beginning of sentence token
16:32
as X zero
16:33
so which is always the same
16:35
and then you turn them into embeddings
16:38
And then what you do is that you
16:40
go through this transformer
16:41
which I'll discuss in the later half of
16:43
the lecture
16:44
So this is a transformer
16:48
For now, you just think of this as a new network
16:50
which does a lot of computation
16:52
machine application, you know, tensions, which I'm going to discuss
16:55
a lot of operations
16:56
and then you output
16:58
some logits of, like, you output some kind of logits for
17:03
at each of the locations
17:08
So
17:08
each of these logits is supposed to describe the conditional distribution
17:12
before the, the softmax. So basically
17:21
what you do is you say
17:23
the probability of, so let's just
17:26
you say each of this U1 is
17:30
let's call this U1 U theta X zero
17:34
and then P of X one is
17:39
modeled as, you know, just the softmax
17:44
Its distribution P of XI is the
17:47
XY is modeled as the softmax of
17:52
F theta X zero
17:55
So the first, distribution, which is not conditional
17:58
you just, get the output from the X zero
18:01
you get some logit
18:02
So this U, UI is in dimension V
18:06
and you take a softmax
18:07
you get a probability vector
18:09
So this is a probability vector
18:15
right, in this, simplex of dimension V
18:18
right? So the sum
18:19
this, this, after taking a softmax
18:21
the sum of the entries is one
18:23
and all the
18:23
all of the entries are non-active
18:25
So then this is your probability for X
18:27
one. This is the distribution
18:28
Technically, I think the distribution of this is equal to softmax
18:31
okay?
18:32
and then the same thing
18:34
you know, when you have X two given X one
18:37
you say this is equals to softmax
18:41
of, U2, where U2 is the output at the second position
18:48
and, and U2, you know
18:49
we'll just call U2 the same as
18:50
you know, so we call U2
18:55
the output, you know, of, this model given X
19:03
zero and X one
19:06
So, so each of these UIs
19:08
you know, is, basically the output
19:15
for, the sequence before it
19:17
right? So UT plus one is
19:19
F theta X zero up to XT
19:23
So, so, so basically generically, you know
19:26
P of XT
19:27
given X one up XT minus one is the softmax
19:34
of, UT
19:39
is it UT or UT plus one?
19:41
UT plus UT, yes, UT
19:44
which is the softmax
19:47
of F theta X zero to X
19:52
T minus
20:00
one
20:02
Okay. Any questions so far?
20:07
So I, I, I particularly chose it to index at one
20:09
yes
20:13
Yes
20:18
U is just a vector
20:19
you know, I think, you know, I can just call U
20:21
I think, it's kind of like
20:23
I can just call U F theta X zero
20:24
it's the same thing
20:25
I'm not, did I answer the question?
20:28
One second
20:43
So, oh, I think maybe, maybe that's the
20:47
So
20:48
so
20:49
I'm just using f theta x zero
20:51
x one as a
20:52
more explicit way to
20:54
represent UI because I'm trying to express the dependency
20:58
So, so basically I'm saying like
21:01
basically I, I think they are the same thing
21:04
Like, it's just there
21:05
there are no difference at all in some sense
21:06
just how you view it
21:07
right? Like in one way you just say
21:08
oh, I have, I need a, a
21:10
a notation for the vector
21:11
and the other way you say I have
21:12
I
21:12
I need to interpret this vector as a
21:14
function of the inputs
21:18
Oh, okay, okay
21:19
okay. So maybe I didn't get the question
22:00
Yes, yes, yes
22:01
So
22:01
I think maybe I'm trying to figure out
22:04
what's the question here. So
22:07
I think maybe what they're asking is the following
22:09
So, so maybe let's take this one
22:12
right? So this is a
22:13
so this is a vector
22:15
This is a vector in dimension V
22:19
So in, in a vocabulary size
22:21
And you have to take a softmax
22:22
it's a, it's a sequence of numbers which are
22:27
all non-negative and they sum to one
22:29
So, and what this really means is that
22:33
P theta of xt is equals to one
22:37
given x one up to xt minus one
22:39
The chance to see the first token under this probabilistic model
22:42
parametric theta, is equals to, this is the vector
22:47
the first entry of this vector
22:49
right, is equals to softmax
22:52
f theta x zero up to xt minus one
22:57
indexed by one. This is the first entry
22:59
of this vector
23:00
And you have many
23:00
many other entries, right? Each entry is
23:02
you have another entry softmax
23:06
f theta, sorry
23:14
two
23:14
and that
23:15
is equals to P theta of xt is equal to two
23:19
the chance of seeing the
23:21
second token given x one
23:23
up to xt minus one
23:25
And you have so on and so forth
23:28
Does that answer the question?
23:31
V, the, yeah
23:32
this V absolute value is the
23:34
size of the vocabulary. Yep
23:45
Yep. So this is the causal model so far
23:47
but I haven't defined the transformer yet
23:49
So, but I'm insisting that this is only
23:52
so guys
23:52
this is one of the reasons why I'm
23:54
using this notation. So I'm insisting that UT
23:57
is only a function of X zero and
23:58
X T minus one
23:59
but not a function of the
24:01
the, the later, inputs
24:04
So, but that will be
24:06
that will be guaranteed when we introduce the
24:08
exact inverse working of this transformer
24:27
Okay, good question. So the first
24:28
the second question is easier to answer
24:30
So the embeddings is only a property of the
24:33
this X, right? You don't have to see any other things
24:35
right? It's just actually
24:36
it's literally, if this is the first token
24:38
then you just read the first row
24:39
of this matrix. If this is the tenth token
24:42
you raise the tenth row
24:43
and that's your embedding. No any other dependencies
24:45
And while we are writing this as
24:47
as functions of the X
24:48
but not the embeddings
24:49
that's just, it's just
24:53
is sometimes the embeddings is the inner working
24:55
to some degree
24:56
I'm just trying to give you more
24:57
I'm trying to unpack it a bit
24:58
right? So, so, so
25:00
so if you don't have to unpack it
25:01
you just work
25:02
you can work with the text
25:03
Yeah. That's just the representation
25:05
Yeah. And it's one-to-one
25:07
so in some sense
25:07
there's no difference
25:13
Okay, so this is a
25:14
a full description of the probabilistic model
25:16
Once you have this probabilistic model
25:17
you can compute the maximum likelihood
25:18
given any data. We can say
25:20
I have a data distribution
25:21
I have a data point
25:22
a sequence of data
25:23
I can see under this probabilistic model
25:25
P theta, what is the chance of seeing this
25:29
this data
25:30
So that's, that's what you need for computing the
25:33
do the training, right? You, you
25:34
you need to compute the likelihood
25:35
and then you maximize the likelihood as a function of theta
25:39
right?
25:39
So, and, let's briefly discuss, you know, for generation, right?
25:44
So suppose you already have the generational decoding
25:50
suppose you already have the model
25:52
then a, a model theta
25:54
then what do you do?
25:55
So the, as you can see
25:56
the most, natural thing is you just keep generating X one
26:00
X two, X three, so on and so forth
26:01
right?
26:02
You just say, I'm gonna generate X one first from this
26:05
you know, P, say, softmax
26:12
Softmax, you know
26:13
f theta
26:14
x one up to x zero up to x t
26:17
x, I guess I just write
26:20
generically, so you generate x t minus one from
26:24
f theta x zero up to x t minus one
26:27
right? And you do this sequentially
26:28
Sometimes people draw this in this way
26:29
so you say, oh, you have x zero
26:31
you first go through the transformer
26:35
and you get x one
26:36
and you feed x one back to this transformer
26:40
right? So, and
26:43
and then you generate x two by sampling from the softmax
26:48
and then you feed it back
26:49
So that's kind of why it's called autoregressive
26:51
because you You have to
26:53
like when you are generating
26:54
you don't have the sequence
26:55
right? When you are computing log likelihood
26:57
you see the whole
26:58
you already see the whole sequence
26:59
You are just trying to figure out
27:00
what's the chance of seeing this sequence
27:02
But when you are generating
27:03
you start with nothing
27:04
and then you have to generate your sequence
27:06
yourself
27:06
And that's why you start with the beginning of sentences token
27:10
and then you generate first token
27:12
and then you
27:12
give the first token back to transformer and compute this softmax
27:16
and then you generate second token
27:19
you do a sampling to generate second token
27:20
and you give it back
27:21
and you keep doing this
27:23
So that's the autoregressive generation
27:26
the
27:28
and sometimes, you know, like
27:30
this generation only start from certain part
27:33
because in many cases
27:34
you are not generating from nothing
27:37
So you are given maybe a few tokens
27:39
maybe you are given
27:40
so the, sometimes it's called prompt
27:43
So prompt is like something
27:44
the models are given. It's maybe like a question
27:46
you know, or instructions
27:47
so on and so forth
27:48
You say you are given
27:50
X amount to X K
27:51
And then the model is supposed to complete the generation
27:55
so it start with X T plus one
27:56
So X T plus one
27:58
so this is given
28:00
And then you start with X T plus one
28:02
you first generate X T plus one from
28:03
this softmax
28:05
of the F theta X one up to X K
28:10
and then you generate X T K plus two
28:14
and then you do the same thing
28:15
So it doesn't have to be always from
28:17
the beginning
28:18
And then there's another small thing
28:20
which is that when you generate
28:22
you don't necessarily have to always follow the same
28:26
model. And there's one thing that you can deviate from it
28:28
which is that you can
28:30
add additional temperature where
28:35
you say, as opposed to generating From the original
28:40
probabilistic model I'm generate
28:42
I'm gonna divide this by a temperature T
28:45
tau, or T sometimes
28:48
so, and, what this ha- what
28:50
what does this mean is that you change the
28:53
you don't change the orders between these
28:55
choices, right? So this is a
28:58
this, this F theta is a vector
28:59
right? It's a
29:00
a V-dimensional vector
29:01
that describes the logis for each of the
29:03
tokens in the vocabulary
29:05
And when you divide by tau
29:07
you don't change their relative orders
29:08
The bigger ones are still bigger
29:09
the smaller ones are small
29:11
still smaller, but you change the
29:13
the scale
29:14
So, so for example, before you could have like a
29:19
so suppose, you know
29:22
F theta X zero up to X K is
29:26
is equals to, let's say, maybe like a two
29:30
one, minus one, something like this
29:34
And if you divide by tau
29:35
let's say suppose you divide by tau
29:37
which is very, very small
29:38
Suppose tau is almost zero
29:40
Then after dividing by tau
29:41
like if you divide by tau
29:44
then what happens is that maybe this becomes two hundred
29:47
this becomes one hundred. I'm
29:48
I'm giving you a concrete example
29:50
which is, a little bit unrealistic
29:51
but, to demonstrate a point
29:53
And what's the difference between the softmax of
29:55
this vector versus the softmax of the readjusted
29:58
vector?
29:59
The difference is that
30:00
the softmax of this is much sharper
30:03
So maybe let me just give you an example here
30:05
So, so if you just have
30:08
let's say, maybe I just have three dimensions
30:10
so that, it's easier to demonstrate a point
30:16
So if you do softmax of this vector
30:22
of two, one, minus one, what happens is that you get
30:28
in some sense you get exponential two
30:30
over exponential two
30:33
Plus exponential one plus exponential minus one
30:37
That's the, after softmax was the probability of the first
30:41
token, right? And the second token will be exponential one
30:46
exponential two plus exponential one plus exponential minus one
30:52
right? And the third one will be
30:54
the, the same denominator, the different numerator, right?
30:57
So that's, that's the softmax of this
31:00
But if you take the softmax of the two hundred
31:02
one hundred minus one hundred
31:03
suppose you do that
31:11
then this will give you
31:13
exponential two hundred
31:16
versus exponential two hundred plus exponential
31:21
one hundred plus exponential minus one hundred
31:25
blah, blah, blah
31:27
So what's the difference here?
31:28
The difference is that in terms of the
31:30
the three numbers, in terms of the order
31:31
it's always like this is bigger than this
31:33
this is bigger than this
31:34
right?
31:34
Because 200 is big
31:35
2 is bigger than 1
31:36
1 is bigger than minus 1
31:37
the order is always maintained
31:39
But the, the, the relativeness
31:44
scale is different
31:45
because here you can see exponential 200 is way
31:48
way bigger than exponential 100
31:50
The difference is like so astronomical
31:53
like it's too, the
31:54
the difference is too
31:55
big so that
31:56
these two numbers are just so small compared
31:57
to exponential 200
31:59
So they are even negligible
32:01
So basically, in this calculation
32:03
these two numbers don't even have to show up here
32:05
like it, it don't even matter
32:06
So you just get what?
32:07
So basically, the first token will just have probably exactly
32:10
almost exactly what, maybe 0.9999 something
32:12
And then if you look at the second entry
32:16
which is exponential 100
32:18
divided by exponential 200 plus exponential 100 plus exponential minus 100
32:27
Again, everything is negligible compared to exponential 200
32:31
So this is not very useful
32:32
this is not very useful
32:33
this is kind of like close to zero
32:35
So basically this entry will be zero
32:37
So that's why after you
32:38
you, you divide by tau to make this bigger
32:41
then this is just approximately one
32:44
zero, and zero
32:45
So that's
32:45
that's what I mean by it's much sharper
32:47
So you, you, you, you sharpen the
32:49
the distribution to make it
32:51
most of the distribution to be
32:53
the, the probability mass to be on the
32:54
the most likely token
32:56
And however here
32:58
exponential two and exponential one are not that
33:00
different
33:00
If you compute this
33:01
you'll get something like maybe
33:03
the first one will be like maybe point seven
33:05
the second one will be point three
33:06
I, I don't know
33:07
like we can use a calculator to see
33:09
But, you know
33:09
it will still be like this one is bigger than this
33:12
and the second one is bigger than the third one
33:13
but it won't be like
33:14
all the mass is on the first token
33:16
So, so
33:17
so basically what I'm trying to say is
33:18
that if tau is a small number
33:20
so that you You
33:22
you amplify the logis
33:24
then you are sharpening the
33:25
resulting distribution to make it more like a
33:27
deterministic distribution
33:28
And then as a result
33:29
you are just generating
33:32
you are just sampling from the
33:34
the most likely token. You are taking the
33:35
max as opposed to sampling from it
33:38
So, and, and, and that's actually what people actually
33:44
do, like they say, basically, like I think actually
33:48
in API call, you have the choice to choose the temperature
33:51
and if you choose a small temperature
33:52
it means that you want more deterministic generation
33:55
if you choose a temperature to be zero
33:56
it means that your generation will just be
33:58
always deterministic every time you take the largest on
34:02
on, on most likely token
34:04
and then keep going on
34:05
right? And of course
34:06
you can also change the temperature to be smaller than
34:09
bigger than one, which makes the, the distribution more softer
34:12
so that there's a
34:13
you, you focus on more the long tail
34:14
like you can generate more
34:17
tokens from the tail
34:18
so you have more stochasticity and
34:20
and certainty in your generation
34:22
Sometimes that's useful because
34:23
you want to encourage some diversity in your generation
34:26
and you, you do that
34:27
especially in the RL
34:28
sometimes you have to do that
34:31
cool
34:32
Okay, sounds good
34:34
And sometimes there are also other ways of sampling
34:36
which is, probably I shouldn't go into details
34:39
there are some notes
34:40
in the lecture notes
34:42
So another way to do it is that you
34:45
this vector right now I have three dimensions
34:47
but actually it's like a
34:48
the dimensionality is V
34:49
right? Like, which is like 250 or something
34:52
You can take top K
34:54
and then drop all of the other
34:55
dimensions. So you only sample from the
34:57
the most likely K tokens
34:59
and you re-normalize the distribution
35:01
So basically the, the first K tokens may not cover the
35:04
all the probabilistic, probabilistic noise
35:07
So
35:08
but still that's probably easier to generate because
35:10
you don't want to generate too rare tokens
35:12
so you just drop everything
35:14
below top K and then generate
35:18
okay, sounds good. So I think I'm probably
35:27
And then, okay, so once you have all
35:29
so this is the generation
35:30
and then for the training
35:32
I think I already kind of like describe it
35:35
a little bit
35:36
so when you do the training
35:40
you just, computer log likelihood, the negative
35:46
log likelihood
35:52
The negative log likelihood
35:55
and you say this is a log
35:57
so if you have example x1 up to xt
36:01
so the minus log p theta x1 up to xt
36:06
capital T, is just a minus log
36:11
the product of all of this
36:13
right, p theta x1, p theta x2 given x1
36:19
so on and so forth
36:27
And this is the
36:29
minus the sum of the log of the
36:31
p of xt
36:33
given x1 up to xt minus one
36:37
And then you change this to
36:40
log of the softmax
36:44
of p1 of this f theta
36:53
And here, this XT is a particular choice of XT
36:56
right, which is seen in the data
36:59
So that means you are looking at the
37:00
XT's entry of this softmax
37:02
This softmax will give you the probability for every possible tokens
37:06
and then you are only interested in the
37:08
probability to see that particular X token XT
37:10
so that's why you're looking at XT's index of that vector
37:15
and that's a single scalar
37:16
You take the log
37:17
and you take the sum over T
37:19
and that's your loss function
37:22
And then you can optimize this loss function by
37:24
for example, some optimizers, like, SGD or IDAM
37:29
we didn't discuss IDAM in this lecture
37:30
which is an extension of the X
37:32
SGD optimizer
37:33
but mostly
37:34
you can just call the
37:35
optimizer as a black box to optimize the
37:37
loss function
37:54
It's actually not the last one
37:55
right? Because this is sum from t
37:57
to one to capital T
38:00
The little t and larger
38:10
Okay, sounds good
38:13
Okay, so then in the next
38:15
thirty minutes, I'm going to tell you what this
38:18
this function f is
38:19
right?
38:20
Or maybe what's hidden in this transformer
38:22
What's the operation we have to do to compute
38:25
the, the, the logic here?
38:29
and it's pretty complicated
38:30
so, it probably takes more than thirty minutes
38:33
but I'll see how much I can do
38:36
So
38:41
any questions before that?
38:43
Okay
38:44
So at a high level
38:46
this transformer has multiple layers
38:49
And, so you start with like EX1 up to EXT
38:54
and you first apply a so-called attention layer
38:58
which I'll define, and then you apply the so-called MLP
39:02
So, first of all, the attention layer
39:05
the attention always takes in
39:08
a sequence of vectors and output
39:09
a sequence of vectors
39:11
So you output a bunch of vectors
39:13
at each position you output one vector
39:15
and then
39:17
each of these vector will go through a
39:18
MLP
39:19
MLP means like a multilayer perception
39:21
which is just a two or three layers
39:23
two or three layer network
39:25
And, each of these vector will go through this MLP
39:31
and then you apply another attention
39:33
This attention will takes in many
39:36
many vectors, and, and then output a sequence of vectors
39:40
and then you do the MLP
39:44
So that's the, the high-level, structure of this, attention
39:49
And eventually, you have to output
39:51
remember, eventually
39:53
you have to output a sequence of vectors
39:54
So basically, you do enough of these alternations between these two
39:58
and eventually, you have this, MLP
40:01
and you output
40:03
some vectors u1 up to ut
40:04
And you alternate between attention and MLP
40:09
That's the high-level idea
40:12
And, and I think we'll focus most of the
40:15
our energy on defining attention
40:17
which is kind of like the most
40:19
new thing, right? Because MLP is just a two-layer network
40:22
with some special kind of gating
40:24
or kind of like
40:25
some kind of like
40:26
special activation functions
40:28
and another small thing is that
40:29
the MLP are sharing the parameters
40:32
and they are applying independently
40:34
So each of these vectors is
40:37
Processed by the MLP in an independent fashion
40:39
there's no
40:40
kind of dependencies between
40:41
positions when you apply the MLP
40:43
They are just very separate operations
40:45
So the only way that you are using
40:48
you are kind of like
40:49
you fuse the input from different positions or
40:52
different time steps is the attention
40:54
So the attention will look at all those vectors and
40:57
and process a little bit and then output
40:58
a sequence of vectors
41:00
And the MLP is
41:00
is very parallelized and parallelizable and independent
41:05
Okay, so now let's, let's define attention
41:15
This entire thing will be treated as a single system
41:17
So, so, so basically once you define the architecture
41:19
you apply the auto-differentiation
41:22
you, you, like, you treat this whole thing as a
41:25
as a gigantic network
41:26
yeah, end-to-end
41:37
Yeah, but auto-differentiation still works
41:39
So, so the auto-differentiation, I think as we discussed
41:43
if you remember, like
41:44
it's kind of like fundamental is claiming that
41:45
if you have a way to compute your forward pass
41:48
as long as there are compositions of many
41:50
many different components, you know
41:52
like, basic components
41:53
you can compute the backward pass the same fast
41:55
the same way
41:56
So here is, is indeed a composition of many
41:59
many small arithmetic operations
42:01
because as you see
42:02
the attention layer has many small matrix modifications
42:06
and the MLP has many matrix modifications
42:08
and you can automatically differentiate
42:14
okay, so
42:17
okay, so now let's, let's talk about attention
42:30
So for attention
42:32
one thing to remember
42:33
first is that the type signature is that your
42:36
your input, a sequence of vectors to the attention
42:39
and they output a sequence of vectors
42:41
And, and
42:43
we're gonna start with the so-called single header
42:45
tension
42:51
So the single header tension is like this
42:53
So you first have
42:56
a input as a
42:58
a sequence of vectors
43:00
let's say H one
43:03
say, in up to H T in
43:07
And, and you want to output
43:09
you know, H, I'm trying to define the
43:11
the type signature
43:13
So you output
43:16
H one out up to H T out
43:20
And what you do is the following
43:21
So
43:23
the first thing is that you turn
43:25
There's input vectors into a set of query key
43:29
and, and value vectors
43:32
So
43:34
so you define, so basically you say H
43:38
one in, you multiply it by a matrix W Q
43:42
this is a weight matrix
43:43
this is a parameter
43:46
And, by the way
43:48
let's treat all of this as row vectors
43:49
and then all the multiplication is on the right-hand side
43:51
as opposed to on the left-hand side
43:53
So every vector is a row vector
43:55
and you multiply on the right-hand side
43:56
you get another row vector
43:58
So, so this H in
44:00
let's say this is like a dimension D
44:02
R D, let's say, say R one by D
44:05
and this W Q is in dimension
44:09
so this is R one by D
44:11
and this W Q is in R D by
44:17
D H
44:18
dh is a parameter
44:20
it's a dimension you choose
44:22
and, and then the resulting thing is called q1
44:28
which is in dimension dh
44:31
maybe one by dh
44:32
depending on how you think about it
44:35
and you can do this for every index
44:37
every time step. Here I'm only doing it
44:38
for time step one
44:39
you can do it for time step t
44:41
so you have qt
44:43
So basically this give you a factor
44:44
a sequence of vectors q1 up to q capital T
44:47
each of them is in dimension
44:50
one by dh
44:52
And this called, these are called queries
44:56
And
44:57
you do the same thing for keys and
44:59
values. So keys Let me define them first
45:02
So it's the same thing you say the
45:05
hT in multiplied by your so-called key projection matrix
45:10
W K, and that's your K T
45:12
And the dimensionality, I think, is the same
45:14
so this is also in the same dimension
45:17
and the K T is in the same
45:18
dimension. And it's the same thing for value func- value vectors
45:21
not value functions, hT in times W V
45:28
So in some sense
45:30
maybe
45:32
maybe this is the right time to
45:34
talk about some of the intuitions
45:35
you know, for the tensions
45:36
which is not that obvious because eventually it just works
45:39
you know
45:39
like nobody really know exactly how it works
45:42
but the, the
45:43
the rough intuition is that you have to
45:44
have some interactions between these inputs to make it meaningful
45:48
right? Because if everything is independent
45:49
then you're just
45:50
dealing with all the time steps independently
45:52
So that's why you have to have to inter
45:55
interactions. And interactions sometimes is about
45:57
how do you kind of like
46:00
retrieve or kind of pay attention to
46:02
each other
46:03
So that's actually how the word attention come from
46:05
So basically, when you are processing the
46:07
maybe this vector
46:09
you probably should attend to
46:11
or pay attention to some other vectors which are
46:13
have relevant information
46:15
And which one you
46:16
you pay attention to
46:17
is what we're going to decide here using
46:19
with this query and key and vectors
46:21
So in some sense
46:22
intuitively, the attention will, what attention will do
46:24
which I didn't show you yet
46:26
is that you try to
46:29
for processing each of the vectors
46:31
you try to figure out
46:32
which are the relevant vectors that has relevant information
46:36
and, and then you pay actual attention to those vectors
46:40
And, and then you compute the output depends on
46:42
depending on those special vectors to some degree
46:46
So that's what we are trying to do
46:48
and
46:49
and to kind of like decide which one
46:50
you pay attention to
46:52
you have to prepare these queries and key and value vectors
46:56
and how to use them
46:57
So use the query and keys as a
46:59
way to decide which one you pay attention to
47:02
So basically, let's say, suppose you have, the T's position, right?
47:08
I want to decide
47:10
what the T's position should pay attention to
47:13
and, and
47:14
and the way to do it is you
47:15
say you compute the query Q T
47:19
this is, the query at T's position
47:22
and you multiply the query
47:24
with the keys at all the other positions
47:28
So maybe I, I will just draw it
47:30
So you have queries Q one up to Q T
47:34
and one of them is Q little t
47:37
and you have Q
47:38
K one up to
47:41
K T
47:43
And you, you multiply Qs with all the Ks
47:49
so you have Q1 times
47:51
you know, K1 transpose, Qt times K2 transpose
47:57
and Qt times K capital T transpose
48:02
and there is a
48:04
sometimes there's a scaling factor here
48:06
to kind of like normalize this in a way that
48:09
makes it a little bit more kind of like well-behaved
48:11
but I think for the moment you probably
48:13
don't have to pay too much attention to the scaling factor
48:15
so
48:16
the important things that you are taking the
48:17
inner product between Qs and
48:19
each of the other Ks
48:20
And this Qt transpose K
48:23
Q, K transpose is actually inner product
48:26
because Q is a
48:28
is a row vector
48:29
and K transpose is a column vector
48:34
And you are taking the inner product between
48:35
these row vectors and the column vectors
48:39
and you get one scalar out of it
48:41
So So this is
48:44
all of these numbers are
48:45
are real numbers, right? So this is in R T
48:47
so you have T real numbers
48:51
and then you take a softmax
48:52
over these T real numbers to turn it
48:54
into a probability vector
48:55
So then you say
48:56
I'm gonna do a softmax
49:01
Say softmax
49:04
of this whole thing
49:14
And now you get what?
49:15
You get a, this is a vector
49:17
the dimensionality here is R T
49:19
and each of this
49:21
this is, the
49:22
the outcome of the softmax is a sequence of numbers
49:24
right?
49:27
And, they are T numbers
49:28
and, and all of these numbers are non-active
49:30
and the sum of them is one
49:32
So it's a probability vector over T positions
49:35
And if you see a large number
49:36
it means that Let's suppose this number is big
49:41
it means that
49:42
this number is somewhat big
49:43
compared to other ones
49:44
It means that the Qs and Qs have
49:46
like more higher inner product
49:48
right?
49:49
And, it means that you should pay more attention to K
49:53
K1, right? So
49:54
so basically if either of these entries is big
49:56
it means that
49:58
intuitively Qt needs to pay more attention to K1
50:03
and
50:03
and that will be reflected by the probability vectors because the
50:06
the corresponding probability will also be high
50:09
And, so I think let's call this
50:11
I guess I have a notation here
50:13
so let's call this
50:14
let's denote this to be Pt1
50:18
Pt2 up to Pt capital T
50:23
So the
50:23
so each of the PtI is
50:26
Each of the pT i is non-negative
50:29
and the sum of the pT i
50:31
over i from one to t
50:33
is equal to one. So this is a probability vector
50:36
and, and if this, if, and if this entries is big
50:38
it means that we should pay more attention
50:40
to that particular position when processing the
50:43
little t's query
50:45
And then how do you turn this
50:46
how does this relate to the final
50:49
outcome?
50:50
So finally, once you have this
50:52
then, you say the, the output at t's position
50:58
which is a vector
50:59
is using these probability vectors to weight
51:03
the values
51:05
The values is, v's, which are we are defining here
51:10
right, v t, sorry, v t
51:11
So you say it's pT
51:13
one times v one plus
51:20
pT
51:20
two times v two plus pT t times
51:22
v t. So basically
51:23
if each of
51:24
any of this
51:25
Probability is
51:27
much higher than you are multiplying this high probability with the
51:31
the corresponding value at that position
51:34
So basically you are using the V
51:36
right, if, for example, if this number is very big
51:38
then you are using V2
51:40
a lot in the output at time T
51:48
So you determine them by multiplying the matrices
51:51
and the matrices will be trained
51:55
like, like, yeah. How
51:56
how they attend to each other is also trained
51:58
Like, so
51:59
so you're only giving a template in some sense
52:01
Like, how, how, you know, like, it may be that
52:03
you know, like
52:05
it could be that all of these are zero
52:07
you know, then you are not doing anything with it
52:09
and then your neural network
52:10
the loss function will be not good
52:12
but it's still a valid neural network
52:14
it's just the loss is not good
52:15
So eventually you are going to train the
52:17
model to behave well
52:20
so that
52:20
so basically
52:22
The training algorithms will choose the right
52:24
W so that this mechanism is
52:27
doing the right thing
52:43
So, let's try to articulate
52:45
you know, I think it's a good question
52:46
but let's try to articulate so that we
52:48
can answer it
52:49
So what's the
52:54
your
53:00
Yes. Why ain't, particular education, like I think this year
53:08
that would be available for the interview when you train it
53:12
and it's like not five minutes
53:13
so we need to
53:14
like, continue this
53:17
So I think you are asking why you need such architecture
53:19
but not just only using MLP
53:21
for example
53:21
right? That's one way to rephrase the question
53:25
I think
53:27
so if you are only using MLP in this way
53:30
suppose you remove all the tensions
53:32
you are only using MLP in this way
53:34
then there's no interactions between different positions whatsoever
53:38
So there's no chance that it's gonna work
53:39
right? You are just applying
53:41
separate networks for each of the tokens
53:44
there's no chance you can understand the
53:46
the, the, the, the dependencies, you know, the, the meanings, right?
53:49
Like, there's no chance I
53:50
I can just give you
53:51
individual words and you can understand a sentence
53:53
There's no chance, right?
53:54
So, so you do have to
53:57
have a model that can read
54:00
all the
54:01
tokens, you know, and, and use
54:03
and study their kind of correlations and combinations
54:05
so on and so forth
54:06
right?
54:06
But, there are other possible
54:09
possible ways to deal with that
54:11
For example, what if you use
54:12
use a gigantic MLP
54:14
that takes in all of this
54:15
maybe you concatenate all of them
54:17
and you take in one MLP
54:19
right? So, so, for example
54:21
you just concatenate all of these tokens
54:23
vectors into one long vector
54:25
and you go through a network
54:26
So there are several issues with that
54:28
One is that, you, will have way too many parameters
54:32
because your sequence length
54:34
Maybe sometimes even millions
54:37
Then if you have a million by million
54:38
matrices in your MLP
54:40
it's going to be way too big
54:41
Right now, the MLP is actually very small
54:44
It only takes in a vector
54:45
for one position. It doesn't even depend on
54:47
T
54:48
And another thing is that you're not only the
54:50
the size of the MLP is big
54:52
but also the size of the MLP is also changes over
54:55
as you change the sequence length
54:57
which is also very undesirable
54:58
Because, for example, if you train with sequence length a thousand
55:01
and then you test with sequence length a thousand and one
55:04
then you have to change your parameter count
55:06
which is not good
55:08
And, and here, if you really look into details
55:10
you'll find out that
55:11
the number of parameters doesn't depend on the sequence
55:15
length
55:33
Exactly, right? So, so, so you, you know that, you know
55:36
if you have sufficient compute
55:38
you know, probably a gigantic MLP will work
55:40
If you have sufficient data so you don't overfit
55:43
and you have sufficient compute
55:44
then there is some MLP that can actually solve this problem
55:47
but that's too expensive
55:49
So you need to prescribe some
55:52
structure so that you can reduce the complexity
55:55
right? And, and this is the
55:57
the, the prescription we have
55:58
but you still have to leave enough flexibilities
56:01
You cannot just say
56:02
oh, for example
56:03
one way to do it is you just
56:04
say let's average
56:05
Then there's no parameters
56:06
you just, prescribe too much, right?
56:07
You, you, you dictate too much
56:09
then that's not good. So it's kind of like an
56:11
you need
56:11
you need to strike a balance between
56:13
how much structure you introduce and how
56:15
much flexibilities you give the models to figure
56:17
out
56:19
Yeah
56:22
yeah. Any other questions?
56:26
By the way, there is actually, in the history, there was
56:28
actually other ways to deal with this sequence
56:30
For example, you can use a convolutional network on this
56:33
which is kind of like MLP
56:34
So you have use a
56:35
use a MLP, but with a, a specific receptive field
56:39
so that each time you only see a
56:41
kind of like a window
56:43
I think there are still papers that show
56:46
says that that's not a terrible idea
56:49
Actually, if you use a convolutional network
56:51
it ac-actually can work to some degree
56:52
It's just not as good as this one
56:55
you know
56:56
and, and, and also you can see
56:58
like, there are also other
57:00
probably we'll talk about that later
57:01
so there are other variants of tensions to
57:03
that does similar stuff
57:05
but have better efficiency
57:07
anyway, so let's continue. We ha-we are not done yet
57:11
actually. This is only a sin-so-called single-head attention
57:14
you know
57:15
actually
57:16
the real thing is a little more complicated
57:19
but Let's see. Before I go into that
57:22
let's see
57:27
So I think
57:29
you can see the notation skills is very messy
57:31
so
57:32
let's try to
57:33
simplify it a little bit
57:34
so that we don't have to always work
57:36
with this messy notation
57:38
So
57:39
one typical way that people simplify this is
57:41
that you
57:41
try to organize all the q-queries
57:44
into a matrix
57:47
We call that each of the queries a row vector
57:49
and if you put them all in your matrix
57:51
then this is a
57:52
a matrix of dimension T by dH
57:56
and you can put all the keys into a matrix
58:02
and you can put all the values in
58:03
in your matrix as well
58:09
right, so, and then you can write this H out
58:22
The C- all two-seating factors in either kind of like
58:28
also as a matrix is
58:29
in some sense
58:30
you do a softmax
58:32
of the row of
58:34
this Q K transpose
58:36
divided by C matrix times V
58:39
So here the, basically this Q K matrix
58:41
Q K transpose matrix is the
58:44
the formula of Q
58:46
I times K J transpose divided by C
58:50
So every entry is of this form
58:53
which is kind of like this guys
58:56
And then if you do a row softmax
58:58
that corresponds to this softmax
59:00
And then if you multiply by V
59:01
it means, it means this linear combination
59:04
I think you, we probably, we don't have time
59:06
enough time to go into the details
59:09
like, but you can double check
59:11
you know
59:11
it's just a very basic linear algebra
59:13
It's just real, real and ranging, like, it's
59:14
it's mo-mostly just notations
59:16
Right
59:16
so this Q case
59:17
divided by C matrix is the inner product
59:19
between the queries and keys
59:21
and then you take a row
59:22
row by softmax to get the probabilities
59:24
So after the softmax
59:25
this matrix becomes, every row is a probability vector
59:29
and then you multiply that with the V to
59:31
take the linear combination of Vs.
59:35
and
59:36
there's one
59:37
thing that we kind of cheat a little bit here
59:39
which is that if you look at this architecture
59:43
then
59:44
it's not true that the T's output
59:49
only depends on everything before it
59:52
There's no this kind of
59:53
like, auto-recursive kind of, like, properties yet, right?
59:56
Because the output at time T Depends on everything
60:03
right? It depends on V capital T
60:05
even the last input
60:07
V capital T
60:07
V capital T is a function of the last input
60:10
And, so
60:12
and the output at time T depends on the input
60:16
depends on V T and it depends on H T
60:18
in
60:19
So
60:20
so basically there's no kind of like this
60:21
dependency. So then what you do is you say
60:25
you introduce the concept of masking to
60:28
make sure that there's no
60:31
this kind of like
60:32
wrong dependency you want. So basically what you
60:34
really want to do is that you just
60:36
in some sense, for H little t
60:39
you want to delete everything after
60:43
time T. So basically
60:44
let's say, before you have this
60:48
you know, P T V T plus
60:55
P T T V T
60:57
P T capital T V capital T
61:00
So basically you say these dependencies are invalid for me
61:03
so I just, don't have them
61:04
So basically you just delete all of this
61:07
And
61:08
and then you renormalize these Ps
61:10
so that the sum of these Ps
61:13
got normalized to one. So it's still a
61:14
linear combination or convex combination of the V
61:17
one up to V T
61:18
So that's pretty much what you do
61:19
So you, you just say if the dependency is
61:22
not good, you just
61:23
in some sense mask them out
61:25
And mathematically what you do
61:27
actually also implementation-wise, what you do is through this masking
61:31
which means that you
61:39
which means that you add
61:43
a masking here
61:45
Maybe let's use a different color to
61:47
show this
61:49
So you add a masking here
61:51
I think they're different by their color
61:59
Okay
62:00
So you add a mask here so that
62:04
you can mask out the
62:07
Let me see. You can mask out half of the
62:09
this matrix
62:11
So, so you have like a Q1
62:14
K1 transpose, Q1, K2 transpose
62:17
so on and so forth
62:19
And Q1, K2 transpose is not valid in some sense
62:22
because you are looking
62:23
you are tending to something you shouldn't attend
62:25
right? Q1 shouldn't be able to see K2
62:28
So basically you add this masking matrix
62:30
what you add is that you add minus infinity
62:35
you add minus infinity here
62:39
And for Q1, K3 transpose, you also add minus infinity
62:45
so on and so forth
62:47
And then in the next row
62:48
you have Q2, K1 transpose, which is good
62:52
You have Q2, K2 transpose, which is good
62:54
And you have Q2
62:55
K3 transpose, which is not good
62:58
So you add minus infinity here
63:00
so on and so forth
63:01
So basically everything here
63:03
will be adding additional minus infinity
63:06
So basically this upper triangle
63:09
part of this matrix will be all minus infinity
63:11
You know, I mean, we'll
63:13
By adding minus infinity
63:14
it just means like you
63:15
you turn it into minus infinity
63:16
right? Any number add
63:18
added by minus infinity is the minus infinity
63:20
And then what happens is that
63:22
or in other words
63:23
it's kind of like you are just
63:25
trying to
63:27
add all of
63:28
minus infinity to all of this part
63:30
And then when you add
63:31
when you turn all of this into minus infinity
63:33
when you do the softmax
63:35
this part just doesn't help
63:37
doesn't change, doesn't
63:39
like, doesn't, by this part, I mean, everything after T, right?
63:44
So everything after little t
63:45
you, you turn them into minus infinity
63:46
and after you do a softmax
63:48
they become zero
63:50
So because exponential to the minus infinity becomes like
63:54
so, so, so, become basically zero
63:56
So that's why in this P
63:59
so everything after PTT
64:04
everything after Q will become zero
64:06
and the
64:07
the sum of the rest will be one
64:09
So that's exactly what you
64:10
how you achieve this
64:11
because
64:12
basically the leading combination of the rest of
64:13
the terms doesn't matter because their coefficients are zero
64:23
Yeah, we are masking this because you shouldn't
64:26
in this autoregressive model
64:28
you don't allow
64:30
the output
64:31
at time t
64:32
to depend on anything that is after t
64:35
So you have to respect this kind of
64:36
like a sequential dependency
64:38
So, you know, in generation, you
64:41
you are not able to see the future
64:43
tokens to generate a current token
64:45
So
64:46
so every
64:47
token can only depends on everything before it
64:50
So that's why we have to mask it out
64:57
Okay, we have, yeah, we are almost done there
65:00
Okay, so
65:06
okay
65:06
so this is the so-called single head attention
65:10
and we have to make it
65:11
more complicated to even finish the attention part
65:14
So then what we do next is that
65:18
maybe let's erase here
65:30
So in some sense
65:31
what we are doing here is like we
65:33
have H1 in
65:35
And hT in
65:37
and we go through this single header tension
65:41
I guess, in the note, I call it
65:43
a tension one h
65:45
and you get this h out
65:50
h one out, up to hT out
65:53
That's what we define so far
65:55
And, in practice
65:57
you have multiple copies of this single header tension with different
66:02
weight matrices
66:03
So you get another copy
66:05
of the single header tension
66:08
Maybe let's call this single header tension one
66:11
this is called single tension
66:12
header tension two, and they have the same
66:17
input, and then you drop
66:19
you get the, another copy of the output
66:25
But here, by copy, I don't really mean literally copy
66:27
because then it's not useful
66:28
So what's, the difference between this and this is that the
66:31
the weight are, the weight matrices are different
66:36
So basically this WQ
66:38
WKWV for these two copies of the tension are different
66:43
So, so, so that's why they're not really literary copies
66:46
they are just, like you have another
66:49
another version of it
66:51
And
66:58
yeah, so
66:59
so in some sense you should index this
67:00
by something else because the output will be different
67:02
right? So maybe you should index this by
67:07
let's say this is
67:09
because this is the first attention
67:10
you say H11 and HT1
67:12
and this is H12
67:14
HT2
67:15
And you also have another one
67:17
and you get H13 up to HT3
67:23
So, so, so when you have so many
67:25
copies, you have, several sequences of vectors
67:28
but eventually you only want one sequence of vectors
67:30
so you have to combine them in some
67:32
way
67:33
And the way to combine them is that you
67:37
you concatenate these three things
67:43
So you concatenate
67:46
H one one out
67:47
H one two out
67:53
one, I think how many
67:56
each of this is called one hat
67:58
So you have, let's say, N H
68:00
N H hat
68:02
So this is a tension N sub H
68:06
So you concatenate all of these vectors
68:09
and you get, and then you multiply that with
68:11
another projection matrix, and then this will give you the final
68:17
H, the output at one
68:21
So that's the final output at position one
68:24
And you do the same thing for position two
68:25
where you concatenate
68:29
maybe these guys, these guys, And
68:33
and then you do a projection
68:34
you get the output at position two
68:37
So in some sense
68:37
this is just give you more power
68:39
to compute different kind of like
68:44
kind of attention. In some sense
68:45
you know, I think actually in practice
68:47
if you
68:48
I think the intuition is that
68:50
maybe each of these attention is doing
68:52
single head attention is trying to pay special type of like
68:57
is doing some special type of job
68:58
right? For example
68:59
maybe here it's trying to find out
69:01
which entity in the path passage matters to me
69:05
and here this is trying to Find out
69:08
you know, which
69:11
what are kind of like the sentiment
69:13
of the past passages
69:15
and, and, and which part of the sentiment matters to me
69:17
so and so forth
69:18
right? So each of these attention probably pay
69:19
attention to different type of information
69:22
and then eventually you aggregate the outcome of
69:24
all of this information into one
69:27
big vector, and you project it into
69:30
a single output
69:39
Cool
69:42
No, no, they are, they are all parallel
69:43
yes
69:45
They are, they are parallel, and then you aggregate
69:46
So
69:47
and all of this is in one of this
69:50
And then
69:51
so basically in one of this you have multiple parallel heads
69:54
and then next round you have another
69:56
number of parallel heads
70:03
I think the number of heads is
70:04
maybe on the order of like maybe And most a hundred
70:07
I guess?
70:10
let me think about this
70:11
I think something on that order
70:14
I think
70:15
Yeah. It also depends on the model size
70:18
of course, right? I think the
70:19
the biggest one, I don't know
70:20
I don't know whether a hundred is
70:22
is for which model size
70:23
I think maybe for like a
70:25
a hundred billion parameter models we have
70:27
have a hundred heads. Yeah
70:28
Yeah
70:52
Yeah. I think, I, I don't know exactly what's your proposal
70:55
but, maybe it, it sounds like a simple
70:58
simplified answer, but like a, like a
71:00
like many of these proposals has been considered in
71:03
in various ways, and, and some of them are For
71:06
on most of our efficiency perspectives
71:08
because eventually you can just have a
71:10
like as we discussed
71:11
you know, like if you really don't care about efficiency
71:13
you can just have a generic
71:15
very gigantic network space. So
71:17
all of these kind of like
71:19
special structures to try to
71:21
use your parameters in the right way so
71:23
that you can use it more efficiently in some sense
71:31
okay, so
71:35
and then, okay, so I guess, let's try to
71:38
let's just give me two or three minutes to talk about
71:41
to talk about some very quick things
71:44
So one thing is that actually when I draw this
71:49
it's actually a bit more complicated than
71:51
Just doing this
71:52
So because there's also a residual part in the
71:56
in this kind of like how to
71:57
combine the MLP with attention
71:59
So, so actually what
72:01
actually you really do is that you say you apply
72:07
so after you do the
72:09
attention, you
72:14
you do, some kind of like normalization here
72:18
So there's a normalization layer
72:20
Often this is using the R
72:21
RSM norm that we
72:22
briefly discussed involved in the network section
72:25
And then you apply MLP
72:26
and then you add the residuals back to this
72:30
You, you
72:31
you kind of like in the residual network
72:32
that we discussed
72:33
And then you go to the next layer
72:35
And there are different ways to kind of
72:36
like add residuals. You can add residuals after you do the
72:40
attention or, or before you do the attention
72:42
So there's something called Pre-norm and the
72:45
and post-norm, thing, so I guess the details you
72:48
you, you can check out the lecture notes
72:50
so there's some kind of like orders on how you
72:52
when you apply your normalization
72:53
residues, and attention, so on and so forth
72:56
And, and finally, I think one important thing
72:59
especially for the next lecture about the system ML
73:03
perspective, is that the, we should try to quickly
73:08
think about the computational efficiency here and the memory foot
73:12
footprint
73:13
So how many operations we have to do
73:15
especially as a function of the
73:17
of T
73:19
as a function of other things
73:20
it's actually relatively easier because as a function
73:22
of the number of parameters
73:23
you know, like all of those is kind of like a
73:25
it's all multiple kind of and linear in
73:27
some sense
73:27
The main important thing is what is the
73:29
the dependency on T
73:31
And you can see that for each of the T's position
73:34
you have to compute all of this inner
73:36
products
73:37
So for each of the position
73:38
you have to compute capital T inner product
73:41
So, and
73:42
and you have capital T of these positions
73:45
So that means that you have to
73:47
basically do a T squared dependency
73:49
right? So, so, or in other words
73:51
you know, even this, just this matrix itself is a T
73:56
T by T matrix
73:57
capital T by capital T matrix
73:58
and each of these entry requires some inner product
74:01
So, so I think the dimension
74:03
if you still remember the QIs
74:05
or the QTs are in
74:07
are dH
74:08
and you need to compute T squared of this inner product
74:11
so basically the
74:13
the number of operations is T squared times
74:15
dH
74:16
The dependence on dH is less relevant
74:18
the, the, you know, less, less important to some degree
74:20
the dependence on T squared is very important
74:22
because this means that if you have
74:24
capital T being a million
74:26
it's gonna take a million times square
74:28
square, operations, which is prohibitive
74:31
So, and this is one of the issues with
74:34
the long context issues
74:35
probably have heard of
74:36
like, like I think we are using ChatGPT or Cloud Code
74:39
you see the context
74:40
and, and after some point they say
74:42
let me compact my context
74:43
right?
74:43
And the reason is that you have to shrink your context
74:45
otherwise your computational efficiency is too bad
74:49
but of course, you know, in the, in the, in the real production system
74:52
it's not like all of this capital T is really
74:54
like there are other ways to
74:55
reduce the dependencies
74:57
I think in practice
74:58
I don't think necessarily is T squared is
74:59
probably more closer to linear in T
75:02
because there are other
75:03
ways
75:04
variants of transformers that can reduce the dependency
75:06
on T
75:07
I think after, in the next Wednesday
75:09
we're going to discuss a few
75:11
things that can
75:12
reduce the dependency on T a little bit
75:15
exactly how we do it
75:16
we don't really know in some sense
75:18
because the there are a lot of open source papers that
75:22
propose different approach, on which
75:25
at least I don't know exactly which one is used
75:28
in the, in OpenAI topic
75:31
and another thing is that what is the
75:33
memory usage here?
75:35
So the memory usage
75:36
you know, as, as, at, at the first side
75:38
you have to spend T squared memory
75:41
because this matrix itself requires T squared memory
75:44
but that actually can be
75:45
reduced because you don't have to actually
75:48
compute the whole matrix
75:50
one by one and then do the multiplication
75:52
with V
75:53
So in some sense
75:53
to some degree
75:54
you can
75:55
do some part of this matrix and multiply
75:56
with part of the V first
75:58
and then you don't have to save that
75:59
part of the matrix
76:00
you can compute the second part
76:02
So, and, and the, the idea is called flash
76:04
flash attention, probably you have heard of
76:06
which is a way to reduce the
76:07
the, the memory
76:10
footprint, in these, systems
76:14
So
76:15
yeah, I think that's, but, but, but, you know, like, some
76:18
some of these kind of
76:18
like, ideas do have a trade-off because when you reduce the
76:24
for example
76:24
if you change your architecture to make the
76:25
dependency on T squared better
76:27
for example, we'll discuss some of this
76:30
on Wednesday. So if you change the architecture
76:32
the dependency on T is better
76:33
but your architecture is less expressive
76:35
and then you may lose some
76:37
performance, to some degree
76:39
Okay, sounds good
76:41
Yeah, thanks
76:47
So I guess, you know, I think you have
76:50
you know, Chris has talked about a linear model
76:52
which is kind of like you have a
76:53
model which is parameterized by theta
76:56
and the input is X
76:57
Let's just recall the notation so that we're
76:59
on the same page about notation
77:01
I think Chris is using Almost exactly the same notation
77:04
you know
77:04
it cannot be exactly the same because sometimes
77:06
it's not a linear model
77:07
right? So, but for, for linear model, recall that, you know
77:10
if you have a linear model
77:12
then the model is kind of like this
77:13
right? You have theta h theta x
77:15
which is equal to theta one
77:17
x one plus theta two x two
77:20
up to theta d x d
77:23
where x is the input
77:25
Which is a d-dimensional vector
77:27
and theta is the parameters
77:28
which are also a d-dimensional vector
77:31
and once you have nonlinear models
77:33
what does that mean?
77:34
You know
77:34
generally nonlinear just means that as long as
77:36
this form is nonlinear
77:38
then it's nonlinear. And
77:40
and here we are mostly talking about nonlinear in the parameters
77:43
because this nonlinear in the
77:45
data is not nonlinear enough in some sense
77:47
So I suppose you have something like
77:49
theta one x one squared plus theta two x two squared
77:54
so on and so forth
77:56
So that dx d squared
77:58
I think we didn't talk about this kind of models
77:59
you know, it, you know
78:00
bef- because we skipped them
78:03
in some sense in the new version
78:04
but this is
78:05
not sufficiently nonlinear to be interesting because you can just
78:09
define new data
78:09
which is the square of the old data
78:11
So suppose you say
78:12
I'm going to replace this by z one
78:14
and replace this by z two
78:16
and then it still becomes a linear model
78:18
And anyway, you are not learning the
78:20
you are not in
78:20
you are not trying to find out what are the
78:22
you know, you are trying to learn the theta
78:24
theta i's
78:24
right? You're going to figure out the
78:26
You don't really have to figure out
78:28
there's no learning on ZI
78:29
right? So whether Z is ZI or XI squared
78:32
it doesn't really matter
78:33
right? So that's why this is not exactly
78:35
what we're talking about
78:35
even though some people will say this is
78:36
still a nonlinear model
78:37
but
78:38
this is not exactly what we are talking
78:40
about. We are talking about cases like you
78:41
have theta one squared
78:42
for example, X one, plus theta two, maybe
78:46
You know, I'm just giving an example
78:47
something like this, right? Whatever nonlinear model you can write
78:49
as long as it's nonlinear
78:50
it's, it's called nonlinear model
78:52
Okay, so, of course, you know, the
78:54
the real model we are talking about is
78:56
not that like this
78:56
this is just one example
78:58
so the model we are
79:00
talk about is neural networks
79:01
which is, more structured and more
79:03
you know, you know, you know, you know, you know, specialized way
79:06
Anyway, so, but before talking about neural networks
79:09
let's talk about how do you learn nonlinear models
79:12
and what's the kind of the framework
79:15
this is in some sense
79:16
mostly just recalling what Chris was talking about
79:19
with a little bit twist about
79:20
you know, the model
79:21
So basically the model can be a general function
79:24
as opposed to, a linear function
79:26
So, let's just quickly recall some of this
79:29
So let's say you have a data set
79:31
I think Chris probably mentions the concept of data set
79:34
which is, a finite data point with
79:37
pairs of inputs and outputs
79:40
So here, let's talk about supervised learning
79:42
by the way
79:43
you know, in other cases, you know
79:45
we have data sets of different format
79:46
but here, if you have a supervised learning data set
79:48
you have a sequence of input and output pairs
79:51
and let's say we have N of this
79:54
and let's say we assume xi
79:57
is in D
79:58
And just recall that in this lecture
80:01
in this whole course
80:02
we are trying to use this notation where
80:04
the superscript is for the indices of the examples
80:08
and the subscript is for the
80:09
the dimensions, the coordinates, right?
80:11
So when I write x sub i
80:13
this is the first coordinate of the vector xi
80:16
and we have superscript is
80:17
is index, is indexing one particular example
80:20
Go
80:23
ahead. Yeah, yeah, that's great
80:37
So if you are only nonlinear in
80:41
x by non-linear theta
80:42
then you c-
80:43
it means that you can replace the x by something else
80:45
right? You can replace xi
80:47
for example, in this example
80:48
you can replace x1 squared by z1
80:51
and then you just say z is my inputs now
80:53
and then I have a linear model
80:55
in both z and theta
80:56
So fundamentally
80:57
it's the same as linear model from a
80:58
learning perspective
81:00
but here you cannot do this
81:01
right? You, you
81:02
there is no way you can
81:03
easily transform it into a linear model
81:05
Actually, there's no way just fundamentally
81:07
So that's why it's
81:08
it's becomes, it requires different kind of like
81:12
slightly different kind of techniques
81:13
right? So some of the linear models techniques
81:15
cannot work anymore
81:17
Yeah
81:20
yeah, you can write, you know, yeah, yeah
81:22
This is just the one
81:24
you know, you can even add more things
81:25
you know, you can take more square roots
81:26
you know, whatever, you can take squares
81:28
you know, yeah. You
81:29
you'll see some real examples
81:31
Yeah
81:40
In which cases, in this case?
81:45
Okay
81:52
Sorry. So you add one of these
82:00
two?
82:03
But then you change the model
82:04
right? Suppose my model is really
82:06
you know
82:06
suppose my model assumption is just literally this
82:10
what my argument was that you can replace
82:12
X Y square by Z one
82:13
so that you maintain the model's functionality
82:16
but, but, but you can reapply the same techniques that
82:21
that is useful for linear models
82:26
Okay, sounds good. So, anyway, so, okay, so how do we
82:31
so now let's just assume we have a general
82:33
you know, okay, so let's talk about the framework
82:35
So, and, And you have a
82:39
let's say we have a regression problem
82:40
I guess, you know
82:41
Chris's lecture talked about regression versus classification
82:44
And for regression problem
82:48
let's say
82:50
one of the
82:51
the distinction between regression classification is that we
82:53
have a regression problem
82:54
the YI, the labels are
82:57
in real number, right? These are real numbers
82:59
And then your model
83:00
let's say, is the H parameterized by theta
83:03
which is a function that maps the input
83:05
which is the, you know, RD to a real number R
83:10
right? And
83:11
you can have a cost function or maybe
83:13
loss function
83:15
I think these days people use the word loss function
83:18
more often
83:19
In the lecture notes
83:20
we still use cost function
83:21
you know, but most researchers understand
83:24
what you are talking about
83:26
so, so this is the most likely Of course
83:31
there are also many different type of mean square
83:33
average functions or loss functions you can
83:36
you can define
83:37
By the way, average loss and
83:38
and cost is the same
83:40
let's just use loss for all the cases
83:44
and the loss function could be something like
83:45
this
83:47
With J for the ith example
83:49
the loss function is a function of the
83:51
parameter because the data is not something you optimize for
83:53
right? It's not vari- It's not a variable
83:55
it's given
83:56
So
83:57
so loss function
83:58
for the j's example is yi
84:03
minus h theta xi. You apply the data
84:06
you apply the model to the data
84:07
you get some prediction
84:09
and you compare that with the real label
84:10
and you take the square
84:13
Right
84:14
Something like this
84:15
And then, you can also have like the
84:18
the average loss
84:19
which is something like one over n
84:22
times the sum of
84:25
all the losses
84:27
So
84:28
and then you try to minimize this function
84:30
over the parameter theta
84:32
So that's for regression. And
84:34
let me give another example
84:35
you know, I think you are
84:36
you know, this, this is mostly just a
84:37
re- like
84:39
repeating some of the stuff we have discussed
84:41
The only thing is here is just that
84:43
this is a nonlinear model as opposed to a linear model
84:45
right? So like, in crisp lecture
84:46
I think this is like just a theta
84:48
one X one plus X two
84:50
so on and so forth
84:51
And, I guess for multi-class classification
84:57
we can also talk about single class classification
84:58
but, which is actually in the lecture notes
85:01
I think I just
85:03
thought to give you an instance of time
85:04
let's just only talk about multi-class classification
85:06
Basically
85:07
you just have to define the loss function
85:08
the same way as we have done
85:10
for, for linear models
85:12
except that you replace the models by nonlinear
85:14
functions. By the way
85:15
I think in most cases
85:16
you don't have to think about the superscript I
85:18
because You know, this is just indexing examples
85:20
right? So if you remove
85:21
if you say, I only have one example
85:23
X and one, and X and Y
85:25
then everything without a superscript still works
85:28
maybe I should just write in that notation
85:30
to simplify it a little bit
85:32
you know, so let's say you have
85:35
so in this case
85:36
the Y, you know
85:37
I can write Y superscript I for the I's label
85:39
or I can just
85:40
use Y to indicate one label
85:42
Let's say, suppose you have one label
85:44
in Mathematica's application, you have, it's a discrete choice of labels
85:48
right? One, two, K
85:51
and here, I guess, you know
85:54
you can say you have a model
85:56
let's use H bar
85:58
which is mapping the input X
86:01
So the X is still in RD
86:03
Think of this as an image
86:04
think of this as some kind of labels
86:06
for the image
86:07
And then the
86:11
which is mapping the input from to K numbers
86:14
and these K numbers are often called logits
86:21
This is the same largest in the
86:23
in the linear case
86:25
So, and, and then you say the
86:29
which is largest, the predicted probability of each class
86:34
of each class, for this example X
86:38
it, for each class, let's say class J
86:41
is the, this kind of like exponential form
86:44
right? You look at the J's
86:47
this is the J's output
86:49
right?
86:50
Index J means like this is the
86:51
the J's output of this K-dimensional vector
86:55
and then divided by
86:58
the sum of the exponential of all the
87:02
this is the so-called softmax
87:03
So this is the so-called softmax
87:15
sorry
87:18
So this is the so-so-called softmax function of this
87:21
h
87:22
and then, so basically you say
87:27
yeah, so basically your model is like this
87:29
right? So this, this is your model
87:30
so p of
87:32
y is equal to j given x
87:35
is this
87:36
This is my parametric model for the prediction
87:41
parameters by theta
87:43
And then I need to compute a log likelihood
87:45
so I'm going to use negative
87:48
log likelihood
87:50
Which is also what we have discussed before
87:52
By the way, here
87:53
this is also log likelihood model if you
87:55
assume there's some Gaussian probability distribution
87:57
I didn't really define
87:58
go through that calculation
87:59
So if you think of that your model
88:02
your probabilistic model is something you add a
88:03
Gaussian to the h theta
88:05
and then you compute the log likelihood under that Gaussian
88:07
then it's the same mean squared error
88:09
Here, I'm just going through more details
88:11
So, I define my probability distribution
88:14
I compute log likelihood on the data
88:16
so this is the minus log p of y given x
88:21
and then my parameterized model
88:23
then this is a minus log of this thing
88:32
So, oh, sorry
88:33
I think one subtle thing here is that this is the
88:37
so this model
88:39
Specifies what's the probability for every choice of J
88:43
but now I have a particular choice of J
88:45
which is equals to my label Y
88:47
so that's why I need to plug in my Y
88:50
so this is, exponential H theta bar X
88:54
This is for the particular label I'm given
88:57
the width index, the, the width corner of that
89:02
logic vector divided by the
89:05
the, the denominator is the same
89:07
just the sum of the expansions
89:10
Oh, I think I have some typo here
89:24
and this is just called
89:26
the entropy loss
89:28
I think if you look at the multi-class classification case
89:30
this is the so-called
89:32
cross entropy loss
89:36
You have many different form of it
89:38
but this is the
89:39
the cross entropy loss
89:41
where the
89:45
you can just write this
89:46
which is theta bar
89:51
So this is the logic
89:52
this is the label
89:52
and you can plug in the cross entropy
89:54
loss. If you use PyTorch to implement it
89:55
I think you can find a module
89:57
which is just called cross entropy loss
89:59
and you give it to the inputs
90:00
and you can compute the loss
90:02
but of, of course, the underlying math is this
90:06
okay, this all sounds somewhat familiar to some degree
90:09
Okay, good
90:10
Any questions?
90:18
is cross entropy
90:21
oh, is there, so, so
90:24
so what's the second part of the question?
90:29
Yeah, I think this is the cross entropy between the
90:34
The, the probability distribution defined by H theta X
90:41
Okay, so I think here's the thing
90:42
So if you
90:43
so the question is why this is called
90:45
cross-entropy loss
90:46
I think if you
90:53
how do I
90:53
how do I write it in some sense?
90:55
Like, I think if you have a
90:59
let's say this distribution that's quite
91:02
maybe P theta
91:04
Okay
91:05
So, and
91:07
I think this cross-entropy loss is actually equals
91:10
to the cross-entropy
91:15
of between the true dis
91:18
I guess you have to take
91:22
if you take the expectation
91:25
of this L cross-entropy loss
91:31
Of this, H bar and Y
91:34
and expecting, so, a Y from the true distribution
91:37
from the true label distribution
91:39
whatever you call it
91:40
and then this is equals to a cross entropy
91:45
between
91:47
this distribution P of theta and a true distribution P star
91:52
which is something like you
91:54
I guess the cross entropy just
91:55
is some
91:56
is actually the probability of one distribution times
91:58
the log of the probability of the other distribution
92:03
yeah, I think the derivation is probably a bit long
92:04
especially, if you're reading this language
92:06
it's a bit cumbersome
92:07
but, but this is a fundamental
92:09
I think actually probably I have it in the lecture notes
92:11
I, I forgot exactly
92:13
yeah. Any other questions?
92:19
Why, what is the one hot vector?
92:22
You know, see, in
92:23
in my case, you know, in my notation
92:25
Y is just a discrete variable
92:28
So that's why I
92:28
you can see I have H theta bar
92:30
theta X indexed by Y
92:32
I
92:32
Y is like two or three or four or K
92:35
So, but if
92:37
you can also encode Y as a one-hot vector
92:39
which is the same thing
92:39
right? So like
92:40
then your math needs to be a little different
92:42
right? Then this is gonna be the inner
92:43
product between this vector times the one-hot vector
92:46
Yeah. But, but here I'm just using Y as really a
92:49
a discrete choice between one to K
92:59
So, can you repeat the question?
93:01
Sorry. So, can you repeat the question?
93:02
Sorry
93:09
Okay, so the question is why for the regression case
93:11
it also is not like
93:12
not, not, not likelihood, right?
93:15
So I think the
93:17
Let me try to see what I can
93:17
do it on the fly
93:18
I think it should be easy
93:20
So I think the
93:21
the modeling
93:22
so if you make the modeling assumption that if you assume
93:26
suppose you assume that the
93:27
your modeling assumption is that Y is generated from
93:32
say, H theta bar, theta X plus some Gaussian noise
93:36
with epsilon is from N zero
93:39
maybe sigma squared, something like this
93:43
then
93:46
the likelihood of Y given X is
93:50
you know, I think one over
93:52
I guess this is challenging my memory of this
93:55
something like this exponential
93:58
Y minus H theta X squared
94:03
over two
94:06
sigma squared, I guess, something like this
94:09
and then you take the negative log likelihood
94:13
So then this is a log of
94:15
P Y given X is equals to
94:18
there's some constant, you take the log
94:19
you get a constant
94:21
which doesn't really matter
94:22
and you got, the log of this
94:24
which is one over two sigma squared Y
94:27
minus H theta X squared
94:30
So then it's, it's kind of like this
94:32
right?
94:32
the scaling doesn't really matter because this
94:34
like, it doesn't change the optimization function fundamentally
94:38
Sounds good
94:44
Any other questions?
94:47
Okay, cool. So
94:53
and, and in this case
94:54
you know, here I only have like one example
94:57
right? Eventually, I'm going to average over many examples
94:59
right? So, I guess maybe just
95:02
let me write here
95:02
so
95:03
If you continue here
95:05
so eventually your loss
95:06
you know, your real loss is something like the
95:08
the final loss
95:10
is the average over all the examples
95:14
You take the cross entropy loss
95:19
of each of the ex- examples X
95:22
i and Y i
95:25
So, and
95:26
and each of these functions is a function of theta
95:28
so basically your total loss is also a
95:30
function of theta
95:32
And now
95:33
the next task is to optimize to find
95:35
theta that makes the loss smaller
95:36
right?
95:37
The same
95:38
I think we also introduced a little bit about gradient descent
95:41
you know, like
95:44
Newton method
95:45
right? So these are all ways to optimize
95:46
things
95:47
So here you can't use
95:48
Newton method because the function is now convex
95:50
so that's why Newton's method is not guaranteed to work
95:54
actually, no methods can guarantee to work for all non-convex functions
95:57
but
95:59
stochastic gradient descent seems to be a good
96:00
idea. So just to quickly recall
96:02
you know, you can use an optimizer
96:07
so, so optimization
96:09
I think one of the options is you
96:10
take gradient descent
96:16
So, so let's say, suppose you have a loss function
96:18
let's just, summarize all the cases as this
96:21
Let's be a little abstract
96:22
So let's say each of the examples have one loss
96:24
have the loss J I of theta
96:27
and you have an average of these loss
96:29
functions. Within descent means that you just say
96:32
I'm gonna ignore the structure here
96:35
I just treat this as a single loss function
96:36
I, I take the update theta is updated to
96:40
I think we use this notation to denote update
96:43
something like eta times gradient of J
96:45
of theta
96:47
So this is gradient descent
96:49
and how do you compute this gradient?
96:51
You, you compute the gradient of each of the loss
96:53
the gradient, sorry
96:54
you compute the gradient of each of the
96:55
loss with respect to parameters
96:56
and you add them up
96:58
Right, so that's gradient descent
96:59
And for stochastic gradient descent
97:08
so what you do is you say
97:10
your first sample, say B
97:16
let's see. So you sample
97:18
I guess they are
97:18
you know, maybe let's say
97:20
the most vanilla version of it is you sample one item
97:23
you sample
97:26
J from one to N uniformly
97:34
and then you run this
97:36
you know
97:38
so of, of, of course there's a for loop
97:41
and you sample
97:44
and then you say theta is updated to theta minus eta
97:48
blah, blah
97:50
The cost of the example J
97:52
the cost of loss of example J
97:54
theta
97:56
And
97:57
the kind of the justification here is that
97:59
if you think about what this is
98:01
so the difference here is that here you
98:02
compute a gradient of the average loss
98:04
and here you only compute a gradient of one single example
98:07
the loss on one single example
98:09
And the justification here is that you say
98:13
Even though this is one example
98:14
it's not actually the loss
98:16
loss of the whole gradient
98:18
but actually if you look at the expectation of the
98:24
the gradient, the per example gradient
98:27
and your randomness come from J
98:29
right, which is sampled from uniform rate from one to N
98:32
this is equals to the gradient of the
98:36
the average gradient, the gradient of the loss function
98:40
So, so in some sense
98:42
this quantity is approximation of this quantity
98:45
and the approximation is in the sense that
98:47
this quantity in average is equal to this
98:49
It's just that sometimes it's biased towards
98:51
it's, you know, it's, it's never approx, you know, let's say
98:54
suppose this quantity is like in this direction
98:56
and this quantity is just a
98:58
it can be any directions
98:59
but if you average out all the possibilities
99:01
you get the
99:02
the original direction on the goal
99:04
So in some sense
99:05
you have like a much larger noise
99:07
right? So like if you do gradient descent
99:09
you, it's kind of like
99:10
I guess maybe let me use two different
99:11
colors
99:17
So suppose you was grading this end
99:19
this is the grading of the
99:21
the average loss, and if you stochastic grading this end
99:24
it's stochastic, you may go here
99:25
you may go here
99:27
or you may go here
99:28
maybe you may
99:28
sometimes you may even go in the reverse direction
99:30
but in average
99:31
all of these
99:32
blue ones will average out in
99:34
the same direction as the black one
99:36
And, and so, and then if you have a trajectory
99:39
right, so suppose the black one goes like this
99:44
the blue one, because it's stochastic, you have noise, right
99:47
so it will go look like this
99:48
right?
99:49
Maybe something like this
99:51
Maybe like even, but, you know, but in average
99:53
it's kind of roughly tracking the grading descent
99:55
Of course, how much this tracks
99:57
you know, how much these two things tracks over time is
100:00
is a very complex kind of like a matter
100:02
Like, it depends on how large the
100:04
the step size is
100:05
you know, how
100:06
You know, kind of like
100:07
depends on many kind of like subtle things
100:08
but con-conceptually you
100:10
you should expect that the blue one sum will track the
100:12
the, the, the black one
100:14
You know
100:14
they are, the, the blue one
100:15
your average is exactly equal to the black
100:17
one in one step
100:18
But the error sometimes can be amplified
100:20
sometimes the error can be shrinked
100:22
So it depends on the situation
100:24
Ideally
100:24
you want them to track to some degree
100:33
I think you can use gradient descent as well
100:35
So, okay, but I think, I think I had a question
100:36
you know, it is a quick
100:38
I think, it sounds like the
100:39
the, the
100:41
the question is really about why you use
100:43
stochastic gradient descent. In what cases you want
100:44
to use stochastic gradient? That is a great
100:46
question. So
100:46
why you even consider stochastic gradient descent?
100:49
The reason is that computing this gradient is
100:51
very costly each time
100:52
because you have so many examples
100:54
N could be, you know, in 2015
100:56
it's kind of one million examples
100:58
In 2026 now, I think it's kind of like
101:01
you know, like one trillion tokens
101:03
right? There's no chance you can compute All the gradients
101:07
even completing one gradient is very costly
101:09
let alone completing all the gradients
101:11
So that's why you have to find an efficient way
101:12
so, so that's why, you want to
101:15
have a sampler which is less costly
101:17
And, and, maybe just one, so, and also another reason you
101:22
you can
101:23
do this is that completing the full gradient sometimes is actually
101:29
wasteful, because maybe it's
101:30
it's the case that all of these individual gradients are actually
101:33
they are correlated, they are highly correlated
101:36
Right? So
101:37
there's no reason to compute each of them to get
101:40
understand which direction to go
101:42
right? So you just only need a subset
101:43
It's kind of like every
101:44
example tells you to go that way
101:47
why you have to compute all of them
101:49
right? You just need a subset
101:50
subset of them
101:51
And another thing is that even this is not entirely true
101:54
in many cases, like
101:56
you don't really have to be very precise in this
101:58
in this dynamics
101:59
It's kind of like
102:02
even you have some mistakes
102:03
you know, it will be corrected later
102:05
So in some, maybe another thing
102:06
way to think about it is that for some cases
102:08
like, for example, if you have a convex function
102:10
right? So in some sense
102:12
like, one way to do it is that you do this
102:15
right?
102:16
But sometimes if you kind of like
102:17
overshoot, you know, or maybe you
102:19
you go in the wrong direction
102:20
still eventually you're going to go down
102:22
It's, how do I say this
102:23
like, you know, like basically, like even you have some perturbations
102:28
and this is not exactly the right example
102:30
but like even you have some perturbations in your dynamics
102:32
then over time, because you are, every time you, you have a shared target
102:37
eventually those kind of perturbations got kind of like washed out
102:40
by the, by the
102:41
by the shared force in some sense
102:42
So I just wanted to ask on a
102:42
non-convex function like a stochastic
102:43
and I believe that this is very stochastic
102:46
are we still evolving through the intersect or
102:47
would it not be? Yeah
102:56
I think that's a
102:57
that's a great question. So
102:58
one of the theory for why stochastic
103:00
is good is that it can
103:02
work better on, on non-convex functions, and maybe let me
103:06
let me first explain what you said
103:07
but, but actually it's not entirely right
103:09
So, so I think, so basically
103:13
suppose you have a non-convex function in one D
103:15
right, let's say, right? So like this
103:17
So if you start here
103:19
you, you keep going down
103:22
This green incentive means you go downhill in
103:23
some sense. So you go to the directions
103:25
which is the going downhill in the
103:27
sometimes it's called steepest descent
103:29
It's basically you are finding the
103:30
the direction which is going downhill the fastest in some sense
103:33
right? So you just go here and you
103:34
get stuck here because you can never leave this
103:36
this place, right? Here the green is zero
103:38
right? The, the, the curvature is zero
103:41
and, and, and
103:45
I think what you are trying to say
103:46
is that if you do stochastic winning sense
103:48
because there's some noise
103:49
maybe even though you want to go this way
103:51
maybe the first step you just
103:53
because of the noise
103:54
you go the other way and you jump
103:55
to another local minimum
103:57
I think this, is plausible, but, generally speaking, you know
104:01
of course the theory here is kind of very limited
104:03
but I think
104:05
the community don't believe that
104:07
this is the case. This is why stochastic
104:09
winning sense is helping
104:10
I don't think people believe that stochastic winning
104:12
sense
104:12
can help you to jump out of a local minimum
104:15
however, on the other hand
104:17
we don't even believe that there are
104:19
Many bad local minimum
104:20
So, so in one
104:21
it's actually very easy to have local minimum
104:23
but in high dimensional cases
104:25
it's not that trivial to have that many local minimum
104:28
So, so, so
104:29
I think at least one in the actu- actually was
104:31
I was involved in developing some of the theory
104:33
Of course, nobody really knows exactly whether they are right
104:36
but, the, the common, let's say hypothesis or
104:40
or hypothesized consensus in some sense is that
104:43
there are very limited number of bad local minimums in this
104:47
in this high dimensional function you're optim-optimizing
104:49
Maybe there's even no bad local minimum
104:51
All the local minimum are actually global minimum
104:53
It's kind of like if you think about
104:54
you are skiing in a mountain
104:57
like
104:59
I think if you just randomly try to go down scales
105:02
skiing downhills in 2D
105:03
right?
105:04
It's probably
105:04
you're probably going to go back to one
105:05
of the parking lots
105:06
which is one of the global minimum
105:08
right? So like, you know, of, but in 2D, I
105:10
I think it's a bit easier to have a
105:12
a local minimum. You probably go back country
105:13
and you got stuck somewhere
105:14
That's possible, but, you know
105:16
but you always find some river going down
105:18
you always find some valley going down eventually
105:20
I think, you know
105:21
Not, not, you know
105:21
I'm just going to give an analogy
105:22
You know, it's not exactly true in 2D
105:24
but in, in, in high dimension, it's
105:25
it's more likely to fi
105:26
always find a way to
105:28
to go down, downhill in some sense
105:33
Yeah, I think, I, I don't
105:47
think there's a
105:48
a very simple way to say it
105:50
I think
105:53
It's not, it's not universally true
105:55
right? So like, so in some senses
105:56
I think maybe the
105:57
you know, if I need
105:58
I can only use one or two sentences
106:00
I think it's kind of like just in high dimension
106:03
it's always, there's always some ways to
106:05
like the local minimum is your Hessian at that point
106:07
it's just all, in all directions positive semi-definite
106:11
That's a very hard thing to satisfy
106:13
So like, so, so, so you find the local
106:15
you, you find, a station, stationary point where the
106:18
the gradient is zero
106:19
and you look at the Hessian
106:20
the Hessian is kind of like
106:23
always have some direction which is negative
106:26
So, and I think, I
106:27
I was thinking about how to formalize this
106:29
you know, like, so for example, if you think about it
106:31
the, the gradient zero is already a pretty strong constraint
106:33
right? You suppose you have
106:34
you already have like
106:35
suppose you have a d-dimensional function
106:36
and you
106:37
add d constraints that
106:39
each of the dimension of the gradient is zero
106:41
that's already restrict yourself a lot
106:43
And if you add another thing
106:45
which is the Hessian is positive semi-definite
106:48
I think that just makes the
106:49
the number of points much smaller
106:51
Yeah, and, and in some cases you can compute
106:53
In, in one of my papers
106:54
I think we try to compute the number
106:55
of local minimums by using this
106:57
I think in, in differential geometry
106:59
there's something called Catswright's formula
107:01
And actually not only my papers
107:02
but many other people's papers
107:04
So in some, certain cases you can compute
107:05
or you can count
107:06
and you find out that
107:07
oh, all the local minimum are actually just those
107:09
number of local minimum is the same as
107:10
number of global minimum
107:12
But, but generally it's very hard to count
107:15
Any other questions?
107:21
Okay, sounds great. So, okay, so going back to here
107:24
so, so that's the algorithm people do
107:26
of course, practically speaking
107:27
you don't also just only sample one example each time
107:31
because sample, sampling one example, you know, Suddenly all, you know
107:34
using all the examples is too inefficient because
107:36
you have to complete all the gradings
107:38
doing one example, the, the downside is that it's too noisy
107:42
right? So sometimes you want to have like a
107:43
a, a trade-off, right? So you
107:45
you want to
107:46
have a few examples and so that you can
107:49
you can be a little bit less noisy
107:50
And another thing is that
107:52
which probably we won't go
107:53
go in detail here
107:54
but another thing is that
107:57
when you have, one example, your GPU allocation is too
108:00
too low. You know
108:02
not, it's not always the case
108:03
but in some cases
108:04
using
108:05
too few of examples make your GPU allocation
108:06
too low. For example
108:07
if you have like ten thousand GPUs
108:09
and you just only complete one example of
108:11
one grading each time
108:12
then most of the GPUs are actually idle
108:14
So
108:15
so that's why you need a so-called batch grading descent
108:17
Oh, sorry. like I think it's called
108:19
it's also still called stochastic gradient descent with
108:21
some kind of like small batches
108:24
it's
108:24
I think HDD just means like
108:27
both the version with batch size one or
108:30
whatever bigger batch. So when you have a bigger batch is
108:32
you know, you can imagine what it is
108:34
Basically, you just say, I'm gonna
108:37
with batches
108:40
I guess let's call it HDD
108:42
So I think you just
108:43
change the algorithm to be sample
108:46
say, B, examples from one to N
108:53
and then, then you say
108:57
theta is equal to theta
109:00
minus eta gradient
109:15
Okay
109:16
Sounds good. So, and, okay, so this is just
109:19
a general introduction to the framework for the
109:22
supervised learning
109:25
and actually in some of the unsupervised learning
109:26
it's the same thing. You define loss function for each examples
109:28
and you average them. The only question is
109:30
when you have unsupervised learning later
109:33
that we'll cover, you don't have the y's
109:35
So
109:35
but still the kind of the framework is
109:36
the same. You define the loss functions
109:38
but the loss functions then involve y
109:40
and then you have average loss functions
109:41
and you run optimizers
109:43
you can choose different type of optimizers
109:47
so, and then pretty much there are two questions
109:51
So, we're going to address that
109:53
So, so the two remaining questions
109:56
question one is that what is this function h theta
109:58
right?
109:59
So we haven't, right, so this whole framework
110:01
you know, works for every function h theta
110:03
and, or h theta bar
110:05
I think I use bar just because
110:08
it's a little bit different
110:08
no
110:10
With, with all bar register
110:11
I just want to distinguish it from the regression case
110:13
but they mean the same thing
110:15
And so, and the question
110:17
so question one is how do we define H theta
110:19
and question two is how do we compute
110:22
how to
110:25
compute the gradient
110:27
the gradient J on each of the example
110:31
And once you know how to compute the
110:32
gradient for each of the example
110:33
you can compute the gradient for all the
110:34
examples or batch examples
110:37
by averaging the gradient for every examples
110:39
So, and that's called
110:41
so this is called back propagation
110:43
The algorithm for this is called back propagation or auto-differentiation
110:46
which is the, the topic for next lecture
110:48
And in the rest of this lecture
110:50
I'm going to define a bunch of
110:51
H theta that people are using
110:53
mostly for like, more classical cases, like vision cases, like superserving
110:59
and, of course, for language case
111:02
you can have even more complicated
111:04
nonlinear functions like transformers
111:07
Okay
111:08
So, okay. So that's the plans
111:12
I'm going to start with
111:13
you know, the very basics, like new networks
111:16
So
111:22
so
111:22
we are still using these kind of housing
111:23
examples
111:25
I know that this example is probably too old
111:27
like it's 20 years old
111:28
So, so let's say, suppose you have like
111:30
to motivate, the, the, the, the introduction of like nonlinear models
111:34
suppose you have like size and
111:36
and, like this is the price
111:39
right?
111:42
That's housing price prediction
111:47
So, you know, if you look at the real data
111:49
most likely something like this
111:52
Maybe some, I don't know exactly what it would look like
111:54
but at the beginning
111:55
it doesn't increase that much as the size goes up
111:57
and then eventually becomes linear
111:59
Let's
111:59
let's say this is the case we will
112:00
see in the data
112:01
right?
112:02
So, you know, if you use a linear model
112:03
and you're going to fit something here
112:05
then you're not be accurate enough
112:06
So that's why, you probably have to have a nonlinear model
112:09
which is something like this
112:11
So you want your model to be able
112:12
to output a curve like this
112:14
Of course
112:15
you don't really know which line is this
112:16
It could be like a
112:17
you know, a probably without seeing the data
112:19
you can have multiple kind of like curves like this
112:21
but at least you should allow
112:23
the model to generate a curve like this
112:24
as opposed to only generating a linear function
112:28
So, and, and what this function is
112:30
right? So this function
112:33
so is actually of this form
112:37
So this blue function is actually of the form
112:41
it's a max of say
112:46
wx plus b, which is a linear function
112:49
and zero
112:51
I guess, you know, especially if you
112:54
Suppose this is zero
112:55
then you just
112:56
maybe I say this is max
112:58
wxp and zero plus c
113:02
So
113:03
max of
113:03
wx plus b
113:04
is kind of like a function like this
113:08
And then if you
113:09
With the... Sorry, this one is like this
113:11
And then if you plus c
113:13
you can shift it
113:14
above the, above or, or below a little bit
113:19
So basically, you just cut off everything that is below zero
113:21
which makes sense because
113:23
you know, for housing price, you know, it cannot be negative
113:25
you know, it has to be positive
113:28
so
113:32
Yeah, okay
113:33
So, and, and, so this actually
113:37
this introduced a primitive that is used very
113:39
often in deep learning these days
113:40
which is this function
113:42
which is max
113:46
of t and zero
113:48
So, I think in, in mathematics, people call it this
113:51
like, people write this, something like
113:55
like this, but, but for us, we just call this value
113:58
I think this is the
113:59
the standard in machine learning
114:02
So, this function is called rectified
114:03
rectified linear units
114:06
which is literally just this function where you
114:08
have linear functions
114:10
on the positive plane and then zero
114:12
on the other side
114:13
So, zero and then linear
114:15
and linear with slope one
114:17
So, this is a very important primitive
114:19
and then you can write this function as
114:21
you know
114:22
this function can be written as
114:24
value of t maybe plus c
114:26
something like this
114:28
Oh, sorry, value of
114:32
Guys
114:32
this one will be written as value
114:34
of W X plus B plus C
114:37
So basically you introduce
114:39
so, you know, it's, see, if you don't have the value
114:41
then it's basically a linear function
114:42
But if you have the value
114:43
then you somehow change the shape a little bit
114:46
And
114:47
and also this value is often called activation
114:49
function
114:52
I think the reason for the word activation is because in
114:55
there's some kind of like
114:56
connections to, biological neural networks
114:59
I, I think in biology
115:01
you know, like, in your brain, in our brain, the neurons kind of like
115:06
how the neurons are fired up is a
115:07
nonlinear function of the stimulus it receives from other neurons
115:11
So, and the, the, it's not exactly this
115:14
but it's something like this
115:15
right? So like, when it's passing from threshold
115:17
you got, the neurons got fired up
115:18
and
115:21
and, and below that threshold
115:22
the neurons just doesn't do anything
115:24
So, so that's why it's called
115:26
activation, right? If it's fired up
115:28
So, but, but I think in biology
115:30
you know, I'm not a expert there
115:31
but, you know, based on my
115:35
knowledge of it, like the, the, the threshold, the function is, is
115:37
is never exactly like this
115:38
Sometimes it's like a sigmoid
115:40
and that's why sigmoid was used at the
115:42
very beginning of deep learning
115:45
and, but
115:47
but then later on people realized that this is actually
115:50
this value is actually a better idea
115:52
for, for neural networks for various computational reasons
115:56
which we'll probably discuss
115:58
in a bit
115:59
Anyway, so for now, I think that's kind of like
116:02
just to remember this concept
116:04
So, so actually we have already defined
116:08
a one-dimensional neural network. So basically this is
116:09
a neural network with one in- one-dimensional input
116:12
and a few parameters
116:14
So that's already a neural network
116:15
So then the rest of the
116:17
the, the, for, for like ten, ten, fifteen minutes
116:20
we are just going to generalize this to multiple
116:24
higher dimensions, you know, like, and
116:25
and you apply this multiple times
116:27
and that will make it a
116:28
a more complex neural networks
116:30
And then we're going to even build more
116:31
on top of that
116:33
So
116:35
okay, so next, what do we do?
116:36
So, so this is only one-dimensional input
116:39
right?
116:40
And
116:40
so now let's try to do a
116:42
higher dimensional case. Let's say
116:43
suppose you
116:44
you have a data with dimension D
116:48
And then how do you kind of like
116:49
imitate this and make it nonlinear?
116:51
So the way to do it is that
116:52
you say I have X in D
116:54
and then I have a parameter W
116:57
which is also in the dimension D
116:59
And often this is called weight
117:03
vector, and actually eventually it will be a weight matrices
117:06
so that's why weight and matrix
117:07
you know
117:08
associate together. Now weight vector
117:10
is a rare cases
117:11
And then you also introduce another number B
117:14
this is called bias
117:17
And then you define your model like this
117:22
Say ReLU of W transpose X plus B
117:26
This is the inner product between two vectors
117:28
and you get the scalar out of it
117:30
And then you add another bias
117:31
which is still a scalar
117:32
so the output of this is a scalar
117:36
And so then basically we have like a
117:40
high-dimensional inputs and low-dimensional outputs
117:42
This is
117:43
this is actually called a one-layer network
117:44
So basically you apply this value
117:47
once
117:48
So, and, and now, let's try to kind of like
117:53
stack multiple, so this is one neuron
117:56
actually, because you, you apply one activation in some sense
117:59
it's one neuron, and then you can have multiple neurons
118:02
So, I mean
118:03
one of the typical examples I used to give
118:04
you know
118:04
maybe this is not exactly the best example
118:06
these days
118:07
For example, let's say, suppose you think about the housing price
118:10
right?
118:10
So it probably depends on a lot of
118:12
intermediate features
118:13
Suppose housing price depends on
118:15
for example, maybe, okay
118:17
maybe let me give you the full example
118:18
just to, to slow down a little bit
118:20
Let's say
118:20
suppose you have this cases where X1 is the size
118:25
X2 is the, the number of bedroom
118:31
and X3 is, let's say, the zip code
118:35
and maybe X4 is
118:37
let's see what's Maybe the
118:44
I don't know what else for
118:46
as far as my nose means
118:47
But anyways, this is your raw inputs
118:48
right?
118:49
So, so, maybe you can think of like
118:52
there are some intermediate things that
118:53
that actually matters more than this raw inputs
118:57
Let's say you have intermediate
119:01
quantities that you
119:03
quantities, like for example, you say
119:09
walkability
119:12
right?
119:13
How, how easy to walk around the house
119:16
right? Maybe there's another one
119:17
which is like the school quality
119:21
So these are, in some sense
119:23
variables that are not present in your original data
119:25
but they matter a lot to the
119:27
the price of the house
119:29
right?
119:30
And, and each of this
119:31
you know, maybe let's call this A1
119:32
this is A2
119:33
each of this is actually a function of
119:35
X1 up to X4
119:36
which can be computed as a nonlinear function
119:39
So
119:39
so you can imagine that A1
119:41
is probably some kind of like a nonlinear function of
119:45
you know
119:47
maybe value of some of these linear combinations
119:49
of this thing
119:54
And you can imagine maybe A2 is also
119:56
a some nonlinear function of this
119:59
of, maybe I say, say the
120:02
I'm using notations in a pretty
120:05
sloppy way just to give you example
120:06
you know
120:07
finally we're going to define things more formally
120:09
And then you can say
120:10
oh
120:10
maybe my final
120:11
Y should be a linear combination of these
120:13
guys. Maybe I have another parameter
120:15
say the nine, A1 plus say the ten
120:18
A2, something like this
120:20
So then this way you
120:21
you kind of like stack
120:22
two layers on top. You have
120:23
two neurons and you stack another linear functions
120:25
on top of it
120:26
So in some sense you are applying this recursively
120:30
So
120:30
so that's kind of like the fundamental idea
120:32
on how do you kind of compose
120:34
So basically you apply this nonlinear functions
120:37
you know, you apply a linear function
120:38
you apply some nonlinear functions
120:40
and then you get some articles
120:41
and then you apply another linear version
120:43
and you can probably apply another nonlinear function here
120:46
and you keep compose
120:47
So that's the, that's the rough idea
120:51
but let's try to make it a little more formal
120:54
so that it's probably more obvious
120:59
Let's see
120:59
what's the right way for me to
121:02
erase this stuff?
121:05
I think we can probably start from the
121:06
left
121:22
So one of the ways that people
121:23
at least used to draw this kind of things
121:25
you know, at least, ten years ago was that they draw
121:29
this kind of like inputs like this
121:31
So suppose you have X one up to X D
121:34
and then you say
121:35
have some intermediate variables A one up to A D
121:39
maybe A M
121:41
and then each of the intermediate variable depend
121:42
on all of this
121:44
and they have some dependencies here
121:46
So let's say A
121:46
M depends on all of the input variables
121:48
and you just draw this kind of edges
121:50
and then you can say you have another
121:51
layer of intermediate variables
121:54
and they connect to each other
121:57
So these drawings are
121:57
you know, once you know what they are talking about
121:59
the drawings are very useful
122:01
but, when you don't really know what they are talking about
122:04
it's kind of like confusing
122:05
So what I really mean here is that
122:07
you know, the way you do it is that you say
122:09
first, let's say, let's do only one layer
122:12
maybe just
122:13
formally. So we have X one up to X D
122:16
right? So then Let's have A1
122:19
The first inconvenient variable
122:22
is defined as maybe value of W
122:28
say, one X one plus B one
122:31
where W one is a vector
122:33
even though I have an in-index one
122:34
but it's the, the first, first vector
122:37
W one transpose
122:39
And X, so X, X is here
122:41
X is a D-dimensional vector
122:43
W one is a vector
122:44
so the inner part together
122:45
you get a scalar
122:45
and you add a scalar B one here
122:48
That's how you get to A one
122:50
which is kind of like what I wrote there
122:52
like A one
122:53
So I use W just because I think
122:56
people use W and B to distinguish the
122:58
the weight matrices and the biases
123:01
And then you have A two
123:03
which is a value of
123:05
W two transpose X plus B two
123:09
where W two is
123:12
R D and B two is in R
123:14
And they have a bunch of this
123:15
let's say you have like
123:18
Up to a n
123:20
Let me see where the n is my
123:21
notation here
123:22
It's very easy to mess up the notation
123:24
in this kind of cases
123:25
Yes
123:26
a n is value of w n
123:29
transpose x plus b n
123:32
And that's how we define a
123:33
a one up to a n
123:35
And each of the a i depends on
123:37
all of the x i's in this particular
123:38
way
123:39
And then you can say
123:40
I'm going to apply another
123:42
my output will just be a linear combination
123:44
of all of these a i's
123:46
Let's say the output is
123:48
some linear combination of the a's plus another
123:54
plus another biases
123:55
So here is the thing
123:56
How do I write a linear combination of
123:58
the a i's? You know
123:59
I can write something like
124:00
say, the one a one
124:02
so on and so forth
124:02
But if I always kind of expand this
124:04
it's too complicated
124:05
So what I'm going to do is I'm
124:05
going to write a vector
124:07
let's say w transpose a plus another b
124:11
But he realized that
124:12
you know, when I write this
124:13
you know
124:13
this W needs to be distinguished from other Ws
124:17
So
124:18
and that's why
124:19
I think I'm going to use this
124:22
two to
124:23
say that this is the
124:24
W on the second layer
124:26
In some sense, these are the Ws on the first
124:27
so there is another
124:30
So all of these AIs are combined to H theta X
124:32
and that's why people write this is the
124:34
W on the second layer
124:36
and these are the Ws on the first layer
124:38
So, so I'm going to write a
124:40
another bracket W A one
124:44
something like this, to distinguish the
124:49
the first layer and the second layer
124:52
So, and this W two
124:55
Super square bracket two is a vector
124:58
of the same dimension as A
124:59
not of the same dimension as X anymore
125:01
because you have to compile
125:02
you have to inner product with A
125:04
So this is a in dimension RM
125:06
because F, M entries here
125:09
and B is always a scalar
125:15
So that's how you define a two-layer network
125:17
So you have a lot of image
125:20
and, and you can see this notation is kind of messy
125:22
right? So like, you know, you probably shouldn't, you know
125:24
when you have like too many super square up
125:27
subscripts, at some point it
125:28
it's kind of like too messy
125:30
So that's why for most of the cases
125:32
you know, some cases you have to live
125:33
live with these types of notations
125:36
so
125:36
but in most cases you can simplify even further
125:39
You say, actually if I write everything as
125:42
as, matrices and vectors
125:43
I don't have to really think about each of these
125:46
the details. So basically I just write
125:49
A as a vector in
125:51
this is a vector in RM
125:53
You can write as value
125:54
of a matrix W times X plus B
125:59
and here W and B are a little bit different
126:02
okay, so W
126:03
I guess I need to write W
126:06
one and B one
126:10
So this is a matrix W one
126:13
If you want to look
126:14
understand the, the correspondence between this notation and this notation
126:18
so W one is this matrix of dimension M by D
126:25
and each row is W one
126:29
one transpose up to W M
126:35
One transpose. Right, so you put all of this row vectors
126:38
because once you transpo... By the way
126:40
all the vectors are supposed to be column vectors
126:42
if you don't do any transpose
126:43
You know, if you really don't understand this
126:45
you, you should try to understand the
126:47
Like, at least the one
126:48
you know, depending on the situations
126:50
right? When, like, but in many cases
126:52
it's good to keep in mind
126:53
like
126:54
whether this vector is a row vector and
126:55
column vector
126:56
So all of the
126:57
these are column vectors
126:58
and then you can transpose this
127:00
into row vectors, so this is a row vector
127:02
and you put all of them
127:03
stack them into a matrix
127:06
and
127:06
and then this B1
127:09
is just the vector of other scalars
127:16
But in many cases
127:17
you actually don't have to
127:18
necessarily remember this correspondence. You just remember this is fine
127:21
right?
127:22
You have a vector in your multiplication matrix
127:23
you get a vector
127:24
and then you add a vector bias
127:25
and you take value entrywise
127:27
Okay, that's probably important. So value applied to a vector
127:30
T, is the same as you applied entrywise
127:40
So, so in many cases
127:41
you just say, oh, one matrix application
127:43
one, layer is just a multiplication matrix
127:45
add a bias vector
127:47
and apply value
127:48
and you get a vector out of it
127:50
So
127:50
you don't have to necessarily understand all the details
127:52
You know, sometimes you do, but
127:53
but most cases these days you don't have to
127:55
So
127:58
and
128:01
And if you count how many parameters are there
128:03
right, like this is M by D
128:04
this, this is the dimension
128:06
So M by D
128:07
right, because the in, your input dimension is D
128:09
the output dimension is M
128:10
and in the matrix that turns the D
128:11
dimensional input to M dimensional outputs
128:14
that's why you need M times D
128:16
parameters
128:17
And then this guy has M parameters
128:19
So, so the total parameters
128:24
parameter count is, M D plus, M
128:30
So D is the input dimension
128:31
M is the output dimension
128:32
and D, this is out, output dimension
128:34
This is B one
128:42
yes
128:45
And this B one is this B one
128:48
So
128:56
any, any questions?
128:58
And I think, you know
128:59
eventually, actually, some
129:01
many of these details don't even matter if
129:02
you don't care. So
129:04
for example, you can even just write this
129:07
like
129:15
for example, you can just say
129:16
oh, I, I even simplify notation here
129:19
you can say
129:23
I think you can say
129:23
let's just, say this is a matrix multiplication
129:32
So let's say you have a
129:33
let's say you have a matrix multiplication
129:35
with W and B
129:37
So W is the weight matrices
129:39
which is this W
129:40
and B is the
129:43
the, the bias, and you apply Z
129:45
and, and this is just equals to WZ plus B
129:49
And, and, and then you can say I'm going to apply
129:54
a sigma
129:59
Sorry, sig-sig, let's say sigmaism, I guess maybe I didn't
130:02
I, probably I go a little too fast
130:04
I see
130:07
So maybe just, before I talk about that
130:11
let's, I think I already allude
130:13
allude to it, so
130:15
this value here can be used also replaced to other
130:19
nonlinear activations
130:25
So you can replace
130:34
so the value was
130:36
you know, some other nonlinear activations
130:38
and all the other activations are functions
130:41
that maps R to R
130:43
so it's always a scalar function
130:45
and, but, you know
130:47
you can apply this to vectors by entrywise applying it
130:51
applying entrywise
130:52
and other activations could be like sigmoid
130:58
sigma looks like this
131:01
you know, let me
131:02
let me just save some time to now define the
131:04
the equations, you know
131:05
you can read the electrons to see the
131:07
the exact equations. There are some
131:09
there are also tan
131:10
tan h, which looks like
131:14
going through zero and like this
131:16
So all of these activations are generally
131:19
increasing functions, but not always the case
131:22
I think there's also Leaky value
131:23
you know
131:24
there are a bunch of different activations functions
131:26
For example, Leaky value looks like
131:29
none of this is actually used that much
131:31
if you want to see what's the
131:33
use the, in the latest network
131:35
I think you have to check the electrons
131:37
I just added a few
131:38
which I don't have it on my notes
131:40
Like, but there's also functions like this
131:42
you know
131:44
anyway, generally it's kind of like some almost increasing functions
131:47
and often on the
131:49
on the left-hand side of the zero is
131:51
kind of close to zero
131:52
But not exactly, right? So for example
131:54
the one that I just draw
131:55
this is called, Jellu
131:58
let me not, write, I think the
132:00
the definition of it is
132:01
the Jellu's definition is this
132:03
just in case you
132:06
you care about the exact equation
132:08
So it's not exactly nonlinear
132:10
it just drop, it
132:11
it dips a little bit
132:13
I think it doesn't go above zero
132:14
I think. Exactly, strictly speaking
132:18
So, it dips a little bit and then goes up
132:20
So
132:21
exactly why you use any of this is
132:23
kind of like a
132:24
a little kind of like a magic
132:25
Like, I think you don't
132:28
I think people seem to
132:30
find out that using some kind of somewhat smooth functions is
132:33
is actually useful. So value is actually not
132:35
necessarily great because it's not smooth enough
132:38
But other than that
132:38
I don't really know
132:40
exactly, there's like a, sometimes it's just a ability thing
132:44
you know, some, sometimes just a trial and error
132:47
Yeah
132:49
so, okay, so that's the activations
132:51
and then, so now I'm trying to
132:55
compact it, you know, I'm trying to kind of have
132:57
have more and more abstract versions
132:59
so you don't have to deal with all these
133:00
notations
133:02
so then, once you have the activations
133:04
you can define this
133:05
right?
133:07
You can
133:07
this is just the sigma of
133:09
WZ plus
133:10
V. This is the abstract version of that
133:12
And then you can compose many times
133:13
You can say, I'm going to apply another one
133:16
so my model can be like
133:17
say, apply a matrix multiplication on top of this sigma
133:21
and then, right, so WBZ, and then apply another thing
133:29
apply another thing, right, so on and so forth
133:33
And, each of this is one layer
133:38
And then if you apply another matrix multiplication
133:40
sigma is not a layer
133:41
and if you do another one
133:42
it's another layer
133:43
The number of layers is
133:44
defined as
133:45
pretty much as the number of matrix multiplication
133:48
Yeah. But here
133:49
one thing I'm going to describe to you
133:50
about is that
133:51
I didn't really tell you what's the parameters
133:52
here in this matrix multiplication
133:54
So technically, if you want to write this formula
133:56
you have to say
133:57
oh, this is W and B
133:58
and this is what?
133:59
Is it still using W and B or
134:01
some other W and some other B?
134:03
So oftentimes you use a different
134:04
W and different B. It's a
134:06
The, the, the, the shape is similar
134:08
but the, the variables are different variables
134:11
So then you should
134:12
then you have to introduce the
134:13
the bracket again, so let's say this is the bracket one
134:16
B one
134:17
and then here the parameters are bracket W bracket two
134:21
B two, so on and so forth
134:22
And then you index each of them
134:24
to fully specify the network
134:34
Which matrix, sorry?
134:38
So, so these matrix?
134:40
Yeah, so, okay
134:41
Yeah, so what these matrix are
134:43
so the only requirement is that you need
134:44
to make sure that dimensionality com- is compatible
134:48
right? So
134:54
so your input, wait, did I write Z?
134:57
Oh, yeah. Okay. So let's say
134:58
suppose your input Z is
135:00
say, dimension D, then you can only apply a matrix
135:05
you know, with, so to make WZ make sense
135:08
you have to make sure
135:09
W is of dimension something by D
135:13
right?
135:14
and then something is what you can choose
135:15
you know, almost freely. Of course
135:17
you know, not all the options are the same
135:19
eventually in terms of the results
135:20
but for the compatibility issue perspective
135:23
you can choose them arbitrarily
135:25
And let's say, suppose you, the first layer you choose
135:28
let's say, suppose you choose, let's say you choose matrix multiplication
135:33
say
135:37
W one, B one, Z plus C
135:43
And suppose this W one is
135:48
M one by D
135:50
So that means that you map
135:51
this D dimensional vector to M
135:53
one dimensional vector. So let's say
135:54
so this output is going to be
135:56
M one dimensional vector
135:58
And then next time
136:00
and if you apply a sigma
136:01
it's still M one dimensional vector
136:03
And next time you apply the matrix application
136:05
you have to make sure that the
136:07
the input dimension of that matrix to be
136:09
compatible with M one
136:10
So basically, if you do another one
136:13
say W two, B two
136:16
then it's required that W two
136:19
is to be something times M one dimension
136:24
and, and something
136:25
the alpha dimension is still what you can choose freely
136:28
but the, the input dimension is not
136:30
And the next, and you have to do this
136:32
you know, again, again. So once you have multiple dimensions
136:34
then, so basically the, the, like, I
136:37
maybe the easy
136:38
way to think about it is that you have
136:40
a matrix, right, times
136:41
say, Z plus B, right?
136:43
So this dimension is what I call the input dimension
136:45
because this dimension got multiplied with the dimension of Z
136:48
and this dimension is the output dimension
136:50
But the output dimension becomes the input dimension
136:52
for the next round
136:53
so that's why you need to make sure all the
136:55
dimensionalities to be compatible
136:58
That make sense?
136:59
And in many cases
137:00
I think, you know, the, the, the
137:04
the original input dimension is
137:06
is something you cannot change
137:07
right? So like that's fixed
137:08
that's based on our data
137:10
And then afterwards
137:11
you don't change dimensions that often
137:13
You typically use the same dimensions for quite a while
137:15
or maybe you change it once or twice
137:17
something like that
137:22
Okay
137:27
And, you know, like, if you
137:29
just in case you read some of the old papers
137:31
you know, which
137:31
and I think even
137:32
these days still the architecture papers sometimes draw like this
137:35
like they, they just draw, you know
137:38
people go more and more abstract
137:39
right? They say, I have an input X
137:41
and then apply a matrix magnification
137:45
and then apply another
137:46
maybe sigma, and apply another matrix magnification
137:50
so on and so forth
137:50
That's how, how people actually draw these networks
137:53
you know, without specifying all details
137:55
And
137:56
and you can even
137:57
merge these two together to call it MLP
137:59
So sometimes people just draw this
138:01
you have X, and if MLP is called multilayer perception
138:05
often it, encapsulates
138:10
one or two layers of this
138:12
a few of these components
138:13
When people call, say, MLP
138:15
I think sometimes it's one matrix multiplication
138:17
one sigma, sometimes it's two matrix multiplication
138:20
so on and so forth
138:35
I think in most cases you get smaller
138:37
dimensions
138:38
the intermediate dimensions become smaller
138:40
and that's often because the input dimension is
138:41
often very big. For example
138:43
for language, it's kind of like your initial
138:47
the, the vocab is kind of like
138:49
the token, you know
138:50
I guess I'm jumping ahead a little bit
138:51
but like your
138:52
your initial dimension could be like 250K or something like that
138:54
and then you, you
138:56
you shrink it to probably 2K and then
138:57
keep using 2K. Yeah
138:59
something like that. And for vision
139:00
I think the same thing
139:01
because initially you have a RGB
139:03
encoded kind of like image
139:05
You have three channels
139:06
three RGB, three numbers
139:07
and each is like 256 by 256 or maybe
139:10
depending on the resolution
139:11
and you have to shrink it a little bit
139:14
yeah
139:17
okay, what else?
139:18
I think one thing is called ResNet
139:20
I think that's still useful
139:20
so probably I should tell you about it
139:22
So, let me raise something here
139:28
So, but eventually, you know
139:30
like you can define many components like this
139:31
Most of these components are machine applications
139:34
activations
139:34
and I'll tell you a few other components
139:36
that are Still largely being used in
139:39
in, in different cases
139:41
and, and, and
139:43
and some of the very more than cases in
139:47
used in like more
139:49
more than components like our attention used in
139:53
language. I'm going to talk about those in a
139:54
in a, in a large language model lectures
139:57
So, so these are some
140:00
so one of the
140:01
the components called residual network
140:05
or, or just a re-residual connections
140:07
if you think about the
140:09
the
140:09
components. So the residual connection is like this
140:12
So instead of define
140:14
instead of using, keep using this, this kind of things
140:18
the residual connection is kind of like
140:20
let's call residual Z
140:22
is this interesting thing where you have
140:27
two applications of this
140:29
alternating between Matrix multiplication and
140:34
and activations
140:35
and then you add
140:36
your input again back to it
140:40
So one thing that is actu-at least
140:42
by definition, it means that you don't change dimension here
140:46
right? You, because to make this compatible
140:49
your, your input dimension, all dimension are exactly the same
140:54
and, of course
140:55
these two matrix applications are not using the
140:56
same weight matrices. I'm not just
140:58
right exactly what they are
140:59
but, they are not using the same weight matrices
141:03
and, and this is called a residual block
141:10
And, the, this was introduced like in twen-twenty
141:13
2015 in this, very famous residual network paper
141:17
I think the kind of the
141:19
the, the rationale was that if
141:23
let's say, suppose By the way
141:25
this Z
141:26
doesn't have to be original raw input
141:27
This could be one intermediate layer
141:29
and then you apply this
141:31
Z, you apply this residual block
141:32
and then you, you can actually apply many
141:34
many residual blocks. So eventually
141:37
the, this is how residual block
141:38
and eventually
141:41
the residual network, the residual net looks like
141:45
you know, with some approximation
141:46
it's kind of like you
141:47
apply a lot of residual net many
141:50
many times
141:53
And at the beginning
141:54
you need to kind of do something maybe
141:55
to change the dimension
141:57
because you cannot work with the original dimension
142:00
So you first change the dimension to some
142:01
dimension of the intermediate dimensions
142:03
and then you apply many
142:04
many residual network
142:05
And maybe you have to apply
142:06
one more at the end
142:07
to change the dimension again
142:09
But largely speaking, it's application of many
142:11
many residual blocks, and that's called residual network
142:14
and, the original motivation was that
142:17
why you apply this?
142:18
That's because suppose you believe that at some
142:20
point you are already get
142:22
getting close to your labels
142:23
right? So suppose
142:26
at some point Z is already approximately equal to Y
142:29
let's say. Y is your target
142:30
and Z is your intermediate variables in some intermediate
142:35
like, layers
142:37
Then
142:40
You probably should model the differences explicitly
142:42
because you are looking up close
142:43
you just probably shouldn't use too much works to model the
142:46
the, the only the differences
142:48
So basically you should probably just model Z
142:49
minus Y as some residual
142:52
I mean Y minus Z
142:53
sorry
142:54
say Y minus Z as some residual of Z
142:57
And, and if you believe in that
143:00
like, sorry, not the residual of C
143:03
Y minus Z to be some kind of
143:05
like a right work
143:06
say something like this
143:09
Plus, right, like this
143:11
Right, so if, if this is your
143:13
your belief, then if you move Z to the other side
143:15
then you have
143:15
Y is equals to sigma M M
143:19
Z plus Z
143:20
And you can see that this is kind
143:21
of like almost the same as here
143:23
The only difference is that
143:24
they found out actually you probably shouldn't use
143:25
a one layer to model the difference
143:27
you should use two layers to model the difference
143:29
So, so they say, oh
143:30
let's say you do two of this
143:33
And then the same thing you just
143:34
have like this
143:38
So this is just a story
143:39
It's not like, any, there's any formal proof there
143:43
and, and actually people have tried to
143:45
you know, ten years ago people have tried
143:47
maybe seven years ago people have tried
143:50
to see how many blocks here you should do
143:53
you know, can do two layers
143:53
three layers, L layers
143:55
and eventually I think two or three layers are the right
143:57
are the best
143:59
also for reasons we don't fundamentally understand necessarily
144:03
yeah, so that's residual network
144:05
I thought
144:05
yeah. I thought like that the term on the right
144:08
that's our hypothesis
144:15
So, so, so, so, so suppose this is the modeling assumption
144:19
you model only the differences
144:20
then, you know, so this implies this
144:23
and they are, they are the same
144:23
right? So like
144:24
which one is the hypothesis depends on what you
144:26
what you believe in
144:27
So, but, but somehow if you
144:28
if you somehow believe that
144:30
it's more beneficial to model directly the residual
144:34
Then you, you, you do this, right?
144:36
And then you change
144:37
and you rearrange together
144:40
That's, that's the only thing I'm saying
144:41
Nothing profound
144:47
And there are other kind of like
144:49
explanations for why this works
144:50
you know. For example
144:51
some of the
144:52
theorists that they try to analyze the optimization
144:54
perspective of different modeling process
144:56
right? So basically
144:57
if you don't add Z here versus you add Z
145:00
you know, in some cases you can argue that
145:04
adding Z will make the optimization easier
145:07
make your optimization more well-conditioned
145:09
including some of my papers
145:12
you know, I think
145:13
I think there are some
145:14
degree of truth there as well
145:15
It's just that it's very hard to fundamentally
145:17
you know, it's like you have many explanations for
145:21
for it. It's
145:22
it's very hard to know exactly which one
145:23
is the right explanation
145:24
Yeah
145:28
maybe, let's see, I have seven minutes
145:30
Any, any question?
145:37
Why the differences in model binding?
145:41
There's no, you know, I
145:42
I got a question or what's the intuition
145:45
but like, maybe I see, so the intuition could be that
145:47
oh, maybe this is already small
145:48
then maybe I should directly model that
145:50
right? As, as opposed to model the
145:51
the original label. So it's a simpler task
145:54
The original label, you know, predicting original label is
145:56
is very hard, but if you predict the difference
145:58
probably it's easier
145:59
So that's why you model easier task with a two-layer networks
146:02
and that's pretty much it
146:05
Like, there's no, nothing more formal than that
146:08
Yeah
146:11
let's see, I have seven minutes
146:13
I think
146:13
Let's see, should I attempt to talk about two things?
146:18
Okay, so maybe let me quickly talk about
146:22
layer norm
146:25
and I can see whether I have time
146:26
to talk about the convolutional network
146:29
So Layer norm is also something that people still use
146:34
in large language models
146:36
So, of course, it's like a different form these days
146:40
So, the original layer norm is the following
146:43
So basically, still, it's, it's a component, right?
146:46
So like, it's a, it's a block, you know
146:47
so you have some input that's called Z
146:48
and
146:50
and this component is trying to normalize Z
146:52
in some way
146:53
And the way to normalize it is that
146:54
you say you first turn Z into
146:57
some normalized version of Z
146:58
which is Z minus
146:59
mu one over sigma one
147:02
comma Z minus mu M sigma M
147:05
I say Z is M dimensional
147:06
Z, sorry, Z one
147:08
Not my bad
147:09
I should, I should read the note
147:10
I should follow the notes more closely
147:11
I think this is
147:13
of the same
147:19
Yeah
147:22
Maybe I should even use
147:31
Yeah
147:31
So let's say, okay, I only remember why I use hat
147:35
but let's use hat
147:37
So this is how you normalize
147:39
where mu hat is the empirical mean
147:43
which is the average of all the ZIs
147:47
And sigma hat is the empirical
147:52
sigma hat squared is the empirical average of the
147:57
the standard deviation
148:00
So this will empirically normalize it to norm one
148:04
right? So you can see it's
148:05
it's pretty sta, and
148:09
and then it's not done yet
148:12
And then you turn this into
148:14
sometimes you find, people found out, oh
148:15
actually mean zero and norm one
148:17
is not that great for whatever reason
148:20
So they say, oh, let's be flexible
148:21
Let's also add
148:27
Of flexible tunable parameters
148:29
beta and gamma. So let's turn
148:30
the next step is turn this into
148:34
beta plus gamma z one minus mu hat
148:39
sigma hat
148:48
Where beta is a
148:49
is a tunable param
148:50
beta is a variable that you can tune
148:51
and gamma is a variable you can tune
148:54
and, so basically this make a mean beta and
148:58
and a standard deviation gamma
149:01
So that's the original layer norm
149:03
And then later people realized
149:05
actually, you probably don't have to do this
149:06
You just, so this is the layer norm
149:09
And now I think there's something called RM-SAM norm
149:12
which is, even simpler
149:14
You don't, subtract the mean anymore
149:18
You just divide by sigma
149:19
So you
149:21
so you just say you do z
149:22
is turned into
149:25
Z one minus sigma one
149:27
sigma hat, say Z M minus sigma hat
149:31
And the sigma hat is also different
149:33
Sigma hat is only the
149:36
the average
149:39
is only kind of like the
149:41
you are only averaging over like the
149:43
you don't have the
149:44
you don't subtract here. So you just try to make the
149:47
so sigma hat is something like one over M
149:54
Z, Z I, sorry, Z I squared
150:04
And, and then you still do the beta thing
150:06
right? And then you turn this into
150:08
beta plus
150:12
Z one over sigma hat
150:17
You know, why
150:18
why this is kind of like more better or not
150:20
you know?
150:22
I think it's mostly empirical
150:24
So basically here you don't really want to make it
150:27
let me think about this
150:28
Wait, sorry, I think you still need gamma
150:30
sorry
150:31
Did I keep gamma or keep the beta?
150:32
I keep the gamma
150:33
sorry, my bad. You do this
150:36
So, and gamma is channel parameter
150:37
So basically this is saying that I don't
150:39
have to care about the
150:39
the shifting mean. I don't really make it mean zero
150:41
but I
150:42
I still
150:42
care about normalizing it in some way
150:44
But I still leave us some degree of flexibility gamma
150:47
which I, you know, it's not really normalizing
150:48
I still allow it to
150:50
to be normalized to something that I
150:51
I will decide later in my training
150:54
Right, so, so this is kind of like the
150:55
the magic here. It's not like you
150:57
you try to force it to not normalize
150:58
to one standard deviation. Why you actually say
151:01
I first normalize to one
151:02
and then I leave a gamma
151:03
parameter there so that
151:05
in the training I can normalize to anything
151:08
and
151:11
I think, let me see
151:13
Yeah, I don't have one then
151:14
So
151:15
so one benefit of all of this is
151:17
that you make it kind of like a
151:20
Scaling environment
151:21
in term
151:21
for both the parameters and the data
151:23
So what does that mean is that if you multiply
151:27
so maybe let's say you call this whole thing called
151:30
let's say Z
151:32
let's just, say Z, right?
151:35
So like, let's say
151:36
suppose this whole operation is called a layer
151:38
norm
151:39
So you apply Z and you'll
151:40
you get a layer norm of Z
151:42
And you can real
151:43
you can find out that there's a scaling environment
151:44
which means that if you apply
151:47
a scalar, let's say you, you multiply a scalar
151:49
say alpha to Z
151:51
and you get L N alpha Z
151:54
but L N alpha Z is actually equals
151:56
to L N Z
152:00
So basically, no matter how you multiply the input
152:02
you don't really change it
152:04
and so that's why no matter how you multiply your
152:08
your matrices or, or you multiply your input data
152:12
you never change the output
152:14
unless you change the
152:15
the, some of the parameters like beta and gamma
152:17
which is, you know, in a test time
152:18
you're not supposed to change them
152:20
So, so basically this makes the
152:22
the model more scaling environment
152:25
and in some cases
152:25
it, it helps, stabilize the model, for example
152:29
At least intuitively, you know, with some difficulty of approximation
152:33
you know, they are, they are, at least in the
152:34
in the past when people tuned all of these models
152:36
it happens that sometimes you apply the
152:39
the, the, these layers many times
152:41
and then the, the output becomes bigger and bigger
152:43
and, and it becomes like a really big
152:46
it's like ten to the twenty or something like that
152:48
and then, and then everything just blows up
152:51
So when you normalize every time
152:52
then you, you never got into this issue
152:54
And also it makes the initialization a little
152:55
more kind of like
152:57
easier so that you don't have to
152:58
you know, before, like if you don't do something right
153:00
then suddenly everything just blows up
153:02
and now like the scaling
153:04
doesn't really matter that much
153:05
but there's also subtlety here
153:07
because when you have
153:08
okay, this might be a too advanced comment
153:10
like one side effect of this is that when the
153:13
when the forward pass is scaling environment
153:15
but actually the, the
153:15
the gradient is not scaling environment still
153:18
So, so, so
153:19
so some of these issues got pushed into the
153:21
the optimization, so, which is, a much more complex matter
153:26
anyway, so I guess, let's see
153:28
So I think I'm done
153:31
The only thing I didn't cover
153:32
is the convolution network
153:34
I think it's in the lecture notes
153:35
So, if I spend one minute on it
153:38
I think it just means that
153:40
instead of multiplying a full matrix
153:42
you actually multiply a
153:44
a topless matrix. I'm not sure you've heard of this word
153:46
like a
153:46
So basically it's a matrix with some structures where like
153:50
So, so when it's, yeah, instead of having this matrix
153:54
convolution network just basically means you have a
153:55
matrix that's kind of like a
153:57
like, look like this
153:58
The non-zeros are only like this
154:01
And, and it's, it's actually the same set of non-zeros
154:03
like you, you, you shift this vector here
154:04
and you put it here
154:05
and you put it here
154:06
and you put it here
154:07
It, it's actually called like a
154:08
a, a, a filter in some sense
154:10
like, so, so this matrix
154:12
all the other parts of the matrix is zero
154:14
and only this, what I drew here is kind of non-zero
154:17
and also they share the parameters
154:18
so basically this vector is the same as
154:20
this vector
154:21
So you apply kind of like a
154:22
a shifting environment, kind of like filter
154:25
to each, to, to the data
154:27
and it's called convolution. Of course
154:29
the real thing is a little bit more complicated
154:30
because when you have 2D and the so and so forth
154:32
it becomes, complicated. But in the lecture notes
154:35
we have that
154:36
But convolution network is not that
154:38
used that much
154:39
except for very hardcore vision cases
154:41
Like even for vision cases
154:42
people use, language models, like, transformers, a lot
154:47
So, so that's why I probably find to keep
154:49
skip them
154:50
Okay, sounds good

Watch these videos on your TV

XTTV brings video content from X to your living room. Download now and start watching on the big screen.