← Back to videos

agentic engineer in 2026. Andrew Ng, the godfather of AI, gave the complete playbook to become one

Dipanshu Kushwaha

Dipanshu Kushwaha

@Dipanshu_AI

November 19, 2026· 105:51

agentic engineer in 2026. Andrew Ng, the godfather of AI, gave the complete playbook to become one from scratch. 1 hour course. Free: • 00:00 - AI agent basics • 12:12 - AI Agentic workflows & design patterns • 53:27 - Practical tips for building AI agents • 1:20:30 - self-improving AI agent loops • 1:30:19 - multi-agent AI systems I watched it last night. Halfway through, I realized I could get into Anthropic in weeks, not years. Bookmark now. Watch it. Then build your own AI agent in

Transcript

When
I coined the term agentic to describe what
0:02
I saw as a
0:03
important and rapidly growing trend in how people
0:06
were building LLM-based applications
0:08
what I did not realize was that a
0:09
bunch of marketers would get hold of this
0:11
term
0:12
and use it as a sticker and put
0:14
this on almost everything in sight
0:16
and
0:16
that has caused hype on agentic AI
0:19
to skyrocket
0:21
The good news, though, is that ignoring the hype
0:23
the number of truly
0:25
valuable and useful applications built
0:27
using agentic AI has also grown very rapidly
0:30
even if
0:31
not quite as rapidly as the hype
0:33
And in this course
0:34
what I'd like to do
0:35
is show you best practices for building agentic AI applications
0:39
and just open up
0:40
a lot of new opportunities
0:42
to you in terms of what you can now build
0:44
Today
0:44
agentic workflows are being used to build applications
0:47
like customer support agents
0:49
or
0:49
to do deep research to help write deeply insightful research reports
0:54
or to process tricky legal documents
0:57
or to look at
0:58
patient input
0:59
and render Or to suggest possible medical diagnoses
1:03
On many of my teams
1:04
a lot of the projects we build just
1:06
would be impossible without agentic workflows
1:08
and so
1:09
knowing how to build applications with them
1:11
is one of the most important and valuable
1:13
skills in AI
1:14
today. It turns out that
1:16
one of the
1:17
biggest differences I've seen between people that really
1:20
know how to build agentic workflows
1:21
compared to people that are less effective at
1:23
it is the ability to drive a disciplined development process
1:27
specifically one focused on evals and error analysis
1:31
And in this course
1:32
I'll tell you what that means and show
1:33
you
1:34
what allows you to be really good at
1:35
building these agentic workflows
1:38
Being able to do this is
1:39
one of the most important skills in AI today
1:42
and will open up a lot more opportunities
1:44
be it job opportunities or
1:46
opportunities to just build amazing software yourself
1:50
With that
1:50
let's go on to the next video
1:52
to dive more into what are agentic workflows
1:56
So
1:56
what is agentic AI and why are agentic
1:58
AI workflows so powerful?
2:01
Let's take a look
2:02
The way that many of us use large
2:03
language models or LLMs today is by prompting it to say
2:07
write an essay for us on a certain
2:09
topic X
2:10
And I think of that as
2:11
akin to going to a human
2:13
or in this case
2:14
going to an AI
2:15
and asking it to please
2:16
type out an essay for me
2:18
by writing from the first word to the
2:20
last word all in one go
2:22
and without ever using backspace
2:24
It turns out that we as people
2:25
we don't do our best writing like that
2:27
by being forced to write in this
2:28
completely linear order and nor do AI models
2:32
But despite the difficulty of being constrained to writing this way
2:36
our LLMs do surprisingly well
2:38
In contrast, with an agentic workflow
2:40
this is what the process might look like
2:42
You may ask it to first
2:43
write an essay online on a certain topic
2:45
then ask if it needs to do any more research
2:48
and
2:48
after doing some web research and maybe downloading some web pages
2:51
then to write the first draft
2:52
and then to read the first draft and
2:54
see what parts need revision or more
2:55
Research, and then revise the draft
2:57
and so on
2:58
And this type of workflow is more akin
3:00
to doing some thinking and some research
3:02
and then doing some revision
3:04
and then doing some more thinking
3:06
and so on. And with this iterative process
3:09
it turns out that an agentic workflow can take longer
3:12
but it delivers a much better work product
3:15
So
3:16
an agentic AI workflow is a process where
3:18
an LM-based app
3:20
executes multiple steps to complete a task
3:23
In this example
3:24
you might use an LM to write the first essay outline
3:27
and then
3:28
you might use an LM to decide
3:30
what search terms to type into a web search engine
3:33
or really what search terms to call
3:36
a
3:36
web search API with in order to get
3:39
back relevant web pages. Based on that
3:42
you can
3:43
feed the downloaded web pages into an LM
3:45
to help you write the first draft
3:47
and then maybe use another LM to
3:49
reflect and decide what needs more revision
3:52
and then depending on how you design this workflow
3:55
Perhaps you may even
3:57
add a
3:57
human in the loop step where
3:59
the OM has the option to request human review
4:03
maybe of some key facts
4:04
and based on that
4:05
it may then revise the draft
4:07
and this process
4:09
results in a much better work output
4:11
One of the key skills you learn in
4:13
this course is how to take a complex task
4:16
like writing an essay
4:18
and
4:19
breaking it down into
4:21
smaller steps
4:22
for agentic workflow to execute one step at
4:24
a time
4:25
to then get the work output that you
4:27
want
4:28
And knowing how to decompose the task into
4:30
steps
4:31
and how to build the components to execute
4:33
the individual steps well
4:34
turns out to be a tricky but important
4:36
skill
4:37
that
4:38
will determine your ability
4:40
to build agentic workflows for a huge range
4:42
of exciting applications. In this course
4:45
a run example that we'll use and something
4:47
that you build alongside me is a
4:50
research
4:50
agent
4:51
So here's an example of what it will
4:53
look like
4:53
You can enter a research topic like How
4:56
do I build a new rocket
4:58
company to compete with SpaceX?
5:03
I don't personally want to compete with SpaceX
5:05
but if you want to
5:05
you can
5:06
try asking a research agent to help your
5:08
background research
5:09
So this agent starts with planning out what research to use
5:14
including calling a web search engine to download some web pages
5:18
and then to synthesize and rank findings
5:21
draft an outline, have an editor agent review for coherence
5:24
and then finally generate a comprehensive markdown report
5:28
which it has done here
5:29
building a new rocket company to compete with SpaceX
5:31
with an intro, background, Findings and so on
5:35
and I take the points of appropriately that
5:38
this is going to be a tough startup
5:39
to build
5:40
so I'm not personally planning to do this
5:43
but if you want to tackle something like this
5:46
maybe a research agent like this could help
5:49
you with some initial research
5:51
And by
5:51
finding and downloading multiple sources and deeply thinking about it
5:56
this actually ends up with a
5:58
much more thoughtful report than
6:01
just prompting an alarm
6:02
to write an essay for you would
6:04
One of the reasons I'm excited about this
6:06
is because in my work
6:07
I've ended up building quite a few specialized research agents
6:11
be it in
6:12
legal documents for conflict legal compliance or for
6:15
some healthcare sectors or some
6:17
business product research areas
6:19
And so I hope that working through this example
6:22
you not only learn how to build agentic
6:24
workflows for many other applications
6:26
but that some of the ideas in building
6:28
research agents will be
6:30
directly useful to you if you ever need
6:32
to build a custom research Search agent yourself
6:35
Now
6:36
one of the
6:37
often discussed areas of
6:40
AI agents is how autonomous are they?
6:43
What you just saw here was a relatively complex
6:45
highly autonomous agentic AI workflow
6:48
but there are also
6:49
other simpler workflows that are incredibly valuable
6:53
Let's go into the next video to talk
6:54
about the degree to which agentic workflows can be autonomous
6:58
and this will give you
6:59
a framework to think about how
7:02
you might go about building different applications and
7:04
how easy or difficult they might be
7:06
See you in the next video
7:09
Agents can be autonomous to different degrees
7:12
A few years ago
7:13
I noticed within the AI community that there
7:15
was a growing controversial debate about what is an agent
7:19
and some people are writing a paper saying about an agent
7:21
and others say, no
7:22
that's not really a true agent
7:24
And I felt this debate was unnecessary
7:27
which is why I started using the term
7:29
agentic because I thought if we use it
7:32
as an adjective rather than a binary
7:34
it's either an agent or not
7:36
then we can have
7:37
just acknowledge that systems can be agentic to different degrees
7:40
and let's just call out agentic and move
7:42
on with the real work of building these
7:44
systems rather than debating
7:46
you know
7:46
is this sufficiently autonomous
7:48
to be an agent or not
7:50
I remember when I prepared a talk on agentic reasoning
7:53
one of my team members actually came to me and said
7:55
hey, Andrew, we don't need yet another word
7:58
You know, we have agent
7:59
Why you make have another word agentic?
8:01
But I decided to use it anyway
8:03
And then later on
8:04
wrote an article in
8:06
given guys newsletter, the back
8:08
And also post on social media
8:10
saying that
8:11
instead of arguing over which words include or
8:13
exclude as being a true agent
8:15
let's
8:15
acknowledge the different degrees to which systems can
8:18
be agentic
8:19
And I think this helped move past the
8:21
debates on
8:22
what is a true agent and
8:24
let us just focus on actually building them
8:27
Some agents can be less autonomous
8:30
So
8:30
take the example of writing an essay
8:32
about black holes
8:34
You can have a relatively simple agents
8:36
to come up with a few
8:38
web search terms or web search queries
8:40
then you can
8:42
hard code in that you call a web search engine
8:45
fetch some web pages
8:46
and then use that to write an essay
8:48
And this would be an example of a
8:49
less autonomous agent with a fully deterministic sequence of steps
8:54
and this will work okay
8:55
In terms of notational convention
8:57
throughout this course
8:58
I'll use the red color
9:00
as you see here on the left
9:01
to denote the user input
9:03
such as a user query in this case
9:05
or in later examples
9:06
maybe the input document. In the agentic workflow
9:09
The gray boxes denote calls to an LM
9:12
and the green boxes
9:14
like the web search
9:16
and the web fetch boxes that you see here
9:18
indicate steps where
9:19
other software is being used to carry out an action
9:22
such as a web search API call or
9:24
executing code to fetch the contents of a
9:26
website. Then an agent can be more autonomous
9:30
where given a request to write an essay about black holes
9:33
perhaps you let the LM decide
9:35
does it want to do a web search
9:36
or does it want to search recent news sources
9:39
or does it want to search for recent
9:41
research papers on the website archive?
9:43
Based on that, maybe in this example
9:45
the LM, not the human engineer
9:47
but the LM chooses in this case
9:49
to call a web search engine
9:51
and then after that
9:52
you may let the LM decide
9:53
how many web pages does it want to fetch
9:55
or if it fetch the PDF
9:58
does it need to
9:59
call a function or also call the tool
10:01
to convert the PDF to text?
10:03
And in this case
10:03
maybe it fetches its top few web pages
10:06
then it can write an essay
10:08
Decide whether or not to reflect and improve
10:10
and maybe even go back to fetch more
10:12
web pages and then to finally produce
10:14
an output
10:15
And so even for this example of a research agent
10:18
we can see that
10:20
some agents can be
10:22
less autonomous
10:23
with a
10:23
linear sequence of steps to be executed determined by a programmer
10:27
and some can be more autonomous where you
10:29
trust the OM to make more decisions
10:31
and the exact sequence of steps that happens
10:33
may be even determined by the OM rather
10:35
than in advance by the programmer
10:37
So for less autonomous systems
10:40
you will usually have all the steps predetermined in advance
10:43
and any functions it calls like web search
10:45
and we'll call that tool use as you
10:47
learn in the third module in this course
10:49
might be hard coded by the human engineer
10:51
by you or me
10:52
and most of the autonomy
10:54
is in what text the OM generates
10:56
At the end of the spectrum would be highly autonomous agents
10:59
where the agent makes many decisions autonomously
11:02
including, for example
11:03
deciding what is the sequence of steps it
11:05
will carry out in order to
11:07
write the essay
11:08
And there's some highly autonomous agents that can
11:10
even write new functions
11:12
or sometimes they
11:13
create new tools that it can then execute
11:16
And somewhere in between are semi-autonomous agents
11:19
We can make some decisions
11:20
choose tools
11:21
but the tools are usually more predefined
11:23
As you look at different examples in this course
11:25
you learn how to build applications anywhere on
11:28
this spectrum of less to more highly autonomous
11:31
And you find that there are tons of
11:32
applications in the less autonomous end of the
11:34
spectrum
11:35
They're very valuable
11:36
being built for tons of businesses today
11:38
And at the same time
11:39
there are also
11:40
applications being worked on at the more
11:42
highly autonomous end of the spectrum
11:44
but those are usually less easily controllable
11:47
a little bit more unpredictable
11:49
and also
11:50
a lot of active research as well to
11:52
figure out how to build
11:53
these more highly autonomous agents
11:56
And with that
11:57
let's go on to the next video
11:59
to
11:59
dive deeper into this
12:01
and to hear
12:02
about some of the benefits of using agents
12:04
and why they allow us
12:05
to do things that just were not possible
12:07
with earlier generations of
12:09
of base applications
12:12
I think the one biggest benefit of agentic
12:14
workflows is that allows you
12:16
to do many tasks effectively that just previously
12:18
were not possible
12:20
But there are other benefits as well
12:22
including parallelism that lets you do certain things quite fast
12:25
as well as modularity that lets you combine
12:27
the best of three components from many different
12:30
places to build an effective workflow
12:32
Let's take a look
12:33
My team collected some data
12:35
on a
12:36
coding benchmark
12:37
that tests the ability of different OLMs to
12:40
write code to carry certain tasks
12:43
The benchmark used in this case is called HumanEval
12:46
and it turns out that GPT-3.5
12:48
this is a model that the first
12:49
hopefully available version of ChatGPT was based on
12:52
if asked to write the code directly
12:54
to just type out the computer program
12:56
gets 48% right on this benchmark
12:59
This is a passe K metric
13:00
GPT-4 is a much better model
13:02
Its performance leaps to
13:04
67% with this also non-agentic workflow
13:08
But it turns out that
13:09
as large as the improvement was from GPT-3.5 to GPT-4
13:13
that improvement is doled by what you can
13:15
achieve by wrapping GPT-3.5 within an agentic workflow
13:20
Using different agentic techniques
13:22
which you learned about late in this course
13:24
you can prompt GPT-3.5 to write code
13:27
and it may be
13:28
reflect on the code and figure out if
13:29
it can improve it. And using techniques like that
13:32
you can actually get GPT-3.5 to get much
13:35
higher levels of performance. And similarly
13:37
GPT-4
13:38
used in the context of an agentic workflow
13:41
also does much better
13:43
So even with today's SLMs and agentic workflow
13:46
lets you get much better performance
13:49
And in fact
13:49
what we
13:50
saw in this example was the improvement from
13:52
one generation of model to another
13:55
which is huge
13:55
is still not as big a difference as
13:58
implementing an agentic workflow on the previous generation
14:01
of model
14:01
Another benefit
14:03
of using agentic workflows is that they can
14:06
paralyze some tasks
14:07
and thus do certain things much faster
14:10
than a human
14:11
For example
14:12
If you ask an
14:13
agentic workflow to write an essay about black holes
14:16
you might be able to have three OMs
14:18
run in parallel to generate ideas
14:20
for web search terms
14:22
to type into search engine
14:23
Based on the first web search
14:25
it may identify, say, three top results to fetch
14:29
and based on the second web search
14:31
it may identify a second set of web pages to fetch
14:34
and so on
14:35
And it turns out that whereas
14:37
a human doing this research
14:38
would have to read these nine web pages
14:41
sequentially or one at a time
14:42
when you're using an agentic workflow
14:44
you can actually parallelize all nine web page downloads
14:49
and then finally feed all these things into
14:51
an LM to write an essay
14:52
So even though agentic workflows
14:54
do take longer than
14:55
purely non-agentic workflows or by direct generation of
14:59
I just prompting a single time
15:00
if you were to compare
15:02
this type of agentic workflow to how a
15:04
human would have to go about the task
15:06
the ability to parallelize downloading lots of web
15:08
pages
15:09
can actually let it do certain tasks much
15:11
faster. Faster than the
15:12
non-parallel sequential way that a single human might
15:16
process this data
15:17
To build on this example
15:19
it turns out
15:19
one of the things I often do when
15:21
building agentic workflows is
15:23
look at the individual components like the
15:25
LM and add or swap out components
15:28
So for example
15:30
maybe I look at a web search engine
15:31
I use up here
15:32
and I might decide that I want to
15:34
swap in a new web search engine
15:36
When building agentic workflows
15:37
there are actually multiple web search engines
15:39
including Google, which you can access by server
15:42
as well as others like Bing
15:44
DuckDuckGo, Tivili
15:45
u.com. There are actually quite a lot of
15:47
options for web search engines designed for OMS
15:50
to use
15:51
Or maybe instead of just doing three web searches
15:53
maybe on this step
15:54
you can swap in a new
15:56
news search engine
15:57
so we can
15:58
find out what's the latest news on
16:00
recent breakthroughs on black hole science
16:04
And lastly
16:05
instead of using the same LM for all
16:07
of the different steps
16:09
I will often try out different
16:11
large models
16:12
and maybe try out different LLM providers
16:15
to see which one
16:16
gives the best result for different
16:18
steps of this system
16:20
So to summarize
16:21
the main reason I use agentic workflows is
16:23
to just gives much better performance on many
16:26
different applications. But in addition
16:29
it can also paralyze
16:30
some tasks that
16:32
humans will otherwise have to do sequentially
16:34
And
16:35
the modular design of many
16:37
agentic workflows also lets us
16:39
add update tools and sometimes swap out models
16:43
We've talked a lot about the key components
16:45
of building agentic workflows
16:47
Let's now take a look
16:48
at the range of agentic AI applications to
16:51
give you a sense of the sorts of
16:52
things people are already building and the sorts
16:54
of things you build to build yourself
16:57
Let's go on to the next video
17:00
People and businesses do a lot of stuff
17:04
How do you take this
17:05
useful stuff that we do
17:06
and break it down into discrete steps
17:09
for an agentic workflow to follow?
17:11
Let's take a look
17:12
Take the example of building a research agent
17:15
If you want
17:17
a AI system to write an essay on the topic X
17:19
one thing you could do is
17:20
prompt an LLM to have it
17:22
generate an output
17:23
directly
17:24
But if you were to do this for
17:26
topics that you want deeply researched
17:28
you may find that the LLM output
17:30
covers only the surface level points or maybe
17:32
covers only the obvious facts
17:34
but doesn't go as deep into
17:36
the subject as you want it to
17:38
In this case
17:39
you might
17:39
then reflect on how you as a human
17:42
would write an essay on a certain topic
17:45
Would you just sit down and start writing
17:46
or would you take multiple steps
17:48
such as first write an essay outline
17:50
and then search the web
17:51
and then based on the input from the web search
17:54
write the essay?
17:55
And as I take a task and decompose it into steps
17:58
one question I'm always asking myself is
18:01
if I look at these steps one
18:02
two, and three
18:03
can each of them be done
18:05
either by an LLM or
18:07
by
18:07
a short piece of code or by a
18:09
function call by a tool
18:11
And so in this case
18:12
I think an LLM
18:13
can maybe write a decent outline on
18:15
many topics that I would want it to
18:18
help me think through. So say
18:20
probably okay on the first step
18:22
And then I know how to
18:23
use an LLM to
18:24
generate search terms to search the web
18:27
So I would say the second step is
18:28
also doable. And then based on web search
18:31
I think an LLM could input the web
18:33
search results and write an essay
18:34
And so this would be a reasonable first
18:36
attempt at an agentic workflow
18:38
for writing an essay
18:39
that goes deeper than just direct generation
18:42
But if
18:43
I were to then implement this agentic workflow
18:45
and look at the results
18:47
maybe
18:47
you find that the results still aren't good
18:49
enough. It's still not yet as deeply thoughtful
18:51
Maybe the essays feel a little bit disjointed
18:54
This has actually happened to me
18:55
Once built a research agent using this sort of workflow
18:58
but when I read the output
18:59
it felt a bit disjointed
19:00
You know
19:00
the start of the article didn't feel completely
19:03
consistent with Middle than if you're completely consistent
19:06
with the end. In this case
19:07
what you might do is then reflect
19:09
on how you would change the workflow if
19:11
you as a human found that the essays
19:13
were a bit
19:13
disjointed. One thing you could do
19:15
is take the third step and further decompose
19:19
write the essay into additional steps
19:22
So instead of writing the essay all in one go
19:25
you might instead have it
19:27
write the first draft
19:28
and then consider what parts need revision
19:30
and then revise the draft
19:32
And this would be
19:33
how I as a human might go about it to
19:36
not just write the final essay at my first attempt
19:39
but write the first draft and then read over it
19:41
which
19:42
is another step that the arm
19:44
is pretty decent at. And then based on
19:46
my own critique of my own essay
19:48
I'll revise the draft
19:49
So the recap, I started off with direct generation
19:53
just one step, decided it wasn't good enough
19:55
and so broke that down into three steps
19:58
and then maybe decide that still isn't good enough
20:00
and took one of the steps and further
20:02
broken it down or decomposed it
20:04
Into three more steps
20:05
resulting in this more complex
20:07
richer process for generating an essay
20:11
And depending on
20:12
how satisfied you are with the results of this process
20:15
you may choose to even modify
20:17
this essay generation process further
20:19
Let's look at a second example of how
20:21
to decompose complex tasks into smaller steps
20:25
Take the example of responding to
20:27
basic customer order
20:29
inquiries
20:30
The first step that the human customer service
20:32
agent might carry out
20:33
might be to first extract the key information
20:36
such as who is this email from
20:38
what do they order
20:39
and what is the order number
20:40
And these are things an OM could do
20:43
so I could just say
20:44
let's have an OM do that
20:45
The second step would be to then
20:47
find the relevant customer records
20:49
So to
20:49
write and generate the relevant database queries to
20:52
pull up the order of what the customer
20:54
had ordered and when it shipped and so
20:56
on
20:56
I think an OM
20:57
with the ability to call a function
21:00
to query
21:01
the orders database should be able to do And lossy
21:05
having pulled up the customer record or the customer order record
21:08
I might then
21:09
write and send a response back to the
21:10
customer. And I think with the information we pulled up
21:13
this third step is also
21:14
doable with an OM. If I give the
21:16
option to
21:17
call an API to send an email
21:19
So this would be another example
21:21
of taking a task
21:22
of responding to
21:23
customer email
21:24
and breaking it down into three individual steps
21:27
where I can look at each of these steps and say
21:29
yep
21:29
I think an OM or an OM
21:31
with the ability to call a function
21:33
to query database or send an email
21:35
should be able to do that
21:36
Just one last example for the invoice processing
21:39
After a PDF invoice has been converted to text
21:42
the first step is to pull out the required information
21:46
the name of the bill
21:46
the address, the due date, the amount due
21:49
and so on. And now it should be
21:50
able to do that
21:51
And then if I want to
21:53
check that
21:53
the information was extracted and save it in
21:56
a new database entry
21:57
then I think an
21:58
OM should be able to
21:59
help me call
22:00
a function to update the database record
22:02
And so to implement this
22:04
We implement an agentic workflow to
22:06
carry out basically these two steps
22:08
When building agentic workflows
22:10
I think of myself
22:12
as having a number of building blocks
22:14
One important building block would be
22:16
large language models or maybe large multimodal models
22:19
if
22:19
I want to
22:20
try to process images or audio as well
22:22
And OMs are good at generating text
22:24
deciding what to call
22:25
maybe extracting information
22:28
For some highly specialized tasks
22:29
I might also use
22:31
some other
22:31
AI models
22:32
such as AI model for converting a PDF
22:35
to text
22:36
or for text-to-speech or for image analysis
22:39
In addition to AI models
22:40
I also have access to a number of software tools
22:44
including different APIs that I can call to do direct search
22:48
to get maybe real-time weather data
22:51
to send emails, check calendar, and so on
22:53
And I might also have tools to retrieve information
22:57
to pull up data
22:58
from a database or
23:00
to implement
23:01
rag or retrieval augmented generation where I can
23:03
look up A large text database and find
23:05
the most relevant text
23:06
or I might also
23:07
have tools to execute code
23:10
And this is a tool that lets an
23:11
LLM
23:12
write code and then run the code on
23:13
your computer to do
23:15
a huge range of things
23:17
In case some of these tools seem a
23:18
bit foreign to you
23:19
don't worry about it. We'll go through
23:21
the most important tools in much greater detail
23:23
in a later module. But
23:25
I think a lot of my work when
23:26
I'm building an agentic workflow as looking at
23:29
the work that a person or business is
23:31
doing and then trying to figure out with these building blocks
23:34
how can I sequence these building blocks together
23:36
in order to carry out the tasks
23:38
that I want my system to carry out
23:41
And this is why
23:42
having a good understanding of what building blocks are available
23:45
which I hope you
23:46
have a better sensor
23:48
by the end of this course as well
23:49
will allow you to better envision what agentic
23:51
workflows you can build by combining these building
23:54
blocks together
23:55
So to summarize
23:57
one of the key skills in building agentic
23:58
workflows is to look at
24:00
a bunch of stuff that maybe someone does
24:02
and to identify The discrete steps
24:05
that it could be implemented with
24:08
And when I'm looking at the individual discrete steps
24:10
one question I'm always asking myself is
24:13
can this step be implemented
24:15
with either an LM
24:17
or with one of the tools such as
24:18
an API or a function call
24:20
that I have access to?
24:22
And in case the answer is no
24:23
I'll then often ask myself
24:25
how would
24:25
I as a human do this step?
24:27
And is it possible to decompose this further
24:29
or break this down into
24:31
even smaller steps that then maybe
24:34
is more amenable to implementation
24:36
with an LM or with one of the
24:38
software tools that I have?
24:40
So
24:40
I hope this gives you a rough sense
24:42
of how to think about task decomposition
24:45
In case you feel like you don't fully have it yet
24:48
don't worry about it. We'll go through many
24:49
more examples in this course
24:50
and you have a
24:51
much better understanding of this
24:53
by the end of this course
24:54
But it turns out that as you build agentic workflows
24:58
you find that often you build an initial task decomposition
25:01
initial agentic workflow
25:03
and then You want to keep on
25:05
iterating and improving on it quite a few
25:07
times until it
25:08
delivers the level of performance that you want
25:11
And to drive this improvement process
25:13
which is found important for many projects
25:16
one of the key skills is to know
25:17
how to evaluate your agentic workflow
25:20
So in the next video
25:22
we'll talk about evaluations or
25:24
evals and just be a key component of
25:27
how you can build
25:28
and then also
25:28
keep on improving your workflows to get the
25:30
performance that you want
25:32
Let's talk about evals in the next video
25:36
I've worked with many different teams on building agentic workflows
25:39
and
25:40
I found that
25:41
one of the biggest predictors
25:42
for whether someone is able to do it
25:44
really well versus be
25:46
less efficient at it is
25:48
whether or not they're able to drive a
25:50
really disciplined evaluation process
25:53
So your ability to drive
25:55
evals for your agentic workflow
25:57
makes a huge difference in your ability
26:00
to build them effectively
26:02
In this video
26:03
we'll take a quick overview of how to build evals
26:06
and this is a subject that we'll actually
26:07
go into
26:08
much deeper
26:09
in a later module in this course
26:11
But let's take a look
26:13
After building an agentic workflow
26:15
like this one for responding to customer order inquiries
26:19
it turns out that it's very difficult to
26:21
know in advance
26:22
what are the things that could go wrong
26:24
And so rather than trying to build evaluations in advance
26:28
what I recommend is you just
26:29
look at the outputs and
26:31
manually look for
26:33
things that you wish it was doing better
26:36
For example
26:37
maybe you read a lot of outputs and
26:38
find that it is unexpectedly
26:41
mentioning your competitors more than it should
26:44
Many businesses don't want their
26:46
agents to mention competitors because it just creates
26:48
an awkward situation. And if you read some of his outputs
26:51
maybe you'll find that there's something that says
26:53
I'm glad you shocked us
26:54
We're much better than our competitor
26:56
CompCo
26:57
Or maybe sometimes say
26:58
sure, we should refund. Unlike RivalCo
27:00
we make returns easy. And you may look
27:02
at this and go
27:03
gee
27:03
I really don't want us to mention competitors
27:07
This would be the example of a problem
27:08
that
27:09
is really hard to anticipate
27:10
in advance of building this agentic
27:13
So the best practice is really to build
27:15
it first
27:15
and then examine it to figure out
27:17
where it is not yet satisfactory
27:20
and then to find ways to evaluate as
27:22
well as improve the system to eliminate the
27:24
ways that it is still not yet satisfactory
27:27
Assuming your business considers it an error
27:31
or a mistake to mention competitors in this way
27:34
then as you work on eliminating these competitor mentions
27:38
one way to track progress would to add
27:40
an evaluation or an eval
27:42
to track how often this error occurs
27:45
So if you have a name list of competitors like CompCo
27:48
RivalCo, the other Co
27:50
then you can actually write codes
27:51
So just search in your own output
27:54
for how often it mentions these competitors by
27:57
name and count up as a number
27:59
as a fraction of the overall responses
28:02
how frequently it mistakenly mentions competitors
28:05
One nice thing about the problem of competitor mentions is
28:09
is a objective metric
28:11
meaning either the competitor That was mentioned or not
28:14
and for objective criteria
28:16
you can write code to check
28:19
for how often this specific error occurs
28:22
But because LLM's output free text
28:24
there also going to be
28:26
criteria
28:27
by which you want to evaluate this output
28:29
that may be more
28:30
subjective and where it's harder
28:32
to just write code to output a black
28:34
and white score. In this case
28:36
using a
28:37
LLM
28:38
as a judge is a common technique
28:40
to evaluate the output. So for example
28:42
ifIf you're building a research agent
28:44
to do research on different topics
28:47
then you can use another LLM
28:49
and prompt it to maybe say
28:51
assign the following essay a quality score between one and five
28:54
where
28:55
one is
28:55
the worst and five is the best essay
28:58
Here I'm using a Python expression to mean
29:01
you know
29:01
copy paste the generated essay into this
29:03
So
29:04
you can prompt the LLM to read the
29:05
essay and assign it a quality score
29:09
Then
29:09
I'm asked the research agent to write a
29:11
number of different research reports
29:13
For example
29:14
on recent developments in black hole science
29:16
or using robots to harvest fruit
29:19
And then in this example
29:21
maybe the judge OM
29:22
assigns the essay on black holes a score of three
29:25
the essay on robot harvesting a score of four
29:28
and as you work on improving your research agent
29:31
hopefully you
29:32
see these scores go up over time
29:34
It turns out, by the way
29:35
that
29:35
OMs are
29:36
actually not that good at these one to
29:39
five scale ratings. You can give it a shot
29:41
like Percy tend not to use this technique that much myself
29:44
but in a later module
29:45
you learn
29:46
some better techniques to have an OM
29:48
output
29:49
more accurate scores than asking it to output
29:51
scores on a one to five scale
29:52
although some people will do this
29:54
maybe an initial cut
29:56
as an OM as judge type of eval
29:58
Just to give a preview of
30:00
some of the
30:01
agentic AI
30:02
evals you learn about later in this course
30:04
You've already heard me talk about how you
30:05
can write codes to evaluate objective criteria
30:09
such as did it mention the competitor or not
30:11
or use an OM as a judge for more objective criteria
30:13
such as what's the quality of this essay
30:15
But later
30:16
you learn about two major types of evals
30:18
One is end-to-end
30:19
where you measure the output quality of the entire agent
30:23
as well as component level evals
30:25
where you might measure the quality of
30:27
the output of a single step in the
30:29
agentic workflow
30:30
And it turns out that these are useful
30:31
for driving different parts of your development process
30:34
And one thing I do
30:36
a lot as well is just examine the intermediate outputs
30:39
or sometimes we call these the traces of the LM
30:42
in order to understand where it is
30:44
falling short of my expectations
30:46
And we call this error analysis
30:48
where we just read through the intermediate outputs
30:50
of every single step to try to spot
30:52
opportunities for improvement
30:54
And it turns out being able to do
30:55
evals and error analysis is a
30:57
really key skill. So with much more
30:59
to say about this in
31:01
the fourth module in the schools
31:03
We're nearly to the end of this first
31:04
module
31:05
Before moving on
31:06
I just want to share with you
31:07
what I think are the most important design
31:09
patterns for building agentic workflows
31:12
Let's We'll take a look at that
31:14
in the next video
31:16
We build agentic workflows by taking building blocks
31:18
and putting them together
31:20
to sequence out these complex workflows
31:23
In this video
31:24
I'd like to share with you a few
31:25
of the key design patterns
31:27
which are patterns for how you can think
31:29
about
31:30
combining these building blocks into more complex workflows
31:34
Let's take a look
31:35
I think four key design patterns for building
31:37
agent workflows are reflection
31:39
two-use, planning, and multi-agent collaboration
31:42
Let me briefly go over what they mean
31:44
and then we'll actually go through
31:46
most of these in-depth later in this course
31:48
as well
31:49
The first of the
31:50
major design patterns is reflection
31:53
So
31:54
I might go to an LM agent and
31:55
ask it to write code
31:57
and it turns out that
31:58
an LM might then generate code like this
32:01
you know
32:01
defines here a Python function to do a
32:03
certain task. I could then construct a prompt
32:05
that looks like this. I can say
32:07
here's code intended for a certain task
32:09
and then copy paste whatever the LM had
32:11
just output back into this prompt
32:13
And then I'll ask it
32:14
check the code carefully for correctness
32:15
style, and efficiency, and give constructive criticism
32:17
And it turns out that the same LM
32:19
model prompted this way
32:20
may be able to point out some problems
32:23
with the code
32:24
And if I then take this
32:25
critique and feed it back to the model to say
32:29
looks, this is a bug
32:30
could you change the code to fix it?
32:32
Then it may
32:33
actually come with a better version
32:35
of the code. To give a preview of tool use
32:39
if you're able to run the code
32:40
and see where the code fails
32:42
then
32:43
feeding that back to the
32:45
LM can also cause it to
32:47
be able to iterate and generate a much better
32:50
say, v3, version 3 of the code
32:52
So reflection is a common design pattern
32:54
where you can ask the LM to examine its own outputs
32:57
or maybe bring in some external sources of information
33:01
such as
33:01
run the code and see if it generates any error messages
33:04
and use that as feedback to
33:06
iterate again
33:07
and come over a better version of its
33:09
output. And this design pattern isn't magic
33:12
It does not result in everything working 100% of the time
33:15
but sometimes it can be a
33:17
nice bump in the performance of your system
33:19
Now
33:19
I've drawn this as if it was a
33:21
single OM that I'm prompting
33:23
To foreshadow multi-agent workflows
33:26
you can also imagine
33:28
instead of having the same model critique itself
33:30
you can imagine having a critic agent
33:33
and all that is is an LLM that's
33:34
been prompted with instructions like
33:37
your role is to critique code
33:38
here's code intended for a task
33:40
check the code carefully
33:41
and so on. And the second critic agent
33:43
may be able to point out errors or run unit tests
33:46
and like having two
33:48
simulator agents where each agent is just an
33:50
LLM prompted to take on a certain persona
33:53
you can have them go back and forth
33:54
to iterate to get a better output
33:57
In addition to reflection pattern
33:59
the second important design pattern is to use
34:02
where today, LLMs can be given tools
34:05
meaning
34:05
functions that they can call in order to
34:07
get work done. For example
34:09
if you ask an LLM
34:10
what's the best coffee maker according to reviewers
34:12
and you give it a web search tool
34:14
then it can actually
34:15
search the internet to find much better answers
34:18
or a code execution tool
34:20
If you ask a math question
34:21
like if I invest $100
34:23
I compound What have you
34:24
it can then write code and execute code
34:27
to compute an answer. Today
34:28
different developers have given
34:30
LLMs many different tools for everything from
34:32
math or data analysis to gather information by
34:36
fetching things from the web or for various databases
34:38
to interface with productivity apps like email
34:40
calendar, and so on
34:42
as well as
34:43
to process images and much more
34:45
And the ability of an LLM
34:47
to decide what tools to use
34:49
meaning what functions to call
34:51
that lets the model
34:52
get a lot more done
34:53
The third of the four design patterns is
34:56
planning. This is an example from a paper called Hugging GPT
35:01
in which if you ask a
35:04
system to
35:05
please generate an image where a girl is reading a book
35:07
and the pose is the same as a
35:08
boy in the image
35:09
then please describe the new image in your
35:10
voice. Then
35:11
a model can automatically decide that to carry out this task
35:15
it first needs to
35:17
find a pose determination model to figure out
35:19
the pose of the boy
35:20
then to pose the image
35:21
to generate a picture of a girl
35:23
and an image. The text
35:24
and then finally, text-to-speech. And so in planning
35:27
an LLM decides
35:29
what is the sequence of actions it needs
35:31
to take. In this case
35:32
it is a sequence of API calls
35:35
so that it can then carry out the
35:37
right sequence of steps in the right order
35:40
in order to carry out the task
35:42
So rather than the developer hard-coding the sequence
35:45
of steps in advance
35:46
this actually lets the
35:47
LLM decide what are the steps to take
35:50
Agents that plan today are
35:52
harder to control and somewhat more experimental
35:54
but sometimes it can give really delightful results
35:57
And then finally, multi-agent workflows
36:00
Just as a human manager might hire a
36:02
number of others to work together on a complex project
36:06
in some cases
36:06
it might make sense for you to hire
36:08
a set of multiple agents
36:10
maybe each of which specializes in a different role
36:13
and have them work together
36:15
to accomplish a complex task
36:17
The picture you see here on the left
36:18
is taken from a project called ChatDev
36:21
which is a
36:21
software framework created By Chen Chen and collaborators
36:25
In ChatDev, multiple agents with different roles
36:28
like chief executive officer
36:30
programmer, tester, designer, and so on
36:33
collaborate together
36:34
as if they were a virtual software company
36:37
and
36:38
can collaboratively complete a range of
36:40
software development tasks
36:42
Let's consider another example
36:44
If you want to write a marketing brochure
36:46
maybe you'd think of hiring a team of three people
36:49
such as a researcher to do online research
36:52
a marketer to write the marketing text
36:54
and then finally an editor to
36:56
edit and polish the text
36:58
And so in a similar way
37:00
you might consider building a multi-agent
37:03
workflow in which you have a simulated research agent
37:06
a simulated marketer agent
37:08
and a simulated editor
37:10
agent that then
37:11
come together
37:12
to carry out this task for you
37:14
Multi-agent workflows are more difficult to control
37:18
since you don't always know ahead of time
37:20
what the agents will do
37:21
but research has shown that they can
37:23
In better outcomes for many complex tasks
37:26
including things like writing biographies or deciding on
37:30
chess moves to make in the game
37:31
You learn more about multi-agentic workflows later in
37:34
this course as well
37:35
And so of that
37:36
I hope you have a sense of what
37:37
agentic workflows can do
37:40
as well as of what are the key
37:41
challenges of finding building blocks and putting them together
37:44
maybe by these design patterns
37:47
in order to implement an agentic workflow
37:49
And of course, also developing evals
37:52
You can see how our system is doing
37:54
and keep on improving on it
37:56
In the next module
37:58
I'd like to share with you
38:00
a deep dive into the first of these design patterns
38:03
that is reflection
38:05
and you find that
38:06
is a
38:07
maybe surprisingly simple to implement technique
38:10
that can give the performance of your system
38:12
sometimes a very nice bump
38:14
So
38:15
let's go on to the next module to
38:17
learn about the reflection design pattern
38:21
In this module
38:22
I'd like to share with you practical tips
38:23
for building agentic AI workflows
38:26
I hope that these tips
38:27
will enable you to be
38:28
much more effective than
38:30
the
38:30
typical developer at building these types of systems
38:34
I find that when developing an agentic AI system
38:37
it's difficult to know in advance
38:39
where it will work and where it won't work so well
38:42
and thus where you should focus your effort
38:45
So very common advice is to
38:47
try to build even a quick and dirty system to start
38:51
so you can then
38:52
try it out and look at it
38:53
to see where it
38:55
may not yet be working as well as you wish
38:57
to then have
38:58
much more focus efforts to develop it even further
39:02
In contrast, I find that it's sometimes less useful to
39:05
you know
39:05
sit around for too many weeks
39:07
theorizing and hypothesizing how to build it
39:10
It's often better to just
39:12
build a quick system in a safe
39:15
reasonable way that doesn't leak data
39:16
kind of do it in a responsible way
39:19
but just build something quickly so you can
39:21
look at it and then use that initial
39:22
prototype to prioritize and drive further development
39:26
Let's start an example of what might happen
39:29
after you've built a prototype
39:31
I want to use as our first example
39:33
the invoice processing workflow that you've seen previously
39:37
where the tasks extract four required fields and
39:39
then they save it to a database record
39:42
After having built such a system
39:45
one thing you might do is find a handful of invoices
39:48
maybe 10 or 20 invoices
39:49
and go through them and just take a
39:51
look at their output and see what went
39:53
well and if there were any mistakes
39:55
So let's say you look through 20 invoices
39:57
you find that invoice one is fine
39:59
the output looks correct. For invoice two
40:01
maybe you're confused, the date of the invoice
40:03
that is when was the invoice issued with
40:05
the due date of the invoice
40:07
And in this task
40:08
we want to extract the due date so
40:09
that we can issue payments on time
40:11
So then I might note down in a
40:13
document or in a spreadsheet that for invoice two
40:16
the dates were mixed up
40:17
Maybe invoice D was fine
40:18
invoice four was fine
40:19
and so on. But as I go through this example
40:22
I find that there are quite a lot
40:23
of examples where I had
40:25
mixed up the dates
40:26
So
40:27
it is based on
40:28
going through a number of examples like this
40:30
that in this This case
40:32
you might conclude that
40:33
one common error mode is that it is
40:35
struggling with the dates. In that case
40:39
one thing you might consider would be to
40:41
of course
40:42
figure out how to improve your system to
40:44
make it extract due dates better
40:46
but also maybe write an eval
40:48
to measure the accuracy with which it is
40:50
extracting due dates. In comparison
40:53
if you had found that it was
40:55
extracting the bullet address incorrectly
40:57
who knows, maybe you have billers with unusual sounding names
41:01
and so maybe it struggles with billers
41:03
or especially if you have international billers whose
41:05
names may not even all be English letters
41:08
then you might instead focus on building an
41:10
eval for the biller address
41:12
So one of the reasons why
41:13
building a quick and dirty system
41:15
and looking at the output is so helpful
41:17
is it even helps you decide
41:19
what do you want to put the most
41:20
effort into evaluating. Now
41:24
If you've decided that you want to
41:26
modify your system to
41:27
improve the accuracy
41:29
with which it is extracting the due date of the invoice
41:31
then to track progress
41:33
it might be a good idea to create an evaluation
41:36
an eval
41:37
to measure the accuracy of data extraction
41:40
There are probably multiple ways one might go about this
41:42
but let me share with you how
41:44
I might go about this
41:45
To create a test set or an evaluation set
41:48
I might
41:49
find 10 to 20 invoices and
41:51
manually write down what is the due date
41:54
So maybe one invoice is a due date of August 20th
41:56
2025, and I write it down in the standard year
41:59
month, date
42:01
format
42:02
And then to make it easier to evaluate in code later
42:06
I would probably write the prompt of the
42:07
alarm to tell it to always format the
42:09
due date in this year month date format
42:12
And with that
42:13
I can then write code
42:14
to extract out the one date that the alarm has output
42:17
which is the due date
42:18
That's the one day we care about
42:19
So this is a regular expression pattern matching
42:22
you know, four numbers for the year
42:23
two for the month
42:24
two for the date
42:25
and then extract that out
42:26
And then I can
42:27
just write code to test if the extract
42:29
date is equal to the actual date
42:32
That is the ground truth annotation
42:34
I had written down. So with an eval set of
42:37
say, 20 or so invoices
42:39
I build up my changes to see if
42:41
the percentage of time
42:43
that it gets the extracted date correct is
42:45
hopefully going up as I
42:46
tweak my prompts or tweak other parts of
42:48
my system. So just to summarize what we've seen so far
42:52
we have built a system
42:53
then look at outputs to discover
42:55
where it may be behaving in an unsatisfactory way
42:58
such as due dates are wrong
42:59
Then to drive improve
43:01
So this important output put in place a
43:03
small eval with say just 20 examples to
43:05
help us track progress
43:07
And this lets me go back to two prompts
43:09
try different albums
43:10
and so on to see if I can
43:11
move up this metric of due date accuracy
43:15
So this is what improving a
43:17
agentic AI workflow will often feel like
43:19
Look at the output
43:20
see what's wrong, then if you know how to fix it
43:22
just fix it. But if you need a
43:23
longer process of improving it
43:24
then put in place an eval and use
43:26
that to drive further development
43:28
One other thing to consider is if after
43:30
working for a while
43:31
if you think those 20 examples you had
43:33
initially aren't good enough
43:34
maybe they don't cover all the cases you want
43:36
or maybe 20 examples is just too few
43:38
then you can always
43:39
add to the eval set over time to
43:41
make sure it better reflects
43:42
your personal judgments on whether or not the
43:45
system's performance is sufficiently satisfactory
43:48
This is just one example
43:49
For the second example
43:50
let's look at building a marketing copy assistant
43:53
for writing captions for Instagram
43:56
where to keep things succinct
43:57
let's say our marketing team tells us that
43:59
they want captions that are at most
44:01
Words long
44:02
So we would have a image of a product
44:04
say a pair of sunglasses that we want to market
44:07
and then have a user query
44:08
like please write a caption to sell these sunglasses
44:12
and then
44:12
have a
44:13
OM or a large multimodal model
44:15
analyze the image and the query
44:17
and generate a description of the sunglasses
44:20
And
44:21
there are lots of different ways that
44:22
a marketing copy assistant may go wrong
44:25
but
44:25
let's say that you look at the output
44:26
and you find that the copy or the
44:28
text generated mostly sounds okay
44:30
but maybe it's just sometimes too long
44:32
So for the sunglasses input
44:33
generate 17 words. If you have a coffee machine
44:36
it's okay. Stylish size is okay
44:37
Blue shirt, 14 words. Blender, 11 words
44:40
So it looks like in this example
44:42
the OM is
44:43
having a hard time adhering to the length guideline
44:46
So again
44:47
there are lots of things that could have
44:48
gone wrong with the marketing copy assistant
44:50
but if you find that it's struggling with
44:52
the length of the output
44:54
then you might build an eval to track
44:56
this
44:57
so that you can
44:58
make improvements
44:59
and make sure it's getting better at it
45:01
To the length guideline
45:03
So to create an eval to measure the text length
45:05
what you might do is create a set of test stars
45:08
so mark the pair of sunglasses
45:09
a coffee machine, and so on
45:11
and maybe create 10 to 20 examples
45:14
Then
45:14
you would run each of them through your
45:16
system
45:17
and write code to measure the word count
45:19
of the
45:20
output. So this is Python code to measure
45:22
the word count of a piece of text
45:25
Then lastly
45:25
you would compare
45:26
the length of the generated text
45:29
to the 10 word target limit
45:32
So if word count is equal to 10
45:33
non-correct plus equals 1
45:35
One difference between this
45:37
and the previous invoice processing example is that
45:40
there is no per example ground truth
45:43
The target is just 10
45:45
same for every single example
45:46
Whereas in contrast, for the invoice processing example
45:49
we had to generate a custom
45:51
target label that is the correct due date of the invoice
45:55
and we're testing the outputs
45:56
against that per example ground truth
45:58
I know I use a very simple workflow
46:00
for Generating these captions
46:02
but
46:02
these type of evals can be applied to
46:04
much more complex generation workflows as well
46:07
Let me touch on one final example in
46:09
which we'll revisit the research agents
46:12
we've been looking at
46:13
If you look at the output of
46:15
the research agents on different input prompts
46:18
let's say that
46:19
when you ask it to write
46:21
an article on recent breakthroughs in black hole science
46:24
you find that it missed some high-profile result
46:27
and a loss of news coverage
46:28
So this is an unsatisfactory result
46:30
Or if you ask it to
46:31
research renting versus buying a home in Seattle
46:33
well, it seems to do a good job
46:35
Or robotics for harvesting fruit
46:37
well, it didn't mention a leading equipment company
46:40
So based on this evaluation
46:42
it looks like sometimes it misses a really
46:44
important point that a human expert writer
46:47
would have captured. So
46:50
then
46:50
I would create an eval to measure how
46:52
often it captures the most important points
46:55
For example
46:56
you might come up with a
46:57
number of example prompts on black holes
47:00
robotic harvesting, and so on
47:01
And for each one
47:02
come up with, let's say
47:03
three to five gold standard discussion points
47:06
for each of these topics
47:08
Notice that here
47:10
we do have a per example
47:11
annotation because the gold standard talking points
47:14
that is the most important talking points
47:16
they are different for each of these examples
47:18
With these ground truth annotations
47:20
you might then use an LMS judge to
47:22
count how many of the gold standard talking
47:25
points were mentioned. And so
47:27
an example prompt might be to say
47:29
determine how many of the five gold standard
47:31
talking points are present in the provided essay
47:34
Here are the original prompts
47:35
the essay text, gold standard points, and so on
47:37
and have it return a JSON object with
47:39
two keys that scores how many of the points
47:42
zero to five, did it score
47:43
as well as an explanation
47:45
And this allows you to get a
47:47
score for each prompt
47:49
in your evaluation set
47:51
In this example
47:52
I'm using
47:53
L as a judge
47:54
to count how many of the
47:56
talking points were mentioned because there's so many
47:58
different ways to talk about these talking points
48:01
and so a
48:01
regular expression or a code for
48:04
simple pattern matching might not work that well
48:06
which is why you might use a
48:08
L as a judge and treat this as
48:10
a slightly more subjective evaluation for whether or not
48:13
say, event horizons were adequately mentioned
48:16
So this is your third example
48:18
of how you might build evals
48:20
In order to
48:21
think about how to build evals for your application
48:25
the evals you build will often
48:27
have to reflect whatever
48:28
you
48:29
see
48:29
or you are worried about going wrong
48:31
in your application
48:32
And it turns out that broadly
48:35
there are two axes of evaluation
48:38
On the top axis is the way you
48:40
evaluate the output
48:43
In some cases
48:43
you evaluate it by writing code with objective evals
48:48
and
48:49
sometimes you use an OM
48:51
as a judge for more subjective evals
48:54
On the other axis
48:55
is whether you have a per example
48:58
ground truth or not
49:00
So for checking invoice data extraction
49:03
we were writing code to evaluate if we
49:05
got the actual date
49:07
and that had a per example ground truth
49:10
because each invoice has a different actual date
49:13
But in the example where we checked marketing copy length
49:16
every example had a length limit of 10
49:20
and so
49:20
there was no per example ground truth for
49:23
that problem
49:24
In contrast, for counting gold standard talking points
49:27
there was a per example ground truth because
49:29
each article had different important talking points
49:32
But we use an OM as a judge
49:34
to read the essay to see if those
49:36
topics were adequately mentioned
49:38
because there's so many different ways
49:39
to mention the talking points
49:41
And
49:42
the last The four quadrants would be
49:44
L message judge
49:45
with no per example ground truth
49:48
And
49:48
one place where we saw that was
49:50
if you are grading charts with a rubric
49:52
this is when we're looking at visualizing the
49:55
coffee machine sales. And if you ask it
49:57
to grade a chart according to a rubric
49:59
such as whether it's clear access labels and so on
50:02
there is the same rubric for every chart
50:05
And that would be using an L message judge
50:06
but without a per example ground truth
50:09
So find this two by two grid as
50:10
maybe a useful way
50:11
to think about the different types of evals
50:13
you might construct for your application
50:16
And by the way
50:17
these are sometimes also called end-to-end evals because
50:20
one end is the impotence
50:21
which is the user query problem
50:23
and the other end is the final output
50:25
And so all of these are evals for
50:28
the entire end-to-end systems performance
50:31
So just to wrap up this video
50:33
I'd like to share a few final tips
50:35
for designing end-to-end evals. First
50:37
quick and dirty evals is fine to get
50:39
started
50:40
I feel like I see
50:42
quite a lot of Teams that are almost
50:44
paralyzed because they think building evals is this massive multi-week effort
50:48
and so
50:49
they take longer than would be ideal to
50:51
get started
50:53
But
50:53
I think just as you
50:54
iterate on an agentic workflow and make it better over time
50:57
you should plan to
50:58
iterate on your evals as well
51:00
So if you put in place 10
51:02
15, 20 examples as your first cut at evals
51:06
and write some code or try prompting them as a judge
51:08
just do something to start to get some metrics
51:11
then that can complement the human eye at
51:13
looking at the output
51:14
and then it's a blend of the two
51:16
that can drive your decision making
51:17
And as the evals become more sophisticated over time
51:20
you can then
51:21
shift more and more of your trust to
51:22
the
51:23
metric-based evals rather than needing to read over
51:26
your hundreds of
51:27
outputs every time you tweak a prompt somewhere
51:29
And as you go through this process
51:31
you'll likely find ways
51:33
to keep on improving your evals as well
51:35
So if you had 20 examples to start
51:38
you may then run into places where
51:40
your evals fail to capture Your judgment
51:44
about what system is better
51:46
So maybe
51:47
you update the system and you look at
51:49
it and you feel like this got to
51:51
be worked much better
51:51
but your eval
51:53
fails to show the new system achieving a
51:55
higher score. If that's the case
51:57
that's often an opportunity to go maybe
52:00
collect the larger eval set or change the
52:02
way you evaluate the output to make it
52:04
correspond better
52:05
to your judgment as to
52:06
what system is actually working better
52:08
And so your evals will get better over time
52:11
And lastly
52:11
in terms of using evals to gain inspiration
52:14
as to what to work on next
52:16
a lot of agentic workflows are being used
52:18
to automate tasks that say humans can do
52:21
And so I find for such applications
52:23
I'll look for places where the performance is
52:25
worse than that of an expert human
52:27
and that often gives me inspiration for where
52:29
to focus my efforts or what are the
52:31
types of examples that I maybe get my
52:33
agentic workflow
52:35
to work better than it is currently
52:37
So
52:38
I hope that after you build that quick and dirty system
52:40
you think about when it would make sense
52:42
to
52:43
start putting in
52:43
some evals to track the potentially problematic aspects of the system
52:48
and that that will then help you
52:49
drive improvements to the system
52:51
In addition to helping you drive improvements
52:54
it turns out that
52:55
there's a method of evals that helps you
52:58
hone in of your entire agentic system
53:01
what are the components most worth focusing
53:04
your attention on?
53:05
Because agentic systems often have many pieces
53:08
So
53:09
which piece is going to be most productive
53:11
for you to spend time working to improve?
53:14
Turns out being able to do this well
53:16
is a really important skill for driving efficient
53:19
development of agentic workflows
53:21
In the next video
53:22
I'd like to deep dive into this topic
53:24
So let's go on to the next video
53:27
Let's say you built an agentic workflow
53:29
and it is not yet working as well
53:31
as you wish. And this happens to be all the time
53:33
by the way. I'll often build a quick and dirty system
53:36
and it doesn't do as well as
53:37
I wish it would
53:38
The question is
53:40
where do you focus your efforts to make
53:41
it better?
53:43
It turns out agentic workflows have many different components
53:46
and working on some of the components could
53:48
be much more fruitful
53:49
than working on some other components
53:51
So your skill at choosing where to focus
53:54
your efforts
53:54
makes a huge difference in the speed with
53:57
which you can make improvements to your system
53:59
And I found that one of the
54:01
biggest predictors for how efficient and how good a team is
54:05
is whether or not
54:06
they're able to drive a disciplined
54:08
error analysis process to tell you
54:11
where to focus your efforts
54:12
So this is an important skill
54:14
Let's take a look
54:15
at how the error analysis
54:17
In the research agent example
54:19
we had carried out an error analysis in the previous video
54:23
and we saw that it was
54:25
often
54:25
missing key Points that a human expert would
54:27
have made in writing essays on certain topics
54:30
So
54:31
now that you've spotted this
54:32
problem that is sometimes missing key points
54:35
how do you know what to work on?
54:37
It turns out that of the many different
54:39
steps in this workflow
54:40
almost any of them could have contributed to
54:43
this problem of missing key points
54:46
For example
54:46
maybe the first LLM was generating search terms that weren't great
54:50
so
54:51
it was just searching for the wrong things
54:53
and did not discover the right articles
54:55
Or maybe use a web search engine that
54:57
just wasn't very good. There are multiple web
54:59
search engines out there. In fact
55:00
there are actually quite a few that I
55:01
tend to use for my LLM-based applications
55:04
and some are better than others
55:06
Or maybe web search was just fine
55:08
but when we gave the list of web
55:10
search results to an LLM
55:11
maybe it didn't do a good job choosing
55:13
the best handful to download
55:16
Maybe web fetch has fewer problems in this case
55:18
assuming you can fetch web pages accurately
55:21
But after dumping the web pages in the LLM
55:23
maybe the LLM is
55:24
ignoring some of the points
55:26
In the documents we had fetched
55:28
So it turns out that there are teams
55:30
that will sometimes look at this and go
55:32
by gut to pick one of these components to work on
55:36
and sometimes that works
55:37
and sometimes that leads to many months of
55:39
work with very little progress in the
55:41
overall performance of the system
55:44
So rather than
55:45
going by gut to decide which of these
55:47
many components to work on
55:49
I think it's much better to
55:51
carry out an error analysis
55:53
to better understand each step in the workflow
55:56
And in particular, I'll often examine the traces
55:58
and that means the intermediate output after each step
56:01
in order to understand which component's performance is subpar
56:06
meaning, say, much worse than what a human expert would do
56:10
because that
56:11
points to where
56:12
there may be room for significant improvement
56:15
Let's look at an example
56:16
If we ask the research agent to write
56:18
an essay about recent news in black hole science
56:21
maybe the outputs search terms like these
56:24
search for black hole theories
56:25
Einstein, event horizon. telescope radio
56:27
and so on
56:28
And I would then have a human expert
56:29
look at these and
56:30
see are these reasonable
56:32
web search terms for
56:34
writing about recent discoveries in black hole science
56:37
And maybe in this case
56:38
an expert says these web searches look okay
56:41
They're, you know
56:42
pretty similar to what
56:43
I would do as a human
56:44
Then I look at the outputs of
56:47
the
56:47
web search and look at the URLs returned
56:51
So web search will return many different web pages
56:54
and maybe
56:55
one web page returns is that an elementary
56:57
school student
56:58
claims to track a 30-year-old black hole mystery
57:00
from Astro Kid News
57:02
And
57:02
this doesn't look like the most rigorous peer
57:05
review article. And maybe examining
57:08
all of the articles that
57:09
web search
57:10
returns causes you to conclude that it's returning
57:13
too many
57:14
blog or popular press types of articles and
57:17
not enough scientific articles to write a
57:19
research report of the quality that you are
57:22
looking for. It'd be good to
57:23
just
57:23
look through the outputs of the other steps
57:25
as well. Maybe the finds the best five sources you can
57:28
and then with AstroKit News
57:30
SpaceBot 2000, SpaceFun News, and so on
57:33
And it is by looking at these
57:35
intermediate outputs
57:36
that you can then try to get a
57:37
sense of the quality of the output of
57:39
each of these steps. To introduce some terminology
57:42
the
57:42
overall
57:43
set of outputs of all of the intermediate
57:45
steps is often called the trace of a
57:48
run of this agent
57:49
And then some terminology you see
57:51
in
57:52
other
57:52
sources as well is the output of a
57:54
single step is sometimes called a span
57:57
This is terminology from the computer observability literature
58:01
where
58:02
people try to figure out what computers are
58:04
doing. And in this course
58:05
I use the word trace quite a bit
58:07
I'll use the word span a little bit less
58:09
but you may see both of these terms
58:11
on the internet
58:12
So by reading the traces
58:14
you start to get an informal sense of
58:16
where might be the most problematic components
58:19
In order to do this in a more systematic way
58:21
it turns out to be useful to focus
58:23
your attention
58:24
on the cases that the system is doing
58:26
poorly on. Maybe it writes some essays just fine
58:29
and the output is completely satisfactory
58:31
So I would put those aside and try
58:32
to come up with a set of examples where
58:34
for whatever reason
58:35
the final output of your research agent is not quite satisfactory
58:39
and just focus on those examples
58:42
So this is one of the reasons we call error analysis
58:44
because we want to focus on the cases
58:46
where the system made an error
58:48
and we want to go
58:49
through to figure out which components
58:51
were most responsible for the error
58:54
in the research agent output
58:56
In order to make this more rigorous
58:58
rather than reading and getting an informal sense
59:01
you might actually build up a spreadsheet to
59:04
more explicitly count up where the errors are
59:07
And by error
59:07
I mean when a step outputs something
59:10
that
59:11
performs significantly worse than maybe what a human
59:14
expert would have given a similar
59:16
input as that component. So
59:18
I'll often do this myself in a spreadsheet
59:20
So I might build a spreadsheet like this
59:22
And so for the first query
59:23
I'll look at recent developments in black hole
59:25
science. And I see that the search results
59:27
says too many blog posts
59:29
popular press articles, not enough scientific papers
59:32
And then based on this
59:34
it is true that the five best sources aren't great
59:37
but here
59:38
I won't say that the five best sources
59:40
did a bad job
59:40
because if the inputs
59:42
to
59:43
the arm for selecting the five best sources
59:45
were all non-rigorous articles
59:47
then I can't blame this
59:48
picking the five best sources for
59:50
not picking better articles because it did the
59:52
best it could have or as
59:53
what did
59:54
nearly as well as any human might have
59:55
given the same selection to choose
59:57
And then you might go through this for
59:59
different prompts
60:00
Renting versus buying in Seattle
60:03
maybe it missed a well-known blog
60:05
robotics for harvesting fruit
60:06
Maybe in this case
60:07
we look at it and say
60:08
oh, the search terms are too generic
60:10
and the search results also weren't good
60:12
and so on. And then based on this
60:14
I would
60:14
count up in my spreadsheet
60:16
how often I observe errors
60:18
in the different components. So in this example
60:21
dissatisfied with the search terms 5% of the time
60:24
but I'm dissatisfied with the search results 45%
60:27
of the time. And if I actually see this
60:29
I might just take a careful look at
60:30
the search terms to make sure that the
60:32
search terms really were okay
60:33
and that poor choice of search terms were
60:35
not what led to poor search results
60:37
But I really think the search terms are fine
60:39
but the search results are not
60:40
then I would take a careful look at
60:42
the web search engine I'm using
60:43
and if there are any parameters I can
60:44
tune to make it bring back more
60:47
relevant or higher quality results
60:49
There's this type of analysis that tells me
60:50
in this example
60:52
that maybe I really should focus
60:53
my attention on fixing the search results and
60:56
not on the other components of this
60:58
agentic workflow
61:00
So to wrap up this video
61:02
I find that it's useful to
61:03
develop a habit of looking at traces
61:06
After you build an agentic workflow
61:07
go ahead and look at the intermediate outputs
61:10
to get a feel for what it's actually
61:11
doing on every step
61:12
so that you can better understand if
61:14
different steps are performing better or worse
61:17
And a more systematic error analysis may be
61:19
done
61:20
with a spreadsheet can let you gather statistics
61:23
or count up which component performs poorly most
61:25
frequently. And so by looking at what components are doing poorly
61:29
as well as
61:30
where I have ideas for efficiently improving different components
61:34
then that will let you prioritize what component
61:37
to work on. So maybe a component is problematic
61:40
but I don't have any ideas for improving
61:42
it. So that would suggest
61:43
maybe not prioritizing that as high
61:45
But if there is a component
61:47
that is generating a lot of errors
61:49
and if I have ideas how to improve that
61:51
then that would be a good reason
61:52
to prioritize working on that component
61:54
And
61:55
I just want to emphasize that
61:57
Error analysis
61:58
is a very helpful output
62:00
for you to decide
62:01
where to focus your efforts because in any complex system
62:05
there are just so many things you could
62:06
work on
62:07
It's too easy
62:08
to pick something to work on and work
62:10
on it for weeks or even months
62:11
only to discover later that that did not
62:14
result in improved performance to your overall system
62:17
And so
62:17
using error analysis to decide where to focus
62:19
your effort turns out to be incredibly useful
62:22
for improving your efficiency. In this video
62:25
we went over error analysis with the research agent example
62:29
but
62:30
I think error analysis is such an important
62:32
topic. I want to go over some additional
62:34
examples with you
62:35
So let's go on to the next video
62:37
where we'll look at more examples of
62:39
error analysis
62:41
An agentic workflow may comprise many different types of components
62:45
and so your tools for improving different components
62:48
will be pretty different
62:49
But I'd like to share with you some
62:51
general patterns I've seen
62:52
Some components, your agentic workflow will be non-OLM based
62:56
so it may be
62:57
something like
62:58
a web search engine or a text retrieval
63:00
component if that's part of your rack or
63:03
retrieval augmented generating system
63:05
something for code execution
63:06
or maybe a separately trained machine learning model
63:09
maybe about speech recognition or detecting people and
63:11
pictures and so on. So
63:13
sometimes
63:14
these non-OLM based components will have parameters or
63:17
hyperparameters that you can tune
63:19
So for web search
63:19
you can tune things like the number of
63:21
results or maybe the date range that you
63:23
also web search engine to consider
63:25
For a rack text retrieval components
63:28
you might change the similarity threshold that determines
63:30
what pieces of text it considers similar or
63:33
the chunk size. Often rack systems will take texting
63:36
chop it up into smaller chunks for matching
63:38
So there may hyperparameters you could use
63:41
Or for people detection
63:42
you might change the detection threshold
63:44
So how sensitive it is and how likely
63:46
it is to declare this found a person
63:48
and just to trade off the false positives
63:50
and false negatives. If they follow all the
63:51
details of the hyperparameters I just discussed
63:53
don't worry about it. The details aren't that important
63:56
but often the components were parameters that
63:58
you can tune. And then of course
63:59
you can also try to replace the component
64:02
I do this a lot in my agentic
64:04
web flows where
64:05
I'll swap in different rack search engines or
64:07
swap in different rack providers and so on
64:09
just to see if
64:10
some of the provider might work better
64:13
Because of the diversity of non-LM based components
64:16
I think the techniques for how to improve
64:18
it will be
64:18
more diverse and dependent of exactly what that
64:21
component is doing. For an LM based component
64:25
here are some options you might consider
64:27
One would be
64:28
to try to improve your prompts
64:30
So
64:30
maybe try to add more explicit instructions
64:33
or if you know what few shot prompting is
64:35
that refers to
64:36
adding
64:37
one or more concrete examples
64:39
of an example of an input
64:41
The design output, and so few-shot prompting
64:43
which you can learn about from some deep
64:45
learning short courses as well
64:47
is a technique that can
64:48
give your LM some examples to hopefully help
64:51
it get better performing outputs written
64:54
Or you can also try a different LM
64:57
So with ASV or other tools
64:59
it could be pretty easy to try multiple LMs
65:02
and then you can use evals
65:03
to pick the best model for your application
65:06
Sometimes
65:07
if a single step is too complex for
65:09
one LM to do
65:10
you can consider if you want to decompose
65:11
the task into smaller steps
65:13
or maybe decompose it into a generation step
65:16
and then a reflection step
65:17
But more generally, if you have instructions that are very complex
65:21
all within one step
65:22
maybe a single LM has a hard time
65:24
following all those instructions
65:25
and if you break the task down to smaller steps
65:27
that may be easier for
65:29
say
65:29
two or three calls to an LM
65:31
to carry out accurately. And lastly
65:34
something to try when the other methods aren't
65:36
working well enough is to consider fine-tuning
65:39
a model
65:40
This tends to be
65:41
quite a bit more complex than the other options
65:43
so it can be quite a bit more
65:44
expensive as well in terms of developer time
65:46
to implement. But if you have some data
65:49
that you can use to fine-tune an LLM on
65:53
that could give you much better performance than
65:55
prompting alone. So
65:57
I tend not to fine-tune a model until
65:59
I've really exhausted the other options
66:01
because fine-tuning tends to be quite complex
66:04
But for applications where after trying everything else
66:07
if I'm still at
66:08
you know, say 90% performance or 95% performance
66:11
and I really need to eke out those
66:13
last few percentage points of improvement
66:15
then sometimes fine-tuning
66:16
my own custom model is a great technique
66:18
to use
66:19
I tend to do this only on the
66:21
more mature
66:22
applications because of how costly it is
66:25
It turns out that when you are
66:26
trying to choose an LLM to use
66:28
one thing that's very hopeful
66:30
for you as a developer is if you
66:32
have good intuitions about how intelligent or how
66:35
capable different large language models are
66:37
One thing you can do is just try
66:38
a lot of models and see what works
66:40
best. But I find that As I work with different models
66:42
I start to hone intuitions about
66:44
which models work best for what types of
66:46
tasks. And when you hone those intuitions
66:48
you can be
66:48
more efficient as well in
66:50
writing good prompts for the model as well
66:52
as choosing good models for your tasks
66:55
So I'd like to share with you
66:56
some
66:57
thoughts on how to hone your intuition
66:59
on
67:00
what models will work well for your application
67:03
Let's illustrate this with an example
67:05
of using an LLM
67:06
to follow instructions to remove or to redact
67:09
PII or personally identifiable information
67:12
So get down to remove private sensitive information
67:15
For example
67:16
if you are using an LLM to summarize customer calls
67:20
then maybe one summary is on July 14th
67:23
2023, Jessica Alvarez with a social security number
67:26
a certain address, a support ticket
67:28
and so on
67:29
So this piece of text has a lot of sensitive
67:31
personally identifiable information. Now
67:34
let's say we want to remove all PII
67:37
from
67:37
such
67:38
summaries because we want to use the data
67:40
for Downstream Cisco analysis of what customers are
67:43
calling about and to protect customer information
67:45
we want to strip out that PII before
67:47
we do that downstream Cisco analysis
67:50
So you might prompt an alarm with
67:52
instructions to
67:53
identify all cases of PII in the text
67:56
below
67:56
and then return the redacted text with redacted
67:59
colon and so on
68:01
It turns out
68:02
that
68:03
the larger frontier models tend to be much
68:06
better at following instructions
68:08
whereas the smaller models
68:10
tend to be pretty good at answering simple factual questions
68:13
but are just not as good at following
68:15
instructions. If you run this prompt on the smaller model
68:18
the OpenWay Llama 3.1 model with abelian parameters
68:21
then it may generate an output like this
68:23
It says, the identified PII is social security number and address
68:27
and then it redacts it as follows
68:29
and so on. And that actually makes a
68:31
few errors. It didn't follow the instructions properly
68:34
It showed the list
68:35
then redacted the text
68:36
then returned another list
68:38
which it wasn't supposed to
68:40
and in this list of PII
68:42
it missed the name
68:43
and then I think it also
68:44
didn't redact part of the address
68:46
So details aren't important
68:47
but didn't follow these instructions perfectly
68:49
and maybe it missed a little bit of PII
68:52
In contrast, if you use a more intelligent model
68:54
one that's better at following instructions
68:56
you may get a better result like this
68:58
where it's actually correctly listed all the PII
69:01
Correctly redacted all of the PII
69:04
And so
69:05
I find that as different OM providers specialize on different tasks
69:09
different models really are better for different tasks
69:12
Some are better at coding
69:13
some are better for the instructions
69:15
some are better at certain niche types of facts
69:18
and if you can hone your intuition for
69:20
what models
69:21
are more or less intelligent
69:23
and what type of instructions they're more or
69:25
less able to follow
69:26
then you'll be able to make better decisions
69:27
as to what models to use
69:29
So to share a couple tips on how to do this
69:32
I encourage you to play with different models
69:34
often. So whenever there's a new model release
69:37
I'll often go try it out and try
69:39
out different queries on it
69:40
both closed way proprietary models as well as
69:43
open way models. And
69:45
I find that sometimes having a personal set
69:46
of evals might also be helpful where there's
69:48
a set of things you ask a lot
69:50
of different models that might help you calibrate
69:52
how well they do
69:53
on different types of tasks
69:56
One of the things that I do a
69:57
lot
69:57
that I hope will be useful to you
69:59
is
70:00
I spend a lot of time reading other
70:01
people's prompts. So sometimes people publish their prompts on the internet
70:06
and I'll often go and read them to
70:08
understand what best practices in prompting looks like
70:11
or I'll often chat to my friends at various companies
70:14
including some of the frontier model companies
70:16
and kind of share my prompts with them
70:17
take a look at how they prompt
70:19
And sometimes I'll also go to open source packages of
70:23
you know
70:23
written by people I really respect
70:24
and download the open source package and dig
70:27
through that open source package
70:29
to find the prompts the authors have written
70:31
in order to read it
70:32
in order the whole mind intuition about how
70:34
to
70:34
write good prompts
70:36
This is one technique that I encourage you
70:38
to consider
70:39
is by reading lots of other people's prompts
70:41
that will help you get better at writing prompts yourself
70:44
And, I certainly do this a lot
70:46
I encourage you to do so too
70:48
And this will hone your intuition about
70:50
what types of instructions models are good at
70:52
following and
70:54
when to say certain things to different models
70:56
In addition to playing with models and reading other people's prompts
70:59
if you try out lots of different models
71:01
in your agentic workflows
71:03
that also lets you hone your intuition
71:05
So you see
71:06
which models work best for which types of
71:08
tasks and either looking at traces to
71:11
get it in a formal sense or looking
71:13
at either component-wise or end-to-end evals can help
71:16
you assess how well different models are working
71:19
Different policy workflow. And then you start to
71:21
hone intuitions about not just performance
71:23
but maybe also
71:25
price and speed trade-offs
71:27
for the use of different models
71:28
And
71:29
one of the reasons I tend to develop
71:30
my agentic workflows of AI Suite is because
71:33
it then makes it easy to quickly swap
71:35
out and try out different models
71:36
And this makes me more efficient in terms
71:38
of trying out and assessing
71:41
which models work best for my workflow
71:43
So we've talked a lot about how to
71:45
improve the performance of different components
71:49
To hopefully improve the overall performance of your
71:52
end-to-end system
71:53
In addition to improving the quality of the output
71:57
one other thing you might want to do
71:58
in your workflows
71:59
is to optimize the latency
72:01
as well as cost
72:03
I find that for a lot of teams
72:04
when you start developing
72:05
usually the number one thing to worry about
72:07
is just are the outputs sufficiently high quality?
72:11
But then
72:11
when the
72:12
system is working well and you put it into production
72:15
then there's often value to make it run
72:17
faster
72:18
as well as run at lower cost as
72:19
well
72:20
So in the next video
72:21
let's take a look
72:22
at some ideas for improving
72:24
cost and latency for agentic workflows
72:28
When building agentic workflows
72:30
I often advise teams to focus on getting
72:33
high quality outputs
72:34
and to optimize cost and latency only later
72:37
It's not that cost and latency don't matter
72:40
but I think
72:41
getting the performance or the
72:43
output quality to be high is usually
72:45
the hardest part
72:47
And then only when it's really working
72:49
then maybe focus on the other things
72:51
One thing that's happened to me a few
72:52
times was
72:53
my team built an agentic workflow and we
72:55
shipped it to users
72:57
and then we're fortunate enough to have so
72:59
many users use it
73:00
that the cost actually became a problem
73:02
And then we had to
73:03
you know
73:03
scramble to bring the cost back down
73:06
But that's a good problem to have
73:07
So I tend to
73:08
worry about cost usually last
73:11
not that I ignore it completely
73:13
but it's just lower down
73:14
my list of things to worry about until
73:15
we have so many users that we really
73:17
need to bring the cost down per user
73:20
And then latency, I tend to worry a bit about it
73:23
but again
73:24
not as much as just
73:25
making sure the output quality is high
73:28
When we do get there
73:29
it will be useful to have tools to
73:31
optimize latency and cost
73:32
Let's take a look at some ideas
73:34
on how to do that
73:35
If you want to optimize the latency of energetic workflow
73:40
one thing I will often do
73:41
is then Benchmark or time the workflow
73:45
So in this research agent
73:47
it takes a number of steps
73:49
and if I were to time each of the steps
73:51
maybe
73:52
takes seven seconds to generate the search terms
73:55
Web search takes five seconds
73:56
this takes three seconds
73:57
this takes 11 seconds
73:58
and then writing the final essay takes 18
74:00
seconds on average
74:02
And it is then by looking at this
74:03
overall timeline that I can get a sense
74:06
of
74:07
which components have the most room for
74:10
making faster
74:12
In this example
74:13
there are maybe multiple things you could try
74:15
If you haven't already taken advantage of parallelism for some steps
74:18
like maybe wet fetch
74:20
maybe it's worth considering
74:22
doing some of these operations in parallel
74:25
Or if you find that some of the
74:26
LM sets are taking too long
74:28
so if this first step takes seven seconds
74:30
this last LM step takes 18 seconds
74:33
I might also consider trying a smaller
74:35
maybe
74:36
slightly less intelligent model to see if it's
74:38
still worthwhile enough for that
74:39
or if I can find a faster LM
74:41
provider. There are lots of APIs online for different ARM interfaces
74:45
and some companies have specialized hardware to allow
74:48
them to serve certain ARMs much faster
74:51
So sometimes it's worth trying different ARM providers
74:53
to see which ones can return tokens
74:56
the fastest
74:57
But at least doing this type of timing
74:59
analysis can give you a sense of which
75:01
components to focus on in terms of
75:03
reducing latency
75:05
In terms of optimizing costs
75:07
a similar calculation where you calculate the cost
75:10
of each step would also let you benchmark
75:12
and decide which steps to focus on
75:14
Many ARM providers charge per
75:16
token based on the input and output length
75:18
Many API providers charge per API call
75:22
and the computational steps
75:23
may have different costs based on how you
75:25
pay for server capacity and how much the
75:27
service costs. And so for a process like this
75:30
you might decide in this example that the
75:33
tokens for this ARM step
75:35
on average cost 0.004 cents
75:38
Each web search API maybe cost 1.6 cents
75:41
Tokens cost This much API call cost this much
75:44
PDF to text cost this much
75:45
tokens for the final essay generation cost this much
75:48
and this would maybe
75:48
again give you a sense of are there
75:50
cheaper components you could use or cheaper arms
75:52
you could use to see where the biggest
75:54
opportunity is for optimizing costs
75:57
And I found that these benchmarking exercises can
76:00
be very clarifying and sometimes they'll clearly tell
76:02
me that certain components are just not worth
76:05
worrying about because they're not that material
76:07
a contributor to either cost or to latency
76:10
So
76:11
I find that wheneither cost or latency becomes
76:13
an issue
76:14
by simply measuring the cost
76:16
and or latency of each step
76:18
that often gives you a basis
76:20
with which to decide which components to focus
76:23
on optimizing
76:25
So we're nearly at the end of this
76:27
module. I know we've covered a lot
76:29
but thank you for sticking with me
76:30
Let's go on to the final video of
76:32
this module to wrap up
76:34
We've gone through a lot of tips for driving a disciplined
76:38
efficient process for building agentic AI systems
76:41
I'd love to wrap up by
76:43
sharing with you
76:43
what it feels like to be going through
76:45
this process
76:47
When I'm building these workflows
76:48
I feel I did two major activities I'm
76:51
often spending time on. One is building
76:53
so writing software
76:54
trying to write code to improve my system
76:56
And the second, which sometimes doesn't feel like progress
77:00
but I think is equally important
77:02
is analysis to help decide where to focus
77:05
my build efforts next
77:06
And I often go back and forth between building and analyzing
77:10
including things like error analysis
77:12
So for example, when building a new agentic workflow
77:15
I'll often start by quickly building an end-to-end system
77:19
maybe even a quick and dirty implementation
77:21
And this lets me then start to
77:23
examine the final outputs of the end-to-end system
77:25
or also
77:26
read through traces to get a sense of
77:28
where it's doing well
77:29
where it's doing poorly. Based on even just looking at traces
77:32
sometimes this would give me a Sense
77:35
of which individual components I might want to
77:37
improve. And so I might go tune some
77:39
individual components or keep tuning
77:42
the overall end-to-end system
77:44
As my system starts to mature a little bit more
77:46
then beyond just
77:48
manually examining a few outputs and reading through traces
77:51
I might start to build evals and have
77:53
a small data set
77:54
maybe just 10, 20 examples to compute metrics
77:57
at least on end-to-end performance
77:59
And this then further
78:01
helps me have a more refined perspective
78:03
on how to improve the end-to-end system or
78:05
how to improve individual components
78:07
As it matures even further
78:09
my analysis then becomes maybe even more disciplined
78:12
where I'll start to do error analysis
78:14
and look through the components and try to
78:16
count up how frequently individual components
78:19
led to
78:20
subpar outputs
78:21
And this more rigorous analysis then lets me
78:23
be
78:24
even more focused in deciding what components to
78:26
work on next or inspire ideas for improving
78:29
the overall end-to-end system. And then eventually
78:31
when it's even more mature
78:32
to drive more efficient improvements at the component level
78:36
that's when I might also build component level
78:38
evals. And so the workflow of building an
78:41
agentic system often goes back and forth
78:43
It's not a linear process
78:45
We sometimes tune the entire system
78:47
then do some error analysis
78:48
then improve a component for a bit
78:50
then tune the component level evals
78:52
and
78:52
I tend to bounce back and forth between
78:54
these two types of techniques
78:56
And what I see
78:57
less experienced teams often do is
78:59
spend a lot of time building
79:01
and probably much less time analyzing with error analysis
79:05
building evals, and so on
79:06
that would be
79:07
ideal because it's this analysis that helps you
79:09
really focus where to spend your time building
79:13
And just one more tip
79:14
there are actually
79:15
quite a few tools out there to help with monitoring traces
79:19
logging runtime, computing costs, and so on
79:21
And those tools can be helpful
79:23
I sometimes use a few of them
79:24
and quite a few of deep-end guys
79:26
short course partners offer those tools
79:28
and they do work well
79:30
I find that for agentic workflows I end up working on
79:33
most agentic Workflows are pretty custom
79:35
and so
79:36
I end up building pretty
79:38
custom evals myself
79:40
because I want to capture the things that
79:42
work incorrectly with my system
79:44
So even though I do use some of those tools
79:46
I also end up building a lot of
79:47
custom evals
79:49
that
79:49
are well fit to my specific application and
79:52
the issues I see with it
79:54
So
79:54
thanks for
79:55
sticking with me
79:55
this far to the end of the
79:57
fourth of five modules
80:00
If you're able to implement
80:02
even a fraction of the ideas from this module
80:06
I think you'll be well ahead of
80:07
the vast majority of developers in terms of
80:10
your sophistication at implementing agentic workflows
80:14
Hope you found these materials useful
80:16
and I look forward to seeing you in
80:18
the final module where we'll talk about some
80:20
more advanced design patterns for building
80:23
highly autonomous agents
80:25
I'll see you in the last module of
80:27
this course
80:29
Welcome to this final module where you learn
80:31
about design patterns that let you build
80:34
highly autonomous agents
80:36
where you don't need to hard code in
80:37
advance the sequence of steps to take
80:40
but it can be more flexible and decide
80:42
for itself what steps it wants to take
80:44
to accomplish a task
80:46
We'll talk about the planning design pattern and
80:48
then later in this module
80:49
how to build multi-agent systems
80:51
Let's dive in
80:52
Suppose you run a sunglasses retail store and
80:56
have information on
80:58
what sunglasses are in your inventory
81:00
stored in a database
81:01
You might want a
81:02
customer service agent to be able to answer questions like
81:05
do you have any round sunglasses in stock
81:07
that are under $100? This is a fairly
81:09
complex query because you have to look through
81:12
the product descriptions to see what sunglasses are round
81:15
then look at what is in stock
81:17
and then finally see what's under $100
81:20
in order to tell the customer
81:22
yes, we have classic sunglasses
81:24
So
81:25
how do you build an agent
81:26
to answer a broad range
81:28
Customer queries like this and many others
81:31
In order to do so
81:32
we're going to give the LLM
81:34
a set of tools to let it get item descriptions
81:37
such as look up if different glasses are round
81:40
check inventory, maybe process item returns
81:43
which is not needed for this query
81:44
but may need it for other queries
81:46
get item price, check past transactions, process item sale, and so on
81:52
In order to
81:53
let an LLM
81:54
figure out what's the right sequence of tools
81:57
to use to respond to the customer request
82:00
you might then write a prompt like this
82:02
You have access to the following tools
82:04
and they give it a description of each of the
82:06
say, six tools or even more tools that the LLM has
82:10
and to then tell it to
82:11
return a step-by-step plan to carry out the
82:14
user's request
82:15
In this case, to answer this particular query
82:18
a reasonable plan that an LLM might output
82:20
might be
82:21
to first use get item descriptions
82:24
to check
82:25
the different descriptions to find the round sunglasses
82:28
Then use check inventory to see if they're in stock
82:30
and use get item price to see if
82:32
the in stock results are less than $100
82:35
After an LM outputs this plan with three steps
82:39
what we can do is then take the step one text
82:43
that is this text written here in red
82:45
and pass that to an LM
82:47
maybe of additional context about what are the tools
82:49
with their user query
82:51
with additional background context
82:53
and toss an LM to carry out
82:55
step one. And in this case
82:57
hopefully the LM will
82:58
choose to call the get item descriptions
83:00
to get the appropriate descriptions of items
83:03
And the output of that first step
83:05
can let it select which are the round sunglasses
83:08
and that output of step one
83:11
is then passed together with the step two instructions
83:14
that would be these instructions I have here in blue
83:16
to an LM to then execute the second
83:18
step of the plan. Hopefully
83:20
it will then take the
83:21
two pairs of round sunglasses we found on
83:23
the previous slide and check the inventory
83:26
and the output of that second step
83:28
Is then used to another OM call
83:31
where you have the output of the second step
83:33
as well as
83:33
the instructions of what to do for step three
83:35
pass the OM
83:36
to have it get the item price
83:38
And finally
83:39
this output is fed back to the OM
83:41
one last time to generate the final answer
83:44
for the user. In this slide
83:46
I've simplified a lot of details a little
83:48
bit
83:48
The actual plan typically written by the OM
83:50
is more detailed than these simple one-line instructions
83:53
but the basic workflow is to have an
83:55
OM write out multiple steps of a plan
83:57
and then task it
83:58
to execute each step of the plan
84:01
in turn
84:01
with some appropriate surrounding context about what is the task
84:05
what are the tools available
84:06
and so on. And the exciting thing
84:09
about using an OM to plan this way
84:11
is that
84:12
we did not have to decide in advance
84:14
what is the sequence in which
84:16
to call tools in order to answer a
84:18
fairly complex
84:20
customer request. If a customer were to make a different request
84:24
such as
84:24
I would like to return the
84:26
gold frame glasses that I purchase
84:29
but not the metal frame ones
84:31
Then you can imagine an OM
84:33
similarly being able to come up with a
84:35
different plan
84:36
to figure out based on what they have purchased previously
84:39
which glasses they had bought based on get item descriptions
84:41
whether gold frame ones they want to return
84:43
and then maybe call process item return
84:46
So with an agent that can plan like this
84:48
it can carry out a much wider
84:50
range of tasks
84:51
that can require calling many different tools and
84:54
many different orders. One more example of planning
84:56
let's take a look at an email assistant
84:59
If you want to be able to tell
85:00
your assistant to please reply to that email
85:02
invitation from Bob wrote in New York
85:03
tell him attend and archive his email
85:05
Then an email assistance may be given tools like this
85:08
to search email, move an email, delete an email
85:11
and send an email
85:12
And you might write a system prompt saying
85:14
you have access to following tools
85:15
and again, please return the step-by-step plan
85:17
In this case
85:18
maybe the OM will
85:19
say
85:19
the steps for this are to use search
85:21
email to find the email from Bob that
85:24
mentioned dinner and New York
85:25
and then
85:26
generate and send an email to confirm Tendence
85:28
and velocity move that email to the archive
85:31
folder. Given this plan
85:32
which looks a reasonable one
85:33
you would then again
85:35
task an OM step by step
85:37
to carry out this plan
85:38
So
85:39
the text from the first step shown here
85:41
in red
85:41
would be fed to the OM
85:43
with additional background context and hopefully it triggers
85:45
search email. Then the output of that
85:48
can be given to
85:49
an OM again with the step two instructions
85:52
to send a appropriate response
85:54
And then finally, assuming the email was sent successfully
85:57
you can take that output
85:58
and have the OM execute the third step
86:00
of moving the email
86:02
from Bob into the archive folder
86:04
The planning design pattern is already used successfully
86:08
in many
86:08
highly agentic coding systems where if you ask
86:11
it to
86:12
write a piece of software to build some fairly complex application
86:16
it might actually come to a plan to build this component
86:19
build this component
86:20
build this component to almost form a checklist
86:22
and then do those steps one at a
86:24
time to build decently complex piece of software
86:28
For many other applications
86:29
the use of planning is still maybe more experimental
86:33
is not in very widespread use
86:35
And one of the challenges of planning is
86:38
it makes the system sometimes
86:39
a little bit harder to control because you as a developer
86:42
you don't really know
86:43
at runtime what plan it will come up
86:45
with. And so
86:46
I think on
86:47
outside highly agentic coding systems where it actually works really well
86:51
adoption of planning is still growing in other sectors
86:55
but it's an exciting technology
86:56
and I think it will keep getting better
86:58
and we'll see it in
86:59
more and more applications
87:01
The cool thing about
87:02
building agents that can plan for themselves is
87:04
you don't need to hard code in advance
87:06
the exact sequence of steps an LLM
87:08
may take to carry out a complex task
87:11
Now
87:12
I know that in this video I've gone
87:13
over the planning process at a fairly high
87:15
level with it
87:16
outputting a list of steps and then tasking
87:18
an LLM
87:19
to
87:20
carry out the steps of the plan
87:21
one step at a time
87:23
But how does this actually work?
87:25
In the next video
87:26
we'll take a deeper dive Look
87:28
further into the guts of
87:30
what these plans actually look like and how
87:32
they're stringed together
87:34
to have an LM plan and execute a
87:35
plan
87:36
for you
87:36
Let's take a look at that in the
87:38
next video
87:39
In this video
87:40
we'll look in detail at how to prompt
87:42
an LM to generate a plan and then how to read
87:45
interpret, and execute that plan
87:47
Let's dive in
87:48
This is a plan that you saw in
87:50
the previous video for the customer service agents
87:53
and I have presented this plan at a
87:55
high level using simple text descriptions
87:57
Let's take a look at how you can
87:59
get an arm to write
88:01
very clear plans that go a little bit
88:03
beyond these simple high-level text descriptions
88:06
It turns out that
88:08
many developers
88:09
will ask an arm to format the plan
88:12
it wants executed in JSON format
88:14
because this allows downstream code
88:16
to parse what exactly are the steps of
88:18
a plan in a relatively clear and unambiguous ways
88:21
and
88:22
all of the leading arms are pretty good
88:24
at generating JSON outputs at this point
88:26
So the system prompt might say something like this
88:29
you have access to file tools
88:30
distribution tools, and then create a step-by-step plan in JSON format
88:34
and
88:34
you might describe the JSON format in enough
88:37
detail with the goal of getting it to
88:39
output a plan
88:40
like that shown here on the right
88:42
So in this JSON output
88:43
it creates a list
88:45
where the first list item has clear keys
88:48
and values. Values that say
88:50
the step one of the plan
88:51
has
88:52
following description
88:53
and it should use the following tool
88:55
with the following arguments passed to that tool
88:58
Then after that
88:58
step two of the plan
89:00
is to
89:00
carry out this task and then to use
89:02
this tool and so on
89:03
So
89:04
this JSON format as opposed to writing the
89:06
plan in
89:07
English allows downstream code to more clearly parse
89:10
out exactly
89:11
what are the steps of the plan
89:13
so that it can be reliably executed
89:15
one step at a time
89:17
Instead of JSON
89:18
I also see
89:19
some developers use XML where you can
89:22
use XML delimiters or use XML tags to
89:25
clearly specify
89:27
what are the steps of the plan and
89:28
what step number it is
89:30
Some developers, I feel like fewer developers will use Markdown
89:33
which is
89:34
just sometimes slightly more ambiguous
89:36
in terms of how we parse it
89:38
And I think plain text is maybe the
89:39
least reliable of these options
89:41
But I think either JSON
89:42
which I'm showing here
89:43
or XML
89:44
would be good options for how to ask
89:46
the LM to format a plan
89:48
unambiguously
89:50
So that's it. By opening plans in JSON
89:53
you can then
89:54
pause it and have
89:55
downstream workflows execute different steps of the plan more systematically
90:00
Now, in terms of giving elements a plan
90:03
it turns out there's one other
90:05
really neat idea
90:06
that lets an element output very complex plans
90:08
and get them executed reliably
90:11
and that's allow them
90:12
write code and to have code express
90:14
the plan
90:15
Let's take a look at this
90:16
in the next video
90:19
We've talked a lot about how to build
90:20
a single agent to complete tasks for you
90:23
In a multi-agent or multi-agentic workflow
90:26
we instead have a collection of multiple agents
90:28
collaborate to do things for you
90:31
When some people hear for the first time about multi-agent systems
90:34
they wonder, why do I need multiple agents?
90:37
It's just the same OM that I'm prompting
90:39
over and over or just one computer
90:41
Why do I need multiple agents?
90:44
I find that
90:45
one useful analogy is
90:47
even though I may do things on a single computer
90:49
we do decompose work
90:51
in a single computer into
90:53
maybe multiple processes or multiple threads
90:56
And as a developer
90:57
thinking even though it's well one CPU on a computer
91:00
say
91:01
thinking about how to take work and decompose
91:03
it into multiple processes and multiple computer programs to run
91:07
that makes it easier
91:08
for me as a developer to write code
91:11
And in a similar way too
91:13
if you have a complex task to carry out
91:16
sometimes instead of thinking about how to
91:19
hire one person to do it for you
91:21
you might think about hiring a team of
91:23
a few people
91:24
to do different pieces
91:25
of the task for you
91:27
And so in practice
91:28
I found that for many developers of agentic systems
91:31
having this mental framework of not asking
91:33
what's the one person I might hire to do something
91:36
but instead
91:37
would it make sense to hire people with
91:39
three or four different roles to do this
91:41
overall task for me?
91:42
That holds
91:43
Give another way
91:44
to take a
91:45
complex thing and decompose it into subtasks and
91:48
to build for those individual subtasks
91:50
one at a time
91:51
Let's take a look at some examples of
91:53
how this works
91:54
Take the task of creating marketing assets
91:57
Say you want to market sunglasses
91:59
Can you
92:00
come up with a marketing brochure for that?
92:02
You might need a researcher on your team
92:04
to look at trends on sunglasses and what
92:06
competitors are offering. You might also
92:09
have a graphic designer on your team to
92:11
render charts or
92:12
nice looking graphics for your sunglasses
92:14
And then also a writer to take the research
92:17
take the graphic assets
92:18
and put it all together into a nice
92:19
looking brochure. Or to write a research article
92:22
you might want a researcher to do online research
92:25
a statistician to calculate statistics
92:27
a lead writer
92:28
and an editor to come with a polished
92:30
report. Or to prepare a legal case
92:32
Real law firms will often have associates
92:34
paralegals, maybe an investigator. And we naturally
92:38
because of the way human teams do work
92:41
can think of
92:42
different ways that
92:43
Complex tasks can be broken down into
92:47
different individuals with different roles
92:50
So these are examples of when a complex
92:53
task
92:53
will already naturally decompose into subtasks that different
92:57
people with different skills can carry out
93:00
Take the example of creating marketing assets
93:03
Look into detail into what a researcher
93:05
graphic designer, and writer might do
93:08
A researcher
93:09
might
93:10
have the task of analyzing market trends
93:12
and researching competitors. And when designing the research agents
93:17
one question to keep in mind is
93:19
what are the tools that the researcher
93:21
may need in order to come up with
93:23
a research report on market trends and what
93:25
competitors are doing. So one natural tool
93:28
that a
93:29
agentic researcher might need to use would be
93:32
web search. Just as a human researcher
93:34
asked to do these tasks might need to
93:35
search online in order to
93:37
come up with their report
93:39
Or for a graphic designer agents
93:42
they might be tasked with creating visual and
93:44
artwork. And so
93:46
what are the
93:46
tools that a agentic software graphic designer might need?
93:50
Well, they may need image generation and manipulation APIs
93:54
or maybe
93:55
similar to what you saw with the coffee machine example
93:59
maybe it needs code execution to generate charts
94:02
And then lastly
94:03
the writer has transformed the research into report
94:05
text and marketing copy
94:07
and in this case
94:08
they don't need any tools other than
94:10
what an LLM can already do
94:12
to generate text.In this and the next video
94:15
I'm going to use these purple boxes to denote an agent
94:19
and the way you build individual agents is
94:21
by prompting an LLM
94:23
to play the role of
94:24
a researcher or a graphic designer or a writer
94:27
depending on which agent it is part of
94:30
So, for example, for the research agents
94:32
you might prompt it to say
94:33
you are a research agent
94:35
expert at analyzing market trends and competitors
94:38
carry out
94:39
online research to analyze market trends for the sunglasses product
94:43
and give Summary as well of what competitors
94:44
are doing. So that
94:46
would allow you to build a researcher agent
94:49
And similarly
94:50
by prompting an LM to act as a
94:52
graphic designer
94:53
with the appropriate tools and to act as a writer
94:56
that's
94:57
how you can build a graphic designer
94:59
as well as a writer agent
95:01
Having built these three agents
95:04
one way to have them work together
95:06
to
95:07
generate your final report would be to
95:09
use a
95:10
Simple
95:11
linear
95:11
ordered
95:12
workflow or a linear plan in this case
95:14
So
95:15
if you want to create a summer marketing campaign for sunglasses
95:19
you might give that prompt to the research
95:20
agents. The research agent then writes a report that says
95:24
here are the current sunglasses trends and competitive
95:26
offerings. This research report can then be fed
95:28
to the graphic designer that looks at
95:31
the data the research has found and creates
95:33
a few data visualizations and artwork options
95:36
All these assets can then be passed
95:38
to the writer
95:39
that then takes the research and the graphic
95:42
output and writes the final marketing brochure
95:46
The advantage of building a multi-agent workflow in
95:48
this case is
95:50
when designing a researcher or graphic designer or writer
95:53
you can focus on one thing at a
95:55
time. So I can spend some time building
95:57
maybe the best graphic designer agents I can
95:59
while maybe
96:00
my collaborators are building research agents and writer
96:03
agents. And then in the end
96:04
we string it all together
96:06
to come up with this multi-agent system
96:08
And in some cases
96:10
I'm seeing developers start to reuse
96:13
some agents as well
96:14
So having built a graphic designer for marketing brochures
96:18
maybe I'll think about if I can build
96:20
a more general graphic designer
96:22
that can help me write marketing brochures as
96:23
well as social media posts
96:25
as well as help me illustrate
96:27
online web pages
96:29
So by
96:30
coming up with what are the
96:32
agents you might hire to do a task
96:34
and this will sometimes correspond to
96:37
who are the
96:37
types of human employees you might hire to do a task
96:41
you can come up with a workflow like this
96:44
with maybe even building agents that you could
96:46
choose to reuse in other applications as well
96:49
Now
96:50
what you see here is a linear plan where one agent
96:53
the researcher does his work
96:55
then the graphic designer
96:56
and then the writer. With agents
96:58
you can also, as an alternative to a linear plan
97:01
you can also have agents
97:03
interact with each other in more complex ways
97:05
Let me illustrate with an example of planning
97:08
using multiple agents. So previously
97:11
you saw how we may give an OM
97:12
a set of tools that they can call
97:15
to carry out different tasks
97:16
In what I want to show you
97:18
we will instead give an OM
97:20
the option to call on different
97:22
agents to ask the different agents to help
97:24
complete different tasks. So in detail
97:27
you might write a prompt
97:28
like you're a marketing manager
97:29
have the following team of agents to work with
97:31
and then give a description to the agents
97:33
And this is very much similar to what
97:35
we're doing with planning using tools
97:37
except the tools, the green boxes, are replaced with agents
97:41
these purple boxes that the OM can call
97:43
on
97:43
And then we'll ask it to return a
97:44
step-by-step plan to carry out the user's requests
97:47
And in this case
97:47
the OM may
97:48
ask the
97:49
researcher to
97:50
research current sunglasses trends and then report back
97:53
Then they'll ask the graphic designer to create
97:55
the images and then report back
97:56
Then ask the writer to create a report
97:58
And then maybe the OM will choose to
97:59
review
98:00
or reflect on and improve the report
98:02
one final time. In executing this plan
98:06
you would then take the step one text of the researcher
98:08
carry out research
98:10
Then pass that to the graphic designer
98:12
pass it to the writer
98:14
and then maybe do one final reflection step
98:16
and then you'd be done
98:18
One interesting view of this workflow is
98:20
as if you have these three agents up here
98:23
but this arm on the left
98:25
is actually like a fourth agent
98:27
There's a marketing manager that is a manager
98:30
of a marketing team
98:31
that is setting direction and then delegating tasks to the researcher
98:34
the graphic designer, and the writer agents
98:36
So this becomes actually a collection of four agents
98:39
for the marketing manager agent coordinating the work of the researcher
98:42
the graphic designer, and the writer
98:45
In this video, you saw two communication patterns
98:48
One was a linear one
98:50
where your agents took
98:51
actions one at a time until you got to the end
98:54
and the second had a marketing manager
98:56
coordinating the activity of a few other agents
99:00
It turns out that one of the
99:01
key design decisions you may end up having
99:04
to make when building multi-agentic systems is
99:06
what is the communication pattern between your different
99:09
agents
99:10
This is an area of hard research
99:12
and there are multiple patterns emerging
99:14
but in the next video
99:15
I want to show you what are some
99:16
of the most common communication patterns for
99:19
getting your agents to work with each other
99:21
Let's go see that in the next video
99:24
When you have a team of people working together
99:28
the patterns
99:29
by which they communicate can be quite complex
99:31
And in fact
99:32
designing a
99:33
organizational chart is actually pretty complex to try
99:36
to figure out what's the best way for people to communicate
99:39
to collaborate
99:40
It turns out designing communication patterns for
99:43
multi-agent systems is also quite complex
99:46
But let me show you some of the
99:47
most common design patterns I see
99:50
used by different teams today
99:51
In a marketing team with a linear plan
99:53
where first the researcher worked
99:55
then the graphic designer
99:56
then the writer, the communication pattern was linear
99:59
The researcher would communicate with the graphic designer
100:02
and then both the research and the graphic designer
100:04
maybe pass the outputs to the writer
100:06
and so this is very linear communication pattern
100:09
This is
100:10
one of the two most common
100:12
communication plans that I see being used today
100:14
The second of the
100:15
two most common communication plans would be
100:19
similar to what you saw in this example
100:22
with planning using multiple agents
100:24
Where there is a
100:25
manager
100:26
that
100:27
communicates with a
100:28
number of team members and coordinates their work
100:31
So in this example
100:32
the marketing manager decides to call on the
100:34
researcher to do some work
100:36
then
100:36
if you think of the marketing manager as
100:38
getting the report back and then sending it
100:40
to the graphic designer
100:41
getting the report back and send it to the writer
100:43
this would be a hierarchical communication pattern
100:47
If you're actually implementing a hierarchical communication pattern
100:50
it would probably be simpler to have the
100:52
researcher pass the report back to the marketing
100:54
manager rather than the researcher pass the results
100:56
directly to the graphic designer and to the
100:58
writer. But so this type of hierarchy
101:01
is also a pretty common way
101:03
to
101:03
plan the communication patterns where you have one
101:05
manager coordinating the work of
101:07
a number of other agents
101:09
And just to share with you
101:10
some more advanced and
101:13
less frequently used but nonetheless sometimes used in practice communication patterns
101:17
one would be a deeper hierarchy where same as before
101:20
if you have a marketing manager
101:22
send tasks to researcher
101:23
graphic designer, Writer
101:24
but maybe the researcher
101:26
has themselves two other agents that they call on
101:29
such as a wet researcher and a fact
101:30
checker. Maybe the graphic designer just works by themselves
101:33
whereas the writer has a
101:35
initial style writer and a citation checker
101:38
So this would be a hierarchical organization
101:41
of agents in which
101:43
some agents might themselves call other
101:45
sub-agents and also see this used in some applications
101:49
but this is much more complex than a one-level hierarchy
101:52
so used less often today
101:54
And then one final pattern that is quite challenging to execute
101:58
but I see a few experimental projects use it
102:02
is the all-to-all communication pattern
102:04
So in this pattern
102:06
anyone's allowed to talk to anyone else at any time
102:08
and the way you implement this is you
102:10
prompt all four of your agents
102:13
in this case
102:14
to tell them that there are
102:16
three other agents it could decide to call on
102:18
and whenever one of your agents decides to
102:20
send a message to another agent
102:22
that message gets added The receiver agent's context
102:27
and then the receiver agent can
102:28
think for a while and decide when to
102:30
get back to that first agent
102:32
And so if you all
102:33
collaborate in a crowd and talk to each
102:34
other for a while until
102:36
say
102:37
each of them declares that it is done with its task
102:40
and then it starts talking
102:41
then maybe when everyone thinks it's done
102:43
or maybe when the writer concludes it's good enough
102:45
that's when you generate the final output
102:47
In practice
102:48
I find the results of all-to-all communication patterns
102:50
a bit hard to predict
102:52
so some applications don't need high control
102:55
and you can run it and see what you get
102:57
and the marketing brochure isn't good
102:58
Maybe that's okay
102:59
you just run it again and see if
103:00
you get a different result
103:01
But I think for
103:02
applications where you're willing to
103:04
tolerate a little bit of chaos and unpredictability
103:07
I do see some developers using this communication pattern
103:10
So that, I hope
103:12
conveys some of the richness of multi-agentic systems
103:16
Today
103:17
there are quite a lot of software frameworks
103:19
as well that support easily building multi-agent systems
103:22
and they also
103:24
Implementing some of these communication patterns relatively easy
103:27
So maybe if you use your own multi-agent system
103:31
you'll find some of these frameworks hopeful
103:33
for exploring these different communication patterns as well
103:36
And so
103:38
that now brings us to the final
103:41
video of this module and of this course
103:45
Let's go on to the final video
103:47
to wrap up
103:49
Welcome to the final video of this course
103:51
It feels like we've been through a lot together
103:54
this you and me
103:54
and we've gone through a lot of topics
103:56
in agentic AI
103:58
Let's take a look
103:59
In the first module
104:00
we talked about what are the applications you
104:02
can build with agentic AI
104:04
that just were not possible before
104:06
And we started then to look at key design patterns
104:11
including the reflection design pattern
104:13
which is a
104:14
simple way to sometimes give your application a nice performance boost
104:18
and then tool use or function calling
104:20
which expands what your LM application can do
104:23
with code execution being one important case of
104:26
that. And then we spent a lot of
104:27
time talking about
104:28
evaluations as well as error analysis
104:31
and how to drive a discipline process
104:33
of building
104:34
as well as analyzing to be efficient in
104:37
how you
104:37
keep on improving the performance of your agentic
104:40
AI system. This fourth module is
104:42
some of the material that I think
104:44
you will find most useful
104:46
as you keep building agentic AI systems
104:48
I hope, for a long time
104:50
And then in this module
104:51
we talk about planning and multi-agent systems that
104:54
can let you build much more powerful
104:57
although sometimes harder to control
104:59
harder to predict and advance types of systems
105:02
So with the skills you learn from this course
105:04
I think you now know how to build
105:06
a lot of cool
105:07
exciting, agentic AI applications
105:10
When my team, or I see other teams as well
105:12
interview people for jobs
105:14
I find interviews often try to
105:16
assess whether or not candidates have
105:19
pretty much the skills you're learning about
105:21
in this course. And so I hope that
105:23
this course will also open up new professional opportunities for you
105:26
and you should be able to do more
105:29
Whether you're doing these things for fun
105:31
or for professional or practical settings
105:34
I think you'll enjoy
105:35
this new set of things you can
105:38
now build
105:39
So just to wrap up
105:41
I want to thank you again for spending
105:43
all this time with me
105:44
and I hope you will take these skills
105:46
use them responsibly, and just go build closer

Watch these videos on your TV

XTTV brings video content from X to your living room. Download now and start watching on the big screen.