← Back to videos

Don't waste 2 years learning to become an AI agentic engineer in 2026. Andrew Ng, the godfather o

shobhanai

shobhanai

@HeyShobhan

November 18, 2026· 105:51

Don't waste 2 years learning to become an AI agentic engineer in 2026. Andrew Ng, the godfather of AI, gave the complete playbook to become one from scratch. 1 hour course. Free: • 00:00 - AI agent basics • 12:12 - AI Agentic workflows & design patterns • 53:27 - Practical tips for building AI agents • 1:20:30 - self-improving AI agent loops • 1:30:19 - multi-agent AI systems I watched it last night. Halfway through, I realized I could get into Anthropic in weeks, not years. Bookmark now. Watch it. Then build your own AI agent

Transcript

When
I coined the term agentic to describe what
0:02
I saw as a
0:03
important and rapidly growing trend in how people
0:06
were building on-base applications
0:08
what I did not realize was that a
0:09
bunch of marketers would get hold of this
0:11
term
0:12
and use it as a sticker and put
0:14
this on almost everything in sight
0:16
And
0:16
that has caused hype on agentic AI
0:19
to skyrocket
0:21
The good news, though, is that ignoring the hype
0:23
the number of truly
0:25
valuable and useful applications built
0:27
using agentic AI has also grown very rapidly
0:30
even if
0:31
not quite as rapidly as the hype
0:33
And in this course
0:34
what I'd like to do
0:35
is show you best practices for building agentic AI applications
0:39
and just open up
0:40
a lot of new opportunities
0:42
to you in terms of what you can now build
0:44
Today
0:44
agentic workflows are being used to build applications
0:47
like customer support agents
0:49
or
0:49
to do deep research to help write deeply insightful research reports
0:54
or to process tricky legal documents
0:57
or to look at
0:58
patient input
0:59
and render Or to suggest possible medical diagnoses
1:03
On many of my teams
1:04
a lot of the projects we built just
1:06
would be impossible without agentic workflows
1:09
and so
1:09
knowing how to build applications with them
1:11
is one of the most important and valuable
1:13
skills in AI
1:15
today. It turns out that
1:17
one of the
1:17
biggest differences I've seen between people that really
1:20
know how to build agentic workflows
1:22
compared to people that are less effective at
1:23
it is the ability to drive a disciplined development process
1:27
specifically one focused on evals and error analysis
1:31
And in this course
1:32
I'll tell you what that means and show
1:33
you
1:34
what allows you to be really good at
1:36
building these agentic workflows
1:38
Being able to do this is
1:40
one of the most important skills in AI today
1:42
and will open up a lot more opportunities
1:44
be it job opportunities or
1:46
opportunities to just build amazing software yourself
1:50
With that
1:50
let's go on to the next video
1:52
to dive more into what are agentic workflows
1:56
So
1:56
what is agentic AI and why are agentic
1:58
AI workflows so powerful?
2:01
Let's take a look
2:02
The way that many of us use large
2:03
language models or LLMs today is by prompting it to
2:06
say
2:07
write an essay for us on a certain
2:09
topic X
2:10
And I think of that as
2:11
akin to going to a human
2:13
or in this case
2:14
going to an AI
2:15
and asking it to please
2:16
type out an essay for me
2:18
by writing from the first word to the
2:20
last word
2:21
all in one go
2:22
and without ever using backspace
2:24
It turns out that we as people
2:25
we don't do our best writing like that
2:27
by being forced to write in this
2:29
completely linear order and nor do AI models
2:32
But despite the difficulty of being constrained to writing this way
2:36
our LLMs do surprisingly well
2:38
In contrast, with an agentic workflow
2:40
this is what the process might look like
2:42
You may ask it to first
2:43
write an essay online and listen to the topic
2:46
then ask if it needs to do any more research
2:48
and
2:48
after doing some web research and maybe downloading some web pages
2:51
then to write the first draft
2:52
and then to read the first draft and
2:54
see what parts need revision or more
2:55
Research, and then revise the draft
2:57
and so on
2:58
And this type of workflow is more akin
3:00
to doing some thinking and some research
3:02
and then doing some revision
3:04
and then doing some more thinking
3:06
and so on. And with this iterative process
3:09
it turns out that an agentic workflow can take longer
3:12
but it delivers a much better work product
3:15
So
3:16
an agentic AI workflow is a process where
3:18
an LM-based app
3:20
executes multiple steps to complete a task
3:23
In this example
3:24
you might use an LM to write the first essay outline
3:28
and then
3:28
you might use an LM to decide
3:30
what search terms to type into a web search engine
3:34
or really what search terms to call a
3:36
web search
3:37
API with in order to get back
3:39
relevant web pages. Based on that
3:42
you can
3:43
feed the downloaded web pages into an LM
3:45
to help you write the first draft
3:47
and then maybe use another LM to
3:49
reflect and decide what needs more revision
3:52
and then depending on how you design this workflow
3:55
Perhaps you may even
3:57
add a human in the loop step where
3:59
the arm has the option to request human review
4:03
maybe of some key facts
4:04
and based on that
4:05
it may then revise the draft
4:07
and this process
4:09
results in a much better work output
4:11
One of the key skills you learn in
4:13
this course is how to take a complex task
4:16
like writing an essay
4:18
and
4:19
breaking it down into
4:21
smaller steps
4:22
for agentic workflow to execute one step at
4:25
a time
4:26
to then get the work output that you
4:28
want
4:28
And knowing how to decompose the task into
4:30
steps
4:31
and how to build the components to execute
4:33
the individual steps well
4:34
turns out to be a tricky but important
4:37
skill
4:37
that will determine your ability
4:40
to build agentic workflows for a huge range
4:42
of exciting applications. In this course
4:45
a run example that we'll use and something
4:47
that you build alongside me is a
4:50
research
4:50
agent
4:51
So here's an example of what it will
4:53
look like
4:53
You can enter a research topic like How
4:56
do I build a new rocket
4:59
company to compete with SpaceX?
5:03
I don't personally want to compete with SpaceX
5:05
but if you want to
5:06
you can
5:06
try asking a research agent to help your
5:08
background research
5:10
So this agent starts with planning out what research to use
5:14
including calling a web search engine to download some web pages
5:18
and then to synthesize and rank findings
5:21
draft an outline, have an editor agent review for coherence
5:24
and then finally generate a comprehensive markdown report
5:28
which it has done here
5:29
building a new rocket company to compete with SpaceX
5:31
with an intro, background, Findings and so on
5:35
and I take the points of appropriately that
5:38
this is going to be a
5:39
tough startup to build
5:40
so I'm not personally planning to do this
5:43
but if you want to tackle something like this
5:46
maybe a research agent like this could help
5:49
you with some initial research
5:51
And by finding
5:53
and downloading multiple sources and deeply thinking about it
5:56
this actually ends up with a
5:58
much more thoughtful report than
6:01
just prompting an alarm
6:02
to write an essay for you would
6:04
One of the reasons I'm excited about this
6:06
is because in my work
6:07
I've ended up building quite a few specialized research agents
6:11
be it in
6:12
legal documents for conflict legal compliance or for
6:15
some healthcare sectors or some
6:17
business product research areas
6:19
And so I hope that working through this example
6:22
you not only learn how to build agentic
6:24
workflows for many other applications
6:26
but that some of the ideas in building
6:28
research agents will be
6:30
directly useful to you if you ever need
6:32
to build a custom research Search agent yourself
6:35
Now
6:36
one of the
6:37
often discussed areas of
6:40
AI agents is how autonomous are they?
6:43
What you just saw here was a relatively complex
6:45
highly autonomous agentic AI workflow
6:48
but there are also
6:49
other simpler workflows that are incredibly valuable
6:53
Let's go into the next video to talk
6:55
about the degree to which agentic workflows can be autonomous
6:58
and this will give you
7:00
a framework to think about how
7:02
you might go about building different applications and
7:04
how easy or difficult they might be
7:06
See you in the next video
7:09
Agents can be autonomous to different degrees
7:12
A few years ago
7:13
I noticed within the AI community that there
7:15
was a growing controversial debate about what is an agent
7:19
and some people are writing a paper saying about an agent
7:22
and others say, no
7:22
that's not really a true agent
7:24
And I felt this debate was unnecessary
7:27
which is why I started using the term
7:29
agentic because I thought if
7:31
we use it as an adjective rather than a binary
7:34
it's either an agent or not
7:36
then we can have
7:37
just acknowledge that systems can be agentic to different degrees
7:41
and let's just call out agentic and move
7:42
on with the real work of building these
7:44
systems rather than debating
7:46
you know
7:47
is this sufficiently autonomous
7:48
to be an agent or not
7:50
I remember when I prepared a talk on agentic reasoning
7:54
one of my team members actually came to me and said
7:55
hey, Andrew, we don't need yet another word
7:58
You know, we have agent
7:59
Why you make have another word agentic?
8:01
But I decided to use it anyway
8:03
And then later on
8:04
wrote an article in
8:06
DeepMind's newsletter
8:07
The Batch. And also
8:09
post on social media saying that
8:11
instead of arguing over which words include or
8:13
exclude as being a true agent
8:15
let's
8:15
acknowledge the different degrees to which systems can
8:18
be agentic
8:19
And I think this helped move past the
8:21
debates on
8:23
what is a true agent and
8:24
let us just focus on actually building them
8:27
Some agents can be less autonomous
8:30
So
8:30
take the example of writing an essay
8:32
about black holes
8:34
You can have a relatively simple agents
8:37
to come up with a few
8:38
web search terms or web search queries
8:41
then you can
8:42
hard code in that you call a web search engine
8:45
fetch some web pages
8:46
and then use that to write an essay
8:48
And this would be an example of a
8:49
less autonomous agent with a fully deterministic sequence of steps
8:54
and this will work okay
8:56
In terms of notational convention
8:57
throughout this course
8:58
I'll use the red color
9:00
as you see here on the left
9:01
to denote the user input
9:03
such as a user query in this case
9:05
or in later examples
9:06
maybe the input document. In the agentic workflow
9:09
The gray boxes denote calls to an LM
9:12
and the green boxes
9:14
like the web search
9:16
and the web fetch boxes that you see here
9:18
indicate steps where
9:20
other software is being used to carry out an action
9:22
such as a web search API call or
9:24
executing code to fetch the contents of a
9:26
website. Then an agent can be more autonomous
9:30
where given a request to write an essay about black holes
9:33
perhaps you let the LM decide
9:35
does it want to do a web search
9:36
or does it want to search recent news sources
9:39
or does it want to search for recent
9:41
research papers on the website archive?
9:43
Based on that, maybe in this example
9:45
the LM, not the human engineer
9:47
but the LM chooses in this case
9:49
to call a web search engine
9:51
and then after that
9:52
you may let the LM decide
9:54
how many web pages does it want to fetch
9:55
or if it fetch the PDF
9:58
does it need to
9:59
call a function or also call the tool
10:01
to convert the PDF to text?
10:03
And in this case
10:03
maybe it fetches its top few web pages
10:07
then it can write an essay
10:08
Decide whether or not to reflect and improve
10:10
and maybe even go back to fetch more
10:12
web pages
10:13
and then to finally produce an output
10:15
And so even for this example of a research agent
10:19
we can see that
10:20
some agents can be
10:22
less autonomous
10:23
with a
10:23
linear sequence of steps to be executed determined by a programmer
10:27
and some can be more autonomous where you
10:29
trust the OM to make more decisions
10:31
and the exact sequence of steps that happens
10:33
may be even determined by the OM rather
10:35
than in advance by the programmer
10:37
So for less autonomous systems
10:40
you will usually have all the steps predetermined in advance
10:43
and any functions it calls like web search
10:46
and we'll call that tool use as you
10:47
learn in the third module in this course
10:49
might be hard coded by the human engineer
10:51
by you or me
10:52
and most of the autonomy
10:54
is in what text the OM generates
10:56
At the end of the spectrum would be highly autonomous agents
10:59
where the agent makes many decisions autonomously
11:02
including, for example
11:03
deciding what is the sequence of steps it
11:05
will carry out in order to
11:07
write the essay
11:08
And there's some highly autonomous agents that can
11:10
even write new functions
11:12
or sometimes they
11:13
create new tools that it can then execute
11:16
And somewhere in between are semi-autonomous agents
11:19
where it can make some decisions
11:20
choose tools
11:21
but the tools are usually more predefined
11:23
As you look at different examples in this course
11:25
you learn how to build applications anywhere on
11:28
this spectrum of less to more highly autonomous
11:31
And you find that there are tons of
11:32
applications in the less autonomous end of the
11:34
spectrum
11:35
They're very valuable
11:36
being built for tons of businesses today
11:38
And at the same time
11:39
there are also
11:40
applications being worked on at the more
11:42
highly autonomous end of the spectrum
11:44
but those are usually less easily controllable
11:47
a little bit more unpredictable
11:49
and also
11:50
a lot of active research as well to
11:52
figure out how to build
11:53
these more highly autonomous agents
11:56
And with that
11:57
let's go on to the next video
11:59
to
11:59
dive deeper into this
12:01
and to hear
12:02
about some of the benefits of using agents
12:04
and why they allow us
12:05
to do things that just were not possible
12:07
with earlier generations of
12:09
of base applications
12:12
I think the one biggest benefit of agentic
12:14
workflows is that allows you
12:16
to do many tasks effectively that just previously
12:19
were not possible
12:20
But there are other benefits as well
12:22
including parallelism that lets you do certain things quite fast
12:25
as well as modularity that lets you combine
12:27
the best of three components from many different
12:30
places to build an effective workflow
12:32
Let's take a look
12:33
My team collected some data
12:35
on a
12:36
coding benchmark
12:37
that tests the ability of different OLMs to
12:40
write code to carry out certain tasks
12:43
The benchmark used in this case is called HumanEval
12:46
and it turns out that GPT-3.5
12:48
this is a model that the first
12:50
hopefully available version of ChatGPT was based on
12:52
if asked to write the code directly
12:54
to just type out the computer program
12:56
gets 48% right on this benchmark
12:59
This is a passe K metric
13:00
GPT-4 is a much better model
13:02
Its performance leaps to
13:04
67% with this also non-agentic workflow
13:08
But it turns out that
13:09
as large as the improvement was from GPT-3.5 to GPT-4
13:14
that improvement is stored by what you can
13:15
achieve by wrapping GPT-3.5 within an agentic workflow
13:21
Using different agentic techniques
13:22
which you learned about later in this course
13:24
you can prompt GPT-3.5 to write code
13:27
and it may be
13:28
reflect on the code and figure out if
13:30
it can improve it. And using techniques like that
13:32
you can actually get GPT-3.5 to get much
13:35
higher levels of performance. And similarly
13:37
GPT-4
13:38
used in the context of an agentic workflow
13:41
also does much better
13:43
So even with today's SLMs
13:45
an agentic workflow lets you get
13:47
much better performance. And in fact
13:49
what we
13:50
saw in this example was the improvement from
13:53
one generation of model to another
13:55
which is huge
13:55
is still not as big a difference as
13:58
implementing an agentic workflow on the previous generation
14:01
of model
14:02
Another benefit
14:03
of using agentic workflows is that they can
14:06
paralyze some tasks
14:07
and thus do certain things much faster
14:10
than a human
14:11
For example
14:12
If you ask an
14:13
agentic workflow to write an essay about black holes
14:16
you might be able to have three OMs
14:18
run in parallel to generate ideas
14:20
for web search terms
14:22
to type into search engine
14:23
Based on the first web search
14:25
it may identify, say, three top results to fetch
14:29
and based on the second web search
14:31
it may identify a second set of web pages to fetch
14:34
and so on
14:35
And it turns out that whereas
14:37
a human doing this research
14:38
would have to read these nine web pages
14:41
sequentially or one at a time
14:42
when you're using an agentic workflow
14:45
you can actually parallelize all nine web page downloads
14:49
and then finally feed all these things into
14:51
an LM to write an essay
14:52
So even though agentic workflows
14:54
do take longer than
14:56
truly non-agentic workflows or by direct generation of
14:59
I just prompting a single time
15:00
if you were to compare
15:02
this type of agentic workflow to how a
15:04
human would have to go about the task
15:06
the ability to parallelize downloading lots of web
15:08
pages
15:09
can actually let it do certain tasks much
15:11
faster. Faster than the
15:13
non-parallel sequential way that a single human might
15:16
process this data. To build on this example
15:19
it turns out
15:20
one of the things I often do when
15:21
building
15:22
agentic workflows is
15:23
look at the individual components like the
15:25
LM and add or swap out components
15:28
So for example
15:30
maybe I look at a web search engine
15:31
I use up here
15:32
and I might decide that
15:33
I want to swap in a new web
15:35
search engine. When building agentic workflows
15:37
there are actually multiple web search engines
15:39
including Google, which you can access via Surfer
15:42
as well as others like Bing
15:44
DuckDuckGo, Tivili
15:45
u.com. There are actually quite a lot of
15:47
options for web search engines designed for LMs
15:50
to use. Or maybe instead of just doing three web searches
15:53
maybe on this step
15:54
you can swap in a new
15:56
news search engine
15:57
so they can find out what's the latest
15:59
news on recent breakthroughs on black hole science
16:04
And lastly
16:05
instead of using the same LM for all
16:07
of the different steps
16:09
I will often try out different
16:11
large Gosh models
16:12
and maybe try out different LLM providers
16:15
to see which one
16:16
gives the best result for different
16:18
steps of this system
16:20
So to summarize
16:21
the main reason I use agentic workflows is
16:24
it just gives much better performance on many
16:26
different applications. But in addition
16:29
it can also paralyze
16:30
some tasks that humans will otherwise have to do sequentially
16:34
and
16:35
the modular design of many
16:37
agentic workflows also lets us
16:39
add or update tools
16:40
and sometimes swap out models
16:43
We've talked a lot about the key components
16:45
of building agentic workflows
16:47
Let's now take a look
16:48
at the range of agentic AI applications to
16:51
give you a sense of the sorts of
16:52
things people are already building and the sorts
16:55
of things you will build yourself
16:57
Let's go on to the next video
17:00
People and businesses do a lot of stuff
17:04
How do you take this
17:05
useful stuff that we do
17:07
and break it down into discrete steps
17:09
for the genetic workflow to follow?
17:11
Let's take a look
17:12
Take the example of building a research agent
17:15
If you want
17:17
a AI system to write an essay on the topic X
17:19
one thing you could do is
17:20
prompt an LLM to have it
17:22
generate an output
17:23
directly
17:24
But
17:24
if you were to do this for
17:26
topics that you want deeply researched
17:28
you may find that the LLM output
17:30
covers only the surface level points or maybe
17:32
covers only the obvious facts
17:34
but doesn't go
17:35
as deep into
17:36
the subject as you want it to
17:38
In this case
17:39
you might
17:40
then reflect on how you as a human
17:42
would write an essay on a certain topic
17:45
Would you just sit down and start writing
17:47
or would you take multiple steps
17:48
such as first write an essay outline
17:50
and then search the web
17:51
and then based on the input from the web search
17:54
write the essay?
17:55
And as I take a task and decompose it into steps
17:58
one question I'm always asking myself is
18:01
if I look at these steps one
18:02
two, and three
18:03
can each of them be done
18:05
either by an LLM or
18:07
by
18:07
a short piece of code or by a
18:09
function call by a tool
18:11
And so in this case
18:12
I think an LLM
18:13
can maybe write a decent outline on
18:16
many topics that I would want it to
18:18
help me think through. So say
18:20
probably okay on the first step
18:22
And then I know how to
18:23
use an LLM to
18:24
generate search terms to search the web
18:27
So I would say the second step is
18:28
also doable. And then based on web search
18:31
I think an LLM could input the web
18:33
search results and write an essay
18:34
And so this would be a reasonable first
18:36
attempt at an agentic workflow
18:38
for writing an essay
18:39
that goes deeper than just direct generation
18:42
But if
18:43
I were to then implement this agentic workflow
18:45
and look at the results
18:47
maybe
18:47
you find that the results still aren't good
18:49
enough. It's still not yet as deeply thoughtful
18:51
Maybe the essays feel a little bit disjointed
18:54
This has actually happened to me
18:55
Once built a research agent using this sort of workflow
18:58
but when I read the output
18:59
it felt a bit disjointed
19:00
You know
19:00
the start of the article didn't feel completely
19:03
consistent with Middle didn't feel completely consistent
19:06
with the end. In this case
19:07
what you might do
19:08
is then reflect on how you would change
19:10
the workflow if you as a human found
19:12
that the essays were a bit disjointed
19:14
One thing you could do
19:15
is take the third step and further decompose
19:19
write the essay into additional steps
19:22
So instead of writing the essay all in one go
19:25
you might instead have it
19:27
write the first draft
19:28
and then consider what parts need revision
19:31
and then revise the draft
19:32
And this would be
19:33
how I as a human might go about it to
19:36
not just write the final essay at my first attempt
19:39
but write the first draft and then read over it
19:41
which
19:42
is another step that the arm
19:44
is pretty decent at. And then based on
19:46
my own critique of my own essay
19:48
I'll revise the draft
19:49
So to recap, I started off with direct generation
19:53
just one step, decided it wasn't good enough
19:55
and so broke that down into three steps
19:58
and then maybe decide that still isn't good enough
20:00
and took one of the steps and further
20:02
broken it down or decompose it
20:04
Into three more steps
20:05
resulting in this more complex
20:08
richer process for generating an essay
20:11
And depending on
20:12
how satisfied you are with the results of this process
20:15
you may choose to even modify
20:17
this essay generation process further
20:19
Let's look at a second example of how
20:21
to decompose complex tasks into smaller steps
20:25
Take the example of responding to
20:27
basic customer order
20:29
inquiries
20:30
The first step that a human customer service
20:32
agent might carry out
20:33
might be to first extract the key information
20:36
such as who is this email from
20:38
what do they order
20:39
and what is the order number
20:40
And these are things an OM could do
20:43
so I could just say
20:44
let's have an OM do that
20:45
The second step would be to then
20:47
find the relevant customer records
20:49
So to
20:49
write and generate the relevant database queries to
20:52
pull up the order
20:54
what the customer had ordered and when it
20:55
shipped and so on
20:56
I think an OM
20:57
with the ability to call a function
21:00
to query the orders database
21:03
should be able to do that
21:04
That, and lossy
21:05
having pulled up the customer record or the customer order record
21:08
I might then
21:09
write and send a response back to the
21:10
customer. And I think with the information we pulled up
21:13
this third step is also
21:14
doable with an OM. If I give it
21:16
the option to
21:17
call an API to send an email
21:19
So this would be another example of taking
21:21
a task
21:22
of responding to
21:23
customer email and breaking it down into three
21:26
individual steps where I can look at each
21:28
of these steps and say
21:29
yep
21:29
I think an OM or an OM
21:31
with the ability to call a function
21:33
to query database or send an email
21:35
should be able to do that
21:36
Just one last example for the invoice processing
21:40
After a PDF invoice has been converted to text
21:42
the first step is to pull out the required information
21:46
the name of the biller
21:46
the address, the due date, the amount due
21:49
and so on. And now it should be
21:50
able to do that
21:51
And then
21:52
if I want to
21:53
check that
21:54
the information was extracted and save it in
21:56
a new database entry
21:57
then I think an
21:58
OM should be able to
21:59
help me call
22:00
a function to update the database record
22:02
And so to implement this
22:04
We'll implement an agentic workflow to
22:06
carry out basically these two steps
22:09
When building agentic workflows
22:10
I think of myself
22:12
as having a number of building blocks
22:14
One important building block would be
22:16
large language models or maybe large multimodal models
22:19
if
22:19
I want to
22:20
try the process images or audio as well
22:22
And OMs are good at generating text
22:24
deciding what to call
22:25
maybe extracting information
22:28
For some highly specialized tasks
22:29
I might also use
22:31
some other
22:32
AI models
22:33
such as AI model for converting a PDF
22:35
to text or for text to speech or
22:37
for image analysis. In addition to AI models
22:41
I also have access to a number of software tools
22:44
including different APIs that I can call to do voice search
22:48
to get maybe real-time weather data
22:51
to send emails, check calendar, and so on
22:54
And I might also have tools to retrieve information
22:57
to pull up data
22:58
from a database or
23:00
to implement
23:01
RAG or retrieval augmented generation where I can
23:03
look up Launch text database and find the most relevant text
23:06
or I might also
23:07
have tools to execute code
23:10
And this is a tool that lets an
23:11
LLM
23:12
write code and then run the code on
23:13
your computer to do
23:15
a huge range of things
23:17
In case some of these tools seem a
23:18
bit foreign to you
23:19
don't worry about it. We'll go through
23:21
the most important tools in much greater detail
23:23
in a later module. But
23:25
I think a lot of my work when
23:27
I'm building an agentic workflow as looking at
23:29
the work that the person or business is
23:31
doing and then trying to figure out with these building blocks
23:34
how can I sequence these building blocks together
23:36
in order to carry out the tasks
23:38
that I want my system to carry out
23:41
And this is why
23:42
having a good understanding of what building blocks are available
23:45
which I hope you
23:47
have a better sense of
23:48
by the end of this course as well
23:49
will allow you to better envision what agentic
23:52
workflows you can build by combining these building
23:54
blocks together
23:55
So to summarize
23:57
one of the key skills in building agentic
23:58
workflows is to look at
24:00
a bunch of stuff that maybe someone does
24:02
and to identify the discrete steps
24:05
that it could be implemented with
24:08
And when I'm looking at the individual discrete steps
24:11
one question I'm always asking myself is
24:13
can this step be implemented
24:15
with either an LM or with one of the tools
24:18
such as an API or a function call
24:20
that I have access to?
24:22
And in case the answer is no
24:23
I'll then often ask myself
24:25
how would I as a human
24:26
do this step?
24:27
And is it possible to decompose this further
24:30
break this down into even smaller steps
24:33
that then maybe
24:34
is more amenable to implementation
24:36
with an LM or with one of the
24:38
software tools that I have?
24:40
So
24:40
I hope this gives you a rough sense
24:42
of how to think about task decomposition
24:45
In case you feel like you don't fully have it yet
24:48
don't worry about it. We'll go through many
24:49
more examples in this course
24:51
and you have a
24:51
much better understanding of this
24:53
by the end of this course
24:54
But it turns out that as you build agentic workflows
24:58
you find that often you build an initial task decomposition
25:01
initial agentic workflow
25:03
and then You want to keep on
25:05
iterating and improving on it quite a few
25:07
times until it
25:08
delivers the level of performance that you want
25:11
And to drive this improvement process
25:13
which is found important for many projects
25:16
one of the key skills is to know
25:17
how to evaluate your agentic workflow
25:21
So in the next video
25:22
we'll talk about evaluations or
25:24
evals and just be a key component of
25:27
how you can build
25:28
and then also
25:29
keep on improving your workflows to get the
25:31
performance that you want
25:32
Let's talk about evals in the next video
25:36
I've worked with many different teams on building agentic workflows
25:39
and
25:40
I found that
25:41
one of the biggest predictors for whether someone
25:44
is able to do it really well
25:45
versus be
25:46
less efficient at it is
25:48
whether or not they're able to drive a
25:50
really disciplined evaluation process
25:53
So your ability to drive
25:55
evals for your agentic workflow
25:57
makes a huge difference in your ability
26:00
to build them effectively
26:02
In this video
26:03
we'll take a quick overview of how to build evals
26:06
and this is a subject that we'll actually
26:07
go into
26:08
much deeper
26:09
in a later module in this course
26:11
But let's take a look
26:13
After building an agentic workflow
26:15
like this one for responding to customer order inquiries
26:19
it turns out that it's very difficult to
26:21
know in advance
26:22
what are the things that could go wrong
26:24
And so rather than trying to build evaluations in advance
26:28
what I recommend is you just
26:29
look at the outputs and
26:31
manually look for
26:33
things that you wish it was doing better
26:36
For example
26:37
maybe you read a lot of outputs and
26:39
find that it is unexpectedly
26:41
mentioning your competitors more than it should
26:44
Many businesses don't want their
26:46
agents to mention competitors because it just creates
26:48
an awkward situation. And if you read some of his outputs
26:51
maybe you find that there's something that says
26:53
I'm glad you shopped with us
26:54
We're much better than our competitor
26:56
CompCo
26:57
Or maybe sometimes say
26:58
sure, we actually refund. Unlike RivalCo
27:00
we make returns easy. And you may look
27:02
at this and go
27:03
gee
27:03
I really don't want this to mention competitors
27:07
This would be the example of a problem
27:08
that
27:09
is really hard to anticipate
27:11
in advance of building this agentic
27:13
So the best practice is really to build
27:15
it first
27:15
and then examine it to figure out
27:18
where it is not yet satisfactory
27:20
and then to find ways to evaluate as
27:22
well as improve the system to eliminate the
27:24
ways that it is still not yet satisfactory
27:27
Assuming your business considers it an error
27:31
or mistake to mention competitors in this way
27:34
then as you work on eliminating these competitor mentions
27:38
one way to track progress would be to
27:39
add an evaluation or an eval
27:42
to track how often this error occurs
27:45
So
27:45
if you have a named list of competitors like comp co
27:48
rival co, the other co
27:50
then you can actually write code
27:51
So just search in your own output
27:54
for how often it mentions these competitors by
27:57
name and count up as a number
27:59
as a fraction of the overall responses
28:02
how frequently it mistakenly mentions competitors
28:06
One nice thing about
28:07
the problem of competitor mentions is is a objective metric
28:11
meaning either the competitor That was mentioned or not
28:14
and for objective criteria
28:16
you can write code to check
28:19
for how often this specific error occurs
28:22
But because LLM's output free text
28:25
there also going to be criteria
28:27
by which you want to evaluate this output
28:29
that may be more
28:30
subjective and where it's harder
28:32
to just write code to output a black
28:34
and white score. In this case
28:36
using a
28:37
LLM
28:38
as a judge is a common technique
28:40
to evaluate the output. So for example
28:42
if you're building a research agent
28:44
to do research on different topics
28:47
then you can use another LLM
28:49
and prompt it to maybe say
28:51
assign the following essay a quality score between one and five
28:54
where one is
28:55
the worst and five is the best essay
28:58
Here I'm using a Python expression to mean
29:01
you know
29:01
copy paste the generated essay into this
29:04
So
29:04
you can prompt the LLM to read the
29:05
essay and assign it a quality score
29:09
Then
29:09
I'm ask the research agent
29:10
to write a number of different research reports
29:13
For example
29:14
on recent developments in black hole science
29:17
or using robots to harvest fruit
29:19
And then in this example
29:21
maybe the judge OM
29:22
assigns the essay on black holes a score of three
29:26
the essay on robot harvesting a score of four
29:28
and as you work on improving your research agent
29:31
hopefully you
29:32
see these scores go up over time
29:34
It turns out, by the way
29:35
that
29:36
OMs are
29:36
actually not that good at these one to
29:39
five scale ratings. You can give it a
29:40
shot. Like Percy tend not to use this
29:42
technique that much myself
29:44
but in a later module
29:45
you learn
29:46
some better techniques to have an OM
29:48
output
29:49
more accurate scores than asking it to output
29:51
scores on a one to five scale
29:52
although some people will do this
29:54
maybe the initial cut
29:56
as an OM as judge type of eval
29:58
Just to give a preview of
30:00
some of the
30:01
agentive AI
30:02
evals that you'll learn about later in this
30:03
course
30:04
You've already heard me talk about how you
30:05
can write codes to evaluate objective criteria
30:09
such as did it mention the competitor or not
30:11
or use an OM as a judge for more Objective criteria
30:14
such as what's the quality of this essay
30:15
But later
30:16
you learn about two major types of evals
30:18
One is end-to-end
30:19
where you measure the output quality of the entire agent
30:23
as well as component level evals
30:25
where you might measure the quality of
30:27
the output of a single step in the
30:29
agentic workflow
30:30
And it turns out that these are useful
30:31
for driving different parts of your development process
30:34
And one thing I do
30:36
a lot as well is just examine the intermediate outputs
30:39
or sometimes you call these the traces of the LM
30:42
in order to understand where it is falling
30:45
short of my expectations. And we call this error analysis
30:48
where we just read through the intermediate outputs
30:50
of every single step to try to spot
30:52
opportunities for improvement
30:54
And it turns out being able to do
30:55
evals and error analysis is a really key
30:58
skill. So with much more
30:59
to say about this in
31:01
the fourth module in the schools
31:03
We're nearly to the end of this first
31:04
module
31:05
Before moving on
31:06
I just want to share with you
31:07
what I think are the most important design
31:09
patterns for building agentic workflows
31:12
Let's We'll take a look at that
31:14
in the next video
31:16
We build agentic workflows by taking building blocks
31:19
and putting them together
31:20
to sequence out these complex workflows
31:24
In this video
31:24
I'd like to share with you a few
31:25
of the key design patterns
31:27
which are patterns for how you can think
31:29
about
31:30
combining these building blocks into more complex workflows
31:34
Let's take a look
31:35
I think four key design patterns for building
31:37
agentic workflows are reflection
31:39
two-use planning, and multi-agent collaboration
31:42
Let me briefly go over what they mean
31:44
and then we'll actually go through
31:46
most of these in-depth later in this course
31:48
as well
31:49
The first of the
31:50
major design patterns is reflection
31:53
So
31:54
I'm a Google ML agent and ask it to write code
31:57
and it turns out that
31:58
an LM might then generate code like this
32:01
you know
32:01
defines here a Python function to do a
32:03
certain task. I could then construct a prompt
32:05
that looks like this. I can say
32:07
here's code intended for a certain task
32:09
and then copy paste whatever the LM had
32:11
just output back into this prompt
32:13
And then ask it
32:14
check the code carefully for correctness
32:15
style, and efficiency, and give constructive criticism
32:17
And turns out that the same LM model
32:19
prompted this way
32:20
may be able to point out some problems
32:23
with the code
32:24
And if I then take this
32:25
critique and feed it back to the model to say
32:29
looks like there's a bug
32:30
could you change the code to fix it?
32:32
Then it may
32:33
actually come with a better version
32:35
of the code. To give a preview of tool use
32:39
if you're able to run the code
32:40
and see where the code fails
32:43
then
32:43
feeding that back to the
32:45
LM can also cause it to
32:47
be able to iterate and generate a much better
32:50
say, V3 version 3 of the code
32:52
So
32:52
reflection is Common design pattern
32:54
where you can ask the arm to examine
32:56
its own outputs or maybe
32:58
bring in some external sources of
33:00
information such as
33:01
run the code and see if it generates
33:03
any error messages
33:04
and use that as feedback to
33:06
iterate again
33:07
and come over a better version of its
33:09
output. And this design pattern isn't magic
33:12
It does not result in
33:14
everything working 100% of the time
33:15
But sometimes it can be a
33:17
nice bump in the performance of your system
33:19
Now
33:19
I've drawn this as if it was a
33:21
single OM that I'm prompting
33:23
but to foreshadow multi-agent work those
33:26
you can also imagine
33:28
instead of having the same model critique itself
33:30
you can imagine having a critic agent and
33:33
all that is is an OM that's been
33:34
prompted with instructions like your role is to critique code
33:38
here's code intended for a task
33:40
check the code carefully and so on
33:41
And the second critic agent may be able
33:43
to point out errors or run unit tests
33:46
and like having two
33:48
simulator agents where each agent is just an
33:50
OM prompted to take on a certain persona
33:53
you can have that Go back and forth
33:54
to iterate to get a better output
33:57
In addition to reflection pattern
33:59
the second important design pattern is tool use
34:02
where today OLMs can be given tools
34:05
meaning
34:05
functions that they can call in order to
34:07
get work done. For example
34:09
if you ask an OLM
34:10
what's the best coffee maker according to reviewers
34:12
and you give it a web search tool
34:14
then it can actually
34:15
search the internet to find much better answers
34:18
Or a code execution tool
34:20
If you ask a math question
34:21
like if I invest $100 in a company interest
34:24
what have you
34:24
it can then write code and execute code
34:27
to compute an answer. Today
34:29
different developers have given OLMs many different tools
34:32
for everything from
34:33
math or data analysis to gather information by
34:36
fetching things from the web or for various databases
34:38
to interface with productivity apps like email calendar and so on
34:42
as well as
34:43
to process images and much more
34:45
And the ability of an OLM
34:47
to decide what tools to use
34:49
meaning what functions to call
34:51
that lets the model
34:52
get a lot more Done
34:54
The third of the four design patterns is
34:56
planning. This is an example from a paper called Hugging GPT
35:01
in which if you ask a
35:04
system to
35:05
please generate an image where a girl is reading a book
35:07
kind of pose is the same as a
35:08
boy in the image
35:09
then please describe the new image in your voice
35:11
then a model can automatically decide that
35:14
to carry out this task
35:15
it first needs to
35:17
find a pose determination model to figure out
35:19
the pose of the boy
35:20
then to pose the image
35:21
to generate a picture of a girl
35:23
and image the text
35:24
and then finally text the speech
35:26
And so in planning
35:27
an LLM decides
35:29
what is the sequence of actions it needs
35:31
to take. In this case
35:32
it is a sequence of API calls
35:35
so that it can then carry out the
35:37
right sequence of steps in the right order
35:40
in order to carry out the task
35:42
So rather than the developer hard coding the
35:44
sequence of steps in advance
35:46
this actually lets the LLM decide what are
35:48
the steps
35:49
to take. Agents that plan today are
35:52
harder to control and somewhat more experimental
35:54
but sometimes it can give really delightful results
35:57
And then finally, multi-agent workflows
36:00
Just as a human manager might hire a
36:02
number of others to work together
36:04
on a complex project. In some cases
36:07
it might make sense for you to hire
36:08
a set of multiple agents
36:10
maybe each of which specializes in a different role
36:13
and have them work together to accomplish
36:16
a complex task
36:17
The picture you see here on the left
36:18
is taken from a project called ChatDev
36:21
which is a
36:22
software framework created by Chen Qian and collaborators
36:25
In ChatDev, multiple agents with different roles
36:28
like chief executive officer
36:30
programmer, tester, designer, and so on
36:33
collaborate together
36:34
as if they were a virtual software company
36:37
and
36:38
can collaboratively complete a range of
36:40
software development tasks
36:42
Let's consider another example
36:44
If you want to write a marketing brochure
36:46
maybe you'd think of hiring a team of three people
36:49
such as a researcher to do online research
36:52
a marketer, To write the marketing text
36:54
and then finally an editor to
36:56
edit and polish the text
36:58
And so in a similar way
37:00
you might consider building a multi-agent
37:03
workflow in which you have a simulated research agent
37:06
a simulated marketer agent
37:08
and a simulated editor
37:10
agent that then
37:11
come together
37:12
to carry out this task for you
37:14
Multi-agent workflows are more difficult to control
37:18
since you don't always know ahead of time
37:20
what the agents will do
37:21
but research has shown that they can result
37:23
in better outcomes for many complex tasks
37:26
including things like writing biographies or deciding on
37:30
chess moves to make in a game
37:31
You learn more about multi-agentic workflows later in
37:34
this course as well
37:35
As so of that
37:36
I hope you have a sense of what
37:37
agentic workflows can do
37:40
as well as of what are the key
37:41
challenges of finding building blocks and putting them together
37:45
maybe by these design patterns
37:47
in order to implement an agentic workflow
37:50
and of course also developing evals
37:52
You can see how our system is
37:54
And keep on improving on it
37:56
In the next module
37:58
I'd like to share with you
38:00
a deep dive into the first of these
38:02
design patterns that is reflection
38:05
and you find that
38:06
is a maybe surprisingly simple to implement technique
38:10
that can give the performance of your system
38:12
sometimes a very nice bump
38:14
So
38:15
let's go on to the next module to
38:17
learn about the reflection design pattern
38:21
In this module
38:22
I'd like to share with you practical tips
38:23
for building agentic AI workflows
38:26
I hope that these tips
38:27
will enable you to be
38:28
much more effective than
38:30
the
38:30
typical developer at building these types of systems
38:34
I find that when developing an agentic AI system
38:37
it's difficult to know in advance
38:39
where it will work and where it won't work so well
38:42
and thus where you should focus your effort
38:45
So very common advice is to
38:47
try to build even a quick and dirty system to start
38:51
so you can then
38:52
try it out and look at it
38:53
to see where it
38:55
may not yet be working as well as you wish
38:57
to then have
38:58
much more focused efforts to develop it even further
39:02
In contrast, I find that it's sometimes less useful to
39:05
you know
39:06
sit around for too many weeks
39:07
theorizing and hypothesizing how to build it
39:10
It's often better to just
39:12
build a quick system in a safe
39:15
reasonable way that doesn't leak data
39:16
kind of do it in a responsible way
39:19
but just build something quickly so you can
39:21
look at it and then use that initial
39:22
prototype to prioritize and drive further development
39:26
Let's start an example of what might happen
39:29
after you've built a prototype
39:31
I want to use as a first example
39:33
the invoice processing workflow that you've seen previously
39:37
where the tasks extract four required fields and
39:39
then they save it to a database record
39:42
After having built such a system
39:45
one thing you might do is find a handful of invoices
39:48
maybe 10 or 20 invoices
39:50
and go through them and just take a
39:51
look at their output
39:52
and see what went well and if there
39:54
were any mistakes. So let's say you look through 20 invoices
39:57
you find that invoice one is fine
39:59
the output looks correct. For invoice two
40:01
maybe you're confused, the date of the invoice
40:04
that is when was the invoice issued with
40:05
the due date of the invoice
40:07
And in this task
40:08
we want to extract the due date so
40:10
that we can issue payments on time
40:11
So then I might note down in a
40:13
document or in a spreadsheet that for invoice two
40:16
the dates were mixed up
40:17
Maybe invoice D was fine
40:19
invoice four was fine
40:20
and so on. But as I go through this example
40:22
I find that there are quite a lot
40:23
of examples where I had
40:25
mixed up the dates
40:27
So
40:27
in a space on
40:28
going through a number of examples like this
40:30
that in this This case
40:32
you might conclude that
40:33
one common error mode is that it is
40:35
struggling with the dates. In that case
40:39
one thing you might consider would be to
40:41
of course
40:42
figure out how to improve your system to
40:44
make it extract due dates better
40:46
but also maybe write an eval
40:48
to measure the accuracy with which it is
40:50
extracting due dates. In comparison
40:53
if you had found that it was
40:55
extracting the bullet address incorrectly
40:58
who knows, maybe you have billers with unusual sounding names
41:01
and so maybe it struggles with billers
41:03
or especially if you have international billers whose
41:05
names may not even all be English letters
41:08
then you might instead focus on building an
41:10
eval for the bullet address
41:12
So one of the reasons why
41:13
building a quick and dirty system
41:15
and looking at the output is so helpful
41:17
is it even helps you decide
41:19
what do you want to put the most
41:20
effort into evaluating. Now
41:24
If you've decided that you want to
41:26
modify your system to
41:27
improve the accuracy
41:29
with which it is extracting the due date of the invoice
41:31
then to track progress
41:33
it might be a good idea to create an evaluation
41:36
an eval
41:37
to measure the accuracy of data extraction
41:40
There are probably multiple ways one might go about this
41:42
but let me share with you how
41:44
I might go about this
41:45
To create a test set or an evaluation set
41:48
I might
41:49
find 10 to 20 invoices and
41:51
manually write down what is the due date
41:54
So maybe one invoice is a due date of August 20th
41:56
2025, and I write it down at a standard year
41:59
month, date
42:01
format
42:02
And then to make it easier to evaluate in code later
42:06
I would probably write the prompt to the
42:07
alarm to tell it to always format the
42:09
due date in this year month date format
42:12
And with that
42:13
I can then write code to
42:14
extract out the one date that the alarm has output
42:17
which is the due date
42:18
That's the one day we care about
42:19
So this is a regular expression pattern matching
42:22
you know, four numbers for the year
42:23
two for the month
42:24
two for the date
42:25
and then extract that out
42:26
And then
42:27
I can
42:27
just write code to test if the extract
42:30
date is equal to the actual date
42:32
That is the ground truth annotation
42:34
I had written down. So with an eval set of
42:37
say, 20 or so invoices
42:39
I build up my changes to see if
42:41
the percentage of time
42:43
that it gets the extracted date correct is
42:45
hopefully going up as I
42:47
tweak my prompts or tweak other parts of
42:48
my system. So just to summarize what we've seen so far
42:52
we have built a system
42:53
then look at outputs to discover
42:55
where it may be behaving in an unsatisfactory way
42:58
such as due dates are wrong
42:59
Then to drive improve
43:01
So this important output put in place a
43:03
small eval with say just 20 examples to
43:05
help us track progress
43:07
And this lets me go back to two prompts
43:09
try different algorithms
43:10
and so on to see if I can
43:11
move up this metric of due date accuracy
43:15
So this is what improving a
43:17
agentic AI workflow will often feel like
43:19
Look at the output
43:20
see what's wrong, then if you know how to fix it
43:22
just fix it. But if you need a
43:23
longer process of improving it
43:24
then put in place an eval and use
43:26
that to drive further development
43:28
One other thing to consider is if after
43:30
working for a while
43:31
if you think those 20 examples you had
43:33
initially aren't good enough
43:34
maybe they don't cover all the cases you want
43:36
or maybe 20 examples is just too few
43:38
then you can always add to the eval
43:40
set over time to make sure it better
43:42
reflects
43:43
your personal judgments on whether or not the
43:45
system's performance is sufficiently satisfactory
43:48
This is just one example
43:49
For the second example
43:50
let's look at building a marketing copy assistant
43:53
for writing captions for Instagram
43:56
where to keep things succinct
43:57
let's say our marketing team tells us that
43:59
they want captions that are at most
44:01
words long
44:02
So we would have a image of a product
44:04
say a pair of sunglasses that we want to market
44:07
and then have a user query
44:08
like please write a caption to sell these sunglasses
44:12
and then have a
44:13
OM or a large multimodal model
44:15
analyze the image and the query
44:17
and generate a description of the sunglasses
44:20
And there are lots of different ways that
44:22
a marketing copy assistance may go wrong
44:25
but
44:25
let's say that you look at the output
44:27
and you find that the copy or the
44:28
text generated mostly sounds okay
44:30
but maybe it's just sometimes too long
44:32
So for the sunglasses input
44:33
generate 17 words. If you have a coffee machine
44:36
it's okay. Stylish eyes
44:37
okay. Blue shirt, 14 words
44:39
Blender, 11 words. So it looks like in this example
44:42
the OM is having a hard time adhering
44:45
to the length guideline. So again
44:47
there are lots of things that could have
44:48
gone wrong with the marketing copy assistant
44:51
but if you find that it's struggling with
44:52
the length of the output
44:54
then you might build an eval to track
44:56
this
44:57
so that you can
44:58
make improvements
44:59
and make sure it's getting better at it
45:02
To the length guideline
45:03
So to create an email to measure the text length
45:06
what you might do is
45:07
create a set of test stars
45:08
So mark the pair of sunglasses
45:09
a coffee machine, and so on
45:11
and maybe create 10 to 20 examples
45:14
Then
45:15
you would run each of them through your
45:16
system
45:17
and write code to measure the word count
45:19
of the
45:20
output. So this is Python code to measure
45:22
the word count of a piece of text
45:25
Then lastly
45:26
you would compare
45:26
the length of the generated text
45:29
to the 10 word target limit
45:32
So if word count is equal to 10
45:33
non-correct plus equals 1
45:35
One difference between this
45:37
and the previous invoice processing example is that
45:40
there is no per example ground truth
45:43
The target is just 10
45:45
same for every single example
45:46
Whereas in contrast, for the invoice processing example
45:49
we had to generate a custom target label
45:52
that is the correct due date of the invoice
45:55
and we're testing the outputs
45:56
against that per example ground truth
45:59
I know I use a very simple workflow
46:00
for Generating these captions
46:02
but
46:02
these type of evals can be applied to
46:04
much more complex generation workflows as well
46:07
Let me touch on one final example in
46:09
which we'll revisit the research agents
46:12
we've been looking at
46:13
If you look at the output of
46:15
the research agents on different input prompts
46:18
let's say that
46:19
when you ask it to write
46:21
an article on recent breakthroughs in black hole science
46:24
you find that it missed some high-profile result
46:27
and a loss of news coverage
46:28
So this is an unsatisfactory result
46:30
Or if you ask it to
46:31
research renting versus buying a home in Seattle
46:33
well, it seems to do a good job
46:35
Or robotics for harvesting fruit
46:37
well, it didn't mention a leading equipment company
46:40
So based on this evaluation
46:42
it looks like sometimes it misses a really
46:44
important point that a human expert writer
46:48
would have captured. So
46:50
then
46:50
I would create an eval to measure how
46:52
often it captures the most important points
46:55
For example
46:56
you might come up with a
46:57
number of example prompts on black holes
47:00
robotic harvesting, and so on
47:02
And for each one
47:02
come up with, let's say
47:04
three to five gold standard discussion points
47:06
for each of these topics
47:08
Notice that here, we do have a per example annotation
47:12
because the gold standard talking points
47:14
that is the most important talking points
47:16
they are different for each of these examples
47:18
With these ground truth annotations
47:20
you might then use an LMS judge to
47:22
count
47:23
how many of the gold standard talking points
47:26
were mentioned. And so an example prompt might be to say
47:29
determine how many of the five gold standard
47:31
talking points are present in the provided essay
47:34
Here are the original prompts
47:35
the essay text, gold standard points, and so on
47:37
and have it return a JSON object with
47:39
two keys that scores how many of the points
47:42
zero to five, did it score
47:43
as well as an explanation
47:45
And this allows you to get a
47:47
score for each prompt
47:49
in your evaluation set
47:51
In this example
47:52
I'm using O as a judge
47:54
to count how many of the
47:56
talking points were mentioned because there are so
47:58
many different ways to talk about these talking points
48:01
and so a
48:02
regular expression or a code for
48:04
simple pattern matching might not work that well
48:07
which is why you might use
48:08
O as a judge and treat this as
48:10
a slightly more subjective evaluation for whether or not
48:13
say, event horizons were adequately mentioned
48:16
So this is your third example
48:18
of how you might build evals
48:20
In order to
48:21
think about how to build evals for your application
48:25
the evals you build will often
48:27
have to reflect whatever
48:28
you
48:29
see
48:29
or you are worried about going wrong in
48:31
your application
48:33
And it turns out that broadly
48:35
there are two axes of evaluation
48:38
On the top axis is the way you
48:40
evaluate the output
48:43
In some cases
48:44
you evaluate it by writing code with objective evals
48:48
and
48:49
sometimes you use an OM
48:51
as a judge for more subjective evals
48:54
On the other axis
48:55
is whether you have a per example
48:58
ground truth or not
49:00
So for checking invoice data extraction
49:03
we were writing code to evaluate if we
49:05
got the actual date
49:07
and that had a per example ground truth
49:10
because each invoice has a different actual date
49:13
But in the example where we checked marketing copy length
49:16
every example had a length limit of 10
49:20
and so
49:20
there was no per example ground truth for
49:23
that problem
49:24
In contrast, for counting gold standard talking points
49:27
there was a per example ground truth because
49:30
each article had different important talking points
49:33
But we use an element as a judge
49:34
to read the essay to see if those
49:36
topics were adequately mentioned
49:38
because there's so many different ways to mention
49:40
the talking points
49:41
And
49:42
the last of the four quadrants would be
49:44
a message judge
49:45
with no per example ground truth
49:48
And
49:48
one place where we saw that was
49:50
if you are grading charts with a rubric
49:52
this is when we're looking at visualizing the
49:55
coffee machine sales. And if you ask it
49:57
to grade a chart according to a rubric
49:59
such as whether it's clear access labels and so on
50:02
there is the same rubric for every chart
50:05
And that would be using an element as a judge
50:07
but with validate per example ground truth
50:09
So I find this two by two grid
50:10
as maybe a useful way
50:12
to think about the different types
50:13
evals you might construct for your application
50:16
And by the way
50:17
these are sometimes also called end-to-end evals because
50:20
one end is the impedance
50:21
which is the user query prompt
50:23
and the other end is the final output
50:26
And so all of these are evals for
50:28
the entire end-to-end system's performance
50:31
So just to wrap up this video
50:33
I'd like to share a few final tips
50:35
for designing end-to-end evals. First
50:37
quick and dirty evals is fine to get
50:39
started
50:40
I feel like I see
50:42
quite a lot of teams that are almost
50:44
paralyzed because they think building evals is this massive multi-week effort
50:48
and so
50:49
they take longer than would be ideal to
50:51
get started
50:53
But
50:53
I think just as you
50:54
iterate on an agentic workflow and make it better over time
50:57
you should plan to iterate on your evals
51:00
as well. So if you put in place 10
51:02
15, 20 examples as your first cut at evals
51:06
and write some code
51:07
or try prompting them as a judge
51:08
just do something to start to get some metrics
51:11
and that can complement the human At looking at the output
51:14
and then it's a blend of the two
51:16
that can drive your decision making
51:17
And as the evals become more sophisticated over time
51:20
you can then
51:21
shift more and more of your trust to
51:22
the metric-based evals rather than needing to read over
51:26
you know
51:26
hundreds of
51:27
outputs every time you tweak a prompt somewhere
51:29
And as you go through this process
51:31
you'll likely find ways
51:33
to keep on improving your evals as well
51:35
So if you had 20 examples to start
51:38
you may then run into places where
51:40
your evals fail to capture your judgment
51:44
about what system is better
51:46
So maybe
51:47
you update the system and you look at
51:49
it and you feel like this has got
51:51
to be worked much better
51:52
but your eval
51:53
fails to show the new system achieving a
51:55
higher score. If that's the case
51:57
that's often an opportunity to go maybe
52:00
collect a larger eval set or change the
52:02
way you evaluate the output to make it
52:04
correspond better to your judgment as to
52:07
what system is actually working better
52:08
And so your evals will get better over time
52:11
And lastly
52:11
in terms of using evals To gain inspiration
52:14
as to what to work on next
52:16
a lot of agent workflows are being used
52:18
to automate tasks that
52:19
say, humans can do
52:21
And so I find for such applications
52:23
I'll look for places where the performance is
52:25
worse than that of an expert human
52:27
and that often gives me inspiration
52:29
for where to focus my efforts or what
52:31
are the types of examples that I may
52:33
relate get my agentic workflow
52:35
to work better than it is currently
52:37
So
52:38
I hope that after you've built that quick and dirty system
52:41
you'll think about when it would make sense
52:42
to
52:43
start putting in
52:44
some evals to track the potentially problematic aspects of the system
52:48
and that that will then help you
52:49
drive improvements to the system
52:52
In addition to helping you drive improvements
52:54
it turns out that
52:55
there's a method of evals that helps you
52:58
hone in Of your entire agentic system
53:01
what are the components most worth focusing
53:04
your attention on? Because agentic systems often have many pieces
53:08
so
53:09
which piece is going to be most productive
53:11
for you to spend time working to improve?
53:14
Turns out being able to do this well
53:16
is a really important skill for driving efficient
53:19
development of agentic workflows
53:21
In the next video
53:22
I'd like to deep dive into this topic
53:24
So let's go on to the next video
53:27
Let's say you built an agentic workflow
53:29
and it is not yet working as well
53:31
as you wish. And this happens to be all the time
53:33
by the way. I'll often build a quick and dirty system
53:36
and it doesn't do as well as
53:37
I wish it would
53:38
The question is
53:40
where do you focus your efforts to make
53:42
it better? It turns out agentic workflows have many different components
53:46
and working on some of the components could
53:48
be much more fruitful
53:49
than working on some other components
53:51
So your skill
53:52
at choosing where to focus your efforts
53:55
makes a huge difference in the speed with
53:57
which you can make improvements to your system
53:59
And I found that one of the
54:01
biggest predictors for how efficient and how good a team is
54:05
is whether or not
54:06
they're able to drive a disciplined
54:08
error analysis process to tell you
54:11
where to focus your efforts
54:12
So this is an important skill
54:14
Let's take a look
54:15
at how the error analysis
54:17
In the research agent example
54:20
we had carried out an error analysis in the previous video
54:23
and we saw that it was
54:25
often
54:25
missing key Points that a human expert would
54:28
have made in writing essays on certain topics
54:30
So
54:31
now that you've spotted this
54:33
problem that is sometimes missing key points
54:35
how do you know what to work on?
54:37
It turns out that of the many different
54:39
steps in this workflow
54:41
almost any of them could have contributed to
54:43
this problem of missing key points
54:46
For example
54:46
maybe the first LLM was generating search terms that weren't great
54:50
so
54:51
it was just searching for the wrong things
54:53
and did not discover the right articles
54:55
Or maybe use a web search engine that
54:57
just wasn't very good. There are multiple web
54:59
search engines out there. In fact
55:00
there are actually quite a few that I
55:01
tend to use for my own based applications
55:04
and some are better than others
55:06
Or maybe web search was just fine
55:09
but when we gave the list of web
55:10
search results to an LLM
55:11
maybe it didn't do a good job choosing
55:13
the best handful to download
55:16
Maybe web fetch has few problems in this case
55:19
assuming you can fetch web pages accurately
55:21
But after dumping the web pages in the LLM
55:23
maybe the LLM is
55:25
ignoring some of the points
55:26
In the documents we had fetched
55:28
So it turns out that there are teams
55:30
that will sometimes look at this and go
55:32
by gut to pick one of these components to work on
55:36
and sometimes that works
55:37
and sometimes that leads to many months of
55:39
work with very little progress in the
55:41
overall performance of the system
55:44
So rather than
55:45
going by gut to decide which of these
55:47
many components to work on
55:49
I think it's much better to
55:51
carry out an error analysis
55:53
to better understand each step in the workflow
55:56
And in particular, I'll often examine the traces
55:59
and that means the intermediate output after each step
56:02
in order to understand which component's performance is subpar
56:06
meaning, say, much worse than what a human expert would do
56:10
because that
56:11
points to where
56:13
there may be room for significant improvement
56:15
Let's look at an example
56:16
If we ask the research agent to write
56:18
an essay about recent news in black hole science
56:21
maybe the outputs search terms like these
56:24
search for black hole theory
56:25
science, event horizon. telescope radio
56:27
and so on
56:28
And I would then have a human expert
56:29
look at these and see
56:31
are these reasonable
56:32
web search terms for
56:34
writing about recent discoveries in black hole science
56:37
And maybe in this case
56:38
an expert says these web searches look okay
56:41
They're, you know
56:42
pretty similar to what
56:43
I would do as a human
56:45
Then I look at
56:46
the outputs of the
56:47
web search and look at the URLs returned
56:51
So web search will return many different web pages
56:54
and maybe
56:55
one web page returns is that an elementary
56:57
school student
56:58
claims to track a 30-year-old
57:00
black hole mystery from Astro Kid News
57:02
And
57:03
this doesn't look like the most rigorous peer
57:05
review article. And maybe examining
57:08
all of the articles that web search
57:10
returns causes you to conclude that it's returning
57:13
too many
57:14
blog or popular press types of articles and
57:17
not enough scientific articles to write a
57:19
research report of the quality that you are
57:22
looking for. It'd be good to
57:23
just
57:23
look through the outputs of the other steps
57:25
as well. Maybe the Find the best five sources you can
57:28
and then with AstroKit News
57:30
SpaceBot 2000, SpaceFun News, and so on
57:33
And it is by looking at these
57:35
intermediate outputs
57:36
that you can then try to get a
57:38
sense of the quality of the output of
57:39
each of these steps. To introduce some terminology
57:42
the
57:42
overall
57:43
set of outputs of all of the intermediate
57:45
steps is often called the trace of a
57:48
run of this agent
57:50
And then some terminology you see
57:51
in
57:52
other
57:52
sources as well is the output of a
57:54
single step is sometimes called a span
57:57
This is terminology from
57:59
the
57:59
computer observability literature where
58:02
people try to figure out what computers are
58:04
doing. And in this course
58:05
I use the word trace quite a bit
58:07
I'll use the word span a little bit less
58:09
but you may see both of these terms
58:11
on the internet
58:12
So by reading the traces
58:14
you start to get an informal sense of
58:16
where might be the most problematic components
58:19
In order to do this in the most systematic way
58:21
it turns out to be useful to focus
58:23
your attention
58:24
on the cases that the system is Doing
58:26
poorly on. Maybe write some essays just fine
58:29
and the output is completely satisfactory
58:31
So I would put those aside and try
58:32
to come up with a set of examples where
58:35
for whatever reason
58:36
the final output of your research agent is not quite satisfactory
58:39
and just focus on those examples
58:42
So just one of the reasons we call error analysis
58:44
because we want to focus on the cases
58:46
where the system made an error
58:48
and we want to go
58:49
through to figure out which components
58:51
were most responsible for the error
58:54
in the research agent output
58:56
In order to make this more rigorous
58:58
rather than reading and getting an informal sense
59:01
you might
59:01
actually build up a spreadsheet to more explicitly
59:05
count up where the errors are
59:07
And by error
59:08
I mean when a step outputs something
59:10
that
59:11
performs significantly worse than maybe what a human
59:14
expert would have given a similar
59:16
input as that component. So
59:18
I'll often do this myself in a spreadsheet
59:20
So I might build a spreadsheet like this
59:22
and so for the first query
59:23
I'll look at recent developments in black hole science
59:26
and I see That the search results has
59:28
too many blog posts
59:29
popular press articles, not enough scientific papers
59:32
And then based on this
59:34
it is true that the five best sources aren't great
59:37
but
59:37
here
59:38
I won't say that the five best sources
59:40
did a bad job because if the
59:41
inputs
59:42
to
59:43
the arm for selecting the five best sources
59:45
were all non-rigorous articles
59:47
then I can't blame this
59:48
picking the five best sources for
59:50
not picking better articles because it did the
59:52
best it could have or as
59:53
what did nearly as well as any human
59:55
might have given the same selection to choose
59:57
from. And then you might go through this for different prompts
60:00
renting versus buying in Seattle
60:03
maybe it missed a well-known blog
60:05
robotics for housing fruit
60:07
maybe in this case we look at it and say
60:08
oh
60:08
the search terms are too generic and the
60:10
search results also weren't good and so on
60:13
And then based on this
60:14
I would
60:14
count up in my spreadsheet
60:16
how often I observe errors
60:19
in the different components. So in this example
60:21
dissatisfied with the search terms
60:23
5% of the time
60:24
but I'm dissatisfied with the search results
60:26
5% of the time. And if I actually see this
60:29
I might just take a careful look at
60:30
the search terms to make sure that the
60:32
search terms really were okay and that poor
60:34
choice of search terms were not what led
60:36
to poor search results. But I really think
60:38
the search terms are fine
60:39
but the search results are not
60:40
Then I would take a careful look at
60:42
the web search engine I'm using and if
60:44
there are any parameters I can tune to
60:45
make it bring back more
60:47
relevant or higher quality results
60:49
There's this type of analysis that tells me
60:51
in this example
60:52
that maybe I really should focus
60:53
my attention on fixing the search results and
60:56
not on the other components of this
60:58
agentic workflow
61:00
So to wrap up this video
61:02
I find that it's useful to
61:03
develop a habit of looking at traces
61:06
After you build an agentic workflow
61:08
go ahead and look at the intermediate outputs
61:10
to get a feel for what it's actually
61:11
doing on every step so that you can
61:13
better understand if
61:14
different steps are performing better or worse
61:17
And a more systematic error analysis may be
61:19
done
61:20
with a spreadsheet can let you gather statistics
61:23
or count up which component performs poorly most
61:25
frequently. And so by looking at what components are doing poorly
61:29
as well as
61:30
where I have ideas for efficiently improving different components
61:35
then that will let you prioritize what component
61:37
to work on. So maybe a component is problematic
61:40
but I don't have any ideas for improving
61:42
it. So that would suggest
61:44
maybe not prioritizing that as high
61:45
But if there is a component
61:47
that is generating a lot of errors
61:49
and if I have ideas how to improve that
61:51
then that would be a good reason
61:52
to prioritize working on that component
61:54
And
61:55
I just want to emphasize that
61:57
error analysis
61:58
is a very helpful output
62:00
for you to decide
62:01
where to focus your efforts
62:02
Because in any complex system
62:05
there are just so many things you could
62:06
work on
62:07
It's too easy
62:08
to pick something to work on and work
62:10
on it for weeks or even months
62:11
only to discover later that that did not
62:14
result in improved performance to your overall system
62:17
And so using error analysis to decide where
62:19
to focus your effort turns out to be
62:20
incredibly useful for improving your efficiency
62:24
In this video, we went over error analysis
62:27
With the research agent example
62:30
But I think error analysis is such an
62:31
important topic. I want to go over some
62:33
additional examples with you
62:35
So let's go on to the next video
62:37
where we'll look at more examples of
62:39
error analysis
62:41
An agentic workflow may comprise many different types of components
62:45
and so your tools to improving different components
62:48
will be pretty different
62:49
But I'd like to share with you some
62:51
general patterns I've seen
62:53
Some components
62:53
your agentic workflow will be non-OLM based
62:56
So it may be
62:57
something like a web search engine or a text retrieval component
63:01
if that's part of your rack or retrieval augmented generating system
63:05
something for code execution
63:06
or maybe a separately trained machine learning model
63:09
maybe with
63:09
speech recognition or digital people and pictures and
63:12
so on. So
63:13
sometimes
63:14
these non-OLM based components will have parameters or
63:17
hyperparameters that you can tune
63:19
So for web search
63:19
you can tune things like the number of
63:21
results or maybe the date range that you
63:23
also web search engine to consider
63:25
For a rack text retrieval components
63:28
you might change the similarity threshold that determines
63:31
what pieces of text it considers similar or
63:33
the chunk size. Often rack systems will take texting
63:36
chop it up into smaller chunks for matching
63:39
So the main hyperparameters you could use Or for people detection
63:42
you might change the detection threshold
63:44
So how sensitive it is and how likely
63:46
it is to declare this found a person
63:48
and just to trade off the false positives
63:50
and false negatives. If they follow all the
63:52
details of the hyperparameters I just discussed
63:54
don't worry about it. The details aren't that important
63:56
but often the components were parameters that
63:58
you can tune. And then of course
63:59
you can also try to replace the component
64:02
I do this a lot in my agentic
64:04
workflows where
64:05
I'll swap in different rack search engines or
64:07
swap in different rack providers and so on
64:09
just to see if some other provider
64:11
might work better. Because of the diversity of non-OM based components
64:16
I think the techniques for how to improve
64:18
it will be
64:18
more diverse and dependent of exactly what that
64:21
component is doing. For an OM based component
64:25
here are some options you might consider
64:27
One would be
64:28
to try to improve your prompts
64:30
So
64:31
maybe try to add more explicit instructions
64:33
or if you know what few shot prompting is
64:35
that refers to
64:36
adding
64:37
one or more concrete examples of an example
64:40
of an input. The design output
64:42
and so few-shot prompting
64:43
which you can learn about from some deep
64:45
learning AI short courses as well
64:47
is a technique that can
64:48
give your LM some examples to hopefully help
64:51
it get better performing outputs written
64:54
Or you can also try a different LM
64:57
So with ASV or other tools
64:59
it could be pretty easy to try multiple LMs
65:02
and then you can use evals
65:03
to pick the best model for your application
65:06
Sometimes
65:07
if a single step is too complex for
65:09
one LM to do
65:10
you can consider if you want to decompose
65:12
the task into smaller steps
65:13
or maybe decompose it into a generation step
65:16
and then a reflection step
65:17
But more generally, if you have instructions that are very complex
65:21
all within one step
65:22
maybe a single LM has a hard time
65:24
following all those instructions
65:25
and if you break the task down to smaller steps
65:27
that may be easier for
65:29
say
65:30
two or three calls to an LM
65:31
to carry out accurately. And lastly
65:34
something to try when the other methods aren't
65:36
working well enough is to consider fine-tuning
65:39
a model. This tends to be
65:41
quite a bit more complex than the other options
65:43
so it can be quite a bit more
65:44
expensive as well in terms of developer time
65:46
to implement. But if you have some data
65:49
that you can use to fine-tune an LLM on
65:53
that could
65:54
give you much better performance than prompting alone
65:57
So
65:57
I tend not to fine-tune a model until
65:59
I've really exhausted the other options
66:01
because fine-tuning tends to be quite complex
66:04
But for applications where after trying everything else
66:07
if I'm still at
66:08
you know, say 90% performance or 95% performance
66:12
and I really need to eke out those
66:13
last few percentage points of improvement
66:15
then sometimes fine-tuning
66:17
my own custom model is a great technique
66:19
to use
66:19
I tend to do this only on the
66:21
more mature
66:22
applications because of how costly it is
66:25
It turns out that when you are
66:27
trying to choose an LLM to use
66:29
one thing that's very hopeful
66:30
for
66:31
you as a developer is if you have
66:32
good intuitions about how intelligent or how capable
66:35
different large language models are
66:37
One thing you can do is just try
66:38
a lot of models and see what works
66:40
best. But I find that As I work with different models
66:42
I start to hone intuitions about
66:44
which models work best for what types of
66:46
tasks. And we hone those intuitions
66:48
you can be
66:49
more efficient as well in
66:50
writing good prompts for the model as well
66:52
as choosing good models for your tasks
66:55
So I'd like to share with you
66:56
some
66:57
thoughts on how to hone your intuition
66:59
on
67:00
what models will work well for your application
67:03
Let's illustrate this with an example
67:05
of using an LLM
67:06
to follow instructions to remove or to redact
67:09
PII or personally identifiable information
67:12
So get down to remove private sensitive information
67:15
For example
67:16
if you are using an LLM to summarize customer calls
67:20
then maybe one summary is on July 14th
67:23
2023
67:24
Jessica Alvarez with a social security number and a certain address
67:27
a support ticket, and so on
67:29
So this piece of text has a lot of sensitive
67:32
personally identifiable information. Now
67:35
let's say we want to remove all PII
67:37
from
67:37
such
67:38
summaries because we want to use the data
67:40
for Downstream Cisco analysis of what customers are calling about
67:44
and to protect customer information
67:45
we want to strip out that PII before
67:47
we do that downstream Cisco analysis
67:50
So you might prompt an alarm with
67:52
instructions to identify all cases of PII in the text below
67:56
and then return the redacted text with redacted
67:59
colon and so on
68:01
It turns out
68:02
that
68:03
the larger frontier models tend to be much
68:06
better at following instructions
68:08
whereas the smaller models
68:10
tend to be pretty good at answering simple factual questions
68:13
but are just not as good at following
68:15
instructions. If you run this prompt on a smaller model
68:18
the OpenWay Llama 3.1 model with A billion parameters
68:21
then it may generate an output like this
68:23
It says, the identified PII is social security number and address
68:27
and then it redacts it as follows
68:29
and so on. And it actually makes a
68:31
few errors. It didn't follow the instructions properly
68:34
It showed the list
68:35
then redacted the text
68:36
then returned another list
68:38
which it wasn't supposed to
68:40
and in this list of PII
68:42
it missed the name
68:43
and then I think it also
68:44
didn't redact part of the address
68:46
So details aren't important
68:47
but didn't follow these instructions perfectly
68:49
and maybe it missed a little bit of PII
68:52
In contrast, if you use a more intelligent model
68:54
one that's better at following instructions
68:56
you may get a better result like this
68:58
where it's actually correctly listed all the PII
69:01
Correctly redacted all of the PII
69:04
And so
69:05
I find that as different OM providers specialize on different tasks
69:09
different models really are better for different tasks
69:12
Some are better at coding
69:13
some are better at following instructions
69:15
some are better at certain niche types of facts
69:18
and we can hone your intuition for
69:20
what models
69:21
are more or less intelligent
69:23
and what type of instructions they're more or
69:25
less able to follow
69:26
then you'll be able to make better decisions
69:28
as to what models to use
69:30
So to share a couple tips on how to do this
69:32
I encourage you to play with different models
69:34
often. So whenever there's a new model release
69:37
I'll often go try it out and try
69:39
out different queries on it
69:40
both closed-weight proprietary models as well as open-weight
69:43
models. And
69:45
I find that sometimes having a personal set
69:46
of evals might also be helpful where there's
69:49
a set of things you ask a lot
69:50
of different models that might help you calibrate
69:52
how well they do
69:53
on different types of tasks
69:56
One of the things that I do a
69:57
lot that I hope will be useful to
69:59
you is
70:00
I spend a lot of time reading other
70:01
people's prompts. So sometimes people publish their prompts on the internet
70:06
and I often go and read them to
70:08
understand what best practices in prompting looks like
70:11
or I often chat to my friends at various companies
70:14
including some of the frontier model companies
70:16
and kind of share my prompts with them
70:17
take a look at how they prompt
70:19
And sometimes I'll also go to open source packages of
70:23
you know
70:23
written by people I really respect
70:25
and download the open source package and dig
70:27
through that open source package
70:29
to find the prompts the authors have written
70:31
in order to read it
70:32
in order to hone my intuition about how
70:34
to
70:35
write good prompts
70:36
This is one technique that I encourage you
70:38
to consider
70:39
is by reading lots of other people's prompts
70:41
that will help you get better at writing prompts yourself
70:44
And, I certainly do this a lot
70:46
and I encourage you to do so too
70:48
And this will hone your intuition about
70:51
what types of instructions models are good at
70:53
following and
70:54
when to say certain things to different models
70:56
In addition to playing with models and reading other people's prompts
70:59
if you try out lots of different models
71:01
in your agentic workflows
71:03
that also lets you hone your intuition
71:05
So you see
71:06
which models work best for which types of tasks
71:09
and either looking at traces to
71:11
get it in a formal sense
71:13
or looking at either component-wise
71:15
or end-to-end evals can help you assess how
71:17
well different models are working
71:19
different policy workflow. And then you start to
71:21
hone intuitions about not just performance
71:23
but maybe also
71:25
price and speed trade-offs
71:27
for the use of different models
71:28
And
71:29
one of the reasons I tend to develop
71:31
my agentic workflows of AI Suite is because
71:33
it then makes it easy to quickly swap
71:35
out and try out different models
71:36
And this makes me more efficient in terms
71:38
of trying out and assessing
71:41
which models work best for my workflow
71:43
So we've talked a lot about how to
71:45
improve the performance of different components
71:49
So hopefully improve the overall performance of your
71:52
end-to-end system
71:53
In addition to improving the quality of the output
71:57
one of the thing you might want to
71:58
do in your workflows
71:59
is to optimize the latency
72:01
as well as cost
72:03
I find that for a lot of teams
72:04
when you start developing
72:05
usually the number one thing to worry about
72:07
is just are the outputs sufficiently high quality
72:11
But then
72:11
when the
72:12
system is working well and you put it into production
72:15
then there's often value to make it run
72:17
faster
72:18
as well as run at lower cost as
72:19
well
72:20
So in the next video
72:21
let's take a look
72:22
at some ideas for improving
72:24
cost and latency for agentic workflows
72:28
When building agentic workflows
72:30
I often advise teams to focus on getting
72:33
high quality outputs
72:34
and to optimize cost and latency only later
72:37
It's not that cost and latency don't matter
72:40
but I think
72:41
getting the performance or the
72:43
output quality to be high is usually
72:45
the hardest part
72:47
And then only when it's really working
72:49
then maybe focus on the other things
72:51
One thing that's happened to me a few
72:52
times was
72:53
my team built an agentic workflow and we
72:55
shipped to the users
72:57
and then we're fortunate enough to have so
72:59
many users use it
73:00
that the cost actually became a problem
73:02
And then we had to
73:03
you know
73:03
scramble to bring the cost back down
73:06
But that's a good problem to have
73:07
So I tend to
73:08
worry about cost usually last
73:11
not that I ignore it completely
73:13
but it's just lower down my list of
73:14
things to worry about until we have so
73:16
many users that we really
73:17
need to bring the cost down per user
73:20
And then latency, I tend to worry a bit about it
73:23
but again
73:24
not as much as just
73:25
making sure the output quality is high
73:28
But when you do get there
73:29
it will be useful to have tools to
73:31
optimize latency and cost
73:33
Let's take a look at some ideas
73:34
on how to do that
73:35
If you want to optimize the latency of energetic workflow
73:40
one thing I will often do
73:41
is then Benchmark or time the workflow
73:45
So in this research agent
73:47
it takes a number of steps
73:49
and if I were to time each of the steps
73:51
maybe it
73:52
takes seven seconds to generate the search terms
73:55
Web search takes five seconds
73:56
this takes three seconds
73:57
this takes eleven seconds
73:58
and then writing the final essay takes eighteen
74:00
seconds on average
74:02
And it is then by looking at this
74:03
overall timeline that I can get a sense
74:06
of
74:07
which components have the most room for
74:10
making faster
74:12
In this example
74:13
there are maybe multiple things you could try
74:15
If you haven't already taken advantage of parallelism for some steps
74:18
like maybe wet fetch
74:20
maybe it's worth considering
74:22
doing some of these operations in parallel
74:25
Or if you find that some of the
74:26
LM sets are taking too long
74:28
so if this first step takes seven seconds
74:30
this last LM step takes eighteen seconds
74:33
I might also consider trying a smaller
74:35
maybe
74:36
slightly less intelligent model to see if it
74:38
still works well enough for that
74:39
or if I can find a faster LM
74:41
provider. There are lots of APIs online for different
74:44
interfaces
74:45
and some companies have specialized hardware to allow
74:48
them to serve certain elements much faster
74:51
So sometimes it's worth trying different element providers
74:53
to see which ones can return tokens
74:56
the fastest
74:57
But at least doing this type of timing
74:59
analysis can give you a sense of which
75:01
components to focus on in terms of
75:03
reducing latency
75:05
In terms of optimizing costs
75:07
a similar calculation where you calculate the cost
75:10
of each step would also let you benchmark
75:12
and decide which steps to focus on
75:14
Many element providers charge per
75:16
token based on the input and output length
75:18
Many API providers charge per API call
75:22
and the computational steps
75:24
may have different costs based on how you
75:25
pay for server capacity and how much the
75:27
service costs. And so for a process like this
75:31
you might decide in this example that the
75:33
tokens for this element step
75:35
on average cost 0.004 cents
75:38
Each web search API maybe cost 1.6 cents
75:41
Tokens cost This much API call cost this much
75:44
PDF to text cost this much
75:45
tokens for the final essay generation cost this much
75:48
and this would maybe
75:48
again give you a sense of are there
75:50
cheaper components you could use or cheaper arms
75:53
you could use to see where the biggest
75:55
opportunity is for optimizing costs
75:57
And I found that these benchmarking exercises can
76:00
be very clarifying and sometimes they'll clearly tell
76:02
me that certain components are just not worth
76:05
worrying about because they're not that material
76:07
or contributor to either cost or to latency
76:10
So
76:11
I find that wheneither cost or latency becomes
76:13
an issue
76:14
by simply measuring the cost
76:17
and or latency of each step
76:18
that often gives you a basis
76:20
with which to decide which components to focus
76:23
on optimizing
76:25
So we're nearly at the end of this
76:27
module. I know we've covered a lot
76:29
but thank you for sticking with me
76:30
Let's go on to the final video of
76:32
this module to wrap up
76:34
We've gone through a lot of tips for driving a disciplined
76:38
efficient process for building agentic AI systems
76:41
I'd love to wrap up by
76:43
sharing with you
76:44
what it feels like to be going through
76:45
this process
76:47
When I'm building these workflows
76:49
I feel like there are two major activities
76:51
I'm often spending time on
76:52
One is building, so writing software
76:54
trying to write code to improve my system
76:56
And the second, which sometimes doesn't feel like progress
77:00
but I think is equally important
77:02
is analysis to help me decide where to
77:05
focus my build efforts next
77:06
And I often go back and forth between building and analyzing
77:10
including things like error analysis
77:12
So for example, when building a new agentic workflow
77:15
I'll often start by quickly building an end-to-end system
77:19
maybe even a quick and dirty implementation
77:21
And this lets me then start to
77:23
examine the final outputs of the end-to-end system
77:25
or also
77:26
read through traces to get a sense of
77:28
where it's doing well
77:29
where it's doing poorly. Based on even just locating traces
77:32
sometimes this will give me Got sense
77:35
of which individual components I might want to
77:38
improve. And so I might go tune some
77:39
individual components or keep tuning
77:42
the overall end-to-end system
77:44
As my system starts to mature a little bit more
77:46
then beyond just
77:48
manually examining a few outputs and reading through traces
77:51
I might start to build evals
77:53
and have a small data set
77:54
maybe just 10, 20 examples to compute metrics
77:57
at least on end-to-end performance
77:59
And this then further
78:01
helps me have a more refined perspective
78:04
on how to improve the end-to-end system or
78:06
how to improve individual components
78:07
As it matures even further
78:09
my analysis then becomes maybe even more disciplined
78:12
where I'll start to do error analysis
78:14
and look through the components and try to
78:16
count up how frequently individual components
78:19
led to
78:20
subpar outputs
78:21
And this more rigorous analysis then lets me
78:23
be
78:24
even more focused in deciding what components to
78:26
work on next or inspire ideas for improving
78:29
the overall end-to-end system. And then eventually
78:31
when it's even more mature
78:32
to drive more efficient Improvements at the component level
78:36
that's when I might also build component level
78:39
evals. And so
78:40
the workflow of building an agentic system often
78:42
goes back and forth. It's not a linear
78:44
process. We sometimes tune the engine system
78:47
then do some error analysis
78:48
then improve a component for a bit
78:50
then tune the component level evals
78:52
and
78:52
I tend to bounce back and forth between
78:54
these two types of techniques
78:56
And what I see
78:57
less experienced teams often do is
78:59
spend a lot of time building
79:02
and probably much less time analyzing with error analysis
79:05
building evals, and so on
79:06
that would be
79:07
ideal because it's this analysis that helps you
79:09
really focus where to spend your time building
79:13
And just one more tip
79:14
there are actually
79:15
quite a few tools out there to help with monitoring traces
79:19
logging runtime, computing costs, and so on
79:21
And those tools can be helpful
79:23
I sometimes use a few of them
79:24
and
79:25
quite a few of deep-endized short course partners offer those tools
79:29
and they do work well
79:30
I find that for agentic workflows I end up working on
79:33
most agentic Workflows are pretty custom
79:36
and so
79:36
I end up building pretty
79:38
custom evals myself
79:40
because I want to capture the things that
79:42
work incorrectly with my system
79:44
So even though I do use some of those tools
79:46
I also end up building a lot of
79:47
custom evals
79:49
that are well fit to my specific application
79:52
and the issues I see with it
79:54
So
79:54
thanks for
79:55
sticking with me
79:55
this far to the end of the
79:57
fourth of five modules
80:00
If you're able to implement
80:02
even a fraction of the ideas from this module
80:06
I think you'll be well ahead of
80:08
the vast majority of developers in terms of
80:10
your sophistication at implementing agentic workflows
80:14
Hope you found these materials useful
80:16
and I look forward to seeing you in
80:18
the final module
80:19
We'll talk about some
80:20
more advanced design patterns for building
80:23
highly autonomous agents
80:25
I'll see you in the last module of
80:27
this course
80:29
Welcome to this final module where you learn
80:31
about design patterns that let you build highly autonomous agents
80:36
where you don't need to hard code in
80:38
advance the sequence of steps to take
80:40
but it can
80:41
be more flexible and decide for itself what
80:43
steps it wants to take
80:44
to accomplish a task
80:46
We'll talk about the planning design pattern
80:48
and then later in this module
80:49
how to build multi-agent systems
80:51
Let's dive in
80:53
Suppose you run a sunglasses
80:54
retail store and have information on
80:58
what sunglasses are in your inventory
81:00
stored in a database
81:01
You might want a
81:02
customer service agent to be able to answer questions like
81:05
do you have any round sunglasses in stock
81:07
that are under $100? This is a fairly
81:09
complex query because you have to look through
81:12
the product descriptions to see what sunglasses are around
81:15
then look at what is in stock
81:17
and then finally see what's under $100
81:20
in order to tell the customer
81:22
yes, we have classic sunglasses
81:24
So how do you build an agent
81:26
to answer a broad range
81:28
Customer queries like this and many others
81:31
In order to do so
81:32
we're going to give the LLM
81:34
a set of tools to let it get item descriptions
81:37
such as look up if different glasses are round
81:40
check inventory, maybe process item returns
81:43
which is not needed for this query
81:45
but may need it for other queries
81:46
get item price, check past transactions, process item sale, and so on
81:52
In order to
81:53
let an LLM
81:54
figure out what's the right sequence of tools
81:57
to use to respond to the customer request
82:00
you might then write a prompt like this
82:02
You have access to the following tools
82:04
and they give it a description of each of the
82:06
say, six tools or even more tools that the LLM has
82:10
and to then tell it to
82:11
return a step-by-step plan to carry out the
82:14
user's request
82:15
In this case, to answer this particular query
82:18
a reasonable plan that an LLM might output
82:21
might be
82:21
to first use get item descriptions
82:24
to check the different descriptions to find the round sunglasses
82:28
and Then use check inventory to see if they're in stock
82:30
and use get item price to see if
82:32
the in stock results are less than $100
82:35
After an LM outputs this plan with three steps
82:39
what we can do is then take the step one text
82:43
that is this text written here in red
82:45
and pass that to an LM
82:47
maybe with additional context about what are the tools
82:49
with their user query
82:51
with additional background context
82:53
and toss an LM to carry out
82:55
step one. And in this case
82:57
hopefully the LM will
82:58
choose to call the get item descriptions
83:00
to get the appropriate descriptions of items
83:03
And the output of that first step
83:05
can let it select which are the round sunglasses
83:09
and that
83:09
output of step one
83:11
is then passed together with the step two instructions
83:14
that would be these instructions I have here in blue
83:16
to an LM to then execute the second
83:18
step of the plan. Hopefully
83:20
it will then take the
83:21
two pairs of round sunglasses we found on
83:23
the previous slide and check the inventory
83:26
and the output
83:27
of that second step Is then used to another OM call
83:31
where you have the output of the second step
83:33
as well as
83:33
the instructions of what to do for step three
83:36
past the OM
83:37
to have it get the item price
83:38
And finally
83:39
this output is fed back to the OM
83:41
one last time to generate the final answer
83:44
for the user. In this slide
83:46
I've simplified a lot of details a little
83:48
bit
83:48
The actual plan typically written by the OM
83:50
is more detailed than these simple one-line instructions
83:53
but the basic workflow is to have an
83:55
OM write out multiple steps of a plan
83:57
and then task it
83:58
to execute each step of the plan
84:01
in turn
84:02
with some appropriate surrounding context about what is the task
84:05
what are the tools available
84:06
and so on. And the exciting thing
84:09
about using an OM to plan this way
84:11
is that
84:12
we did not have to decide in advance
84:14
what is the sequence in which
84:16
to call tools in order to answer a
84:18
fairly complex customer request
84:21
If a customer were to make a different request
84:24
such as
84:25
I would like to return the
84:26
gold frame glasses that
84:28
I purchase but not the metal frame ones
84:31
Then you can imagine an OM
84:33
similarly being able to come up with a
84:35
different plan
84:36
to figure out based on what they have purchased previously
84:39
which glasses they had bought based on get item descriptions
84:42
whether gold frame ones they want to return
84:44
and then maybe call process item return
84:46
So with an agent that can plan like this
84:48
it can carry out a much wider
84:50
range of tasks
84:51
that can require calling many different tools and
84:54
many different orders. One more example of planning
84:57
let's take a look at an email assistant
84:59
If you want to be able to tell
85:00
your assistant to please reply to that email
85:02
invitation from Bob Molden
85:03
New York
85:04
tell him attend and archive his email
85:05
Then an email assistant may be given tools like this
85:09
to search email, move an email, delete an email
85:11
and send an email
85:12
And you might write a system prompt saying
85:14
you have access to following tools
85:15
and again, please return the step-by-step plan
85:17
In this case
85:18
maybe the OM will
85:19
say
85:20
the steps for this are to use search
85:21
email to find the email from Bob that
85:24
mentioned dinner and New York
85:26
and then
85:26
generate and send an email to confirm And
85:28
then send the last email to the archive
85:31
folder. Given this plan
85:32
which looks a reasonable one
85:34
you would then again
85:35
task an LM step by step
85:37
to carry out this plan
85:38
So the text from the first step
85:40
shown here in red
85:41
would be fed to the LM with additional background context
85:44
and hopefully it triggers search email
85:46
Then the output of that
85:48
can be given to
85:49
an LM again with the step two instructions
85:52
to send a appropriate response
85:55
And then finally, assuming the email was sent successfully
85:57
you can take that output
85:58
and have the LM execute the third step
86:01
of moving the email
86:02
from Bob into the archive folder
86:05
The planning design pattern is already used successfully
86:08
in many highly agentic coding systems
86:11
where if you ask it to write a piece of software
86:13
to build some fairly complex application
86:16
it might actually come to a plan to build this component
86:19
build this component, build this component, to almost form a checklist
86:22
and then do those steps one at a
86:24
time to build decently complex piece of software
86:28
For many other applications
86:29
the use of planning is still maybe more experimental
86:33
is not in very widespread use
86:35
And one of the challenges of planning is
86:38
it makes the system sometimes
86:39
a little bit harder to control because you as a developer
86:42
you don't really know
86:43
at runtime what plan it will come up
86:45
with. And so I think
86:47
also highly agentic coding systems where it actually works really well
86:51
adoption of planning is still growing in other sectors
86:55
but this is exciting technology
86:56
and
86:57
I think it will keep getting better and
86:58
we'll see it in more and more applications
87:01
The cool thing about
87:02
building agents that can plan for themselves is
87:04
you don't need to hard code in advance
87:06
the exact sequence of steps an LLM
87:08
may take to carry out a complex task
87:11
Now, I know that in this video
87:13
I've gone over the planning process at a fairly high level
87:16
where we're
87:16
outputting a list of steps and then tasking
87:18
an LLM
87:19
to
87:20
carry out the steps of the plan
87:22
one step at a time
87:23
But how does this actually work?
87:25
In the next video
87:26
we'll take a deeper dive to look
87:28
further into the guts of
87:30
what these plans actually look like and how
87:32
to string it together
87:34
to have an LLM plan and execute a
87:35
plan
87:36
for you
87:37
Let's take a look at that in the
87:38
next video
87:40
In this video
87:40
we'll look in detail at how to prompt
87:42
an LLM to generate a plan and then how to read
87:45
interpret, and execute that plan
87:47
Let's dive in
87:48
This is a plan that you saw in
87:50
the previous video for the customer service agents
87:53
and I have presented this plan at a
87:55
high level using simple text descriptions
87:57
Let's take a look at how you can
87:59
get an arm to write
88:01
very clear plans that go a little bit
88:03
beyond these simple high-level text descriptions
88:06
It turns out that
88:08
many developers
88:09
will ask an arm to format the plan
88:12
it wants executed in JSON format
88:14
because this allows downstream code
88:16
to parse what exactly are the steps of
88:18
the plan in a relatively clear and unambiguous ways
88:21
and
88:22
all of the leading arms are pretty good
88:24
at generating JSON outputs at this point
88:26
So the system prompt might say something like this
88:29
you have access to file tools
88:30
distribution tools, and then create a step-by-step plan in JSON format
88:34
and
88:34
you might describe the JSON format in enough
88:37
detail with the goal of getting it to
88:39
output a plan
88:40
like that shown here on the right
88:42
So in this JSON output
88:43
it creates a
88:44
list
88:45
where the first list item has clear keys
88:48
and values. Values that say
88:50
the step one of the plan
88:51
has
88:52
following description
88:53
and it should use the following tool with
88:55
the following arguments passed to that tool
88:58
Then after that
88:58
step two of the plan
89:00
is to
89:00
carry out this task and then to use
89:02
this tool and so on
89:03
So this JSON format
89:05
as opposed to writing the plan in English
89:08
allows downstream code to more clearly parse out
89:11
exactly
89:12
what are the steps of the plan
89:13
so that it can be reliably executed
89:15
one step at a time
89:17
Instead of JSON
89:18
I also see
89:19
some developers use XML where you can
89:22
use XML delimiters or use XML tags to
89:25
clearly specify
89:27
what are the steps of the plan and
89:28
what step number it is
89:30
Some developers, I feel like fewer developers
89:32
will use Markdown
89:33
which is
89:34
just sometimes slightly more ambiguous
89:36
in terms of how we parse it
89:38
And I think plain text is maybe the
89:39
least reliable of these options
89:41
But I think either JSON
89:42
which I'm showing here
89:43
or XML
89:44
would be good options for how to ask
89:46
the LM to format a plan
89:48
Unambiguously
89:50
So that's it. By opening plans in JSON
89:53
you can then
89:54
pause it and have
89:55
downstream workflows execute different steps of the plan more systematically
90:00
Now, in terms of giving elements a plan
90:03
it turns out there's one other
90:05
really neat idea
90:06
that lets an element output very complex plans
90:08
and get them executed reliably
90:11
and that's allow them
90:12
write code and to have code express
90:14
the plan
90:15
Let's take a look at this
90:16
in the next video
90:19
We've talked a lot about how to build
90:20
a single agent to complete tasks for you
90:23
In a multi-agent or multi-agentic workflow
90:26
we instead have a collection of multiple agents
90:28
collaborate to do things for you
90:31
When some people hear for the first time about multi-agent systems
90:34
they wonder, why do I need multiple agents?
90:37
It's just the same OM that I'm prompting over and over
90:40
or just one computer
90:41
Why do I need multiple agents?
90:44
I find that
90:45
one useful analogy is
90:47
even though I may do things on a single computer
90:50
we do decompose work
90:51
in a single computer into
90:53
maybe multiple processes or multiple threads
90:56
And as a developer
90:57
thinking, even though it's, well, one CPU on a computer
91:00
say
91:01
thinking about how to take work and decompose
91:03
it into multiple processes or multiple computer programs to run
91:07
that makes it easier
91:08
for me as a developer to write code
91:11
And in a similar way too
91:13
if you have a complex task to carry out
91:16
sometimes instead of thinking about how to
91:19
hire one person to do it for you
91:21
you might think about hiring a team of
91:23
a few people
91:24
to do different pieces of the task for
91:26
you
91:27
And so in practice
91:28
I found that for many developers of agentic systems
91:31
having this mental framework of not asking
91:34
what's the one person I might hire to do something
91:36
but instead
91:37
would it make sense to hire people with
91:39
three or four different roles to do this
91:41
overall task for me?
91:42
That holds
91:43
give another way
91:45
to take a
91:45
complex thing and decompose it into subtasks and
91:48
to build for those individual subtasks
91:50
one at a time
91:51
Let's take a look at some examples of
91:53
how this works
91:54
Take the task of creating marketing assets
91:57
Say you want to market sunglasses
91:59
Can you
92:00
come up with a marketing brochure for that?
92:02
You might need a researcher on your team
92:04
to look at trends on sunglasses and what
92:06
competitors are offering. You might also
92:09
have a graphic designer on your team to
92:11
render
92:12
charts or nice looking graphics of your sunglasses
92:14
And then also a writer to take the research
92:17
take the graphic assets and put it all
92:19
together into a nice looking brochure
92:21
Or to write a research article
92:22
you might want a researcher to do online research
92:25
a statistician to calculate statistics
92:27
a lead writer
92:28
and an editor to cover the polished report
92:30
Or to prepare a legal case
92:32
real law firms will often have associates
92:34
paralegals, maybe an investigator. And we naturally
92:38
because of the way human teams do work
92:41
can think of
92:42
different ways that
92:43
complex tasks can be broken down into
92:47
different individuals with different roles
92:50
So these are examples. Of when a complex
92:53
task
92:53
will already naturally decompose into subtasks that different
92:57
people with different skills can carry out
93:00
Take the example of creating marketing assets
93:03
Look into detail into what a researcher
93:05
graphic designer, and writer might do
93:08
A researcher
93:09
might
93:10
have the task of analyzing market trends
93:12
and researching competitors. And when designing the research agents
93:17
one question to keep in mind is
93:19
what are the tools that the researcher
93:21
may need in order to come up with
93:23
a research report on market trends and what
93:25
competitors are doing. So one natural tool
93:28
that a
93:29
agentic researcher might need to use would be
93:32
web search. Just as a human researcher
93:34
asked to do these tasks might need to
93:36
search online in order to
93:37
come up with their report
93:39
Or for a graphic designer agents
93:42
they might be tasked with creating visualizations and
93:44
artwork. And so
93:46
what are the
93:46
tools that a agentic software graphic designer might need?
93:50
Well, they may need a Generation and manipulation APIs
93:54
or maybe
93:55
similar to what you saw with the coffee machine example
93:59
maybe it needs code execution to generate charts
94:02
And then lastly
94:03
the writer has transformed the research into report
94:05
text and marketing copy
94:07
and in this case
94:08
they don't need any tools other than
94:10
what an LLM can already do
94:12
to generate text. In this and the next video
94:15
I'm going to use these purple boxes to denote an agent
94:19
and the way you build individual agents is
94:21
by prompting an LLM
94:23
to play the role of
94:25
a researcher or a graphic designer or a writer
94:27
depending on which agent it is part of
94:30
So for example, for the researcher agents
94:32
you might prompt it to say
94:33
you are a research agent
94:35
expert at analyzing market trends and competitors
94:38
carry out
94:39
online research to analyze market trends for the sunglasses product
94:43
and give a summary as well of what
94:44
competitors are doing. So that
94:46
would allow you to build a researcher agent
94:49
And similarly
94:50
by prompting an LLM To act as a
94:52
graphic designer with your proper tools
94:54
and to act as a writer
94:56
that's
94:57
how you can build a graphic designer
94:59
as well as a writer agent
95:01
Having built these three agents
95:04
one way to have them work together
95:06
to
95:07
generate your final report would be to
95:09
use a
95:10
Simple
95:11
linear
95:12
ordered
95:12
workflow or a linear plan in this case
95:14
So
95:15
if you want to create a summer marketing campaign for sunglasses
95:19
you might give that prompt to the research
95:21
agents. The research agent then writes a report that says
95:24
share the current sunglasses trends and competitive offerings
95:27
This research report can then be fed to
95:29
the graphic designer that looks at
95:31
the data the research has found and creates
95:33
a few data visualizations and artwork options
95:36
All these assets can then be passed
95:38
to the writer
95:40
that then takes the research and the graphic
95:42
output and writes the final marketing brochure
95:46
The advantage of building a multi-agent workflow in
95:48
this case is
95:50
when designing a researcher or graphic designer or writer
95:53
you can focus on one thing at a
95:55
time. So I can spend some time building
95:57
maybe the best graphic designer agents I can
96:00
while maybe
96:00
my collaborators are building research agents and writer
96:03
agents. And then in the end
96:04
we string it all together
96:06
to come up with this multi-agent system
96:09
And in some cases
96:10
I'm seeing developers start to reuse
96:13
some agents as well
96:15
So having built a graphic designer for marketing brochures
96:18
maybe I'll think about if I can build
96:20
a more general graphic designer
96:22
that can help me write marketing brochures as
96:24
well as social media posts
96:26
as well as help me illustrate
96:27
online web pages
96:29
So by
96:30
coming up with what are the
96:32
agents you might hire to do a task
96:34
and this will sometimes correspond to
96:37
who are the
96:37
types of human employees you might hire to do a task
96:41
you can come up with a workflow like this
96:44
with maybe even building agents that you could
96:46
choose to reuse in other applications as well
96:49
Now
96:50
what you see here is a linear plan where one agent
96:53
the researcher does his work
96:55
then the graphic designer
96:56
and then the writer. With agents
96:58
you can also, as an alternative to a linear plan
97:01
you can also have agents
97:03
interact with each other in more complex ways
97:05
Let me illustrate with an example of planning
97:08
using multiple agents. So previously
97:11
you saw how we may give an OM
97:13
a set of tools that they can call
97:15
to carry out different tasks
97:16
In what I want to show you
97:18
we will instead give an OM
97:20
the option to call on different
97:22
agents to ask the different agents to help
97:24
complete different tasks. So in detail
97:27
you might write a prompt
97:28
like you're a marketing manager
97:29
have the following team of agents to work with
97:31
and then give a description to agents
97:33
And this is very much similar to what
97:35
we're doing with planning using tools
97:37
except the tools, the green boxes, are replaced with agents
97:41
these purple boxes that the OM can call
97:43
on
97:43
And then we'll ask it to return a
97:44
step-by-step plan to carry out the user's request
97:47
And in this case
97:47
the OM may
97:48
ask the
97:49
researcher to
97:50
research current sunglasses trends and then report back
97:53
Then they'll ask the graphic designer to create
97:55
the images and then report back
97:56
Then ask the writer to create a report
97:58
And then maybe the OM will choose to
97:59
review
98:00
or reflect on and improve the report
98:02
one final Time. In executing this plan
98:06
you would then take the set one text to the researcher
98:09
carry out research, then pass that to the graphic designer
98:12
pass it to the writer
98:14
and then maybe do one final reflection step
98:17
and then you'd be done
98:18
One interesting view of this workflow is
98:20
as if you have these three agents up here
98:23
but this arm on the left
98:25
is actually like a fourth agent
98:27
There's a marketing manager that is a manager
98:30
of a marketing team
98:31
that is setting direction and then delegating tasks to the researcher
98:34
the graphic designer, and the writer agents
98:36
So this becomes actually a collection of four agents
98:39
both marketing manager agent coordinating the work of the researcher
98:42
the graphic designer, and the writer
98:45
In this video, you saw two communication patterns
98:48
One was a linear one where your agents
98:51
took
98:51
actions one at a time until you got to the end
98:54
and the second had a marketing manager
98:56
coordinating the activity of a few other agents
99:00
It turns out that one of the
99:01
key design decisions you made And that having
99:04
a make when building multi-agentic systems is
99:07
what is the communication pattern between your different
99:09
agents
99:10
This is an area of hot research
99:12
and there are multiple patterns emerging
99:14
but in the next video
99:15
I want to show you what are some
99:16
of the most common communication patterns for
99:19
getting your agents to work with each other
99:21
Let's go see that in the next video
99:24
When you have a team of people working together
99:28
the patterns
99:29
by which they communicate can be quite complex
99:31
And in fact
99:32
designing a
99:33
organizational chart is actually pretty complex to try
99:36
to figure out what's the best way for people to communicate
99:39
to collaborate
99:40
It turns out designing communication patterns for
99:43
multi-agent systems is also quite complex
99:46
But let me show you some of the
99:47
most common design patterns I see
99:50
used by different teams today
99:51
In a marketing team with a linear plan
99:53
where first the researcher worked
99:55
then the graphic designer
99:56
then the writer, the communication pattern was linear
99:59
The researcher would communicate with the graphic designer
100:02
and then both the research and the graphic designer
100:04
maybe pass the outputs to the writer
100:06
and so this very linear communication pattern
100:09
This is
100:10
one of the two most common
100:12
communication plans that I see being used today
100:14
The second of the
100:15
two most common communication plans would be
100:19
similar to what you saw in this example
100:22
with planning using multiple agents
100:24
Where there is a
100:25
manager
100:26
that
100:27
communicates with a
100:28
number of team members and coordinates their work
100:31
So in this example
100:32
the marketing manager decides to call on the
100:34
researcher to do some work
100:36
then
100:36
if you think of the marketing manager as
100:38
getting the report back and then sending it
100:40
to the graphic designer
100:41
getting the report back and sending it to the writer
100:43
this would be a hierarchical communication pattern
100:47
If you're actually implementing a hierarchical communication pattern
100:50
it would probably be simpler
100:51
to have the researcher pass the report back
100:53
to the marketing manager rather than the researcher
100:56
pass the results directly to the graphic designer
100:58
and to the writer
100:59
But so this type of hierarchy
101:01
is also a pretty common way to
101:03
plan the communication patterns where you have one
101:05
manager coordinating the work of
101:08
a number of other agents
101:09
And just to share with you some more
101:11
advanced and less frequently used
101:14
but nonetheless sometimes used in practice communication patterns
101:17
one would be a deeper hierarchy where same as before
101:20
if you have a marketing manager
101:22
send tasks to researcher
101:23
graphic designer, Writer
101:24
but maybe the researcher
101:26
has themselves two other agents that they call on
101:29
such as a wet researcher and a fact
101:30
checker. Maybe the graphic designer just works by themselves
101:34
whereas the writer has a
101:36
initial style writer and a citation checker
101:38
So this would be a hierarchical organization
101:41
of agents in which some agents might themselves call other sub-agents
101:46
and also see this used in some applications
101:49
but this is much more complex than a one-level hierarchy
101:52
so used less often today
101:55
And then one final pattern that is quite challenging to execute
101:58
but I see a few experimental projects use it
102:02
is the all-to-all communication pattern
102:04
So in this pattern
102:06
anyone's allowed to talk to anyone else at any time
102:09
and the way you implement this is you
102:11
prompt all four of your agents
102:13
in this case
102:14
to tell them that there are
102:16
three other agents it could decide to call on
102:18
and whenever one of your agents decides to
102:20
send a message to another agent
102:22
that message gets added. The receiver agent's context
102:27
and then a receiver agent can
102:28
think for a while and decide when to
102:30
get back to that first agent
102:32
And so if you all
102:33
collaborate in a crowd and talk to each
102:35
other for a while until
102:37
say
102:37
each of them declares that it is done with its task
102:40
and then it starts talking
102:41
then maybe when everyone thinks it's done
102:43
or maybe when the writer concludes it's good enough
102:45
that's when you generate the final output
102:47
In practice
102:48
I find the results of all-to-all communication patterns
102:50
a bit hard to predict
102:52
So some applications don't need high control
102:55
and you can run it and see what you get
102:57
and the marketing brochure isn't good
102:58
Maybe that's okay
102:59
just run it again and see if you
103:00
get a different result. But I think for
103:03
applications where you're willing to
103:04
tolerate a little bit of chaos and unpredictability
103:07
I do see some developers using this communication pattern
103:10
So that, I hope
103:12
conveys some of the richness of multi-agent systems
103:17
Today
103:17
there are quite a lot of software frameworks
103:19
as well that support easily building multi-agent systems
103:23
and they also
103:24
Implementing some of these communication patterns relatively easy
103:27
So maybe if you use your own multi-agent system
103:31
you'll find some of these frameworks hopeful
103:33
for exploring these different communication patterns as well
103:37
And so
103:38
that now brings us to the final
103:41
video of this module and of this course
103:45
Let's go on to the final video
103:47
to wrap up
103:49
Welcome to the final video of this course
103:51
It feels like we've been through a lot together
103:54
just you and me
103:55
and we've gone through a lot of topics
103:56
in agentic AI
103:58
Let's take a look
103:59
In the first module
104:00
we talked about what are the applications you
104:02
can build with agentic AI that just were not possible before
104:06
and we started then to look at key design patterns
104:11
including the reflection design pattern
104:13
which is a
104:14
simple way to sometimes give your application a nice performance boost
104:18
and then tool use or function calling
104:20
which expands what your LM application can do
104:23
with code execution being one important case of
104:26
that. And then we spent a lot of
104:27
time talking about
104:28
evaluations as well as error analysis
104:31
and how to drive a disciplined process
104:34
of building
104:35
as well as analyzing to be efficient in
104:37
how you
104:38
keep on improving the performance of your agentic
104:40
AI system
104:41
This fourth module is
104:42
some of the material that I think
104:44
you will find most useful
104:46
as you keep building agentic AI systems
104:48
I hope, for a long time
104:50
And then in this module
104:51
we talk about planning and multi-agent systems that
104:54
can let you build much more powerful
104:57
although sometimes
104:58
harder to control
104:59
How to predict and advance types of systems
105:02
So with the skills you learn from this course
105:04
I think you now know how to build
105:06
a lot of cool
105:07
exciting, agentic AI applications
105:10
When my team, or I see other teams as well
105:13
interview people for jobs
105:14
I find interviews often try to
105:17
assess whether or not candidates have
105:19
pretty much the skills you're learning about
105:22
in this course. And so I hope that
105:23
this course will also open up new professional opportunities for you
105:27
and you should be able to do more
105:29
Whether you're doing these things for fun
105:31
or for professional or practical settings
105:34
I think you enjoy
105:35
this new set of things you can
105:38
now build
105:39
So just to wrap up
105:41
I want to thank you again for spending
105:43
all this time with me
105:44
and I hope you will take these skills
105:47
use them responsibly
105:48
and just go build cool stuff

Watch these videos on your TV

XTTV brings video content from X to your living room. Download now and start watching on the big screen.