← Back to videos
Don't waste 2 years learning to become an AI agentic engineer in 2026. Andrew Ng, the godfather o

shobhanai
@HeyShobhan
November 18, 2026· 105:51
Don't waste 2 years learning to become an AI agentic engineer in 2026. Andrew Ng, the godfather of AI, gave the complete playbook to become one from scratch. 1 hour course. Free: • 00:00 - AI agent basics • 12:12 - AI Agentic workflows & design patterns • 53:27 - Practical tips for building AI agents • 1:20:30 - self-improving AI agent loops • 1:30:19 - multi-agent AI systems I watched it last night. Halfway through, I realized I could get into Anthropic in weeks, not years. Bookmark now. Watch it. Then build your own AI agent
Transcript
- When
- I coined the term agentic to describe what
- 0:02
- I saw as a
- 0:03
- important and rapidly growing trend in how people
- 0:06
- were building on-base applications
- 0:08
- what I did not realize was that a
- 0:09
- bunch of marketers would get hold of this
- 0:11
- term
- 0:12
- and use it as a sticker and put
- 0:14
- this on almost everything in sight
- 0:16
- And
- 0:16
- that has caused hype on agentic AI
- 0:19
- to skyrocket
- 0:21
- The good news, though, is that ignoring the hype
- 0:23
- the number of truly
- 0:25
- valuable and useful applications built
- 0:27
- using agentic AI has also grown very rapidly
- 0:30
- even if
- 0:31
- not quite as rapidly as the hype
- 0:33
- And in this course
- 0:34
- what I'd like to do
- 0:35
- is show you best practices for building agentic AI applications
- 0:39
- and just open up
- 0:40
- a lot of new opportunities
- 0:42
- to you in terms of what you can now build
- 0:44
- Today
- 0:44
- agentic workflows are being used to build applications
- 0:47
- like customer support agents
- 0:49
- or
- 0:49
- to do deep research to help write deeply insightful research reports
- 0:54
- or to process tricky legal documents
- 0:57
- or to look at
- 0:58
- patient input
- 0:59
- and render Or to suggest possible medical diagnoses
- 1:03
- On many of my teams
- 1:04
- a lot of the projects we built just
- 1:06
- would be impossible without agentic workflows
- 1:09
- and so
- 1:09
- knowing how to build applications with them
- 1:11
- is one of the most important and valuable
- 1:13
- skills in AI
- 1:15
- today. It turns out that
- 1:17
- one of the
- 1:17
- biggest differences I've seen between people that really
- 1:20
- know how to build agentic workflows
- 1:22
- compared to people that are less effective at
- 1:23
- it is the ability to drive a disciplined development process
- 1:27
- specifically one focused on evals and error analysis
- 1:31
- And in this course
- 1:32
- I'll tell you what that means and show
- 1:33
- you
- 1:34
- what allows you to be really good at
- 1:36
- building these agentic workflows
- 1:38
- Being able to do this is
- 1:40
- one of the most important skills in AI today
- 1:42
- and will open up a lot more opportunities
- 1:44
- be it job opportunities or
- 1:46
- opportunities to just build amazing software yourself
- 1:50
- With that
- 1:50
- let's go on to the next video
- 1:52
- to dive more into what are agentic workflows
- 1:56
- So
- 1:56
- what is agentic AI and why are agentic
- 1:58
- AI workflows so powerful?
- 2:01
- Let's take a look
- 2:02
- The way that many of us use large
- 2:03
- language models or LLMs today is by prompting it to
- 2:06
- say
- 2:07
- write an essay for us on a certain
- 2:09
- topic X
- 2:10
- And I think of that as
- 2:11
- akin to going to a human
- 2:13
- or in this case
- 2:14
- going to an AI
- 2:15
- and asking it to please
- 2:16
- type out an essay for me
- 2:18
- by writing from the first word to the
- 2:20
- last word
- 2:21
- all in one go
- 2:22
- and without ever using backspace
- 2:24
- It turns out that we as people
- 2:25
- we don't do our best writing like that
- 2:27
- by being forced to write in this
- 2:29
- completely linear order and nor do AI models
- 2:32
- But despite the difficulty of being constrained to writing this way
- 2:36
- our LLMs do surprisingly well
- 2:38
- In contrast, with an agentic workflow
- 2:40
- this is what the process might look like
- 2:42
- You may ask it to first
- 2:43
- write an essay online and listen to the topic
- 2:46
- then ask if it needs to do any more research
- 2:48
- and
- 2:48
- after doing some web research and maybe downloading some web pages
- 2:51
- then to write the first draft
- 2:52
- and then to read the first draft and
- 2:54
- see what parts need revision or more
- 2:55
- Research, and then revise the draft
- 2:57
- and so on
- 2:58
- And this type of workflow is more akin
- 3:00
- to doing some thinking and some research
- 3:02
- and then doing some revision
- 3:04
- and then doing some more thinking
- 3:06
- and so on. And with this iterative process
- 3:09
- it turns out that an agentic workflow can take longer
- 3:12
- but it delivers a much better work product
- 3:15
- So
- 3:16
- an agentic AI workflow is a process where
- 3:18
- an LM-based app
- 3:20
- executes multiple steps to complete a task
- 3:23
- In this example
- 3:24
- you might use an LM to write the first essay outline
- 3:28
- and then
- 3:28
- you might use an LM to decide
- 3:30
- what search terms to type into a web search engine
- 3:34
- or really what search terms to call a
- 3:36
- web search
- 3:37
- API with in order to get back
- 3:39
- relevant web pages. Based on that
- 3:42
- you can
- 3:43
- feed the downloaded web pages into an LM
- 3:45
- to help you write the first draft
- 3:47
- and then maybe use another LM to
- 3:49
- reflect and decide what needs more revision
- 3:52
- and then depending on how you design this workflow
- 3:55
- Perhaps you may even
- 3:57
- add a human in the loop step where
- 3:59
- the arm has the option to request human review
- 4:03
- maybe of some key facts
- 4:04
- and based on that
- 4:05
- it may then revise the draft
- 4:07
- and this process
- 4:09
- results in a much better work output
- 4:11
- One of the key skills you learn in
- 4:13
- this course is how to take a complex task
- 4:16
- like writing an essay
- 4:18
- and
- 4:19
- breaking it down into
- 4:21
- smaller steps
- 4:22
- for agentic workflow to execute one step at
- 4:25
- a time
- 4:26
- to then get the work output that you
- 4:28
- want
- 4:28
- And knowing how to decompose the task into
- 4:30
- steps
- 4:31
- and how to build the components to execute
- 4:33
- the individual steps well
- 4:34
- turns out to be a tricky but important
- 4:37
- skill
- 4:37
- that will determine your ability
- 4:40
- to build agentic workflows for a huge range
- 4:42
- of exciting applications. In this course
- 4:45
- a run example that we'll use and something
- 4:47
- that you build alongside me is a
- 4:50
- research
- 4:50
- agent
- 4:51
- So here's an example of what it will
- 4:53
- look like
- 4:53
- You can enter a research topic like How
- 4:56
- do I build a new rocket
- 4:59
- company to compete with SpaceX?
- 5:03
- I don't personally want to compete with SpaceX
- 5:05
- but if you want to
- 5:06
- you can
- 5:06
- try asking a research agent to help your
- 5:08
- background research
- 5:10
- So this agent starts with planning out what research to use
- 5:14
- including calling a web search engine to download some web pages
- 5:18
- and then to synthesize and rank findings
- 5:21
- draft an outline, have an editor agent review for coherence
- 5:24
- and then finally generate a comprehensive markdown report
- 5:28
- which it has done here
- 5:29
- building a new rocket company to compete with SpaceX
- 5:31
- with an intro, background, Findings and so on
- 5:35
- and I take the points of appropriately that
- 5:38
- this is going to be a
- 5:39
- tough startup to build
- 5:40
- so I'm not personally planning to do this
- 5:43
- but if you want to tackle something like this
- 5:46
- maybe a research agent like this could help
- 5:49
- you with some initial research
- 5:51
- And by finding
- 5:53
- and downloading multiple sources and deeply thinking about it
- 5:56
- this actually ends up with a
- 5:58
- much more thoughtful report than
- 6:01
- just prompting an alarm
- 6:02
- to write an essay for you would
- 6:04
- One of the reasons I'm excited about this
- 6:06
- is because in my work
- 6:07
- I've ended up building quite a few specialized research agents
- 6:11
- be it in
- 6:12
- legal documents for conflict legal compliance or for
- 6:15
- some healthcare sectors or some
- 6:17
- business product research areas
- 6:19
- And so I hope that working through this example
- 6:22
- you not only learn how to build agentic
- 6:24
- workflows for many other applications
- 6:26
- but that some of the ideas in building
- 6:28
- research agents will be
- 6:30
- directly useful to you if you ever need
- 6:32
- to build a custom research Search agent yourself
- 6:35
- Now
- 6:36
- one of the
- 6:37
- often discussed areas of
- 6:40
- AI agents is how autonomous are they?
- 6:43
- What you just saw here was a relatively complex
- 6:45
- highly autonomous agentic AI workflow
- 6:48
- but there are also
- 6:49
- other simpler workflows that are incredibly valuable
- 6:53
- Let's go into the next video to talk
- 6:55
- about the degree to which agentic workflows can be autonomous
- 6:58
- and this will give you
- 7:00
- a framework to think about how
- 7:02
- you might go about building different applications and
- 7:04
- how easy or difficult they might be
- 7:06
- See you in the next video
- 7:09
- Agents can be autonomous to different degrees
- 7:12
- A few years ago
- 7:13
- I noticed within the AI community that there
- 7:15
- was a growing controversial debate about what is an agent
- 7:19
- and some people are writing a paper saying about an agent
- 7:22
- and others say, no
- 7:22
- that's not really a true agent
- 7:24
- And I felt this debate was unnecessary
- 7:27
- which is why I started using the term
- 7:29
- agentic because I thought if
- 7:31
- we use it as an adjective rather than a binary
- 7:34
- it's either an agent or not
- 7:36
- then we can have
- 7:37
- just acknowledge that systems can be agentic to different degrees
- 7:41
- and let's just call out agentic and move
- 7:42
- on with the real work of building these
- 7:44
- systems rather than debating
- 7:46
- you know
- 7:47
- is this sufficiently autonomous
- 7:48
- to be an agent or not
- 7:50
- I remember when I prepared a talk on agentic reasoning
- 7:54
- one of my team members actually came to me and said
- 7:55
- hey, Andrew, we don't need yet another word
- 7:58
- You know, we have agent
- 7:59
- Why you make have another word agentic?
- 8:01
- But I decided to use it anyway
- 8:03
- And then later on
- 8:04
- wrote an article in
- 8:06
- DeepMind's newsletter
- 8:07
- The Batch. And also
- 8:09
- post on social media saying that
- 8:11
- instead of arguing over which words include or
- 8:13
- exclude as being a true agent
- 8:15
- let's
- 8:15
- acknowledge the different degrees to which systems can
- 8:18
- be agentic
- 8:19
- And I think this helped move past the
- 8:21
- debates on
- 8:23
- what is a true agent and
- 8:24
- let us just focus on actually building them
- 8:27
- Some agents can be less autonomous
- 8:30
- So
- 8:30
- take the example of writing an essay
- 8:32
- about black holes
- 8:34
- You can have a relatively simple agents
- 8:37
- to come up with a few
- 8:38
- web search terms or web search queries
- 8:41
- then you can
- 8:42
- hard code in that you call a web search engine
- 8:45
- fetch some web pages
- 8:46
- and then use that to write an essay
- 8:48
- And this would be an example of a
- 8:49
- less autonomous agent with a fully deterministic sequence of steps
- 8:54
- and this will work okay
- 8:56
- In terms of notational convention
- 8:57
- throughout this course
- 8:58
- I'll use the red color
- 9:00
- as you see here on the left
- 9:01
- to denote the user input
- 9:03
- such as a user query in this case
- 9:05
- or in later examples
- 9:06
- maybe the input document. In the agentic workflow
- 9:09
- The gray boxes denote calls to an LM
- 9:12
- and the green boxes
- 9:14
- like the web search
- 9:16
- and the web fetch boxes that you see here
- 9:18
- indicate steps where
- 9:20
- other software is being used to carry out an action
- 9:22
- such as a web search API call or
- 9:24
- executing code to fetch the contents of a
- 9:26
- website. Then an agent can be more autonomous
- 9:30
- where given a request to write an essay about black holes
- 9:33
- perhaps you let the LM decide
- 9:35
- does it want to do a web search
- 9:36
- or does it want to search recent news sources
- 9:39
- or does it want to search for recent
- 9:41
- research papers on the website archive?
- 9:43
- Based on that, maybe in this example
- 9:45
- the LM, not the human engineer
- 9:47
- but the LM chooses in this case
- 9:49
- to call a web search engine
- 9:51
- and then after that
- 9:52
- you may let the LM decide
- 9:54
- how many web pages does it want to fetch
- 9:55
- or if it fetch the PDF
- 9:58
- does it need to
- 9:59
- call a function or also call the tool
- 10:01
- to convert the PDF to text?
- 10:03
- And in this case
- 10:03
- maybe it fetches its top few web pages
- 10:07
- then it can write an essay
- 10:08
- Decide whether or not to reflect and improve
- 10:10
- and maybe even go back to fetch more
- 10:12
- web pages
- 10:13
- and then to finally produce an output
- 10:15
- And so even for this example of a research agent
- 10:19
- we can see that
- 10:20
- some agents can be
- 10:22
- less autonomous
- 10:23
- with a
- 10:23
- linear sequence of steps to be executed determined by a programmer
- 10:27
- and some can be more autonomous where you
- 10:29
- trust the OM to make more decisions
- 10:31
- and the exact sequence of steps that happens
- 10:33
- may be even determined by the OM rather
- 10:35
- than in advance by the programmer
- 10:37
- So for less autonomous systems
- 10:40
- you will usually have all the steps predetermined in advance
- 10:43
- and any functions it calls like web search
- 10:46
- and we'll call that tool use as you
- 10:47
- learn in the third module in this course
- 10:49
- might be hard coded by the human engineer
- 10:51
- by you or me
- 10:52
- and most of the autonomy
- 10:54
- is in what text the OM generates
- 10:56
- At the end of the spectrum would be highly autonomous agents
- 10:59
- where the agent makes many decisions autonomously
- 11:02
- including, for example
- 11:03
- deciding what is the sequence of steps it
- 11:05
- will carry out in order to
- 11:07
- write the essay
- 11:08
- And there's some highly autonomous agents that can
- 11:10
- even write new functions
- 11:12
- or sometimes they
- 11:13
- create new tools that it can then execute
- 11:16
- And somewhere in between are semi-autonomous agents
- 11:19
- where it can make some decisions
- 11:20
- choose tools
- 11:21
- but the tools are usually more predefined
- 11:23
- As you look at different examples in this course
- 11:25
- you learn how to build applications anywhere on
- 11:28
- this spectrum of less to more highly autonomous
- 11:31
- And you find that there are tons of
- 11:32
- applications in the less autonomous end of the
- 11:34
- spectrum
- 11:35
- They're very valuable
- 11:36
- being built for tons of businesses today
- 11:38
- And at the same time
- 11:39
- there are also
- 11:40
- applications being worked on at the more
- 11:42
- highly autonomous end of the spectrum
- 11:44
- but those are usually less easily controllable
- 11:47
- a little bit more unpredictable
- 11:49
- and also
- 11:50
- a lot of active research as well to
- 11:52
- figure out how to build
- 11:53
- these more highly autonomous agents
- 11:56
- And with that
- 11:57
- let's go on to the next video
- 11:59
- to
- 11:59
- dive deeper into this
- 12:01
- and to hear
- 12:02
- about some of the benefits of using agents
- 12:04
- and why they allow us
- 12:05
- to do things that just were not possible
- 12:07
- with earlier generations of
- 12:09
- of base applications
- 12:12
- I think the one biggest benefit of agentic
- 12:14
- workflows is that allows you
- 12:16
- to do many tasks effectively that just previously
- 12:19
- were not possible
- 12:20
- But there are other benefits as well
- 12:22
- including parallelism that lets you do certain things quite fast
- 12:25
- as well as modularity that lets you combine
- 12:27
- the best of three components from many different
- 12:30
- places to build an effective workflow
- 12:32
- Let's take a look
- 12:33
- My team collected some data
- 12:35
- on a
- 12:36
- coding benchmark
- 12:37
- that tests the ability of different OLMs to
- 12:40
- write code to carry out certain tasks
- 12:43
- The benchmark used in this case is called HumanEval
- 12:46
- and it turns out that GPT-3.5
- 12:48
- this is a model that the first
- 12:50
- hopefully available version of ChatGPT was based on
- 12:52
- if asked to write the code directly
- 12:54
- to just type out the computer program
- 12:56
- gets 48% right on this benchmark
- 12:59
- This is a passe K metric
- 13:00
- GPT-4 is a much better model
- 13:02
- Its performance leaps to
- 13:04
- 67% with this also non-agentic workflow
- 13:08
- But it turns out that
- 13:09
- as large as the improvement was from GPT-3.5 to GPT-4
- 13:14
- that improvement is stored by what you can
- 13:15
- achieve by wrapping GPT-3.5 within an agentic workflow
- 13:21
- Using different agentic techniques
- 13:22
- which you learned about later in this course
- 13:24
- you can prompt GPT-3.5 to write code
- 13:27
- and it may be
- 13:28
- reflect on the code and figure out if
- 13:30
- it can improve it. And using techniques like that
- 13:32
- you can actually get GPT-3.5 to get much
- 13:35
- higher levels of performance. And similarly
- 13:37
- GPT-4
- 13:38
- used in the context of an agentic workflow
- 13:41
- also does much better
- 13:43
- So even with today's SLMs
- 13:45
- an agentic workflow lets you get
- 13:47
- much better performance. And in fact
- 13:49
- what we
- 13:50
- saw in this example was the improvement from
- 13:53
- one generation of model to another
- 13:55
- which is huge
- 13:55
- is still not as big a difference as
- 13:58
- implementing an agentic workflow on the previous generation
- 14:01
- of model
- 14:02
- Another benefit
- 14:03
- of using agentic workflows is that they can
- 14:06
- paralyze some tasks
- 14:07
- and thus do certain things much faster
- 14:10
- than a human
- 14:11
- For example
- 14:12
- If you ask an
- 14:13
- agentic workflow to write an essay about black holes
- 14:16
- you might be able to have three OMs
- 14:18
- run in parallel to generate ideas
- 14:20
- for web search terms
- 14:22
- to type into search engine
- 14:23
- Based on the first web search
- 14:25
- it may identify, say, three top results to fetch
- 14:29
- and based on the second web search
- 14:31
- it may identify a second set of web pages to fetch
- 14:34
- and so on
- 14:35
- And it turns out that whereas
- 14:37
- a human doing this research
- 14:38
- would have to read these nine web pages
- 14:41
- sequentially or one at a time
- 14:42
- when you're using an agentic workflow
- 14:45
- you can actually parallelize all nine web page downloads
- 14:49
- and then finally feed all these things into
- 14:51
- an LM to write an essay
- 14:52
- So even though agentic workflows
- 14:54
- do take longer than
- 14:56
- truly non-agentic workflows or by direct generation of
- 14:59
- I just prompting a single time
- 15:00
- if you were to compare
- 15:02
- this type of agentic workflow to how a
- 15:04
- human would have to go about the task
- 15:06
- the ability to parallelize downloading lots of web
- 15:08
- pages
- 15:09
- can actually let it do certain tasks much
- 15:11
- faster. Faster than the
- 15:13
- non-parallel sequential way that a single human might
- 15:16
- process this data. To build on this example
- 15:19
- it turns out
- 15:20
- one of the things I often do when
- 15:21
- building
- 15:22
- agentic workflows is
- 15:23
- look at the individual components like the
- 15:25
- LM and add or swap out components
- 15:28
- So for example
- 15:30
- maybe I look at a web search engine
- 15:31
- I use up here
- 15:32
- and I might decide that
- 15:33
- I want to swap in a new web
- 15:35
- search engine. When building agentic workflows
- 15:37
- there are actually multiple web search engines
- 15:39
- including Google, which you can access via Surfer
- 15:42
- as well as others like Bing
- 15:44
- DuckDuckGo, Tivili
- 15:45
- u.com. There are actually quite a lot of
- 15:47
- options for web search engines designed for LMs
- 15:50
- to use. Or maybe instead of just doing three web searches
- 15:53
- maybe on this step
- 15:54
- you can swap in a new
- 15:56
- news search engine
- 15:57
- so they can find out what's the latest
- 15:59
- news on recent breakthroughs on black hole science
- 16:04
- And lastly
- 16:05
- instead of using the same LM for all
- 16:07
- of the different steps
- 16:09
- I will often try out different
- 16:11
- large Gosh models
- 16:12
- and maybe try out different LLM providers
- 16:15
- to see which one
- 16:16
- gives the best result for different
- 16:18
- steps of this system
- 16:20
- So to summarize
- 16:21
- the main reason I use agentic workflows is
- 16:24
- it just gives much better performance on many
- 16:26
- different applications. But in addition
- 16:29
- it can also paralyze
- 16:30
- some tasks that humans will otherwise have to do sequentially
- 16:34
- and
- 16:35
- the modular design of many
- 16:37
- agentic workflows also lets us
- 16:39
- add or update tools
- 16:40
- and sometimes swap out models
- 16:43
- We've talked a lot about the key components
- 16:45
- of building agentic workflows
- 16:47
- Let's now take a look
- 16:48
- at the range of agentic AI applications to
- 16:51
- give you a sense of the sorts of
- 16:52
- things people are already building and the sorts
- 16:55
- of things you will build yourself
- 16:57
- Let's go on to the next video
- 17:00
- People and businesses do a lot of stuff
- 17:04
- How do you take this
- 17:05
- useful stuff that we do
- 17:07
- and break it down into discrete steps
- 17:09
- for the genetic workflow to follow?
- 17:11
- Let's take a look
- 17:12
- Take the example of building a research agent
- 17:15
- If you want
- 17:17
- a AI system to write an essay on the topic X
- 17:19
- one thing you could do is
- 17:20
- prompt an LLM to have it
- 17:22
- generate an output
- 17:23
- directly
- 17:24
- But
- 17:24
- if you were to do this for
- 17:26
- topics that you want deeply researched
- 17:28
- you may find that the LLM output
- 17:30
- covers only the surface level points or maybe
- 17:32
- covers only the obvious facts
- 17:34
- but doesn't go
- 17:35
- as deep into
- 17:36
- the subject as you want it to
- 17:38
- In this case
- 17:39
- you might
- 17:40
- then reflect on how you as a human
- 17:42
- would write an essay on a certain topic
- 17:45
- Would you just sit down and start writing
- 17:47
- or would you take multiple steps
- 17:48
- such as first write an essay outline
- 17:50
- and then search the web
- 17:51
- and then based on the input from the web search
- 17:54
- write the essay?
- 17:55
- And as I take a task and decompose it into steps
- 17:58
- one question I'm always asking myself is
- 18:01
- if I look at these steps one
- 18:02
- two, and three
- 18:03
- can each of them be done
- 18:05
- either by an LLM or
- 18:07
- by
- 18:07
- a short piece of code or by a
- 18:09
- function call by a tool
- 18:11
- And so in this case
- 18:12
- I think an LLM
- 18:13
- can maybe write a decent outline on
- 18:16
- many topics that I would want it to
- 18:18
- help me think through. So say
- 18:20
- probably okay on the first step
- 18:22
- And then I know how to
- 18:23
- use an LLM to
- 18:24
- generate search terms to search the web
- 18:27
- So I would say the second step is
- 18:28
- also doable. And then based on web search
- 18:31
- I think an LLM could input the web
- 18:33
- search results and write an essay
- 18:34
- And so this would be a reasonable first
- 18:36
- attempt at an agentic workflow
- 18:38
- for writing an essay
- 18:39
- that goes deeper than just direct generation
- 18:42
- But if
- 18:43
- I were to then implement this agentic workflow
- 18:45
- and look at the results
- 18:47
- maybe
- 18:47
- you find that the results still aren't good
- 18:49
- enough. It's still not yet as deeply thoughtful
- 18:51
- Maybe the essays feel a little bit disjointed
- 18:54
- This has actually happened to me
- 18:55
- Once built a research agent using this sort of workflow
- 18:58
- but when I read the output
- 18:59
- it felt a bit disjointed
- 19:00
- You know
- 19:00
- the start of the article didn't feel completely
- 19:03
- consistent with Middle didn't feel completely consistent
- 19:06
- with the end. In this case
- 19:07
- what you might do
- 19:08
- is then reflect on how you would change
- 19:10
- the workflow if you as a human found
- 19:12
- that the essays were a bit disjointed
- 19:14
- One thing you could do
- 19:15
- is take the third step and further decompose
- 19:19
- write the essay into additional steps
- 19:22
- So instead of writing the essay all in one go
- 19:25
- you might instead have it
- 19:27
- write the first draft
- 19:28
- and then consider what parts need revision
- 19:31
- and then revise the draft
- 19:32
- And this would be
- 19:33
- how I as a human might go about it to
- 19:36
- not just write the final essay at my first attempt
- 19:39
- but write the first draft and then read over it
- 19:41
- which
- 19:42
- is another step that the arm
- 19:44
- is pretty decent at. And then based on
- 19:46
- my own critique of my own essay
- 19:48
- I'll revise the draft
- 19:49
- So to recap, I started off with direct generation
- 19:53
- just one step, decided it wasn't good enough
- 19:55
- and so broke that down into three steps
- 19:58
- and then maybe decide that still isn't good enough
- 20:00
- and took one of the steps and further
- 20:02
- broken it down or decompose it
- 20:04
- Into three more steps
- 20:05
- resulting in this more complex
- 20:08
- richer process for generating an essay
- 20:11
- And depending on
- 20:12
- how satisfied you are with the results of this process
- 20:15
- you may choose to even modify
- 20:17
- this essay generation process further
- 20:19
- Let's look at a second example of how
- 20:21
- to decompose complex tasks into smaller steps
- 20:25
- Take the example of responding to
- 20:27
- basic customer order
- 20:29
- inquiries
- 20:30
- The first step that a human customer service
- 20:32
- agent might carry out
- 20:33
- might be to first extract the key information
- 20:36
- such as who is this email from
- 20:38
- what do they order
- 20:39
- and what is the order number
- 20:40
- And these are things an OM could do
- 20:43
- so I could just say
- 20:44
- let's have an OM do that
- 20:45
- The second step would be to then
- 20:47
- find the relevant customer records
- 20:49
- So to
- 20:49
- write and generate the relevant database queries to
- 20:52
- pull up the order
- 20:54
- what the customer had ordered and when it
- 20:55
- shipped and so on
- 20:56
- I think an OM
- 20:57
- with the ability to call a function
- 21:00
- to query the orders database
- 21:03
- should be able to do that
- 21:04
- That, and lossy
- 21:05
- having pulled up the customer record or the customer order record
- 21:08
- I might then
- 21:09
- write and send a response back to the
- 21:10
- customer. And I think with the information we pulled up
- 21:13
- this third step is also
- 21:14
- doable with an OM. If I give it
- 21:16
- the option to
- 21:17
- call an API to send an email
- 21:19
- So this would be another example of taking
- 21:21
- a task
- 21:22
- of responding to
- 21:23
- customer email and breaking it down into three
- 21:26
- individual steps where I can look at each
- 21:28
- of these steps and say
- 21:29
- yep
- 21:29
- I think an OM or an OM
- 21:31
- with the ability to call a function
- 21:33
- to query database or send an email
- 21:35
- should be able to do that
- 21:36
- Just one last example for the invoice processing
- 21:40
- After a PDF invoice has been converted to text
- 21:42
- the first step is to pull out the required information
- 21:46
- the name of the biller
- 21:46
- the address, the due date, the amount due
- 21:49
- and so on. And now it should be
- 21:50
- able to do that
- 21:51
- And then
- 21:52
- if I want to
- 21:53
- check that
- 21:54
- the information was extracted and save it in
- 21:56
- a new database entry
- 21:57
- then I think an
- 21:58
- OM should be able to
- 21:59
- help me call
- 22:00
- a function to update the database record
- 22:02
- And so to implement this
- 22:04
- We'll implement an agentic workflow to
- 22:06
- carry out basically these two steps
- 22:09
- When building agentic workflows
- 22:10
- I think of myself
- 22:12
- as having a number of building blocks
- 22:14
- One important building block would be
- 22:16
- large language models or maybe large multimodal models
- 22:19
- if
- 22:19
- I want to
- 22:20
- try the process images or audio as well
- 22:22
- And OMs are good at generating text
- 22:24
- deciding what to call
- 22:25
- maybe extracting information
- 22:28
- For some highly specialized tasks
- 22:29
- I might also use
- 22:31
- some other
- 22:32
- AI models
- 22:33
- such as AI model for converting a PDF
- 22:35
- to text or for text to speech or
- 22:37
- for image analysis. In addition to AI models
- 22:41
- I also have access to a number of software tools
- 22:44
- including different APIs that I can call to do voice search
- 22:48
- to get maybe real-time weather data
- 22:51
- to send emails, check calendar, and so on
- 22:54
- And I might also have tools to retrieve information
- 22:57
- to pull up data
- 22:58
- from a database or
- 23:00
- to implement
- 23:01
- RAG or retrieval augmented generation where I can
- 23:03
- look up Launch text database and find the most relevant text
- 23:06
- or I might also
- 23:07
- have tools to execute code
- 23:10
- And this is a tool that lets an
- 23:11
- LLM
- 23:12
- write code and then run the code on
- 23:13
- your computer to do
- 23:15
- a huge range of things
- 23:17
- In case some of these tools seem a
- 23:18
- bit foreign to you
- 23:19
- don't worry about it. We'll go through
- 23:21
- the most important tools in much greater detail
- 23:23
- in a later module. But
- 23:25
- I think a lot of my work when
- 23:27
- I'm building an agentic workflow as looking at
- 23:29
- the work that the person or business is
- 23:31
- doing and then trying to figure out with these building blocks
- 23:34
- how can I sequence these building blocks together
- 23:36
- in order to carry out the tasks
- 23:38
- that I want my system to carry out
- 23:41
- And this is why
- 23:42
- having a good understanding of what building blocks are available
- 23:45
- which I hope you
- 23:47
- have a better sense of
- 23:48
- by the end of this course as well
- 23:49
- will allow you to better envision what agentic
- 23:52
- workflows you can build by combining these building
- 23:54
- blocks together
- 23:55
- So to summarize
- 23:57
- one of the key skills in building agentic
- 23:58
- workflows is to look at
- 24:00
- a bunch of stuff that maybe someone does
- 24:02
- and to identify the discrete steps
- 24:05
- that it could be implemented with
- 24:08
- And when I'm looking at the individual discrete steps
- 24:11
- one question I'm always asking myself is
- 24:13
- can this step be implemented
- 24:15
- with either an LM or with one of the tools
- 24:18
- such as an API or a function call
- 24:20
- that I have access to?
- 24:22
- And in case the answer is no
- 24:23
- I'll then often ask myself
- 24:25
- how would I as a human
- 24:26
- do this step?
- 24:27
- And is it possible to decompose this further
- 24:30
- break this down into even smaller steps
- 24:33
- that then maybe
- 24:34
- is more amenable to implementation
- 24:36
- with an LM or with one of the
- 24:38
- software tools that I have?
- 24:40
- So
- 24:40
- I hope this gives you a rough sense
- 24:42
- of how to think about task decomposition
- 24:45
- In case you feel like you don't fully have it yet
- 24:48
- don't worry about it. We'll go through many
- 24:49
- more examples in this course
- 24:51
- and you have a
- 24:51
- much better understanding of this
- 24:53
- by the end of this course
- 24:54
- But it turns out that as you build agentic workflows
- 24:58
- you find that often you build an initial task decomposition
- 25:01
- initial agentic workflow
- 25:03
- and then You want to keep on
- 25:05
- iterating and improving on it quite a few
- 25:07
- times until it
- 25:08
- delivers the level of performance that you want
- 25:11
- And to drive this improvement process
- 25:13
- which is found important for many projects
- 25:16
- one of the key skills is to know
- 25:17
- how to evaluate your agentic workflow
- 25:21
- So in the next video
- 25:22
- we'll talk about evaluations or
- 25:24
- evals and just be a key component of
- 25:27
- how you can build
- 25:28
- and then also
- 25:29
- keep on improving your workflows to get the
- 25:31
- performance that you want
- 25:32
- Let's talk about evals in the next video
- 25:36
- I've worked with many different teams on building agentic workflows
- 25:39
- and
- 25:40
- I found that
- 25:41
- one of the biggest predictors for whether someone
- 25:44
- is able to do it really well
- 25:45
- versus be
- 25:46
- less efficient at it is
- 25:48
- whether or not they're able to drive a
- 25:50
- really disciplined evaluation process
- 25:53
- So your ability to drive
- 25:55
- evals for your agentic workflow
- 25:57
- makes a huge difference in your ability
- 26:00
- to build them effectively
- 26:02
- In this video
- 26:03
- we'll take a quick overview of how to build evals
- 26:06
- and this is a subject that we'll actually
- 26:07
- go into
- 26:08
- much deeper
- 26:09
- in a later module in this course
- 26:11
- But let's take a look
- 26:13
- After building an agentic workflow
- 26:15
- like this one for responding to customer order inquiries
- 26:19
- it turns out that it's very difficult to
- 26:21
- know in advance
- 26:22
- what are the things that could go wrong
- 26:24
- And so rather than trying to build evaluations in advance
- 26:28
- what I recommend is you just
- 26:29
- look at the outputs and
- 26:31
- manually look for
- 26:33
- things that you wish it was doing better
- 26:36
- For example
- 26:37
- maybe you read a lot of outputs and
- 26:39
- find that it is unexpectedly
- 26:41
- mentioning your competitors more than it should
- 26:44
- Many businesses don't want their
- 26:46
- agents to mention competitors because it just creates
- 26:48
- an awkward situation. And if you read some of his outputs
- 26:51
- maybe you find that there's something that says
- 26:53
- I'm glad you shopped with us
- 26:54
- We're much better than our competitor
- 26:56
- CompCo
- 26:57
- Or maybe sometimes say
- 26:58
- sure, we actually refund. Unlike RivalCo
- 27:00
- we make returns easy. And you may look
- 27:02
- at this and go
- 27:03
- gee
- 27:03
- I really don't want this to mention competitors
- 27:07
- This would be the example of a problem
- 27:08
- that
- 27:09
- is really hard to anticipate
- 27:11
- in advance of building this agentic
- 27:13
- So the best practice is really to build
- 27:15
- it first
- 27:15
- and then examine it to figure out
- 27:18
- where it is not yet satisfactory
- 27:20
- and then to find ways to evaluate as
- 27:22
- well as improve the system to eliminate the
- 27:24
- ways that it is still not yet satisfactory
- 27:27
- Assuming your business considers it an error
- 27:31
- or mistake to mention competitors in this way
- 27:34
- then as you work on eliminating these competitor mentions
- 27:38
- one way to track progress would be to
- 27:39
- add an evaluation or an eval
- 27:42
- to track how often this error occurs
- 27:45
- So
- 27:45
- if you have a named list of competitors like comp co
- 27:48
- rival co, the other co
- 27:50
- then you can actually write code
- 27:51
- So just search in your own output
- 27:54
- for how often it mentions these competitors by
- 27:57
- name and count up as a number
- 27:59
- as a fraction of the overall responses
- 28:02
- how frequently it mistakenly mentions competitors
- 28:06
- One nice thing about
- 28:07
- the problem of competitor mentions is is a objective metric
- 28:11
- meaning either the competitor That was mentioned or not
- 28:14
- and for objective criteria
- 28:16
- you can write code to check
- 28:19
- for how often this specific error occurs
- 28:22
- But because LLM's output free text
- 28:25
- there also going to be criteria
- 28:27
- by which you want to evaluate this output
- 28:29
- that may be more
- 28:30
- subjective and where it's harder
- 28:32
- to just write code to output a black
- 28:34
- and white score. In this case
- 28:36
- using a
- 28:37
- LLM
- 28:38
- as a judge is a common technique
- 28:40
- to evaluate the output. So for example
- 28:42
- if you're building a research agent
- 28:44
- to do research on different topics
- 28:47
- then you can use another LLM
- 28:49
- and prompt it to maybe say
- 28:51
- assign the following essay a quality score between one and five
- 28:54
- where one is
- 28:55
- the worst and five is the best essay
- 28:58
- Here I'm using a Python expression to mean
- 29:01
- you know
- 29:01
- copy paste the generated essay into this
- 29:04
- So
- 29:04
- you can prompt the LLM to read the
- 29:05
- essay and assign it a quality score
- 29:09
- Then
- 29:09
- I'm ask the research agent
- 29:10
- to write a number of different research reports
- 29:13
- For example
- 29:14
- on recent developments in black hole science
- 29:17
- or using robots to harvest fruit
- 29:19
- And then in this example
- 29:21
- maybe the judge OM
- 29:22
- assigns the essay on black holes a score of three
- 29:26
- the essay on robot harvesting a score of four
- 29:28
- and as you work on improving your research agent
- 29:31
- hopefully you
- 29:32
- see these scores go up over time
- 29:34
- It turns out, by the way
- 29:35
- that
- 29:36
- OMs are
- 29:36
- actually not that good at these one to
- 29:39
- five scale ratings. You can give it a
- 29:40
- shot. Like Percy tend not to use this
- 29:42
- technique that much myself
- 29:44
- but in a later module
- 29:45
- you learn
- 29:46
- some better techniques to have an OM
- 29:48
- output
- 29:49
- more accurate scores than asking it to output
- 29:51
- scores on a one to five scale
- 29:52
- although some people will do this
- 29:54
- maybe the initial cut
- 29:56
- as an OM as judge type of eval
- 29:58
- Just to give a preview of
- 30:00
- some of the
- 30:01
- agentive AI
- 30:02
- evals that you'll learn about later in this
- 30:03
- course
- 30:04
- You've already heard me talk about how you
- 30:05
- can write codes to evaluate objective criteria
- 30:09
- such as did it mention the competitor or not
- 30:11
- or use an OM as a judge for more Objective criteria
- 30:14
- such as what's the quality of this essay
- 30:15
- But later
- 30:16
- you learn about two major types of evals
- 30:18
- One is end-to-end
- 30:19
- where you measure the output quality of the entire agent
- 30:23
- as well as component level evals
- 30:25
- where you might measure the quality of
- 30:27
- the output of a single step in the
- 30:29
- agentic workflow
- 30:30
- And it turns out that these are useful
- 30:31
- for driving different parts of your development process
- 30:34
- And one thing I do
- 30:36
- a lot as well is just examine the intermediate outputs
- 30:39
- or sometimes you call these the traces of the LM
- 30:42
- in order to understand where it is falling
- 30:45
- short of my expectations. And we call this error analysis
- 30:48
- where we just read through the intermediate outputs
- 30:50
- of every single step to try to spot
- 30:52
- opportunities for improvement
- 30:54
- And it turns out being able to do
- 30:55
- evals and error analysis is a really key
- 30:58
- skill. So with much more
- 30:59
- to say about this in
- 31:01
- the fourth module in the schools
- 31:03
- We're nearly to the end of this first
- 31:04
- module
- 31:05
- Before moving on
- 31:06
- I just want to share with you
- 31:07
- what I think are the most important design
- 31:09
- patterns for building agentic workflows
- 31:12
- Let's We'll take a look at that
- 31:14
- in the next video
- 31:16
- We build agentic workflows by taking building blocks
- 31:19
- and putting them together
- 31:20
- to sequence out these complex workflows
- 31:24
- In this video
- 31:24
- I'd like to share with you a few
- 31:25
- of the key design patterns
- 31:27
- which are patterns for how you can think
- 31:29
- about
- 31:30
- combining these building blocks into more complex workflows
- 31:34
- Let's take a look
- 31:35
- I think four key design patterns for building
- 31:37
- agentic workflows are reflection
- 31:39
- two-use planning, and multi-agent collaboration
- 31:42
- Let me briefly go over what they mean
- 31:44
- and then we'll actually go through
- 31:46
- most of these in-depth later in this course
- 31:48
- as well
- 31:49
- The first of the
- 31:50
- major design patterns is reflection
- 31:53
- So
- 31:54
- I'm a Google ML agent and ask it to write code
- 31:57
- and it turns out that
- 31:58
- an LM might then generate code like this
- 32:01
- you know
- 32:01
- defines here a Python function to do a
- 32:03
- certain task. I could then construct a prompt
- 32:05
- that looks like this. I can say
- 32:07
- here's code intended for a certain task
- 32:09
- and then copy paste whatever the LM had
- 32:11
- just output back into this prompt
- 32:13
- And then ask it
- 32:14
- check the code carefully for correctness
- 32:15
- style, and efficiency, and give constructive criticism
- 32:17
- And turns out that the same LM model
- 32:19
- prompted this way
- 32:20
- may be able to point out some problems
- 32:23
- with the code
- 32:24
- And if I then take this
- 32:25
- critique and feed it back to the model to say
- 32:29
- looks like there's a bug
- 32:30
- could you change the code to fix it?
- 32:32
- Then it may
- 32:33
- actually come with a better version
- 32:35
- of the code. To give a preview of tool use
- 32:39
- if you're able to run the code
- 32:40
- and see where the code fails
- 32:43
- then
- 32:43
- feeding that back to the
- 32:45
- LM can also cause it to
- 32:47
- be able to iterate and generate a much better
- 32:50
- say, V3 version 3 of the code
- 32:52
- So
- 32:52
- reflection is Common design pattern
- 32:54
- where you can ask the arm to examine
- 32:56
- its own outputs or maybe
- 32:58
- bring in some external sources of
- 33:00
- information such as
- 33:01
- run the code and see if it generates
- 33:03
- any error messages
- 33:04
- and use that as feedback to
- 33:06
- iterate again
- 33:07
- and come over a better version of its
- 33:09
- output. And this design pattern isn't magic
- 33:12
- It does not result in
- 33:14
- everything working 100% of the time
- 33:15
- But sometimes it can be a
- 33:17
- nice bump in the performance of your system
- 33:19
- Now
- 33:19
- I've drawn this as if it was a
- 33:21
- single OM that I'm prompting
- 33:23
- but to foreshadow multi-agent work those
- 33:26
- you can also imagine
- 33:28
- instead of having the same model critique itself
- 33:30
- you can imagine having a critic agent and
- 33:33
- all that is is an OM that's been
- 33:34
- prompted with instructions like your role is to critique code
- 33:38
- here's code intended for a task
- 33:40
- check the code carefully and so on
- 33:41
- And the second critic agent may be able
- 33:43
- to point out errors or run unit tests
- 33:46
- and like having two
- 33:48
- simulator agents where each agent is just an
- 33:50
- OM prompted to take on a certain persona
- 33:53
- you can have that Go back and forth
- 33:54
- to iterate to get a better output
- 33:57
- In addition to reflection pattern
- 33:59
- the second important design pattern is tool use
- 34:02
- where today OLMs can be given tools
- 34:05
- meaning
- 34:05
- functions that they can call in order to
- 34:07
- get work done. For example
- 34:09
- if you ask an OLM
- 34:10
- what's the best coffee maker according to reviewers
- 34:12
- and you give it a web search tool
- 34:14
- then it can actually
- 34:15
- search the internet to find much better answers
- 34:18
- Or a code execution tool
- 34:20
- If you ask a math question
- 34:21
- like if I invest $100 in a company interest
- 34:24
- what have you
- 34:24
- it can then write code and execute code
- 34:27
- to compute an answer. Today
- 34:29
- different developers have given OLMs many different tools
- 34:32
- for everything from
- 34:33
- math or data analysis to gather information by
- 34:36
- fetching things from the web or for various databases
- 34:38
- to interface with productivity apps like email calendar and so on
- 34:42
- as well as
- 34:43
- to process images and much more
- 34:45
- And the ability of an OLM
- 34:47
- to decide what tools to use
- 34:49
- meaning what functions to call
- 34:51
- that lets the model
- 34:52
- get a lot more Done
- 34:54
- The third of the four design patterns is
- 34:56
- planning. This is an example from a paper called Hugging GPT
- 35:01
- in which if you ask a
- 35:04
- system to
- 35:05
- please generate an image where a girl is reading a book
- 35:07
- kind of pose is the same as a
- 35:08
- boy in the image
- 35:09
- then please describe the new image in your voice
- 35:11
- then a model can automatically decide that
- 35:14
- to carry out this task
- 35:15
- it first needs to
- 35:17
- find a pose determination model to figure out
- 35:19
- the pose of the boy
- 35:20
- then to pose the image
- 35:21
- to generate a picture of a girl
- 35:23
- and image the text
- 35:24
- and then finally text the speech
- 35:26
- And so in planning
- 35:27
- an LLM decides
- 35:29
- what is the sequence of actions it needs
- 35:31
- to take. In this case
- 35:32
- it is a sequence of API calls
- 35:35
- so that it can then carry out the
- 35:37
- right sequence of steps in the right order
- 35:40
- in order to carry out the task
- 35:42
- So rather than the developer hard coding the
- 35:44
- sequence of steps in advance
- 35:46
- this actually lets the LLM decide what are
- 35:48
- the steps
- 35:49
- to take. Agents that plan today are
- 35:52
- harder to control and somewhat more experimental
- 35:54
- but sometimes it can give really delightful results
- 35:57
- And then finally, multi-agent workflows
- 36:00
- Just as a human manager might hire a
- 36:02
- number of others to work together
- 36:04
- on a complex project. In some cases
- 36:07
- it might make sense for you to hire
- 36:08
- a set of multiple agents
- 36:10
- maybe each of which specializes in a different role
- 36:13
- and have them work together to accomplish
- 36:16
- a complex task
- 36:17
- The picture you see here on the left
- 36:18
- is taken from a project called ChatDev
- 36:21
- which is a
- 36:22
- software framework created by Chen Qian and collaborators
- 36:25
- In ChatDev, multiple agents with different roles
- 36:28
- like chief executive officer
- 36:30
- programmer, tester, designer, and so on
- 36:33
- collaborate together
- 36:34
- as if they were a virtual software company
- 36:37
- and
- 36:38
- can collaboratively complete a range of
- 36:40
- software development tasks
- 36:42
- Let's consider another example
- 36:44
- If you want to write a marketing brochure
- 36:46
- maybe you'd think of hiring a team of three people
- 36:49
- such as a researcher to do online research
- 36:52
- a marketer, To write the marketing text
- 36:54
- and then finally an editor to
- 36:56
- edit and polish the text
- 36:58
- And so in a similar way
- 37:00
- you might consider building a multi-agent
- 37:03
- workflow in which you have a simulated research agent
- 37:06
- a simulated marketer agent
- 37:08
- and a simulated editor
- 37:10
- agent that then
- 37:11
- come together
- 37:12
- to carry out this task for you
- 37:14
- Multi-agent workflows are more difficult to control
- 37:18
- since you don't always know ahead of time
- 37:20
- what the agents will do
- 37:21
- but research has shown that they can result
- 37:23
- in better outcomes for many complex tasks
- 37:26
- including things like writing biographies or deciding on
- 37:30
- chess moves to make in a game
- 37:31
- You learn more about multi-agentic workflows later in
- 37:34
- this course as well
- 37:35
- As so of that
- 37:36
- I hope you have a sense of what
- 37:37
- agentic workflows can do
- 37:40
- as well as of what are the key
- 37:41
- challenges of finding building blocks and putting them together
- 37:45
- maybe by these design patterns
- 37:47
- in order to implement an agentic workflow
- 37:50
- and of course also developing evals
- 37:52
- You can see how our system is
- 37:54
- And keep on improving on it
- 37:56
- In the next module
- 37:58
- I'd like to share with you
- 38:00
- a deep dive into the first of these
- 38:02
- design patterns that is reflection
- 38:05
- and you find that
- 38:06
- is a maybe surprisingly simple to implement technique
- 38:10
- that can give the performance of your system
- 38:12
- sometimes a very nice bump
- 38:14
- So
- 38:15
- let's go on to the next module to
- 38:17
- learn about the reflection design pattern
- 38:21
- In this module
- 38:22
- I'd like to share with you practical tips
- 38:23
- for building agentic AI workflows
- 38:26
- I hope that these tips
- 38:27
- will enable you to be
- 38:28
- much more effective than
- 38:30
- the
- 38:30
- typical developer at building these types of systems
- 38:34
- I find that when developing an agentic AI system
- 38:37
- it's difficult to know in advance
- 38:39
- where it will work and where it won't work so well
- 38:42
- and thus where you should focus your effort
- 38:45
- So very common advice is to
- 38:47
- try to build even a quick and dirty system to start
- 38:51
- so you can then
- 38:52
- try it out and look at it
- 38:53
- to see where it
- 38:55
- may not yet be working as well as you wish
- 38:57
- to then have
- 38:58
- much more focused efforts to develop it even further
- 39:02
- In contrast, I find that it's sometimes less useful to
- 39:05
- you know
- 39:06
- sit around for too many weeks
- 39:07
- theorizing and hypothesizing how to build it
- 39:10
- It's often better to just
- 39:12
- build a quick system in a safe
- 39:15
- reasonable way that doesn't leak data
- 39:16
- kind of do it in a responsible way
- 39:19
- but just build something quickly so you can
- 39:21
- look at it and then use that initial
- 39:22
- prototype to prioritize and drive further development
- 39:26
- Let's start an example of what might happen
- 39:29
- after you've built a prototype
- 39:31
- I want to use as a first example
- 39:33
- the invoice processing workflow that you've seen previously
- 39:37
- where the tasks extract four required fields and
- 39:39
- then they save it to a database record
- 39:42
- After having built such a system
- 39:45
- one thing you might do is find a handful of invoices
- 39:48
- maybe 10 or 20 invoices
- 39:50
- and go through them and just take a
- 39:51
- look at their output
- 39:52
- and see what went well and if there
- 39:54
- were any mistakes. So let's say you look through 20 invoices
- 39:57
- you find that invoice one is fine
- 39:59
- the output looks correct. For invoice two
- 40:01
- maybe you're confused, the date of the invoice
- 40:04
- that is when was the invoice issued with
- 40:05
- the due date of the invoice
- 40:07
- And in this task
- 40:08
- we want to extract the due date so
- 40:10
- that we can issue payments on time
- 40:11
- So then I might note down in a
- 40:13
- document or in a spreadsheet that for invoice two
- 40:16
- the dates were mixed up
- 40:17
- Maybe invoice D was fine
- 40:19
- invoice four was fine
- 40:20
- and so on. But as I go through this example
- 40:22
- I find that there are quite a lot
- 40:23
- of examples where I had
- 40:25
- mixed up the dates
- 40:27
- So
- 40:27
- in a space on
- 40:28
- going through a number of examples like this
- 40:30
- that in this This case
- 40:32
- you might conclude that
- 40:33
- one common error mode is that it is
- 40:35
- struggling with the dates. In that case
- 40:39
- one thing you might consider would be to
- 40:41
- of course
- 40:42
- figure out how to improve your system to
- 40:44
- make it extract due dates better
- 40:46
- but also maybe write an eval
- 40:48
- to measure the accuracy with which it is
- 40:50
- extracting due dates. In comparison
- 40:53
- if you had found that it was
- 40:55
- extracting the bullet address incorrectly
- 40:58
- who knows, maybe you have billers with unusual sounding names
- 41:01
- and so maybe it struggles with billers
- 41:03
- or especially if you have international billers whose
- 41:05
- names may not even all be English letters
- 41:08
- then you might instead focus on building an
- 41:10
- eval for the bullet address
- 41:12
- So one of the reasons why
- 41:13
- building a quick and dirty system
- 41:15
- and looking at the output is so helpful
- 41:17
- is it even helps you decide
- 41:19
- what do you want to put the most
- 41:20
- effort into evaluating. Now
- 41:24
- If you've decided that you want to
- 41:26
- modify your system to
- 41:27
- improve the accuracy
- 41:29
- with which it is extracting the due date of the invoice
- 41:31
- then to track progress
- 41:33
- it might be a good idea to create an evaluation
- 41:36
- an eval
- 41:37
- to measure the accuracy of data extraction
- 41:40
- There are probably multiple ways one might go about this
- 41:42
- but let me share with you how
- 41:44
- I might go about this
- 41:45
- To create a test set or an evaluation set
- 41:48
- I might
- 41:49
- find 10 to 20 invoices and
- 41:51
- manually write down what is the due date
- 41:54
- So maybe one invoice is a due date of August 20th
- 41:56
- 2025, and I write it down at a standard year
- 41:59
- month, date
- 42:01
- format
- 42:02
- And then to make it easier to evaluate in code later
- 42:06
- I would probably write the prompt to the
- 42:07
- alarm to tell it to always format the
- 42:09
- due date in this year month date format
- 42:12
- And with that
- 42:13
- I can then write code to
- 42:14
- extract out the one date that the alarm has output
- 42:17
- which is the due date
- 42:18
- That's the one day we care about
- 42:19
- So this is a regular expression pattern matching
- 42:22
- you know, four numbers for the year
- 42:23
- two for the month
- 42:24
- two for the date
- 42:25
- and then extract that out
- 42:26
- And then
- 42:27
- I can
- 42:27
- just write code to test if the extract
- 42:30
- date is equal to the actual date
- 42:32
- That is the ground truth annotation
- 42:34
- I had written down. So with an eval set of
- 42:37
- say, 20 or so invoices
- 42:39
- I build up my changes to see if
- 42:41
- the percentage of time
- 42:43
- that it gets the extracted date correct is
- 42:45
- hopefully going up as I
- 42:47
- tweak my prompts or tweak other parts of
- 42:48
- my system. So just to summarize what we've seen so far
- 42:52
- we have built a system
- 42:53
- then look at outputs to discover
- 42:55
- where it may be behaving in an unsatisfactory way
- 42:58
- such as due dates are wrong
- 42:59
- Then to drive improve
- 43:01
- So this important output put in place a
- 43:03
- small eval with say just 20 examples to
- 43:05
- help us track progress
- 43:07
- And this lets me go back to two prompts
- 43:09
- try different algorithms
- 43:10
- and so on to see if I can
- 43:11
- move up this metric of due date accuracy
- 43:15
- So this is what improving a
- 43:17
- agentic AI workflow will often feel like
- 43:19
- Look at the output
- 43:20
- see what's wrong, then if you know how to fix it
- 43:22
- just fix it. But if you need a
- 43:23
- longer process of improving it
- 43:24
- then put in place an eval and use
- 43:26
- that to drive further development
- 43:28
- One other thing to consider is if after
- 43:30
- working for a while
- 43:31
- if you think those 20 examples you had
- 43:33
- initially aren't good enough
- 43:34
- maybe they don't cover all the cases you want
- 43:36
- or maybe 20 examples is just too few
- 43:38
- then you can always add to the eval
- 43:40
- set over time to make sure it better
- 43:42
- reflects
- 43:43
- your personal judgments on whether or not the
- 43:45
- system's performance is sufficiently satisfactory
- 43:48
- This is just one example
- 43:49
- For the second example
- 43:50
- let's look at building a marketing copy assistant
- 43:53
- for writing captions for Instagram
- 43:56
- where to keep things succinct
- 43:57
- let's say our marketing team tells us that
- 43:59
- they want captions that are at most
- 44:01
- words long
- 44:02
- So we would have a image of a product
- 44:04
- say a pair of sunglasses that we want to market
- 44:07
- and then have a user query
- 44:08
- like please write a caption to sell these sunglasses
- 44:12
- and then have a
- 44:13
- OM or a large multimodal model
- 44:15
- analyze the image and the query
- 44:17
- and generate a description of the sunglasses
- 44:20
- And there are lots of different ways that
- 44:22
- a marketing copy assistance may go wrong
- 44:25
- but
- 44:25
- let's say that you look at the output
- 44:27
- and you find that the copy or the
- 44:28
- text generated mostly sounds okay
- 44:30
- but maybe it's just sometimes too long
- 44:32
- So for the sunglasses input
- 44:33
- generate 17 words. If you have a coffee machine
- 44:36
- it's okay. Stylish eyes
- 44:37
- okay. Blue shirt, 14 words
- 44:39
- Blender, 11 words. So it looks like in this example
- 44:42
- the OM is having a hard time adhering
- 44:45
- to the length guideline. So again
- 44:47
- there are lots of things that could have
- 44:48
- gone wrong with the marketing copy assistant
- 44:51
- but if you find that it's struggling with
- 44:52
- the length of the output
- 44:54
- then you might build an eval to track
- 44:56
- this
- 44:57
- so that you can
- 44:58
- make improvements
- 44:59
- and make sure it's getting better at it
- 45:02
- To the length guideline
- 45:03
- So to create an email to measure the text length
- 45:06
- what you might do is
- 45:07
- create a set of test stars
- 45:08
- So mark the pair of sunglasses
- 45:09
- a coffee machine, and so on
- 45:11
- and maybe create 10 to 20 examples
- 45:14
- Then
- 45:15
- you would run each of them through your
- 45:16
- system
- 45:17
- and write code to measure the word count
- 45:19
- of the
- 45:20
- output. So this is Python code to measure
- 45:22
- the word count of a piece of text
- 45:25
- Then lastly
- 45:26
- you would compare
- 45:26
- the length of the generated text
- 45:29
- to the 10 word target limit
- 45:32
- So if word count is equal to 10
- 45:33
- non-correct plus equals 1
- 45:35
- One difference between this
- 45:37
- and the previous invoice processing example is that
- 45:40
- there is no per example ground truth
- 45:43
- The target is just 10
- 45:45
- same for every single example
- 45:46
- Whereas in contrast, for the invoice processing example
- 45:49
- we had to generate a custom target label
- 45:52
- that is the correct due date of the invoice
- 45:55
- and we're testing the outputs
- 45:56
- against that per example ground truth
- 45:59
- I know I use a very simple workflow
- 46:00
- for Generating these captions
- 46:02
- but
- 46:02
- these type of evals can be applied to
- 46:04
- much more complex generation workflows as well
- 46:07
- Let me touch on one final example in
- 46:09
- which we'll revisit the research agents
- 46:12
- we've been looking at
- 46:13
- If you look at the output of
- 46:15
- the research agents on different input prompts
- 46:18
- let's say that
- 46:19
- when you ask it to write
- 46:21
- an article on recent breakthroughs in black hole science
- 46:24
- you find that it missed some high-profile result
- 46:27
- and a loss of news coverage
- 46:28
- So this is an unsatisfactory result
- 46:30
- Or if you ask it to
- 46:31
- research renting versus buying a home in Seattle
- 46:33
- well, it seems to do a good job
- 46:35
- Or robotics for harvesting fruit
- 46:37
- well, it didn't mention a leading equipment company
- 46:40
- So based on this evaluation
- 46:42
- it looks like sometimes it misses a really
- 46:44
- important point that a human expert writer
- 46:48
- would have captured. So
- 46:50
- then
- 46:50
- I would create an eval to measure how
- 46:52
- often it captures the most important points
- 46:55
- For example
- 46:56
- you might come up with a
- 46:57
- number of example prompts on black holes
- 47:00
- robotic harvesting, and so on
- 47:02
- And for each one
- 47:02
- come up with, let's say
- 47:04
- three to five gold standard discussion points
- 47:06
- for each of these topics
- 47:08
- Notice that here, we do have a per example annotation
- 47:12
- because the gold standard talking points
- 47:14
- that is the most important talking points
- 47:16
- they are different for each of these examples
- 47:18
- With these ground truth annotations
- 47:20
- you might then use an LMS judge to
- 47:22
- count
- 47:23
- how many of the gold standard talking points
- 47:26
- were mentioned. And so an example prompt might be to say
- 47:29
- determine how many of the five gold standard
- 47:31
- talking points are present in the provided essay
- 47:34
- Here are the original prompts
- 47:35
- the essay text, gold standard points, and so on
- 47:37
- and have it return a JSON object with
- 47:39
- two keys that scores how many of the points
- 47:42
- zero to five, did it score
- 47:43
- as well as an explanation
- 47:45
- And this allows you to get a
- 47:47
- score for each prompt
- 47:49
- in your evaluation set
- 47:51
- In this example
- 47:52
- I'm using O as a judge
- 47:54
- to count how many of the
- 47:56
- talking points were mentioned because there are so
- 47:58
- many different ways to talk about these talking points
- 48:01
- and so a
- 48:02
- regular expression or a code for
- 48:04
- simple pattern matching might not work that well
- 48:07
- which is why you might use
- 48:08
- O as a judge and treat this as
- 48:10
- a slightly more subjective evaluation for whether or not
- 48:13
- say, event horizons were adequately mentioned
- 48:16
- So this is your third example
- 48:18
- of how you might build evals
- 48:20
- In order to
- 48:21
- think about how to build evals for your application
- 48:25
- the evals you build will often
- 48:27
- have to reflect whatever
- 48:28
- you
- 48:29
- see
- 48:29
- or you are worried about going wrong in
- 48:31
- your application
- 48:33
- And it turns out that broadly
- 48:35
- there are two axes of evaluation
- 48:38
- On the top axis is the way you
- 48:40
- evaluate the output
- 48:43
- In some cases
- 48:44
- you evaluate it by writing code with objective evals
- 48:48
- and
- 48:49
- sometimes you use an OM
- 48:51
- as a judge for more subjective evals
- 48:54
- On the other axis
- 48:55
- is whether you have a per example
- 48:58
- ground truth or not
- 49:00
- So for checking invoice data extraction
- 49:03
- we were writing code to evaluate if we
- 49:05
- got the actual date
- 49:07
- and that had a per example ground truth
- 49:10
- because each invoice has a different actual date
- 49:13
- But in the example where we checked marketing copy length
- 49:16
- every example had a length limit of 10
- 49:20
- and so
- 49:20
- there was no per example ground truth for
- 49:23
- that problem
- 49:24
- In contrast, for counting gold standard talking points
- 49:27
- there was a per example ground truth because
- 49:30
- each article had different important talking points
- 49:33
- But we use an element as a judge
- 49:34
- to read the essay to see if those
- 49:36
- topics were adequately mentioned
- 49:38
- because there's so many different ways to mention
- 49:40
- the talking points
- 49:41
- And
- 49:42
- the last of the four quadrants would be
- 49:44
- a message judge
- 49:45
- with no per example ground truth
- 49:48
- And
- 49:48
- one place where we saw that was
- 49:50
- if you are grading charts with a rubric
- 49:52
- this is when we're looking at visualizing the
- 49:55
- coffee machine sales. And if you ask it
- 49:57
- to grade a chart according to a rubric
- 49:59
- such as whether it's clear access labels and so on
- 50:02
- there is the same rubric for every chart
- 50:05
- And that would be using an element as a judge
- 50:07
- but with validate per example ground truth
- 50:09
- So I find this two by two grid
- 50:10
- as maybe a useful way
- 50:12
- to think about the different types
- 50:13
- evals you might construct for your application
- 50:16
- And by the way
- 50:17
- these are sometimes also called end-to-end evals because
- 50:20
- one end is the impedance
- 50:21
- which is the user query prompt
- 50:23
- and the other end is the final output
- 50:26
- And so all of these are evals for
- 50:28
- the entire end-to-end system's performance
- 50:31
- So just to wrap up this video
- 50:33
- I'd like to share a few final tips
- 50:35
- for designing end-to-end evals. First
- 50:37
- quick and dirty evals is fine to get
- 50:39
- started
- 50:40
- I feel like I see
- 50:42
- quite a lot of teams that are almost
- 50:44
- paralyzed because they think building evals is this massive multi-week effort
- 50:48
- and so
- 50:49
- they take longer than would be ideal to
- 50:51
- get started
- 50:53
- But
- 50:53
- I think just as you
- 50:54
- iterate on an agentic workflow and make it better over time
- 50:57
- you should plan to iterate on your evals
- 51:00
- as well. So if you put in place 10
- 51:02
- 15, 20 examples as your first cut at evals
- 51:06
- and write some code
- 51:07
- or try prompting them as a judge
- 51:08
- just do something to start to get some metrics
- 51:11
- and that can complement the human At looking at the output
- 51:14
- and then it's a blend of the two
- 51:16
- that can drive your decision making
- 51:17
- And as the evals become more sophisticated over time
- 51:20
- you can then
- 51:21
- shift more and more of your trust to
- 51:22
- the metric-based evals rather than needing to read over
- 51:26
- you know
- 51:26
- hundreds of
- 51:27
- outputs every time you tweak a prompt somewhere
- 51:29
- And as you go through this process
- 51:31
- you'll likely find ways
- 51:33
- to keep on improving your evals as well
- 51:35
- So if you had 20 examples to start
- 51:38
- you may then run into places where
- 51:40
- your evals fail to capture your judgment
- 51:44
- about what system is better
- 51:46
- So maybe
- 51:47
- you update the system and you look at
- 51:49
- it and you feel like this has got
- 51:51
- to be worked much better
- 51:52
- but your eval
- 51:53
- fails to show the new system achieving a
- 51:55
- higher score. If that's the case
- 51:57
- that's often an opportunity to go maybe
- 52:00
- collect a larger eval set or change the
- 52:02
- way you evaluate the output to make it
- 52:04
- correspond better to your judgment as to
- 52:07
- what system is actually working better
- 52:08
- And so your evals will get better over time
- 52:11
- And lastly
- 52:11
- in terms of using evals To gain inspiration
- 52:14
- as to what to work on next
- 52:16
- a lot of agent workflows are being used
- 52:18
- to automate tasks that
- 52:19
- say, humans can do
- 52:21
- And so I find for such applications
- 52:23
- I'll look for places where the performance is
- 52:25
- worse than that of an expert human
- 52:27
- and that often gives me inspiration
- 52:29
- for where to focus my efforts or what
- 52:31
- are the types of examples that I may
- 52:33
- relate get my agentic workflow
- 52:35
- to work better than it is currently
- 52:37
- So
- 52:38
- I hope that after you've built that quick and dirty system
- 52:41
- you'll think about when it would make sense
- 52:42
- to
- 52:43
- start putting in
- 52:44
- some evals to track the potentially problematic aspects of the system
- 52:48
- and that that will then help you
- 52:49
- drive improvements to the system
- 52:52
- In addition to helping you drive improvements
- 52:54
- it turns out that
- 52:55
- there's a method of evals that helps you
- 52:58
- hone in Of your entire agentic system
- 53:01
- what are the components most worth focusing
- 53:04
- your attention on? Because agentic systems often have many pieces
- 53:08
- so
- 53:09
- which piece is going to be most productive
- 53:11
- for you to spend time working to improve?
- 53:14
- Turns out being able to do this well
- 53:16
- is a really important skill for driving efficient
- 53:19
- development of agentic workflows
- 53:21
- In the next video
- 53:22
- I'd like to deep dive into this topic
- 53:24
- So let's go on to the next video
- 53:27
- Let's say you built an agentic workflow
- 53:29
- and it is not yet working as well
- 53:31
- as you wish. And this happens to be all the time
- 53:33
- by the way. I'll often build a quick and dirty system
- 53:36
- and it doesn't do as well as
- 53:37
- I wish it would
- 53:38
- The question is
- 53:40
- where do you focus your efforts to make
- 53:42
- it better? It turns out agentic workflows have many different components
- 53:46
- and working on some of the components could
- 53:48
- be much more fruitful
- 53:49
- than working on some other components
- 53:51
- So your skill
- 53:52
- at choosing where to focus your efforts
- 53:55
- makes a huge difference in the speed with
- 53:57
- which you can make improvements to your system
- 53:59
- And I found that one of the
- 54:01
- biggest predictors for how efficient and how good a team is
- 54:05
- is whether or not
- 54:06
- they're able to drive a disciplined
- 54:08
- error analysis process to tell you
- 54:11
- where to focus your efforts
- 54:12
- So this is an important skill
- 54:14
- Let's take a look
- 54:15
- at how the error analysis
- 54:17
- In the research agent example
- 54:20
- we had carried out an error analysis in the previous video
- 54:23
- and we saw that it was
- 54:25
- often
- 54:25
- missing key Points that a human expert would
- 54:28
- have made in writing essays on certain topics
- 54:30
- So
- 54:31
- now that you've spotted this
- 54:33
- problem that is sometimes missing key points
- 54:35
- how do you know what to work on?
- 54:37
- It turns out that of the many different
- 54:39
- steps in this workflow
- 54:41
- almost any of them could have contributed to
- 54:43
- this problem of missing key points
- 54:46
- For example
- 54:46
- maybe the first LLM was generating search terms that weren't great
- 54:50
- so
- 54:51
- it was just searching for the wrong things
- 54:53
- and did not discover the right articles
- 54:55
- Or maybe use a web search engine that
- 54:57
- just wasn't very good. There are multiple web
- 54:59
- search engines out there. In fact
- 55:00
- there are actually quite a few that I
- 55:01
- tend to use for my own based applications
- 55:04
- and some are better than others
- 55:06
- Or maybe web search was just fine
- 55:09
- but when we gave the list of web
- 55:10
- search results to an LLM
- 55:11
- maybe it didn't do a good job choosing
- 55:13
- the best handful to download
- 55:16
- Maybe web fetch has few problems in this case
- 55:19
- assuming you can fetch web pages accurately
- 55:21
- But after dumping the web pages in the LLM
- 55:23
- maybe the LLM is
- 55:25
- ignoring some of the points
- 55:26
- In the documents we had fetched
- 55:28
- So it turns out that there are teams
- 55:30
- that will sometimes look at this and go
- 55:32
- by gut to pick one of these components to work on
- 55:36
- and sometimes that works
- 55:37
- and sometimes that leads to many months of
- 55:39
- work with very little progress in the
- 55:41
- overall performance of the system
- 55:44
- So rather than
- 55:45
- going by gut to decide which of these
- 55:47
- many components to work on
- 55:49
- I think it's much better to
- 55:51
- carry out an error analysis
- 55:53
- to better understand each step in the workflow
- 55:56
- And in particular, I'll often examine the traces
- 55:59
- and that means the intermediate output after each step
- 56:02
- in order to understand which component's performance is subpar
- 56:06
- meaning, say, much worse than what a human expert would do
- 56:10
- because that
- 56:11
- points to where
- 56:13
- there may be room for significant improvement
- 56:15
- Let's look at an example
- 56:16
- If we ask the research agent to write
- 56:18
- an essay about recent news in black hole science
- 56:21
- maybe the outputs search terms like these
- 56:24
- search for black hole theory
- 56:25
- science, event horizon. telescope radio
- 56:27
- and so on
- 56:28
- And I would then have a human expert
- 56:29
- look at these and see
- 56:31
- are these reasonable
- 56:32
- web search terms for
- 56:34
- writing about recent discoveries in black hole science
- 56:37
- And maybe in this case
- 56:38
- an expert says these web searches look okay
- 56:41
- They're, you know
- 56:42
- pretty similar to what
- 56:43
- I would do as a human
- 56:45
- Then I look at
- 56:46
- the outputs of the
- 56:47
- web search and look at the URLs returned
- 56:51
- So web search will return many different web pages
- 56:54
- and maybe
- 56:55
- one web page returns is that an elementary
- 56:57
- school student
- 56:58
- claims to track a 30-year-old
- 57:00
- black hole mystery from Astro Kid News
- 57:02
- And
- 57:03
- this doesn't look like the most rigorous peer
- 57:05
- review article. And maybe examining
- 57:08
- all of the articles that web search
- 57:10
- returns causes you to conclude that it's returning
- 57:13
- too many
- 57:14
- blog or popular press types of articles and
- 57:17
- not enough scientific articles to write a
- 57:19
- research report of the quality that you are
- 57:22
- looking for. It'd be good to
- 57:23
- just
- 57:23
- look through the outputs of the other steps
- 57:25
- as well. Maybe the Find the best five sources you can
- 57:28
- and then with AstroKit News
- 57:30
- SpaceBot 2000, SpaceFun News, and so on
- 57:33
- And it is by looking at these
- 57:35
- intermediate outputs
- 57:36
- that you can then try to get a
- 57:38
- sense of the quality of the output of
- 57:39
- each of these steps. To introduce some terminology
- 57:42
- the
- 57:42
- overall
- 57:43
- set of outputs of all of the intermediate
- 57:45
- steps is often called the trace of a
- 57:48
- run of this agent
- 57:50
- And then some terminology you see
- 57:51
- in
- 57:52
- other
- 57:52
- sources as well is the output of a
- 57:54
- single step is sometimes called a span
- 57:57
- This is terminology from
- 57:59
- the
- 57:59
- computer observability literature where
- 58:02
- people try to figure out what computers are
- 58:04
- doing. And in this course
- 58:05
- I use the word trace quite a bit
- 58:07
- I'll use the word span a little bit less
- 58:09
- but you may see both of these terms
- 58:11
- on the internet
- 58:12
- So by reading the traces
- 58:14
- you start to get an informal sense of
- 58:16
- where might be the most problematic components
- 58:19
- In order to do this in the most systematic way
- 58:21
- it turns out to be useful to focus
- 58:23
- your attention
- 58:24
- on the cases that the system is Doing
- 58:26
- poorly on. Maybe write some essays just fine
- 58:29
- and the output is completely satisfactory
- 58:31
- So I would put those aside and try
- 58:32
- to come up with a set of examples where
- 58:35
- for whatever reason
- 58:36
- the final output of your research agent is not quite satisfactory
- 58:39
- and just focus on those examples
- 58:42
- So just one of the reasons we call error analysis
- 58:44
- because we want to focus on the cases
- 58:46
- where the system made an error
- 58:48
- and we want to go
- 58:49
- through to figure out which components
- 58:51
- were most responsible for the error
- 58:54
- in the research agent output
- 58:56
- In order to make this more rigorous
- 58:58
- rather than reading and getting an informal sense
- 59:01
- you might
- 59:01
- actually build up a spreadsheet to more explicitly
- 59:05
- count up where the errors are
- 59:07
- And by error
- 59:08
- I mean when a step outputs something
- 59:10
- that
- 59:11
- performs significantly worse than maybe what a human
- 59:14
- expert would have given a similar
- 59:16
- input as that component. So
- 59:18
- I'll often do this myself in a spreadsheet
- 59:20
- So I might build a spreadsheet like this
- 59:22
- and so for the first query
- 59:23
- I'll look at recent developments in black hole science
- 59:26
- and I see That the search results has
- 59:28
- too many blog posts
- 59:29
- popular press articles, not enough scientific papers
- 59:32
- And then based on this
- 59:34
- it is true that the five best sources aren't great
- 59:37
- but
- 59:37
- here
- 59:38
- I won't say that the five best sources
- 59:40
- did a bad job because if the
- 59:41
- inputs
- 59:42
- to
- 59:43
- the arm for selecting the five best sources
- 59:45
- were all non-rigorous articles
- 59:47
- then I can't blame this
- 59:48
- picking the five best sources for
- 59:50
- not picking better articles because it did the
- 59:52
- best it could have or as
- 59:53
- what did nearly as well as any human
- 59:55
- might have given the same selection to choose
- 59:57
- from. And then you might go through this for different prompts
- 60:00
- renting versus buying in Seattle
- 60:03
- maybe it missed a well-known blog
- 60:05
- robotics for housing fruit
- 60:07
- maybe in this case we look at it and say
- 60:08
- oh
- 60:08
- the search terms are too generic and the
- 60:10
- search results also weren't good and so on
- 60:13
- And then based on this
- 60:14
- I would
- 60:14
- count up in my spreadsheet
- 60:16
- how often I observe errors
- 60:19
- in the different components. So in this example
- 60:21
- dissatisfied with the search terms
- 60:23
- 5% of the time
- 60:24
- but I'm dissatisfied with the search results
- 60:26
- 5% of the time. And if I actually see this
- 60:29
- I might just take a careful look at
- 60:30
- the search terms to make sure that the
- 60:32
- search terms really were okay and that poor
- 60:34
- choice of search terms were not what led
- 60:36
- to poor search results. But I really think
- 60:38
- the search terms are fine
- 60:39
- but the search results are not
- 60:40
- Then I would take a careful look at
- 60:42
- the web search engine I'm using and if
- 60:44
- there are any parameters I can tune to
- 60:45
- make it bring back more
- 60:47
- relevant or higher quality results
- 60:49
- There's this type of analysis that tells me
- 60:51
- in this example
- 60:52
- that maybe I really should focus
- 60:53
- my attention on fixing the search results and
- 60:56
- not on the other components of this
- 60:58
- agentic workflow
- 61:00
- So to wrap up this video
- 61:02
- I find that it's useful to
- 61:03
- develop a habit of looking at traces
- 61:06
- After you build an agentic workflow
- 61:08
- go ahead and look at the intermediate outputs
- 61:10
- to get a feel for what it's actually
- 61:11
- doing on every step so that you can
- 61:13
- better understand if
- 61:14
- different steps are performing better or worse
- 61:17
- And a more systematic error analysis may be
- 61:19
- done
- 61:20
- with a spreadsheet can let you gather statistics
- 61:23
- or count up which component performs poorly most
- 61:25
- frequently. And so by looking at what components are doing poorly
- 61:29
- as well as
- 61:30
- where I have ideas for efficiently improving different components
- 61:35
- then that will let you prioritize what component
- 61:37
- to work on. So maybe a component is problematic
- 61:40
- but I don't have any ideas for improving
- 61:42
- it. So that would suggest
- 61:44
- maybe not prioritizing that as high
- 61:45
- But if there is a component
- 61:47
- that is generating a lot of errors
- 61:49
- and if I have ideas how to improve that
- 61:51
- then that would be a good reason
- 61:52
- to prioritize working on that component
- 61:54
- And
- 61:55
- I just want to emphasize that
- 61:57
- error analysis
- 61:58
- is a very helpful output
- 62:00
- for you to decide
- 62:01
- where to focus your efforts
- 62:02
- Because in any complex system
- 62:05
- there are just so many things you could
- 62:06
- work on
- 62:07
- It's too easy
- 62:08
- to pick something to work on and work
- 62:10
- on it for weeks or even months
- 62:11
- only to discover later that that did not
- 62:14
- result in improved performance to your overall system
- 62:17
- And so using error analysis to decide where
- 62:19
- to focus your effort turns out to be
- 62:20
- incredibly useful for improving your efficiency
- 62:24
- In this video, we went over error analysis
- 62:27
- With the research agent example
- 62:30
- But I think error analysis is such an
- 62:31
- important topic. I want to go over some
- 62:33
- additional examples with you
- 62:35
- So let's go on to the next video
- 62:37
- where we'll look at more examples of
- 62:39
- error analysis
- 62:41
- An agentic workflow may comprise many different types of components
- 62:45
- and so your tools to improving different components
- 62:48
- will be pretty different
- 62:49
- But I'd like to share with you some
- 62:51
- general patterns I've seen
- 62:53
- Some components
- 62:53
- your agentic workflow will be non-OLM based
- 62:56
- So it may be
- 62:57
- something like a web search engine or a text retrieval component
- 63:01
- if that's part of your rack or retrieval augmented generating system
- 63:05
- something for code execution
- 63:06
- or maybe a separately trained machine learning model
- 63:09
- maybe with
- 63:09
- speech recognition or digital people and pictures and
- 63:12
- so on. So
- 63:13
- sometimes
- 63:14
- these non-OLM based components will have parameters or
- 63:17
- hyperparameters that you can tune
- 63:19
- So for web search
- 63:19
- you can tune things like the number of
- 63:21
- results or maybe the date range that you
- 63:23
- also web search engine to consider
- 63:25
- For a rack text retrieval components
- 63:28
- you might change the similarity threshold that determines
- 63:31
- what pieces of text it considers similar or
- 63:33
- the chunk size. Often rack systems will take texting
- 63:36
- chop it up into smaller chunks for matching
- 63:39
- So the main hyperparameters you could use Or for people detection
- 63:42
- you might change the detection threshold
- 63:44
- So how sensitive it is and how likely
- 63:46
- it is to declare this found a person
- 63:48
- and just to trade off the false positives
- 63:50
- and false negatives. If they follow all the
- 63:52
- details of the hyperparameters I just discussed
- 63:54
- don't worry about it. The details aren't that important
- 63:56
- but often the components were parameters that
- 63:58
- you can tune. And then of course
- 63:59
- you can also try to replace the component
- 64:02
- I do this a lot in my agentic
- 64:04
- workflows where
- 64:05
- I'll swap in different rack search engines or
- 64:07
- swap in different rack providers and so on
- 64:09
- just to see if some other provider
- 64:11
- might work better. Because of the diversity of non-OM based components
- 64:16
- I think the techniques for how to improve
- 64:18
- it will be
- 64:18
- more diverse and dependent of exactly what that
- 64:21
- component is doing. For an OM based component
- 64:25
- here are some options you might consider
- 64:27
- One would be
- 64:28
- to try to improve your prompts
- 64:30
- So
- 64:31
- maybe try to add more explicit instructions
- 64:33
- or if you know what few shot prompting is
- 64:35
- that refers to
- 64:36
- adding
- 64:37
- one or more concrete examples of an example
- 64:40
- of an input. The design output
- 64:42
- and so few-shot prompting
- 64:43
- which you can learn about from some deep
- 64:45
- learning AI short courses as well
- 64:47
- is a technique that can
- 64:48
- give your LM some examples to hopefully help
- 64:51
- it get better performing outputs written
- 64:54
- Or you can also try a different LM
- 64:57
- So with ASV or other tools
- 64:59
- it could be pretty easy to try multiple LMs
- 65:02
- and then you can use evals
- 65:03
- to pick the best model for your application
- 65:06
- Sometimes
- 65:07
- if a single step is too complex for
- 65:09
- one LM to do
- 65:10
- you can consider if you want to decompose
- 65:12
- the task into smaller steps
- 65:13
- or maybe decompose it into a generation step
- 65:16
- and then a reflection step
- 65:17
- But more generally, if you have instructions that are very complex
- 65:21
- all within one step
- 65:22
- maybe a single LM has a hard time
- 65:24
- following all those instructions
- 65:25
- and if you break the task down to smaller steps
- 65:27
- that may be easier for
- 65:29
- say
- 65:30
- two or three calls to an LM
- 65:31
- to carry out accurately. And lastly
- 65:34
- something to try when the other methods aren't
- 65:36
- working well enough is to consider fine-tuning
- 65:39
- a model. This tends to be
- 65:41
- quite a bit more complex than the other options
- 65:43
- so it can be quite a bit more
- 65:44
- expensive as well in terms of developer time
- 65:46
- to implement. But if you have some data
- 65:49
- that you can use to fine-tune an LLM on
- 65:53
- that could
- 65:54
- give you much better performance than prompting alone
- 65:57
- So
- 65:57
- I tend not to fine-tune a model until
- 65:59
- I've really exhausted the other options
- 66:01
- because fine-tuning tends to be quite complex
- 66:04
- But for applications where after trying everything else
- 66:07
- if I'm still at
- 66:08
- you know, say 90% performance or 95% performance
- 66:12
- and I really need to eke out those
- 66:13
- last few percentage points of improvement
- 66:15
- then sometimes fine-tuning
- 66:17
- my own custom model is a great technique
- 66:19
- to use
- 66:19
- I tend to do this only on the
- 66:21
- more mature
- 66:22
- applications because of how costly it is
- 66:25
- It turns out that when you are
- 66:27
- trying to choose an LLM to use
- 66:29
- one thing that's very hopeful
- 66:30
- for
- 66:31
- you as a developer is if you have
- 66:32
- good intuitions about how intelligent or how capable
- 66:35
- different large language models are
- 66:37
- One thing you can do is just try
- 66:38
- a lot of models and see what works
- 66:40
- best. But I find that As I work with different models
- 66:42
- I start to hone intuitions about
- 66:44
- which models work best for what types of
- 66:46
- tasks. And we hone those intuitions
- 66:48
- you can be
- 66:49
- more efficient as well in
- 66:50
- writing good prompts for the model as well
- 66:52
- as choosing good models for your tasks
- 66:55
- So I'd like to share with you
- 66:56
- some
- 66:57
- thoughts on how to hone your intuition
- 66:59
- on
- 67:00
- what models will work well for your application
- 67:03
- Let's illustrate this with an example
- 67:05
- of using an LLM
- 67:06
- to follow instructions to remove or to redact
- 67:09
- PII or personally identifiable information
- 67:12
- So get down to remove private sensitive information
- 67:15
- For example
- 67:16
- if you are using an LLM to summarize customer calls
- 67:20
- then maybe one summary is on July 14th
- 67:23
- 2023
- 67:24
- Jessica Alvarez with a social security number and a certain address
- 67:27
- a support ticket, and so on
- 67:29
- So this piece of text has a lot of sensitive
- 67:32
- personally identifiable information. Now
- 67:35
- let's say we want to remove all PII
- 67:37
- from
- 67:37
- such
- 67:38
- summaries because we want to use the data
- 67:40
- for Downstream Cisco analysis of what customers are calling about
- 67:44
- and to protect customer information
- 67:45
- we want to strip out that PII before
- 67:47
- we do that downstream Cisco analysis
- 67:50
- So you might prompt an alarm with
- 67:52
- instructions to identify all cases of PII in the text below
- 67:56
- and then return the redacted text with redacted
- 67:59
- colon and so on
- 68:01
- It turns out
- 68:02
- that
- 68:03
- the larger frontier models tend to be much
- 68:06
- better at following instructions
- 68:08
- whereas the smaller models
- 68:10
- tend to be pretty good at answering simple factual questions
- 68:13
- but are just not as good at following
- 68:15
- instructions. If you run this prompt on a smaller model
- 68:18
- the OpenWay Llama 3.1 model with A billion parameters
- 68:21
- then it may generate an output like this
- 68:23
- It says, the identified PII is social security number and address
- 68:27
- and then it redacts it as follows
- 68:29
- and so on. And it actually makes a
- 68:31
- few errors. It didn't follow the instructions properly
- 68:34
- It showed the list
- 68:35
- then redacted the text
- 68:36
- then returned another list
- 68:38
- which it wasn't supposed to
- 68:40
- and in this list of PII
- 68:42
- it missed the name
- 68:43
- and then I think it also
- 68:44
- didn't redact part of the address
- 68:46
- So details aren't important
- 68:47
- but didn't follow these instructions perfectly
- 68:49
- and maybe it missed a little bit of PII
- 68:52
- In contrast, if you use a more intelligent model
- 68:54
- one that's better at following instructions
- 68:56
- you may get a better result like this
- 68:58
- where it's actually correctly listed all the PII
- 69:01
- Correctly redacted all of the PII
- 69:04
- And so
- 69:05
- I find that as different OM providers specialize on different tasks
- 69:09
- different models really are better for different tasks
- 69:12
- Some are better at coding
- 69:13
- some are better at following instructions
- 69:15
- some are better at certain niche types of facts
- 69:18
- and we can hone your intuition for
- 69:20
- what models
- 69:21
- are more or less intelligent
- 69:23
- and what type of instructions they're more or
- 69:25
- less able to follow
- 69:26
- then you'll be able to make better decisions
- 69:28
- as to what models to use
- 69:30
- So to share a couple tips on how to do this
- 69:32
- I encourage you to play with different models
- 69:34
- often. So whenever there's a new model release
- 69:37
- I'll often go try it out and try
- 69:39
- out different queries on it
- 69:40
- both closed-weight proprietary models as well as open-weight
- 69:43
- models. And
- 69:45
- I find that sometimes having a personal set
- 69:46
- of evals might also be helpful where there's
- 69:49
- a set of things you ask a lot
- 69:50
- of different models that might help you calibrate
- 69:52
- how well they do
- 69:53
- on different types of tasks
- 69:56
- One of the things that I do a
- 69:57
- lot that I hope will be useful to
- 69:59
- you is
- 70:00
- I spend a lot of time reading other
- 70:01
- people's prompts. So sometimes people publish their prompts on the internet
- 70:06
- and I often go and read them to
- 70:08
- understand what best practices in prompting looks like
- 70:11
- or I often chat to my friends at various companies
- 70:14
- including some of the frontier model companies
- 70:16
- and kind of share my prompts with them
- 70:17
- take a look at how they prompt
- 70:19
- And sometimes I'll also go to open source packages of
- 70:23
- you know
- 70:23
- written by people I really respect
- 70:25
- and download the open source package and dig
- 70:27
- through that open source package
- 70:29
- to find the prompts the authors have written
- 70:31
- in order to read it
- 70:32
- in order to hone my intuition about how
- 70:34
- to
- 70:35
- write good prompts
- 70:36
- This is one technique that I encourage you
- 70:38
- to consider
- 70:39
- is by reading lots of other people's prompts
- 70:41
- that will help you get better at writing prompts yourself
- 70:44
- And, I certainly do this a lot
- 70:46
- and I encourage you to do so too
- 70:48
- And this will hone your intuition about
- 70:51
- what types of instructions models are good at
- 70:53
- following and
- 70:54
- when to say certain things to different models
- 70:56
- In addition to playing with models and reading other people's prompts
- 70:59
- if you try out lots of different models
- 71:01
- in your agentic workflows
- 71:03
- that also lets you hone your intuition
- 71:05
- So you see
- 71:06
- which models work best for which types of tasks
- 71:09
- and either looking at traces to
- 71:11
- get it in a formal sense
- 71:13
- or looking at either component-wise
- 71:15
- or end-to-end evals can help you assess how
- 71:17
- well different models are working
- 71:19
- different policy workflow. And then you start to
- 71:21
- hone intuitions about not just performance
- 71:23
- but maybe also
- 71:25
- price and speed trade-offs
- 71:27
- for the use of different models
- 71:28
- And
- 71:29
- one of the reasons I tend to develop
- 71:31
- my agentic workflows of AI Suite is because
- 71:33
- it then makes it easy to quickly swap
- 71:35
- out and try out different models
- 71:36
- And this makes me more efficient in terms
- 71:38
- of trying out and assessing
- 71:41
- which models work best for my workflow
- 71:43
- So we've talked a lot about how to
- 71:45
- improve the performance of different components
- 71:49
- So hopefully improve the overall performance of your
- 71:52
- end-to-end system
- 71:53
- In addition to improving the quality of the output
- 71:57
- one of the thing you might want to
- 71:58
- do in your workflows
- 71:59
- is to optimize the latency
- 72:01
- as well as cost
- 72:03
- I find that for a lot of teams
- 72:04
- when you start developing
- 72:05
- usually the number one thing to worry about
- 72:07
- is just are the outputs sufficiently high quality
- 72:11
- But then
- 72:11
- when the
- 72:12
- system is working well and you put it into production
- 72:15
- then there's often value to make it run
- 72:17
- faster
- 72:18
- as well as run at lower cost as
- 72:19
- well
- 72:20
- So in the next video
- 72:21
- let's take a look
- 72:22
- at some ideas for improving
- 72:24
- cost and latency for agentic workflows
- 72:28
- When building agentic workflows
- 72:30
- I often advise teams to focus on getting
- 72:33
- high quality outputs
- 72:34
- and to optimize cost and latency only later
- 72:37
- It's not that cost and latency don't matter
- 72:40
- but I think
- 72:41
- getting the performance or the
- 72:43
- output quality to be high is usually
- 72:45
- the hardest part
- 72:47
- And then only when it's really working
- 72:49
- then maybe focus on the other things
- 72:51
- One thing that's happened to me a few
- 72:52
- times was
- 72:53
- my team built an agentic workflow and we
- 72:55
- shipped to the users
- 72:57
- and then we're fortunate enough to have so
- 72:59
- many users use it
- 73:00
- that the cost actually became a problem
- 73:02
- And then we had to
- 73:03
- you know
- 73:03
- scramble to bring the cost back down
- 73:06
- But that's a good problem to have
- 73:07
- So I tend to
- 73:08
- worry about cost usually last
- 73:11
- not that I ignore it completely
- 73:13
- but it's just lower down my list of
- 73:14
- things to worry about until we have so
- 73:16
- many users that we really
- 73:17
- need to bring the cost down per user
- 73:20
- And then latency, I tend to worry a bit about it
- 73:23
- but again
- 73:24
- not as much as just
- 73:25
- making sure the output quality is high
- 73:28
- But when you do get there
- 73:29
- it will be useful to have tools to
- 73:31
- optimize latency and cost
- 73:33
- Let's take a look at some ideas
- 73:34
- on how to do that
- 73:35
- If you want to optimize the latency of energetic workflow
- 73:40
- one thing I will often do
- 73:41
- is then Benchmark or time the workflow
- 73:45
- So in this research agent
- 73:47
- it takes a number of steps
- 73:49
- and if I were to time each of the steps
- 73:51
- maybe it
- 73:52
- takes seven seconds to generate the search terms
- 73:55
- Web search takes five seconds
- 73:56
- this takes three seconds
- 73:57
- this takes eleven seconds
- 73:58
- and then writing the final essay takes eighteen
- 74:00
- seconds on average
- 74:02
- And it is then by looking at this
- 74:03
- overall timeline that I can get a sense
- 74:06
- of
- 74:07
- which components have the most room for
- 74:10
- making faster
- 74:12
- In this example
- 74:13
- there are maybe multiple things you could try
- 74:15
- If you haven't already taken advantage of parallelism for some steps
- 74:18
- like maybe wet fetch
- 74:20
- maybe it's worth considering
- 74:22
- doing some of these operations in parallel
- 74:25
- Or if you find that some of the
- 74:26
- LM sets are taking too long
- 74:28
- so if this first step takes seven seconds
- 74:30
- this last LM step takes eighteen seconds
- 74:33
- I might also consider trying a smaller
- 74:35
- maybe
- 74:36
- slightly less intelligent model to see if it
- 74:38
- still works well enough for that
- 74:39
- or if I can find a faster LM
- 74:41
- provider. There are lots of APIs online for different
- 74:44
- interfaces
- 74:45
- and some companies have specialized hardware to allow
- 74:48
- them to serve certain elements much faster
- 74:51
- So sometimes it's worth trying different element providers
- 74:53
- to see which ones can return tokens
- 74:56
- the fastest
- 74:57
- But at least doing this type of timing
- 74:59
- analysis can give you a sense of which
- 75:01
- components to focus on in terms of
- 75:03
- reducing latency
- 75:05
- In terms of optimizing costs
- 75:07
- a similar calculation where you calculate the cost
- 75:10
- of each step would also let you benchmark
- 75:12
- and decide which steps to focus on
- 75:14
- Many element providers charge per
- 75:16
- token based on the input and output length
- 75:18
- Many API providers charge per API call
- 75:22
- and the computational steps
- 75:24
- may have different costs based on how you
- 75:25
- pay for server capacity and how much the
- 75:27
- service costs. And so for a process like this
- 75:31
- you might decide in this example that the
- 75:33
- tokens for this element step
- 75:35
- on average cost 0.004 cents
- 75:38
- Each web search API maybe cost 1.6 cents
- 75:41
- Tokens cost This much API call cost this much
- 75:44
- PDF to text cost this much
- 75:45
- tokens for the final essay generation cost this much
- 75:48
- and this would maybe
- 75:48
- again give you a sense of are there
- 75:50
- cheaper components you could use or cheaper arms
- 75:53
- you could use to see where the biggest
- 75:55
- opportunity is for optimizing costs
- 75:57
- And I found that these benchmarking exercises can
- 76:00
- be very clarifying and sometimes they'll clearly tell
- 76:02
- me that certain components are just not worth
- 76:05
- worrying about because they're not that material
- 76:07
- or contributor to either cost or to latency
- 76:10
- So
- 76:11
- I find that wheneither cost or latency becomes
- 76:13
- an issue
- 76:14
- by simply measuring the cost
- 76:17
- and or latency of each step
- 76:18
- that often gives you a basis
- 76:20
- with which to decide which components to focus
- 76:23
- on optimizing
- 76:25
- So we're nearly at the end of this
- 76:27
- module. I know we've covered a lot
- 76:29
- but thank you for sticking with me
- 76:30
- Let's go on to the final video of
- 76:32
- this module to wrap up
- 76:34
- We've gone through a lot of tips for driving a disciplined
- 76:38
- efficient process for building agentic AI systems
- 76:41
- I'd love to wrap up by
- 76:43
- sharing with you
- 76:44
- what it feels like to be going through
- 76:45
- this process
- 76:47
- When I'm building these workflows
- 76:49
- I feel like there are two major activities
- 76:51
- I'm often spending time on
- 76:52
- One is building, so writing software
- 76:54
- trying to write code to improve my system
- 76:56
- And the second, which sometimes doesn't feel like progress
- 77:00
- but I think is equally important
- 77:02
- is analysis to help me decide where to
- 77:05
- focus my build efforts next
- 77:06
- And I often go back and forth between building and analyzing
- 77:10
- including things like error analysis
- 77:12
- So for example, when building a new agentic workflow
- 77:15
- I'll often start by quickly building an end-to-end system
- 77:19
- maybe even a quick and dirty implementation
- 77:21
- And this lets me then start to
- 77:23
- examine the final outputs of the end-to-end system
- 77:25
- or also
- 77:26
- read through traces to get a sense of
- 77:28
- where it's doing well
- 77:29
- where it's doing poorly. Based on even just locating traces
- 77:32
- sometimes this will give me Got sense
- 77:35
- of which individual components I might want to
- 77:38
- improve. And so I might go tune some
- 77:39
- individual components or keep tuning
- 77:42
- the overall end-to-end system
- 77:44
- As my system starts to mature a little bit more
- 77:46
- then beyond just
- 77:48
- manually examining a few outputs and reading through traces
- 77:51
- I might start to build evals
- 77:53
- and have a small data set
- 77:54
- maybe just 10, 20 examples to compute metrics
- 77:57
- at least on end-to-end performance
- 77:59
- And this then further
- 78:01
- helps me have a more refined perspective
- 78:04
- on how to improve the end-to-end system or
- 78:06
- how to improve individual components
- 78:07
- As it matures even further
- 78:09
- my analysis then becomes maybe even more disciplined
- 78:12
- where I'll start to do error analysis
- 78:14
- and look through the components and try to
- 78:16
- count up how frequently individual components
- 78:19
- led to
- 78:20
- subpar outputs
- 78:21
- And this more rigorous analysis then lets me
- 78:23
- be
- 78:24
- even more focused in deciding what components to
- 78:26
- work on next or inspire ideas for improving
- 78:29
- the overall end-to-end system. And then eventually
- 78:31
- when it's even more mature
- 78:32
- to drive more efficient Improvements at the component level
- 78:36
- that's when I might also build component level
- 78:39
- evals. And so
- 78:40
- the workflow of building an agentic system often
- 78:42
- goes back and forth. It's not a linear
- 78:44
- process. We sometimes tune the engine system
- 78:47
- then do some error analysis
- 78:48
- then improve a component for a bit
- 78:50
- then tune the component level evals
- 78:52
- and
- 78:52
- I tend to bounce back and forth between
- 78:54
- these two types of techniques
- 78:56
- And what I see
- 78:57
- less experienced teams often do is
- 78:59
- spend a lot of time building
- 79:02
- and probably much less time analyzing with error analysis
- 79:05
- building evals, and so on
- 79:06
- that would be
- 79:07
- ideal because it's this analysis that helps you
- 79:09
- really focus where to spend your time building
- 79:13
- And just one more tip
- 79:14
- there are actually
- 79:15
- quite a few tools out there to help with monitoring traces
- 79:19
- logging runtime, computing costs, and so on
- 79:21
- And those tools can be helpful
- 79:23
- I sometimes use a few of them
- 79:24
- and
- 79:25
- quite a few of deep-endized short course partners offer those tools
- 79:29
- and they do work well
- 79:30
- I find that for agentic workflows I end up working on
- 79:33
- most agentic Workflows are pretty custom
- 79:36
- and so
- 79:36
- I end up building pretty
- 79:38
- custom evals myself
- 79:40
- because I want to capture the things that
- 79:42
- work incorrectly with my system
- 79:44
- So even though I do use some of those tools
- 79:46
- I also end up building a lot of
- 79:47
- custom evals
- 79:49
- that are well fit to my specific application
- 79:52
- and the issues I see with it
- 79:54
- So
- 79:54
- thanks for
- 79:55
- sticking with me
- 79:55
- this far to the end of the
- 79:57
- fourth of five modules
- 80:00
- If you're able to implement
- 80:02
- even a fraction of the ideas from this module
- 80:06
- I think you'll be well ahead of
- 80:08
- the vast majority of developers in terms of
- 80:10
- your sophistication at implementing agentic workflows
- 80:14
- Hope you found these materials useful
- 80:16
- and I look forward to seeing you in
- 80:18
- the final module
- 80:19
- We'll talk about some
- 80:20
- more advanced design patterns for building
- 80:23
- highly autonomous agents
- 80:25
- I'll see you in the last module of
- 80:27
- this course
- 80:29
- Welcome to this final module where you learn
- 80:31
- about design patterns that let you build highly autonomous agents
- 80:36
- where you don't need to hard code in
- 80:38
- advance the sequence of steps to take
- 80:40
- but it can
- 80:41
- be more flexible and decide for itself what
- 80:43
- steps it wants to take
- 80:44
- to accomplish a task
- 80:46
- We'll talk about the planning design pattern
- 80:48
- and then later in this module
- 80:49
- how to build multi-agent systems
- 80:51
- Let's dive in
- 80:53
- Suppose you run a sunglasses
- 80:54
- retail store and have information on
- 80:58
- what sunglasses are in your inventory
- 81:00
- stored in a database
- 81:01
- You might want a
- 81:02
- customer service agent to be able to answer questions like
- 81:05
- do you have any round sunglasses in stock
- 81:07
- that are under $100? This is a fairly
- 81:09
- complex query because you have to look through
- 81:12
- the product descriptions to see what sunglasses are around
- 81:15
- then look at what is in stock
- 81:17
- and then finally see what's under $100
- 81:20
- in order to tell the customer
- 81:22
- yes, we have classic sunglasses
- 81:24
- So how do you build an agent
- 81:26
- to answer a broad range
- 81:28
- Customer queries like this and many others
- 81:31
- In order to do so
- 81:32
- we're going to give the LLM
- 81:34
- a set of tools to let it get item descriptions
- 81:37
- such as look up if different glasses are round
- 81:40
- check inventory, maybe process item returns
- 81:43
- which is not needed for this query
- 81:45
- but may need it for other queries
- 81:46
- get item price, check past transactions, process item sale, and so on
- 81:52
- In order to
- 81:53
- let an LLM
- 81:54
- figure out what's the right sequence of tools
- 81:57
- to use to respond to the customer request
- 82:00
- you might then write a prompt like this
- 82:02
- You have access to the following tools
- 82:04
- and they give it a description of each of the
- 82:06
- say, six tools or even more tools that the LLM has
- 82:10
- and to then tell it to
- 82:11
- return a step-by-step plan to carry out the
- 82:14
- user's request
- 82:15
- In this case, to answer this particular query
- 82:18
- a reasonable plan that an LLM might output
- 82:21
- might be
- 82:21
- to first use get item descriptions
- 82:24
- to check the different descriptions to find the round sunglasses
- 82:28
- and Then use check inventory to see if they're in stock
- 82:30
- and use get item price to see if
- 82:32
- the in stock results are less than $100
- 82:35
- After an LM outputs this plan with three steps
- 82:39
- what we can do is then take the step one text
- 82:43
- that is this text written here in red
- 82:45
- and pass that to an LM
- 82:47
- maybe with additional context about what are the tools
- 82:49
- with their user query
- 82:51
- with additional background context
- 82:53
- and toss an LM to carry out
- 82:55
- step one. And in this case
- 82:57
- hopefully the LM will
- 82:58
- choose to call the get item descriptions
- 83:00
- to get the appropriate descriptions of items
- 83:03
- And the output of that first step
- 83:05
- can let it select which are the round sunglasses
- 83:09
- and that
- 83:09
- output of step one
- 83:11
- is then passed together with the step two instructions
- 83:14
- that would be these instructions I have here in blue
- 83:16
- to an LM to then execute the second
- 83:18
- step of the plan. Hopefully
- 83:20
- it will then take the
- 83:21
- two pairs of round sunglasses we found on
- 83:23
- the previous slide and check the inventory
- 83:26
- and the output
- 83:27
- of that second step Is then used to another OM call
- 83:31
- where you have the output of the second step
- 83:33
- as well as
- 83:33
- the instructions of what to do for step three
- 83:36
- past the OM
- 83:37
- to have it get the item price
- 83:38
- And finally
- 83:39
- this output is fed back to the OM
- 83:41
- one last time to generate the final answer
- 83:44
- for the user. In this slide
- 83:46
- I've simplified a lot of details a little
- 83:48
- bit
- 83:48
- The actual plan typically written by the OM
- 83:50
- is more detailed than these simple one-line instructions
- 83:53
- but the basic workflow is to have an
- 83:55
- OM write out multiple steps of a plan
- 83:57
- and then task it
- 83:58
- to execute each step of the plan
- 84:01
- in turn
- 84:02
- with some appropriate surrounding context about what is the task
- 84:05
- what are the tools available
- 84:06
- and so on. And the exciting thing
- 84:09
- about using an OM to plan this way
- 84:11
- is that
- 84:12
- we did not have to decide in advance
- 84:14
- what is the sequence in which
- 84:16
- to call tools in order to answer a
- 84:18
- fairly complex customer request
- 84:21
- If a customer were to make a different request
- 84:24
- such as
- 84:25
- I would like to return the
- 84:26
- gold frame glasses that
- 84:28
- I purchase but not the metal frame ones
- 84:31
- Then you can imagine an OM
- 84:33
- similarly being able to come up with a
- 84:35
- different plan
- 84:36
- to figure out based on what they have purchased previously
- 84:39
- which glasses they had bought based on get item descriptions
- 84:42
- whether gold frame ones they want to return
- 84:44
- and then maybe call process item return
- 84:46
- So with an agent that can plan like this
- 84:48
- it can carry out a much wider
- 84:50
- range of tasks
- 84:51
- that can require calling many different tools and
- 84:54
- many different orders. One more example of planning
- 84:57
- let's take a look at an email assistant
- 84:59
- If you want to be able to tell
- 85:00
- your assistant to please reply to that email
- 85:02
- invitation from Bob Molden
- 85:03
- New York
- 85:04
- tell him attend and archive his email
- 85:05
- Then an email assistant may be given tools like this
- 85:09
- to search email, move an email, delete an email
- 85:11
- and send an email
- 85:12
- And you might write a system prompt saying
- 85:14
- you have access to following tools
- 85:15
- and again, please return the step-by-step plan
- 85:17
- In this case
- 85:18
- maybe the OM will
- 85:19
- say
- 85:20
- the steps for this are to use search
- 85:21
- email to find the email from Bob that
- 85:24
- mentioned dinner and New York
- 85:26
- and then
- 85:26
- generate and send an email to confirm And
- 85:28
- then send the last email to the archive
- 85:31
- folder. Given this plan
- 85:32
- which looks a reasonable one
- 85:34
- you would then again
- 85:35
- task an LM step by step
- 85:37
- to carry out this plan
- 85:38
- So the text from the first step
- 85:40
- shown here in red
- 85:41
- would be fed to the LM with additional background context
- 85:44
- and hopefully it triggers search email
- 85:46
- Then the output of that
- 85:48
- can be given to
- 85:49
- an LM again with the step two instructions
- 85:52
- to send a appropriate response
- 85:55
- And then finally, assuming the email was sent successfully
- 85:57
- you can take that output
- 85:58
- and have the LM execute the third step
- 86:01
- of moving the email
- 86:02
- from Bob into the archive folder
- 86:05
- The planning design pattern is already used successfully
- 86:08
- in many highly agentic coding systems
- 86:11
- where if you ask it to write a piece of software
- 86:13
- to build some fairly complex application
- 86:16
- it might actually come to a plan to build this component
- 86:19
- build this component, build this component, to almost form a checklist
- 86:22
- and then do those steps one at a
- 86:24
- time to build decently complex piece of software
- 86:28
- For many other applications
- 86:29
- the use of planning is still maybe more experimental
- 86:33
- is not in very widespread use
- 86:35
- And one of the challenges of planning is
- 86:38
- it makes the system sometimes
- 86:39
- a little bit harder to control because you as a developer
- 86:42
- you don't really know
- 86:43
- at runtime what plan it will come up
- 86:45
- with. And so I think
- 86:47
- also highly agentic coding systems where it actually works really well
- 86:51
- adoption of planning is still growing in other sectors
- 86:55
- but this is exciting technology
- 86:56
- and
- 86:57
- I think it will keep getting better and
- 86:58
- we'll see it in more and more applications
- 87:01
- The cool thing about
- 87:02
- building agents that can plan for themselves is
- 87:04
- you don't need to hard code in advance
- 87:06
- the exact sequence of steps an LLM
- 87:08
- may take to carry out a complex task
- 87:11
- Now, I know that in this video
- 87:13
- I've gone over the planning process at a fairly high level
- 87:16
- where we're
- 87:16
- outputting a list of steps and then tasking
- 87:18
- an LLM
- 87:19
- to
- 87:20
- carry out the steps of the plan
- 87:22
- one step at a time
- 87:23
- But how does this actually work?
- 87:25
- In the next video
- 87:26
- we'll take a deeper dive to look
- 87:28
- further into the guts of
- 87:30
- what these plans actually look like and how
- 87:32
- to string it together
- 87:34
- to have an LLM plan and execute a
- 87:35
- plan
- 87:36
- for you
- 87:37
- Let's take a look at that in the
- 87:38
- next video
- 87:40
- In this video
- 87:40
- we'll look in detail at how to prompt
- 87:42
- an LLM to generate a plan and then how to read
- 87:45
- interpret, and execute that plan
- 87:47
- Let's dive in
- 87:48
- This is a plan that you saw in
- 87:50
- the previous video for the customer service agents
- 87:53
- and I have presented this plan at a
- 87:55
- high level using simple text descriptions
- 87:57
- Let's take a look at how you can
- 87:59
- get an arm to write
- 88:01
- very clear plans that go a little bit
- 88:03
- beyond these simple high-level text descriptions
- 88:06
- It turns out that
- 88:08
- many developers
- 88:09
- will ask an arm to format the plan
- 88:12
- it wants executed in JSON format
- 88:14
- because this allows downstream code
- 88:16
- to parse what exactly are the steps of
- 88:18
- the plan in a relatively clear and unambiguous ways
- 88:21
- and
- 88:22
- all of the leading arms are pretty good
- 88:24
- at generating JSON outputs at this point
- 88:26
- So the system prompt might say something like this
- 88:29
- you have access to file tools
- 88:30
- distribution tools, and then create a step-by-step plan in JSON format
- 88:34
- and
- 88:34
- you might describe the JSON format in enough
- 88:37
- detail with the goal of getting it to
- 88:39
- output a plan
- 88:40
- like that shown here on the right
- 88:42
- So in this JSON output
- 88:43
- it creates a
- 88:44
- list
- 88:45
- where the first list item has clear keys
- 88:48
- and values. Values that say
- 88:50
- the step one of the plan
- 88:51
- has
- 88:52
- following description
- 88:53
- and it should use the following tool with
- 88:55
- the following arguments passed to that tool
- 88:58
- Then after that
- 88:58
- step two of the plan
- 89:00
- is to
- 89:00
- carry out this task and then to use
- 89:02
- this tool and so on
- 89:03
- So this JSON format
- 89:05
- as opposed to writing the plan in English
- 89:08
- allows downstream code to more clearly parse out
- 89:11
- exactly
- 89:12
- what are the steps of the plan
- 89:13
- so that it can be reliably executed
- 89:15
- one step at a time
- 89:17
- Instead of JSON
- 89:18
- I also see
- 89:19
- some developers use XML where you can
- 89:22
- use XML delimiters or use XML tags to
- 89:25
- clearly specify
- 89:27
- what are the steps of the plan and
- 89:28
- what step number it is
- 89:30
- Some developers, I feel like fewer developers
- 89:32
- will use Markdown
- 89:33
- which is
- 89:34
- just sometimes slightly more ambiguous
- 89:36
- in terms of how we parse it
- 89:38
- And I think plain text is maybe the
- 89:39
- least reliable of these options
- 89:41
- But I think either JSON
- 89:42
- which I'm showing here
- 89:43
- or XML
- 89:44
- would be good options for how to ask
- 89:46
- the LM to format a plan
- 89:48
- Unambiguously
- 89:50
- So that's it. By opening plans in JSON
- 89:53
- you can then
- 89:54
- pause it and have
- 89:55
- downstream workflows execute different steps of the plan more systematically
- 90:00
- Now, in terms of giving elements a plan
- 90:03
- it turns out there's one other
- 90:05
- really neat idea
- 90:06
- that lets an element output very complex plans
- 90:08
- and get them executed reliably
- 90:11
- and that's allow them
- 90:12
- write code and to have code express
- 90:14
- the plan
- 90:15
- Let's take a look at this
- 90:16
- in the next video
- 90:19
- We've talked a lot about how to build
- 90:20
- a single agent to complete tasks for you
- 90:23
- In a multi-agent or multi-agentic workflow
- 90:26
- we instead have a collection of multiple agents
- 90:28
- collaborate to do things for you
- 90:31
- When some people hear for the first time about multi-agent systems
- 90:34
- they wonder, why do I need multiple agents?
- 90:37
- It's just the same OM that I'm prompting over and over
- 90:40
- or just one computer
- 90:41
- Why do I need multiple agents?
- 90:44
- I find that
- 90:45
- one useful analogy is
- 90:47
- even though I may do things on a single computer
- 90:50
- we do decompose work
- 90:51
- in a single computer into
- 90:53
- maybe multiple processes or multiple threads
- 90:56
- And as a developer
- 90:57
- thinking, even though it's, well, one CPU on a computer
- 91:00
- say
- 91:01
- thinking about how to take work and decompose
- 91:03
- it into multiple processes or multiple computer programs to run
- 91:07
- that makes it easier
- 91:08
- for me as a developer to write code
- 91:11
- And in a similar way too
- 91:13
- if you have a complex task to carry out
- 91:16
- sometimes instead of thinking about how to
- 91:19
- hire one person to do it for you
- 91:21
- you might think about hiring a team of
- 91:23
- a few people
- 91:24
- to do different pieces of the task for
- 91:26
- you
- 91:27
- And so in practice
- 91:28
- I found that for many developers of agentic systems
- 91:31
- having this mental framework of not asking
- 91:34
- what's the one person I might hire to do something
- 91:36
- but instead
- 91:37
- would it make sense to hire people with
- 91:39
- three or four different roles to do this
- 91:41
- overall task for me?
- 91:42
- That holds
- 91:43
- give another way
- 91:45
- to take a
- 91:45
- complex thing and decompose it into subtasks and
- 91:48
- to build for those individual subtasks
- 91:50
- one at a time
- 91:51
- Let's take a look at some examples of
- 91:53
- how this works
- 91:54
- Take the task of creating marketing assets
- 91:57
- Say you want to market sunglasses
- 91:59
- Can you
- 92:00
- come up with a marketing brochure for that?
- 92:02
- You might need a researcher on your team
- 92:04
- to look at trends on sunglasses and what
- 92:06
- competitors are offering. You might also
- 92:09
- have a graphic designer on your team to
- 92:11
- render
- 92:12
- charts or nice looking graphics of your sunglasses
- 92:14
- And then also a writer to take the research
- 92:17
- take the graphic assets and put it all
- 92:19
- together into a nice looking brochure
- 92:21
- Or to write a research article
- 92:22
- you might want a researcher to do online research
- 92:25
- a statistician to calculate statistics
- 92:27
- a lead writer
- 92:28
- and an editor to cover the polished report
- 92:30
- Or to prepare a legal case
- 92:32
- real law firms will often have associates
- 92:34
- paralegals, maybe an investigator. And we naturally
- 92:38
- because of the way human teams do work
- 92:41
- can think of
- 92:42
- different ways that
- 92:43
- complex tasks can be broken down into
- 92:47
- different individuals with different roles
- 92:50
- So these are examples. Of when a complex
- 92:53
- task
- 92:53
- will already naturally decompose into subtasks that different
- 92:57
- people with different skills can carry out
- 93:00
- Take the example of creating marketing assets
- 93:03
- Look into detail into what a researcher
- 93:05
- graphic designer, and writer might do
- 93:08
- A researcher
- 93:09
- might
- 93:10
- have the task of analyzing market trends
- 93:12
- and researching competitors. And when designing the research agents
- 93:17
- one question to keep in mind is
- 93:19
- what are the tools that the researcher
- 93:21
- may need in order to come up with
- 93:23
- a research report on market trends and what
- 93:25
- competitors are doing. So one natural tool
- 93:28
- that a
- 93:29
- agentic researcher might need to use would be
- 93:32
- web search. Just as a human researcher
- 93:34
- asked to do these tasks might need to
- 93:36
- search online in order to
- 93:37
- come up with their report
- 93:39
- Or for a graphic designer agents
- 93:42
- they might be tasked with creating visualizations and
- 93:44
- artwork. And so
- 93:46
- what are the
- 93:46
- tools that a agentic software graphic designer might need?
- 93:50
- Well, they may need a Generation and manipulation APIs
- 93:54
- or maybe
- 93:55
- similar to what you saw with the coffee machine example
- 93:59
- maybe it needs code execution to generate charts
- 94:02
- And then lastly
- 94:03
- the writer has transformed the research into report
- 94:05
- text and marketing copy
- 94:07
- and in this case
- 94:08
- they don't need any tools other than
- 94:10
- what an LLM can already do
- 94:12
- to generate text. In this and the next video
- 94:15
- I'm going to use these purple boxes to denote an agent
- 94:19
- and the way you build individual agents is
- 94:21
- by prompting an LLM
- 94:23
- to play the role of
- 94:25
- a researcher or a graphic designer or a writer
- 94:27
- depending on which agent it is part of
- 94:30
- So for example, for the researcher agents
- 94:32
- you might prompt it to say
- 94:33
- you are a research agent
- 94:35
- expert at analyzing market trends and competitors
- 94:38
- carry out
- 94:39
- online research to analyze market trends for the sunglasses product
- 94:43
- and give a summary as well of what
- 94:44
- competitors are doing. So that
- 94:46
- would allow you to build a researcher agent
- 94:49
- And similarly
- 94:50
- by prompting an LLM To act as a
- 94:52
- graphic designer with your proper tools
- 94:54
- and to act as a writer
- 94:56
- that's
- 94:57
- how you can build a graphic designer
- 94:59
- as well as a writer agent
- 95:01
- Having built these three agents
- 95:04
- one way to have them work together
- 95:06
- to
- 95:07
- generate your final report would be to
- 95:09
- use a
- 95:10
- Simple
- 95:11
- linear
- 95:12
- ordered
- 95:12
- workflow or a linear plan in this case
- 95:14
- So
- 95:15
- if you want to create a summer marketing campaign for sunglasses
- 95:19
- you might give that prompt to the research
- 95:21
- agents. The research agent then writes a report that says
- 95:24
- share the current sunglasses trends and competitive offerings
- 95:27
- This research report can then be fed to
- 95:29
- the graphic designer that looks at
- 95:31
- the data the research has found and creates
- 95:33
- a few data visualizations and artwork options
- 95:36
- All these assets can then be passed
- 95:38
- to the writer
- 95:40
- that then takes the research and the graphic
- 95:42
- output and writes the final marketing brochure
- 95:46
- The advantage of building a multi-agent workflow in
- 95:48
- this case is
- 95:50
- when designing a researcher or graphic designer or writer
- 95:53
- you can focus on one thing at a
- 95:55
- time. So I can spend some time building
- 95:57
- maybe the best graphic designer agents I can
- 96:00
- while maybe
- 96:00
- my collaborators are building research agents and writer
- 96:03
- agents. And then in the end
- 96:04
- we string it all together
- 96:06
- to come up with this multi-agent system
- 96:09
- And in some cases
- 96:10
- I'm seeing developers start to reuse
- 96:13
- some agents as well
- 96:15
- So having built a graphic designer for marketing brochures
- 96:18
- maybe I'll think about if I can build
- 96:20
- a more general graphic designer
- 96:22
- that can help me write marketing brochures as
- 96:24
- well as social media posts
- 96:26
- as well as help me illustrate
- 96:27
- online web pages
- 96:29
- So by
- 96:30
- coming up with what are the
- 96:32
- agents you might hire to do a task
- 96:34
- and this will sometimes correspond to
- 96:37
- who are the
- 96:37
- types of human employees you might hire to do a task
- 96:41
- you can come up with a workflow like this
- 96:44
- with maybe even building agents that you could
- 96:46
- choose to reuse in other applications as well
- 96:49
- Now
- 96:50
- what you see here is a linear plan where one agent
- 96:53
- the researcher does his work
- 96:55
- then the graphic designer
- 96:56
- and then the writer. With agents
- 96:58
- you can also, as an alternative to a linear plan
- 97:01
- you can also have agents
- 97:03
- interact with each other in more complex ways
- 97:05
- Let me illustrate with an example of planning
- 97:08
- using multiple agents. So previously
- 97:11
- you saw how we may give an OM
- 97:13
- a set of tools that they can call
- 97:15
- to carry out different tasks
- 97:16
- In what I want to show you
- 97:18
- we will instead give an OM
- 97:20
- the option to call on different
- 97:22
- agents to ask the different agents to help
- 97:24
- complete different tasks. So in detail
- 97:27
- you might write a prompt
- 97:28
- like you're a marketing manager
- 97:29
- have the following team of agents to work with
- 97:31
- and then give a description to agents
- 97:33
- And this is very much similar to what
- 97:35
- we're doing with planning using tools
- 97:37
- except the tools, the green boxes, are replaced with agents
- 97:41
- these purple boxes that the OM can call
- 97:43
- on
- 97:43
- And then we'll ask it to return a
- 97:44
- step-by-step plan to carry out the user's request
- 97:47
- And in this case
- 97:47
- the OM may
- 97:48
- ask the
- 97:49
- researcher to
- 97:50
- research current sunglasses trends and then report back
- 97:53
- Then they'll ask the graphic designer to create
- 97:55
- the images and then report back
- 97:56
- Then ask the writer to create a report
- 97:58
- And then maybe the OM will choose to
- 97:59
- review
- 98:00
- or reflect on and improve the report
- 98:02
- one final Time. In executing this plan
- 98:06
- you would then take the set one text to the researcher
- 98:09
- carry out research, then pass that to the graphic designer
- 98:12
- pass it to the writer
- 98:14
- and then maybe do one final reflection step
- 98:17
- and then you'd be done
- 98:18
- One interesting view of this workflow is
- 98:20
- as if you have these three agents up here
- 98:23
- but this arm on the left
- 98:25
- is actually like a fourth agent
- 98:27
- There's a marketing manager that is a manager
- 98:30
- of a marketing team
- 98:31
- that is setting direction and then delegating tasks to the researcher
- 98:34
- the graphic designer, and the writer agents
- 98:36
- So this becomes actually a collection of four agents
- 98:39
- both marketing manager agent coordinating the work of the researcher
- 98:42
- the graphic designer, and the writer
- 98:45
- In this video, you saw two communication patterns
- 98:48
- One was a linear one where your agents
- 98:51
- took
- 98:51
- actions one at a time until you got to the end
- 98:54
- and the second had a marketing manager
- 98:56
- coordinating the activity of a few other agents
- 99:00
- It turns out that one of the
- 99:01
- key design decisions you made And that having
- 99:04
- a make when building multi-agentic systems is
- 99:07
- what is the communication pattern between your different
- 99:09
- agents
- 99:10
- This is an area of hot research
- 99:12
- and there are multiple patterns emerging
- 99:14
- but in the next video
- 99:15
- I want to show you what are some
- 99:16
- of the most common communication patterns for
- 99:19
- getting your agents to work with each other
- 99:21
- Let's go see that in the next video
- 99:24
- When you have a team of people working together
- 99:28
- the patterns
- 99:29
- by which they communicate can be quite complex
- 99:31
- And in fact
- 99:32
- designing a
- 99:33
- organizational chart is actually pretty complex to try
- 99:36
- to figure out what's the best way for people to communicate
- 99:39
- to collaborate
- 99:40
- It turns out designing communication patterns for
- 99:43
- multi-agent systems is also quite complex
- 99:46
- But let me show you some of the
- 99:47
- most common design patterns I see
- 99:50
- used by different teams today
- 99:51
- In a marketing team with a linear plan
- 99:53
- where first the researcher worked
- 99:55
- then the graphic designer
- 99:56
- then the writer, the communication pattern was linear
- 99:59
- The researcher would communicate with the graphic designer
- 100:02
- and then both the research and the graphic designer
- 100:04
- maybe pass the outputs to the writer
- 100:06
- and so this very linear communication pattern
- 100:09
- This is
- 100:10
- one of the two most common
- 100:12
- communication plans that I see being used today
- 100:14
- The second of the
- 100:15
- two most common communication plans would be
- 100:19
- similar to what you saw in this example
- 100:22
- with planning using multiple agents
- 100:24
- Where there is a
- 100:25
- manager
- 100:26
- that
- 100:27
- communicates with a
- 100:28
- number of team members and coordinates their work
- 100:31
- So in this example
- 100:32
- the marketing manager decides to call on the
- 100:34
- researcher to do some work
- 100:36
- then
- 100:36
- if you think of the marketing manager as
- 100:38
- getting the report back and then sending it
- 100:40
- to the graphic designer
- 100:41
- getting the report back and sending it to the writer
- 100:43
- this would be a hierarchical communication pattern
- 100:47
- If you're actually implementing a hierarchical communication pattern
- 100:50
- it would probably be simpler
- 100:51
- to have the researcher pass the report back
- 100:53
- to the marketing manager rather than the researcher
- 100:56
- pass the results directly to the graphic designer
- 100:58
- and to the writer
- 100:59
- But so this type of hierarchy
- 101:01
- is also a pretty common way to
- 101:03
- plan the communication patterns where you have one
- 101:05
- manager coordinating the work of
- 101:08
- a number of other agents
- 101:09
- And just to share with you some more
- 101:11
- advanced and less frequently used
- 101:14
- but nonetheless sometimes used in practice communication patterns
- 101:17
- one would be a deeper hierarchy where same as before
- 101:20
- if you have a marketing manager
- 101:22
- send tasks to researcher
- 101:23
- graphic designer, Writer
- 101:24
- but maybe the researcher
- 101:26
- has themselves two other agents that they call on
- 101:29
- such as a wet researcher and a fact
- 101:30
- checker. Maybe the graphic designer just works by themselves
- 101:34
- whereas the writer has a
- 101:36
- initial style writer and a citation checker
- 101:38
- So this would be a hierarchical organization
- 101:41
- of agents in which some agents might themselves call other sub-agents
- 101:46
- and also see this used in some applications
- 101:49
- but this is much more complex than a one-level hierarchy
- 101:52
- so used less often today
- 101:55
- And then one final pattern that is quite challenging to execute
- 101:58
- but I see a few experimental projects use it
- 102:02
- is the all-to-all communication pattern
- 102:04
- So in this pattern
- 102:06
- anyone's allowed to talk to anyone else at any time
- 102:09
- and the way you implement this is you
- 102:11
- prompt all four of your agents
- 102:13
- in this case
- 102:14
- to tell them that there are
- 102:16
- three other agents it could decide to call on
- 102:18
- and whenever one of your agents decides to
- 102:20
- send a message to another agent
- 102:22
- that message gets added. The receiver agent's context
- 102:27
- and then a receiver agent can
- 102:28
- think for a while and decide when to
- 102:30
- get back to that first agent
- 102:32
- And so if you all
- 102:33
- collaborate in a crowd and talk to each
- 102:35
- other for a while until
- 102:37
- say
- 102:37
- each of them declares that it is done with its task
- 102:40
- and then it starts talking
- 102:41
- then maybe when everyone thinks it's done
- 102:43
- or maybe when the writer concludes it's good enough
- 102:45
- that's when you generate the final output
- 102:47
- In practice
- 102:48
- I find the results of all-to-all communication patterns
- 102:50
- a bit hard to predict
- 102:52
- So some applications don't need high control
- 102:55
- and you can run it and see what you get
- 102:57
- and the marketing brochure isn't good
- 102:58
- Maybe that's okay
- 102:59
- just run it again and see if you
- 103:00
- get a different result. But I think for
- 103:03
- applications where you're willing to
- 103:04
- tolerate a little bit of chaos and unpredictability
- 103:07
- I do see some developers using this communication pattern
- 103:10
- So that, I hope
- 103:12
- conveys some of the richness of multi-agent systems
- 103:17
- Today
- 103:17
- there are quite a lot of software frameworks
- 103:19
- as well that support easily building multi-agent systems
- 103:23
- and they also
- 103:24
- Implementing some of these communication patterns relatively easy
- 103:27
- So maybe if you use your own multi-agent system
- 103:31
- you'll find some of these frameworks hopeful
- 103:33
- for exploring these different communication patterns as well
- 103:37
- And so
- 103:38
- that now brings us to the final
- 103:41
- video of this module and of this course
- 103:45
- Let's go on to the final video
- 103:47
- to wrap up
- 103:49
- Welcome to the final video of this course
- 103:51
- It feels like we've been through a lot together
- 103:54
- just you and me
- 103:55
- and we've gone through a lot of topics
- 103:56
- in agentic AI
- 103:58
- Let's take a look
- 103:59
- In the first module
- 104:00
- we talked about what are the applications you
- 104:02
- can build with agentic AI that just were not possible before
- 104:06
- and we started then to look at key design patterns
- 104:11
- including the reflection design pattern
- 104:13
- which is a
- 104:14
- simple way to sometimes give your application a nice performance boost
- 104:18
- and then tool use or function calling
- 104:20
- which expands what your LM application can do
- 104:23
- with code execution being one important case of
- 104:26
- that. And then we spent a lot of
- 104:27
- time talking about
- 104:28
- evaluations as well as error analysis
- 104:31
- and how to drive a disciplined process
- 104:34
- of building
- 104:35
- as well as analyzing to be efficient in
- 104:37
- how you
- 104:38
- keep on improving the performance of your agentic
- 104:40
- AI system
- 104:41
- This fourth module is
- 104:42
- some of the material that I think
- 104:44
- you will find most useful
- 104:46
- as you keep building agentic AI systems
- 104:48
- I hope, for a long time
- 104:50
- And then in this module
- 104:51
- we talk about planning and multi-agent systems that
- 104:54
- can let you build much more powerful
- 104:57
- although sometimes
- 104:58
- harder to control
- 104:59
- How to predict and advance types of systems
- 105:02
- So with the skills you learn from this course
- 105:04
- I think you now know how to build
- 105:06
- a lot of cool
- 105:07
- exciting, agentic AI applications
- 105:10
- When my team, or I see other teams as well
- 105:13
- interview people for jobs
- 105:14
- I find interviews often try to
- 105:17
- assess whether or not candidates have
- 105:19
- pretty much the skills you're learning about
- 105:22
- in this course. And so I hope that
- 105:23
- this course will also open up new professional opportunities for you
- 105:27
- and you should be able to do more
- 105:29
- Whether you're doing these things for fun
- 105:31
- or for professional or practical settings
- 105:34
- I think you enjoy
- 105:35
- this new set of things you can
- 105:38
- now build
- 105:39
- So just to wrap up
- 105:41
- I want to thank you again for spending
- 105:43
- all this time with me
- 105:44
- and I hope you will take these skills
- 105:47
- use them responsibly
- 105:48
- and just go build cool stuff
Watch these videos on your TV
XTTV brings video content from X to your living room. Download now and start watching on the big screen.