Lauren Tan, the engineer at SpaceXAI, put 2,500 PRs into production last month and broke down the se

Chrome
@0xchromium
Lauren Tan, the engineer at SpaceXAI, put 2,500 PRs into production last month and broke down the secret that lets her bots merge their own code while she sleeps the timestamps that matter: > 18:53 - agents kept rewriting the same check, so she put it in the skill > 40:08 - 30 bug reports arrive at once and one coordinator splits them up > 45:16 - a bot that logs bad patterns to a doc instead of fixing them > 48:33 - agents work all night, she checks the commits by morning > 59:07 - a skill that carries an old chat's context into a new one the work that survives the night is the work that got written down before the next agent picked it up if your own bots keep losing the thread between steps, the note each one should leave for the next waits in the article below
Transcript
- Thank you so much for joining
- 0:02
- Thanks for having me. Yeah
- 0:03
- very excited to be here
- 0:04
- Big fan of yours
- 0:06
- And a huge fan of yours
- 0:07
- I think people have been talking about this like
- 0:09
- it's like the meeting of the skill minds
- 0:12
- Yes. The skill Mount Olympus or something
- 0:14
- Because both of us have very popular skill libraries
- 0:16
- I'm not, as I was saying before we started
- 0:19
- I've not used a ton of yours and like
- 0:22
- I
- 0:22
- want to get all of the juice out
- 0:24
- of your brain so that I can go
- 0:25
- and use it properly and use it better
- 0:27
- And I think
- 0:28
- where I want to start with this is
- 0:30
- you gave a talk
- 0:31
- pretty recently, like
- 0:33
- about 10 days ago and posted it on X
- 0:35
- which went absolutely nuts
- 0:37
- as about how I shipped 2,500 PRs last
- 0:39
- month to production
- 0:41
- Got about
- 0:41
- 3 million views or something on X
- 0:44
- And
- 0:45
- I
- 0:45
- watched it and I loved it and I
- 0:46
- recommended it
- 0:48
- And
- 0:48
- I kind of want to run this as
- 0:49
- almost like a Q&A of that talk
- 0:51
- basically, of giving you Cause I just
- 0:54
- I just
- 0:55
- had tons of questions about it and I
- 0:56
- wanted to dive into it
- 0:58
- And I think where I want to start is
- 1:00
- you talk about a trust ladder with agents
- 1:04
- where you, as you trust agents more
- 1:06
- you can get them to do
- 1:08
- better and better things
- 1:10
- and or scale them
- 1:11
- up to use more and more agents
- 1:13
- So
- 1:14
- what is your story of how you climbed
- 1:16
- the trust ladder and
- 1:18
- how did that work when like you got
- 1:20
- SpaceX and started climbing more and more?
- 1:23
- So I think this
- 1:24
- the, the journey sort of began even before I joined Cursor
- 1:28
- which is now SpaceX AI
- 1:31
- so the story is
- 1:34
- after Meta, so I
- 1:36
- I used to work
- 1:37
- at Meta on the React team
- 1:39
- I took a month off
- 1:40
- cause I was feeling kind of burnt out
- 1:42
- And of course, when, what
- 1:44
- what do you do when you're burnt out?
- 1:45
- You go and start a new side project
- 1:48
- and so I started a side project
- 1:49
- you know, I was, of course using AI to
- 1:52
- to write code
- 1:54
- but then I started to realize
- 1:56
- you know
- 1:56
- I was spending like so many hours just micromanaging one agent
- 2:00
- right? And, you know, at the time
- 2:02
- this was back in February
- 2:04
- maybe February, very early February or January
- 2:08
- you know
- 2:08
- people were really obsessed with this idea of
- 2:09
- like orchestration. This was like
- 2:11
- you know, before, you know, things like cursor, you know, like the agents window was
- 2:15
- had become popular
- 2:17
- So people were still in like
- 2:18
- like 2E land, you know, in their terminal
- 2:21
- and they were all talking about
- 2:22
- oh, here, you know, I built a custom orchestrator
- 2:24
- right?
- 2:25
- And so, of course, I had
- 2:26
- I was a bit nerd sniped by that
- 2:28
- And, you know, as I was building my toy project
- 2:32
- I got nerd sniped by
- 2:33
- oh
- 2:33
- how do I make my AI coding setup
- 2:35
- more efficient?
- 2:37
- And so, you know, I
- 2:38
- I kind of started the journey there where I just
- 2:42
- you know, took a step back and realized
- 2:44
- you know
- 2:44
- I was spending all this time micromanaging a single agent
- 2:47
- you know, I was creating skills
- 2:49
- and I was, like
- 2:50
- finding it quite difficult to measure the output or the
- 2:53
- the results
- 2:54
- the impact of the skill as well
- 2:57
- So I was kind of flying blind
- 2:58
- but I was, you know, iterating really fast, and
- 3:04
- so that project eventually sort of became the basis of PSAC
- 3:09
- even though I didn't know it at the
- 3:10
- time
- 3:11
- and a lot of
- 3:13
- some tricks I had learned
- 3:14
- like building that early set of skills
- 3:17
- actually it's still open source if you wanna
- 3:19
- if anybody wants to take a look
- 3:20
- it's on my GitHub
- 3:21
- like potato slash, noodle
- 3:25
- n o o d l e
- 3:26
- and in there you will see some skills
- 3:28
- and a brain directory
- 3:30
- And so I was really interested in this
- 3:31
- idea of how do I You know
- 3:34
- extract my own ability
- 3:37
- if that makes sense
- 3:38
- and give it to the agent
- 3:39
- right? Because I was
- 3:40
- I, I realized that, you know
- 3:42
- all I was trying to do was trying
- 3:43
- to teach the agent to write code more like me
- 3:45
- you know, do, do, you do workflows more like me
- 3:49
- So, you know
- 3:49
- the skills were like
- 3:51
- an entry point to doing that
- 3:54
- and then, you know, after I joined Cursor
- 3:57
- I was starting to work on the agent's window
- 3:59
- and, it had a lot of performance issues
- 4:03
- it was, it was, it was pretty laggy
- 4:05
- and so since I had experience working on React
- 4:08
- I was asked like
- 4:09
- hey, do you want to come and help out
- 4:11
- with the agent's window?
- 4:13
- and so the, the
- 4:15
- the
- 4:15
- beginning of the Cursor journey was very manual
- 4:20
- I was
- 4:21
- deep
- 4:21
- in like looking at like flame graphs
- 4:23
- and heap snapshots and trying to see like
- 4:26
- why exactly is the app so slow?
- 4:29
- but then coming back to the same realization that
- 4:31
- you know, I was sort of the bottleneck
- 4:33
- I was doing everything manually
- 4:35
- I was sort of the meat proxy
- 4:36
- in a way
- 4:37
- I was the meat proxy between my agent
- 4:39
- and Chrome DevTools
- 4:41
- and I was like really annoyed by that
- 4:43
- And what month of the year is that
- 4:45
- let's say?
- 4:46
- Where are we in the timeline?
- 4:48
- so I joined Cursor in March
- 4:50
- So this was like early
- 4:52
- early April, probably
- 4:53
- Early April is when
- 4:54
- you know
- 4:56
- I joined and I didn't have any skills
- 4:57
- I had, I
- 4:58
- I sort of abandoned my
- 5:00
- personal skills cause I didn't think they'd be
- 5:01
- relevant anymore
- 5:03
- but then working on the agents window
- 5:06
- and now working on Grokbot
- 5:09
- I sort of realized that a lot of
- 5:10
- the lessons I had learned from those skill
- 5:13
- time building the, the initial set of skills were very relevant
- 5:17
- especially around things like verification
- 5:21
- you know, being very rigorous in your work
- 5:25
- because I think agents from my experience
- 5:28
- even the
- 5:29
- the frontier ones tend to
- 5:32
- Tend to take shortcuts
- 5:34
- they tend to do the easy thing
- 5:38
- so
- 5:39
- a lot of the skills that I've built
- 5:40
- have been around
- 5:42
- how do I make the
- 5:43
- easy thing the right thing?
- 5:44
- You know
- 5:45
- how do I make that the best thing?
- 5:47
- The idea of sort of distilling
- 5:49
- your expertise and turning what you do every day into processes
- 5:55
- that's something that feels super familiar to me
- 5:57
- That's exactly what I've been doing with the
- 5:59
- skills
- 6:00
- And I suppose
- 6:01
- there's something in that which is
- 6:03
- a lot of people think domain expertise is
- 6:05
- getting less useful now as people
- 6:08
- start to rely more on AI
- 6:10
- Where, what do you think about that?
- 6:12
- Just as a sort of
- 6:13
- vibe check before we start talking about it
- 6:16
- I actually feel like domain expertise is
- 6:18
- is more important than ever
- 6:20
- You know
- 6:22
- I think I wrote this on my ex at some point
- 6:24
- but You know, at times I sometimes think of
- 6:28
- you know, AI as this, like
- 6:30
- especially as the models get smarter and smarter and more capable
- 6:33
- and the frontier models are just getting so good
- 6:36
- Like, I love Opus 5.5
- 6:37
- by the way
- 6:40
- you know, as the models get really
- 6:42
- really, really good
- 6:43
- it almost becomes like the bottleneck is no longer the agent
- 6:47
- right? It becomes
- 6:48
- your ability to express your intent and your
- 6:51
- goals in a clear way
- 6:53
- that the agent can understand and
- 6:57
- actually carry out
- 6:58
- And that's where I think
- 6:59
- you know, like
- 6:59
- people with a lot of domain expertise are extremely Have a
- 7:04
- have a huge advantage in my opinion
- 7:06
- especially if you're a little bit like
- 7:07
- you know, tech, techno curious, you know, so I, I think of people like
- 7:11
- you know, like, like a doctor or a lawyer or
- 7:15
- you know, someone who
- 7:16
- who has a deep expertise in a particular non-engineering domain
- 7:21
- and if they're actually just a little bit
- 7:23
- tech savvy and they can figure out how to use agents
- 7:26
- they can actually build really
- 7:27
- really great products, right? If they
- 7:29
- if they have a clear enough
- 7:31
- vision in their head
- 7:33
- and they can articulate it in a way
- 7:34
- that the agent can build it
- 7:36
- you know, I think that that is really the
- 7:40
- the bottleneck these days is
- 7:42
- is like the transfer of your intent
- 7:45
- right, and your vision to the agent
- 7:48
- Yeah, I've been obsessed with language basically since agents
- 7:52
- dropped. I just obsessed with thinking constantly about the
- 7:56
- the composition of words
- 7:58
- how I can make things sharper
- 8:00
- what
- 8:01
- what might be hidden in the phrases that
- 8:03
- I'm using
- 8:04
- And it's, and finding
- 8:06
- what I love is when you
- 8:07
- find a word
- 8:08
- that the agent then hooks onto and then goes
- 8:11
- okay, I'm going to reinforce that word
- 8:13
- I'm going to reuse that in my thinking traces
- 8:15
- You know, I found that with
- 8:17
- TDD was an early example of that
- 8:19
- A lot of chat about TDD recently of like
- 8:21
- you know, people say
- 8:22
- should you use TDD with agents?
- 8:23
- It doesn't matter. What you're doing is you're
- 8:25
- getting the agent to think about TDD
- 8:27
- getting it to write tests
- 8:28
- getting it to prioritize things in a different
- 8:30
- way than it did before
- 8:32
- And that's why sort of grilling
- 8:33
- I think, works effectively
- 8:35
- Because grilling is a
- 8:36
- yeah, yeah. It draws those words out of you
- 8:39
- right? Or
- 8:40
- or at least it helps the agent
- 8:42
- Understand your
- 8:43
- thinking so that it can propose those words
- 8:45
- to you and you can pick up and say
- 8:47
- yes, exactly that
- 8:49
- I've actually copied some of the
- 8:50
- the tips that you've shared as well
- 8:51
- where, you know
- 8:52
- one of my favorite ones that you've shared recently or
- 8:54
- or not
- 8:55
- or like maybe in the past couple of weeks is about
- 8:58
- reducing or eliminating tautological tests
- 9:02
- Like one of my pet peeves of agents
- 9:04
- is like all of the useless tests that
- 9:05
- they write
- 9:06
- And so, you know, that was one thing where
- 9:09
- you know, the word tautology, right, is, is
- 9:11
- is I guess, you know, not many people necessarily know that
- 9:14
- if, if, especially if English isn't your first language
- 9:17
- but there's a lot of meaning to that word and it
- 9:20
- it's like It's almost like compressed
- 9:22
- right? Like you compress a lot of
- 9:24
- intent and meaning into words
- 9:27
- And so I, I
- 9:28
- I totally agree with you
- 9:29
- Like I think language
- 9:30
- I've always been interested in language actually
- 9:33
- like programming languages
- 9:35
- natural human languages and how they came to
- 9:37
- be
- 9:38
- And it's so interesting that now with agents
- 9:40
- it's sort of like this meeting of
- 9:43
- natural language with programming language
- 9:45
- but it's all
- 9:46
- it's all language out of the hood
- 9:47
- It's all communication
- 9:48
- Totally. I did a drama degree
- 9:50
- right? So, you know
- 9:50
- I've been
- 9:51
- thinking about language and Shakespeare and stuff for
- 9:53
- a long time. And so this
- 9:54
- all feels very familiar. Yeah
- 9:57
- so okay, there's sort of, before we get into like
- 10:01
- cause I think the thing I want from
- 10:02
- you is like software factory stuff
- 10:05
- right? Software factory is the big buzzword
- 10:07
- Software factory is the thing that I'm thinking
- 10:09
- about too. I'm sort of releasing a course
- 10:10
- in that direction too
- 10:12
- And
- 10:13
- It's this sort of scaling yourself up to
- 10:17
- unrealistic numbers of PRs basically
- 10:19
- or
- 10:19
- PR numbers that sound ridiculous to people who
- 10:22
- don't understand how this works
- 10:24
- So where I want to get to
- 10:25
- is sort of from people who are doing
- 10:27
- kind of like one to five agents a day up to
- 10:30
- you know
- 10:31
- hundreds of agents running at once and how
- 10:32
- that sort of functions
- 10:34
- And so
- 10:35
- I'd love to hear about your metaphor
- 10:37
- of the Michelin kitchen instead of the software factory
- 10:40
- because I think that says a bit about
- 10:41
- the way you think about this stuff
- 10:43
- Yes
- 10:45
- Yeah, I, I, I've
- 10:46
- I've never really liked the term software factory
- 10:49
- not because, you know, it's not accurate, but I think
- 10:53
- I think the, a lot of people when they think factory
- 10:55
- right, they don't necessarily equate that with quality or craft
- 11:00
- right? Things which are very important to me
- 11:02
- and a lot of people and technologists who work
- 11:06
- you know, building products, we care about the user experience
- 11:09
- we care about the things we're building
- 11:12
- So while
- 11:12
- so while I do think software factory is an apt term
- 11:16
- it also, I guess, maybe conjures up negative
- 11:19
- you know, maybe sometimes negative connotations
- 11:22
- So
- 11:22
- Michelin kitchen is the thing that I've sort of landed on
- 11:25
- where it's much more
- 11:27
- I feel like it's much more aspirational and
- 11:30
- I like the metaphor a lot because
- 11:31
- you know, I like food
- 11:32
- I'm called potato, of course
- 11:34
- and I like cooking
- 11:36
- And I see a lot of parallels
- 11:37
- right?
- 11:38
- Like with food, right, when you're cooking a meal for yourself
- 11:41
- for example, it's both utilitarian
- 11:44
- like you're trying to just feed yourself
- 11:45
- right, and, and survive
- 11:48
- but it can actually be transformed into art
- 11:50
- right? And that's what
- 11:51
- what a Michelin
- 11:53
- starred chef or even just a chef or
- 11:55
- a cook can do with food
- 11:57
- is take something
- 11:58
- very ordinary and turn it into a delicious meal that
- 12:01
- you know
- 12:02
- takes you back to your childhood days or
- 12:04
- something like that
- 12:07
- And so it almost like mirrors that trust
- 12:09
- ladder that I talk about
- 12:10
- where
- 12:12
- you can sort of imagine your own journey
- 12:13
- as a home cook
- 12:15
- right?
- 12:16
- as a home cook
- 12:16
- you are doing all of the food
- 12:18
- the cooking yourself
- 12:19
- You cut all the vegetables
- 12:21
- you do all the prep work
- 12:22
- you do all the cleanup
- 12:24
- you know, you are the one man or one woman show
- 12:28
- really
- 12:31
- and it's an interesting thought experiment
- 12:33
- like, okay, if you were to cook a meal
- 12:36
- and then you add people
- 12:37
- right? Your, your, your partner joining to you
- 12:40
- your brother, your sister
- 12:41
- and now suddenly you have your whole family
- 12:43
- in the kitchen. I think most people would
- 12:45
- get very stressed by that
- 12:46
- right? The thought of
- 12:47
- oh
- 12:47
- so many people are just mucking around in
- 12:49
- my kitchen
- 12:50
- They have no
- 12:50
- no idea where all the utensils are
- 12:52
- They're mucking around. Yeah
- 12:54
- exactly
- 12:56
- So I feel like that
- 12:57
- that's really out because
- 12:59
- When you ask yourself that question of how
- 13:00
- do I go from being a solo cook
- 13:02
- right, to having an army or even not even
- 13:06
- not even an army
- 13:06
- but a few sous chefs
- 13:09
- right, that
- 13:09
- that are helping me in the kitchen
- 13:10
- How do I
- 13:11
- think about
- 13:12
- dividing the work in a way that makes sense?
- 13:15
- You know
- 13:15
- I'm not dividing work just for the sake of it
- 13:18
- but in a way that actually makes the sum the total
- 13:21
- the, the better than, you know
- 13:22
- the total of its parts
- 13:25
- And so the Michelin kitchen metaphor to me
- 13:27
- like, works really well in that regard because
- 13:30
- you know, as a chef, you're
- 13:33
- you know, if you become a chef
- 13:34
- you're in a position where
- 13:36
- you're not necessarily cooking all the food yourself
- 13:38
- anymore
- 13:39
- But you are thinking
- 13:41
- you're almost like the tech lead
- 13:42
- right, for the kitchen, where, you know
- 13:45
- chefs have to think about
- 13:46
- you know, not just cooking
- 13:47
- but they have to basically organize the whole kitchen
- 13:50
- and they're like the CEO of the kitchen
- 13:52
- They have to think about when to order ingredients
- 13:55
- how do you store them
- 13:56
- how do you prepare them
- 13:57
- when do they have to be prepared
- 13:59
- You know, it's a whole, it's a whole job
- 14:00
- right, that's not just cooking
- 14:03
- and I think that
- 14:03
- again, it mirrors so much of how engineers write code today
- 14:08
- where you are not writing the code yourself anymore
- 14:11
- you have agents, right? But you, as the human
- 14:14
- are still responsible for the final outcome
- 14:16
- right? Your name still is associated with the
- 14:19
- work that you do
- 14:20
- your reputation, and, you know
- 14:22
- so how you set up your kitchen
- 14:25
- right, and how you set up your skills
- 14:27
- your environment, your code base
- 14:29
- I think are ultimately the new ingredients that go into
- 14:35
- building product
- 14:37
- Yeah
- 14:38
- I think what I love about your approach
- 14:40
- is the amount of focus that you put
- 14:41
- into the environment that the agent operates in
- 14:44
- right? Because I think a lot of people
- 14:45
- they think, right, the agent is good
- 14:48
- I'm
- 14:49
- probably not going to be able to make
- 14:50
- it better
- 14:51
- Let's just trust what these
- 14:53
- magic model people have put
- 14:54
- into the harness and the model combination
- 14:57
- There's nothing I can really do
- 14:59
- Like I can't mess about with Claude Code
- 15:00
- internals or something or whatever you're using
- 15:04
- But what I love about your approach
- 15:07
- and it's something I advocate for too
- 15:08
- is that you can change the environment the agent operates in
- 15:11
- right?
- 15:12
- You can
- 15:13
- make changes in the code base
- 15:14
- and also give it tools for verification
- 15:17
- as well
- 15:18
- and allow it to verify its own work
- 15:20
- So the thing I
- 15:22
- I loved about watching that talk is
- 15:23
- the amount of focus you put in verification
- 15:26
- and like that is the lever that you
- 15:28
- can start to generate trust
- 15:30
- Can you talk about that and what that
- 15:32
- concretely looks like? Let's start like looking at
- 15:34
- practical ways that people can improve their own processes
- 15:37
- their own kitchens
- 15:39
- Yeah, I've, I've, I've said this a lot actually that
- 15:42
- you know, even if you don't use PSAC or
- 15:45
- you know, your skills
- 15:46
- I think that
- 15:47
- the single most important skill
- 15:49
- that should be in your toolkit is verification
- 15:53
- Because without verification, and for, for
- 15:56
- by the way, for those watching, don't know what that means
- 15:59
- it's this idea that you can give
- 16:01
- you can sort of give your agent hands and eyes
- 16:05
- in a way, that's the, the analogy I like
- 16:08
- where the agent is able to run the code
- 16:11
- right, and actually
- 16:13
- interact with it like a normal human user would
- 16:17
- and also do things like
- 16:18
- you know, debug it, you know, take traces and snapshots
- 16:23
- and, funnily enough, like
- 16:25
- that was actually the first skill I built
- 16:26
- when I joined Cursor
- 16:28
- that gave me a lot of
- 16:29
- that was
- 16:29
- that was the thing that actually started to
- 16:31
- let me
- 16:32
- ascend the trust ladder a little bit in
- 16:34
- a way that
- 16:34
- some of the other skills I had looked
- 16:36
- at
- 16:37
- or built had not really let me do
- 16:40
- Because no matter how good
- 16:42
- you know, some of the other skills were
- 16:43
- like the how skill
- 16:45
- the why skill, the unslop skill were
- 16:48
- I was still relying on me
- 16:50
- right, as the proxy between my agent and the output
- 16:54
- so that, you know
- 16:55
- if the agent can actually see the result of its work
- 16:57
- there's no way it can actually iterate
- 17:00
- Right
- 17:00
- and so this is where people start to talk about loops
- 17:02
- this idea of a loop
- 17:04
- And really, I think the term loop
- 17:06
- you know, seems kind of, almost abstract
- 17:10
- Like people are like
- 17:11
- what, what, what is a loop?
- 17:12
- What is an agent loop?
- 17:13
- But really, to me, like
- 17:15
- the most important part of a loop
- 17:16
- that allows it to be a loop is the verification part
- 17:20
- because the agent is able to
- 17:23
- to verify its own work
- 17:25
- And, you know
- 17:26
- that takes you out of the equation
- 17:28
- where now I can actually do something like
- 17:31
- So the very
- 17:31
- one of the very first use cases I
- 17:32
- had for verification was
- 17:34
- you know
- 17:34
- like the performance work that I was doing
- 17:36
- on Cursor's agent window
- 17:38
- And I wanted
- 17:39
- I wanted to get to a point where
- 17:40
- I could do something called hill climbing
- 17:42
- which is a term that I
- 17:44
- I think the labs
- 17:45
- talk about a lot
- 17:47
- which is this idea that
- 17:48
- you know
- 17:48
- you have some kind of
- 17:49
- rubric or a way to judge or score something
- 17:53
- And now, because you have a loop
- 17:55
- you can have an agent
- 17:56
- continually try to make improvements to that
- 18:00
- I think Carpathy, Andre Carpathy also famously released
- 18:04
- something called AutoResearch that has a lot of
- 18:06
- these ideas
- 18:09
- but yeah, verification, I would say
- 18:10
- is probably the most important skill in PSAC
- 18:15
- and many other, you know, tool sets
- 18:18
- and I think it's
- 18:20
- the most important thing to focus on
- 18:22
- So a lot of the
- 18:24
- a lot of, I spent a lot of time actually
- 18:26
- you know, tuning the verification, the creative verification skill
- 18:31
- and also internally the
- 18:33
- the, we have so many verification skills now
- 18:36
- like every app that Cursor has or SpaceX AI has
- 18:39
- has a, verification skill that is auto-maintained as well
- 18:45
- and it's become critical infrastructure for our team
- 18:47
- because everybody uses it
- 18:49
- And you went pretty far with that too
- 18:51
- right? Like you had a
- 18:53
- in your talk
- 18:53
- I saw that you actually built a custom
- 18:55
- CLI for that too
- 18:56
- So what does that CLI do?
- 18:57
- Like how does it execute things and why
- 18:59
- did you I mean
- 19:00
- that's proof of how deep you're going
- 19:02
- right? Of how much you're pushing that
- 19:04
- Yeah
- 19:05
- so this is actually a tip I learned early on where
- 19:12
- I guess, you know, back in January or
- 19:14
- or, or late last year
- 19:16
- the thing that people were concerned about was context window
- 19:19
- right? That was the big
- 19:20
- the big topic at the time was
- 19:22
- how do I, you know, manage the context window?
- 19:24
- Because, you know, compaction
- 19:26
- summarization wasn't really that good yet
- 19:29
- And people were always
- 19:30
- people had this
- 19:31
- there was almost this meme in the community that
- 19:34
- you know, once your agent summarized or compacted once
- 19:38
- it would become sort of stupid
- 19:40
- right? For the rest of your session
- 19:42
- So there was a lot of thinking around
- 19:44
- like, you know
- 19:46
- being very efficient with your context usage
- 19:48
- And so
- 19:49
- that was actually the inspiration for
- 19:51
- some of the
- 19:53
- the CLI work inside of the verification skills
- 19:57
- I guess now it's less so about context because
- 20:01
- you know
- 20:01
- agents are
- 20:03
- much better or harnesses have gotten a lot
- 20:05
- better with summarization
- 20:09
- I still think there's some benefits to
- 20:10
- you know, having a clean context window
- 20:15
- so the CLI is really just more of
- 20:16
- a way for me to
- 20:18
- take the
- 20:19
- deterministic parts of what the skill does
- 20:22
- and encode that into a script or a CLI to reduce
- 20:25
- to kind of take away the judgment
- 20:29
- that would otherwise unnecessarily be used
- 20:32
- Because with judgment, so in, I also think of
- 20:35
- you know, agents and skills in sort of like
- 20:38
- it's like a gradient
- 20:40
- You have some parts of the work
- 20:57
- Refactor some code, right, from one pattern to another
- 21:00
- that's very mechanical, right? You don't
- 21:02
- you don't need an agent to
- 21:04
- think about it and come up with it
- 21:06
- in a novel way each time
- 21:09
- right? And so that
- 21:09
- that was really the inspiration for the CLI
- 21:12
- and you'll see this in a lot of
- 21:13
- the other skills that I've built
- 21:14
- is like I try to extract
- 21:16
- out the deterministic parts and turn that into code
- 21:20
- and just leave only the parts that actually
- 21:22
- require judgment to the agent
- 21:25
- So in a way
- 21:25
- I think of the skill as kind of like a wrapper
- 21:28
- right? It's a wrapper with some light instructions
- 21:30
- on how to use these custom tools that
- 21:32
- are inside of the skill
- 21:35
- Oh yeah
- 21:36
- I don't think the CLI is really that
- 21:37
- interesting in its own
- 21:38
- really. It's not like a novel piece of
- 21:40
- software
- 21:41
- It's just something that interacts with
- 21:42
- like
- 21:43
- Playwright and the Chrome DevTools protocol and
- 21:46
- calls a bunch of APIs
- 21:47
- It's like, it's just a bunch of glue
- 21:49
- No, it's fascinating
- 21:50
- because it's a way of hiding information from the skill
- 21:52
- right? It's a way of conserving the skill
- 21:54
- keeping the skill quite small
- 21:56
- I imagine
- 21:56
- and then you're able to delegate more of
- 21:58
- the complicated deterministic stuff
- 22:00
- into a script within the skill
- 22:02
- So it's almost
- 22:03
- you're compressing
- 22:04
- information and making the agent do
- 22:07
- more consistent things more consistently
- 22:10
- Yeah. It's fascinating
- 22:12
- Yeah, and it also helps
- 22:14
- I guess, if you care about context window
- 22:16
- it does help, because now the agent doesn't need to
- 22:18
- you know
- 22:20
- reinvent things. Because
- 22:23
- one thing I had noticed
- 22:24
- early on when we didn't have a CLI
- 22:27
- was that while the
- 22:29
- the agent would try to verify
- 22:31
- it would basically rebuild the world each time
- 22:34
- and then every agent did it differently
- 22:36
- And I was starting to notice
- 22:37
- like, that's very inefficient, right?
- 22:38
- I was wasting, it
- 22:40
- it was actually not just about context usage
- 22:42
- but also speed, right? Like
- 22:43
- cause now an agent had to actually go
- 22:44
- off and write the scripts or the CLI and test it
- 22:47
- and, you know, and it doesn't work
- 22:49
- And the last agent did it
- 22:51
- and it worked, but it discarded it
- 22:52
- So it was just very obvious at that point
- 22:54
- like
- 22:55
- I should just turn this into a CLI
- 22:56
- and put that instead of the skill
- 22:59
- so that every agent that uses it now
- 23:02
- benefits from that same piece
- 23:05
- but I
- 23:06
- I also think, like, you know
- 23:07
- it's a good push
- 23:08
- for people to think about
- 23:10
- is
- 23:10
- how much of your skills and rules could
- 23:12
- actually be deterministic
- 23:16
- that's like another core thing or
- 23:18
- or
- 23:18
- one of my core principles that I like
- 23:20
- to think about is
- 23:22
- yeah
- 23:22
- how do I
- 23:24
- make very efficient use of determinism and nondeterminism and
- 23:30
- you know, let agents shine at the nondeterministic parts
- 23:33
- right, because that's what they're trained to do
- 23:36
- and the other parts which are much more mechanical or
- 23:40
- you know, straightforward can be just pure determinism
- 23:44
- and you'll see this as well for things like doing migrations
- 23:48
- which is another big thing that I've
- 23:50
- I've talked about is
- 23:52
- you know, going from one technology to another
- 23:56
- especially one that is better for agents
- 23:58
- right? And a
- 23:59
- a lot of how you can do that migration is
- 24:01
- I think, through things like scripts and CLIs
- 24:04
- like the deterministic parts
- 24:05
- like code mods, you know, like crawling the AST
- 24:09
- the abstract syntax tree
- 24:10
- and, and transforming code literally mechanically
- 24:13
- right? Like a script does it for you
- 24:14
- instead of the agent
- 24:17
- Totally makes sense
- 24:18
- I, I mean, I think what, there's another thing there
- 24:22
- which is
- 24:24
- you're taking stuff away from the agent
- 24:26
- and you're
- 24:27
- kind of putting it in the environment too a little bit
- 24:30
- which is, let's say you have a
- 24:32
- a thing that you notice the agent always gets wrong
- 24:35
- you want to make that
- 24:37
- just impossible within the environment
- 24:40
- And that sort of comes down to
- 24:42
- code quality as well. I mean
- 24:45
- I talk about a lot like having a
- 24:47
- what a good code base means
- 24:49
- right? What is a good code base?
- 24:51
- And there's a definition I like
- 24:52
- which is
- 24:53
- a good code base is a code base
- 24:54
- that's easy to make changes in
- 24:56
- right?
- 24:57
- Easy to change stuff without things screwing up
- 25:01
- And that means that you have a lot of guardrails
- 25:03
- that you have a lot of
- 25:05
- the
- 25:06
- agent or the human is constrained to very
- 25:08
- narrow paths
- 25:09
- And that's, again
- 25:09
- something you talk about in your talk
- 25:12
- And
- 25:12
- you talk about this not only
- 25:13
- on the kind of Sort of automated checks side of things
- 25:17
- so linting and time checking
- 25:18
- blah, blah, blah
- 25:19
- But also in the way you design abstractions
- 25:22
- and you guys even
- 25:23
- I think
- 25:23
- built a framework
- 25:24
- for your agent to work into
- 25:27
- I think what I'd love to hear is
- 25:29
- you obviously think of that as very important
- 25:31
- right? And that's
- 25:33
- how important is that compared to other things
- 25:35
- you could be doing
- 25:36
- like building features or shipping work?
- 25:39
- Yeah, I think that's a
- 25:41
- I almost feel like the new job of
- 25:43
- the engineer is really to
- 25:45
- to spend time on the environment
- 25:48
- I almost actually wrote a tweet about this yesterday
- 25:49
- but I, but I didn't
- 25:51
- But I think that
- 25:55
- I think that, you know, if you
- 25:56
- if you haven't really spent time
- 25:58
- you know
- 25:58
- building trust in your agents and building skills and tools
- 26:01
- you can get stuck in this mode where
- 26:03
- you're very low on that trust ladder
- 26:06
- right? You don't have a lot of trust
- 26:07
- in your agent's work
- 26:08
- And so The only way to cope in that
- 26:12
- when you're in that situation
- 26:13
- is just to kind of lock in and micromanage your agents
- 26:17
- and that's very time consuming
- 26:18
- And when you're stuck in that mode
- 26:19
- you don't really have the luxury to think about
- 26:23
- you know, higher level things, like, like
- 26:27
- like making yourself more productive
- 26:30
- In the same way that
- 26:33
- I guess analogy would be like
- 26:34
- if you've never taken the time to learn
- 26:36
- like, your tools, or as a developer
- 26:38
- when you were writing code yourself
- 26:40
- and, you know, you, you have
- 26:42
- you've never heard of VS Code
- 26:43
- you've never heard of Vim
- 26:44
- you only knew about Notepad
- 26:47
- and you hadn't, hadn't even heard about Git
- 26:49
- That's sort of the analogy
- 26:50
- It's like, you
- 26:50
- you haven't spent the time sharpening your own knives
- 26:54
- right? And so, of course, if you have a dull knife
- 26:57
- then everything can take a long time
- 27:00
- and you are not like
- 27:01
- but you're, you're just going to be
- 27:03
- and, and especially if, you know, deadlines are looming, then you don't have the
- 27:07
- now you're stuck in this rut
- 27:08
- right? Where, where you, you, you don't have sharp knives
- 27:12
- you don't have good tools
- 27:13
- but you're under all this pressure to ship
- 27:15
- right? And so all you can do is
- 27:16
- just focus on that
- 27:18
- But I do think that
- 27:20
- you know
- 27:20
- if you can find yourself
- 27:21
- the time to actually spend time thinking about your setup
- 27:24
- it's, again, going back to the cooking
- 27:27
- you know, analogy
- 27:29
- it's like, you know, if you, for example, if, if cutting, cutting
- 27:34
- cutting the garlic is like super slow
- 27:36
- right? There are garlic mashers
- 27:37
- right? You can
- 27:38
- you can buy it and you put it
- 27:39
- in the thing and you like squeeze it out
- 27:41
- right? It's super fast
- 27:43
- machines and tools are invented for a reason
- 27:46
- right?
- 27:47
- And so if you're operating a Michelin kitchen and your
- 27:50
- your, your cooks had no tools
- 27:52
- then of course everything is going to be extremely inefficient
- 27:55
- very, very, you know, every, every
- 27:57
- every cook is going to make something up
- 27:59
- of their own
- 28:00
- So I think the tools and the determinism to me are
- 28:04
- you know, taking that part away
- 28:06
- And, and just like you said about constraints as well
- 28:09
- it's, the constraints are, are to me as well like
- 28:13
- actually a slight tangent on that is
- 28:17
- I think we should talk about TypeScript
- 28:19
- Cause like we
- 28:20
- we actually both share like a background in TypeScript where
- 28:22
- you know
- 28:23
- you obviously have done a lot of work
- 28:24
- with TypeScript and total TypeScript and
- 28:26
- you know, you're a leader in that space
- 28:28
- And I
- 28:29
- had adopted
- 28:30
- TypeScript pretty early
- 28:32
- and I had given like a talk or two at TypeScriptConf
- 28:35
- many years ago
- 28:37
- And so one of the
- 28:38
- the talk that I did actually was about
- 28:39
- type systems and constraining the
- 28:43
- constraining types. Like
- 28:44
- one of my most favorite
- 28:46
- things about type theory is actually type narrowing
- 28:48
- right? This idea that you go from a very broad type
- 28:51
- right, that could be anything
- 28:53
- and then you, through type guards and
- 28:55
- you know, type narrowing and, you know, runtime checks
- 28:59
- you can actually narrow the space and say like
- 29:01
- oh, this isn't just a string
- 29:03
- this is a very special type of string
- 29:05
- It's a constant, right? Like, I, I
- 29:07
- I determine that through the type system
- 29:10
- And in a way
- 29:10
- it's like, there's a lot of parallels
- 29:12
- I think, to that with constraints in your code base
- 29:16
- where is it you're
- 29:17
- you're constraining the space
- 29:20
- right? If you, if you think about category theory as well
- 29:23
- you know, you're constraining the, the number of possible types
- 29:27
- right, that can, can exist
- 29:30
- And you're saying there's only one type
- 29:32
- right? And for, for us
- 29:34
- like that framework that I'm building called Dune
- 29:38
- it's not an open source framework
- 29:39
- It's the, the way I describe it to people
- 29:42
- it's
- 29:42
- it's kind of like
- 29:43
- our internal next.js for our electron apps
- 29:47
- but it comes with a lot of really
- 29:49
- really restrictive lint rules
- 29:51
- and the code base is designed in a
- 29:53
- way that
- 29:54
- there's really only one way to do something
- 29:56
- So we make use of
- 29:57
- we make use of a lot of conventional
- 30:00
- patterns
- 30:01
- So like features all go into a specific directory
- 30:05
- Well, every feature has its own directory
- 30:07
- As an example
- 30:08
- you know, there's like a
- 30:09
- a thing that discovers features like through a
- 30:11
- registry
- 30:12
- and like crawling the code base and stuff
- 30:14
- like that
- 30:15
- But this
- 30:17
- conventional pattern
- 30:18
- and the lint rules
- 30:20
- make for an environment where it's actually very
- 30:23
- hard to write bad code
- 30:26
- And that sort of frees up the It
- 30:29
- it both frees up your own mental
- 30:32
- you know
- 30:34
- capacity as well as the agent
- 30:36
- sort of doesn't have to think about that anymore
- 30:39
- where it's just like
- 30:40
- oh, there's only, there's, I should just
- 30:41
- if I want to add a new feature
- 30:42
- it just goes in the feature
- 30:44
- the new feature directory
- 30:45
- and all the code goes in there
- 30:46
- and I'm not going to append to a God file
- 30:50
- right? That was really actually the inspiration for
- 30:52
- those feature directories is
- 30:54
- the very first couple of versions of Grokbot
- 30:57
- were composed of like eight God files
- 31:01
- which were like at least 10,000 lines long
- 31:03
- if not longer, and so I
- 31:05
- I kind of had to break it up
- 31:06
- into smaller pieces
- 31:09
- but it was just observing
- 31:10
- you know
- 31:11
- I think that's another important part is observing how agents fail
- 31:15
- and then every time you see a mistake
- 31:17
- every time you see something that could be done better
- 31:20
- you think, you step back and think
- 31:22
- how do I turn this into a lint
- 31:23
- rule? How do I make it so that
- 31:25
- the code base makes this impossible
- 31:27
- right?
- 31:28
- And it comes back to me for my
- 31:29
- you know, my background learning TypeScript and types
- 31:33
- type systems is how do I constrain the space so that
- 31:39
- you know, I know precisely what I'm working with
- 31:41
- and I think, yeah
- 31:41
- there's a lot of parallels there
- 31:43
- Totally makes sense
- 31:44
- And don't, I mean
- 31:46
- it's funny that you mentioned TypeScript and God
- 31:48
- files in the same sentence
- 31:49
- because TypeScript famously has a 25,000 line type
- 31:52
- Yeah, I realize that already
- 31:53
- Although I
- 31:55
- don't know if they've rewritten that in Go
- 31:57
- They probably have, right?
- 32:00
- Okay, so the environment is important
- 32:03
- You should watch your agent like a hawk
- 32:04
- to make sure that any mistakes it makes
- 32:07
- you turn them into things in the environment
- 32:10
- And the benefit of the environment is you're
- 32:11
- not overloading your agent
- 32:13
- right? In terms of rules
- 32:14
- in terms of things it has to remember
- 32:16
- it's just in the environment
- 32:17
- And so it stumbles into the rules and
- 32:21
- you know
- 32:21
- bounces off them and hits them at the right moment
- 32:24
- So, okay
- 32:25
- we still haven't talked about the 2,500 PRs
- 32:27
- Where do those come from?
- 32:28
- Like, how do you, you've built your trust ladder
- 32:31
- you've worked on your environment
- 32:33
- and you understand, okay
- 32:35
- I now want to scale up
- 32:37
- So what are the mechanics of that scaling?
- 32:40
- Are you, initiating 2,500, like, chats per month?
- 32:46
- That can't be right. So there must be
- 32:48
- are there any kind of automated triggers
- 32:50
- that trigger stuff in your repo?
- 32:53
- Like
- 32:53
- how do you get the software factory kind
- 32:55
- of triggering work by itself?
- 32:57
- Right
- 33:00
- I'll definitely say that the prerequisite to
- 33:03
- you know, something like a, a very high volume of
- 33:06
- of pull requests, is the environment, you know, the, the stuff we just talked about
- 33:12
- where I definitely would not have been able
- 33:14
- to do this if I had not spent the time
- 33:17
- you know, thinking about the kitchen
- 33:18
- right
- 33:18
- and the knives and the tools for my agents
- 33:21
- And so, in a way, I
- 33:23
- I think of this as
- 33:25
- I've spent a time building one kitchen
- 33:27
- and one restaurant
- 33:29
- And now I
- 33:30
- am in a position where
- 33:31
- I don't actually have to be there anymore because the environment
- 33:36
- you know, the same analogy, right?
- 33:38
- It works really well
- 33:39
- Yeah, I know, that's not
- 33:42
- Yeah, exactly
- 33:42
- Yeah, exactly
- 33:43
- It's like you're Gordon Ramsay and
- 33:45
- you know, you've taught your executive chef
- 33:48
- like, all the tricks of coming up with great menu
- 33:52
- and, like, the kitchen is set up really well
- 33:54
- everything's just perfect
- 33:56
- and you're now in a position where you
- 33:57
- can open your second
- 33:58
- your third restaurant
- 34:00
- And
- 34:01
- I guess I sort of see each
- 34:03
- project that I work on
- 34:04
- like, each big chat is sort of like a restaurant
- 34:07
- right? And
- 34:09
- I have multiple of them operating at the same time
- 34:12
- and I'm sort of
- 34:13
- like, helicoptering between them, sometimes some more than others
- 34:18
- depending on how in the loop I am
- 34:20
- But yeah
- 34:20
- definitely I think
- 34:22
- there are external triggers and context that those projects don't have
- 34:29
- that for a long time
- 34:32
- I was the proxy for that
- 34:33
- So, the best example I have is like
- 34:35
- you know, you have a project that's working on a feature
- 34:38
- or you're trying to fix a bug
- 34:40
- and you're getting bug reports
- 34:42
- but the bug reports are going to things
- 34:43
- like Slack or Linear or X
- 34:47
- right?
- 34:48
- And these are
- 34:49
- external systems that aren't connected to your inner loop
- 34:53
- So I
- 34:53
- I like to talk about this
- 34:54
- outer loop and the inner loop
- 34:57
- I don't know if I'm using the definition correctly
- 34:59
- but to me, my inner loop is like basically my engineers
- 35:03
- my agent engineers working on the code to an
- 35:07
- building towards an intent
- 35:10
- Or snapshot of my intent
- 35:12
- right?
- 35:12
- And
- 35:13
- the thing about that is that the snapshot can go stale
- 35:16
- right? New information comes to light
- 35:18
- that I then have to be the proxy of and
- 35:21
- you know, transfer that context to my agent
- 35:23
- So, you know, if you
- 35:24
- if you don't have
- 35:25
- these triggers pulling information back into the inner loop
- 35:29
- then you sort of have to play that role where you're
- 35:33
- you're off, you know, in Slack or X or
- 35:36
- or whatever, and you're gathering context
- 35:38
- right?
- 35:39
- You're getting context about bug reports
- 35:41
- about feature requests, about, you know, something someone said about
- 35:45
- you know, our backend infrastructure has some limitation
- 35:48
- you know, all that information
- 35:50
- you have to ferry that across
- 35:52
- to your agent
- 35:53
- So that's where I think like
- 35:55
- tools like Grokbot
- 35:56
- are really good because they help you
- 35:59
- automate the outer loop as well
- 36:01
- And when you connect those two loops
- 36:03
- it's very
- 36:04
- very powerful because now all of a sudden
- 36:06
- your agents have the ability to get context for their
- 36:10
- for themselves, right?
- 36:11
- If for example, you know, either through
- 36:15
- just as a simple example
- 36:17
- like maybe you have the Slack MCP
- 36:19
- right? Or you have
- 36:21
- your own harness
- 36:22
- right? That you've built
- 36:23
- a Slack subscription into for a particular Slack
- 36:26
- channel
- 36:27
- Now all of a sudden you can tell your agents
- 36:29
- okay, subscribe to the Slack channel
- 36:32
- Every time there's a
- 36:34
- you know, bug report about something
- 36:37
- go off and triage that thing
- 36:39
- right? Go reproduce the issue
- 36:41
- right? Using the verification skills that we've already
- 36:43
- spent time building
- 36:45
- and all of those other skills that we've
- 36:46
- set up
- 36:47
- So that I have a lot of trust
- 36:49
- right? I have a lot of trust that
- 36:50
- these agents can actually go off and understand the bug
- 36:54
- you know, verify that the bug actually still exists on main
- 36:58
- and it wasn't something about
- 37:01
- you know, maybe the user's setup or their data
- 37:04
- or maybe, I don't know
- 37:06
- they didn't install a dependency or something like that
- 37:08
- Like, basically, I think
- 37:12
- creating that, yeah
- 37:14
- creating those two loops and connecting them
- 37:16
- is really a very important part of the job these days
- 37:20
- especially if you are thinking about how to scale yourself
- 37:24
- So, a big theme here is really just
- 37:26
- like, always thinking about, like, what
- 37:29
- where am I the bottleneck in this process?
- 37:32
- Why do my agents need me
- 37:33
- you know, to answer this question?
- 37:35
- I, I always like to think about that
- 37:37
- and so
- 37:37
- I try to think about how do I
- 37:38
- actually get the agent to answer its own question
- 37:41
- right? But not by hallucinating
- 37:43
- not by guessing, but actually real data
- 37:46
- And, you know
- 37:47
- a lot of people talk about this idea
- 37:48
- of a company brain
- 37:49
- right, or a context graph
- 37:52
- I feel like those terms aren't necessarily complex
- 37:57
- or even abstract
- 37:58
- To me, it's just about
- 38:00
- how do I
- 38:01
- take information that my agent needs that
- 38:04
- I would otherwise have to go and
- 38:06
- pass it myself and just teach it how to do it
- 38:09
- right? And that removes me from the equation
- 38:12
- And so how I arrive at 2000
- 38:14
- or however many PRs
- 38:16
- is the fact that I have all these loops set up
- 38:20
- right? And so, it allows me to open chain restaurants
- 38:25
- right? I can, I can really parallelize myself
- 38:28
- So yeah, I'm not sitting there creating 2500 chats
- 38:31
- right, of course
- 38:32
- It's really like these projects are
- 38:35
- actually Cursor has a new feature called projects
- 38:37
- which are these like coordinator agents
- 38:42
- and so the coordinator
- 38:43
- coordinator agents are really good at
- 38:45
- Sort of delegating and not doing work of their own
- 38:48
- but they manage and supervise like almost a
- 38:50
- list of tasks
- 38:51
- and they sponsor agents to go and do
- 38:53
- them
- 38:54
- And so I'm just constantly
- 38:55
- feeding context or teaching the agents how to
- 38:57
- get their own context and then they're going
- 38:59
- off and doing the work for me
- 39:02
- and really the big
- 39:03
- the last thing I'll say to this is
- 39:04
- like the big unlock
- 39:05
- for me for getting to 2000 PRs
- 39:08
- is
- 39:09
- starting from the question
- 39:10
- and working backwards of
- 39:12
- how do I
- 39:13
- get to the point where my agent can
- 39:15
- merge its own code?
- 39:16
- Because the obvious thing people ask me when
- 39:20
- when they, when I tell them
- 39:21
- oh, I shipped 2000 and 2500 pull requests last month
- 39:24
- they'll be like, how did you review that?
- 39:27
- Right? That
- 39:27
- that's just a lot of PRs to review
- 39:29
- Like your team must hate you
- 39:30
- Do you
- 39:31
- do you mind if we go there in a second?
- 39:32
- Yes, absolutely. Good question about that
- 39:34
- Yeah, yeah, yeah. I want to
- 39:36
- like, this analogy is great
- 39:37
- I want to like deepen it a bit
- 39:38
- which is Before, if you're like manually initiating all those chats
- 39:43
- it's like you're bringing the orders to your chefs manually
- 39:46
- right?
- 39:47
- Whereas if you've got an agent
- 39:49
- sort of like doing the expo
- 39:50
- then you're able to sort of run it yourself
- 39:53
- What does
- 39:54
- what does that concretely look like then?
- 39:56
- You've got these sort of Grok bots that
- 39:57
- are subscribing to channels
- 39:59
- pulling in Slack messages
- 40:00
- and you, it sounds like
- 40:02
- have a couple of coordinator agents or like
- 40:04
- chief of staff agents that
- 40:07
- like monitor that or something
- 40:08
- Like when you look at your computer to manage your agents
- 40:12
- what does it look like?
- 40:15
- Yeah, so, so this is, I guess, somewhat confusing, but we're working on
- 40:19
- you know, simplifying and unifying
- 40:21
- But so there's Grokbot
- 40:24
- which, or, you know, you can use other tools
- 40:26
- of course, as well
- 40:27
- But I
- 40:28
- I largely think of these tools as like
- 40:29
- your outer loop. These are tools like
- 40:32
- you know, Grokbot that have connectors
- 40:34
- These are connectors, I guess they
- 40:36
- a lot of people call them personal agents
- 40:39
- but they are connectors to things like your email
- 40:41
- your calendar, Slack, Plaid, I don't know
- 40:46
- like all these different services
- 40:48
- And they are a great source of pulling
- 40:49
- context in to your work
- 40:53
- So the same way that a human
- 40:55
- like, you know, if I were
- 40:56
- if I was a manager
- 40:58
- and I was leading a team of engineers
- 41:02
- you know, like when I used to work in Netflix
- 41:04
- one of the biggest things that managers would
- 41:06
- talk about was this idea of context
- 41:08
- not control, which funnily enough, you know, has so much
- 41:13
- has so much, carry over to the agent's world
- 41:17
- of, you know, you, you know, you
- 41:19
- you of course can drive to an outcome
- 41:21
- you want by control
- 41:22
- right? Like by micromanaging
- 41:24
- but what you want is to provide context instead
- 41:27
- right? Like teach the agent
- 41:28
- teach your engineers how to be self-sufficient
- 41:31
- and then you don't have to micromanage them
- 41:35
- and so I see a lot of parallels there
- 41:37
- but yeah, GrokBot, so concretely
- 41:39
- I have some GrokBots that look at my Slack channels
- 41:43
- look at my X
- 41:45
- or my emails, or, or linear, and they're just constantly
- 41:49
- they have routines that subscribe
- 41:51
- so they're constantly watching
- 41:53
- and I have, I'll tell them things like
- 41:55
- you know, I'll watch for issues with
- 41:58
- bugs in the GraphBot. One of the really
- 42:07
- cool things about GraphBot is it connects to
- 42:09
- Cursor
- 42:10
- So Cursor has
- 42:12
- like I, I just mentioned, this new feature called Projects
- 42:15
- and a Project is really a
- 42:18
- again, like a
- 42:18
- you get a coordinator agent that's in the cloud
- 42:21
- it has its own computer
- 42:23
- and all it really does is
- 42:24
- like, it's a manager of agents
- 42:26
- It's like your executive chef
- 42:27
- right, your, your chief of staff
- 42:29
- It doesn't do the work itself
- 42:31
- It delegates, right
- 42:33
- and orchestrates and manages the work of other sub-agents to
- 42:37
- you know, that report to your chief
- 42:41
- your chief, of staff
- 42:43
- And it
- 42:44
- basically is responsible for driving the work forward
- 42:47
- and managing things and
- 42:50
- passing context to them
- 42:52
- So if you get a sudden burst of issues
- 42:53
- let's say you get 30 issues at once
- 42:55
- in one payload or something
- 42:56
- or very quickly
- 42:57
- the coordinator agent can figure it out and delegate
- 42:59
- Yeah, exactly. It gets like
- 43:01
- you know, 30, the
- 43:02
- the 30 or so payloads and
- 43:03
- spawns a sub-agent or a single coordinator agent
- 43:06
- It can actually do a
- 43:07
- a bunch of different topologies of agents
- 43:10
- and it will sort of figure out the best way to
- 43:12
- you know
- 43:13
- efficiently distribute the tasks to your
- 43:17
- team of agents
- 43:21
- so I use pro
- 43:22
- cursor projects a lot
- 43:24
- and I also use GraphBot a lot
- 43:27
- and
- 43:27
- cursor projects are my inner loop and GraphBot
- 43:29
- is my outer loop
- 43:31
- GraphBot takes all the context
- 43:33
- external context
- 43:35
- Gives it to the projects
- 43:36
- because it can actually just send messages to those projects
- 43:39
- right? You don't even have to open cursor
- 43:41
- You can just tell your Grok
- 43:42
- okay, create a project, right
- 43:43
- for these series of tasks
- 43:46
- They're all related, right? Maybe as an example
- 43:49
- you know, you've had a
- 43:52
- a big burst of issues that are all about performance
- 43:54
- right? Your app is slow
- 43:57
- and they're all connected
- 43:59
- right? Maybe some of them even have a similar fix
- 44:02
- right? But
- 44:03
- and you can certainly go off and just
- 44:04
- spawn one agent per task
- 44:07
- but then you've lost that sort of thread between them
- 44:10
- right? And, and you may duplicate work
- 44:12
- or you may not
- 44:12
- really think about the higher level problem
- 44:14
- You know, sometimes when you, you solve bugs, you know
- 44:17
- it helps to have multiple bug reports that are
- 44:20
- are slightly different, because it helps you really
- 44:22
- you know, zoom out and see
- 44:23
- actually, you know, the problem
- 44:25
- when I looked at this one report
- 44:26
- I thought the bug was here
- 44:27
- but actually when, when I see the other multitude of bugs
- 44:30
- it's actually up here
- 44:32
- Yeah
- 44:33
- got you. So
- 44:33
- that's why you have so many agents in that loop then
- 44:36
- right? Because it's not just you have
- 44:38
- like you have a bug report comes in
- 44:40
- you spawn a single agent to look at
- 44:41
- that bug report. That
- 44:43
- that single agent will be duplicating work with other
- 44:46
- other agents
- 44:47
- right? Because if there are multiple bug reports
- 44:49
- coming in through the same thing
- 44:50
- that can be duplicated work
- 44:51
- Fascinating
- 44:53
- That's really fascinating
- 44:55
- Okay, and so this, just this endless series of triggers
- 44:58
- coming from real users reporting real reports
- 45:02
- builds up this sort of
- 45:04
- and accelerates the factory
- 45:05
- sort of adds more orders in
- 45:08
- Other than bug reports
- 45:09
- are there any other sources that you use for
- 45:11
- like, accelerating for pushing these PRs?
- 45:17
- well, funnily enough, it's, some of it comes from
- 45:20
- reading the code too
- 45:22
- So I guess I have sort of
- 45:25
- well, I, so To clarify that
- 45:27
- you know, the 2,500 PRs, they're not obviously like 2,500 features
- 45:32
- right? They are
- 45:33
- a lot of the work actually is spent on gardening
- 45:37
- like another term that I really love
- 45:41
- where
- 45:42
- so
- 45:43
- I guess this is more important
- 45:44
- when you have a big team of engineers
- 45:46
- human engineers that you work with
- 45:48
- where
- 45:50
- and it also just goes back a little
- 45:51
- bit to what I was talking about with the environment
- 45:53
- you know
- 45:53
- setting up a really good environment doesn't just
- 45:55
- help you and your agents
- 45:56
- but everybody on your team
- 45:57
- right?
- 45:58
- Think of a new hire who doesn't have
- 45:59
- a lot of context
- 46:00
- on all of your engineering practices joining your
- 46:03
- your team
- 46:04
- And if you have a really good environment
- 46:06
- they can be productive from day one
- 46:08
- right? They can, they don't have to like
- 46:09
- you know, make, open a bunch of low quality PRs
- 46:12
- they can start, you know
- 46:14
- they can start just churning out really good code
- 46:19
- and, yeah, I think I, I
- 46:22
- I sort of lost my train of thought
- 46:23
- I've got a, I've got a follow-up
- 46:25
- which is what, what are the mechanics of like triggering
- 46:29
- like how, when, when do you trigger a
- 46:31
- a, a, a agent to go and look at the code
- 46:34
- right? Because some people might say
- 46:36
- well, let's just do that every hour or something
- 46:39
- or like on a cron job
- 46:40
- or what's. Oh, yeah, yeah, yeah
- 46:43
- I
- 46:43
- yeah. I saw some of your recent tweets
- 46:45
- as well about like
- 46:45
- you know, the, some of the tweets you've been doing
- 46:47
- which are great for setting up these routines
- 46:52
- I have some routines like that as well
- 46:55
- so, one of them is
- 47:00
- like looking through, just another simple example is
- 47:05
- you know, React has a lot of foot guns
- 47:08
- so, as, as I'm sure you're aware
- 47:10
- And so I have an agent that's just
- 47:12
- constantly looking for bad patterns
- 47:14
- And
- 47:15
- The interesting thing about that one is that
- 47:17
- I don't actually tell it to fix the
- 47:18
- issue first. I tell it to append it
- 47:20
- to a document
- 47:22
- And then every
- 47:23
- couple of days I look at it and I say
- 47:24
- actually, these are all the same thing
- 47:26
- you know?
- 47:27
- And so that gives me
- 47:28
- you know, you almost want like a buffer
- 47:30
- a queue
- 47:31
- Sometimes that's actually more effective than
- 47:34
- just spawning off a couple of
- 47:35
- like
- 47:35
- a lot of sub-agents to fix every single
- 47:37
- thing
- 47:38
- Because when you are in
- 47:40
- kind of pure execution mode and just trying to like
- 47:44
- you know
- 47:45
- execute on the orders that are coming in very fast
- 47:48
- you sometimes miss the big picture
- 47:50
- So sometimes having a buffer
- 47:52
- forces you to think about the big picture because you
- 47:54
- you
- 47:55
- you have these
- 47:56
- Artifacts and things that you can look at as a human
- 48:00
- and sort of use your own human judgment to
- 48:03
- or
- 48:04
- I guess you can use an agent to
- 48:05
- do that as well
- 48:06
- but you give the agent
- 48:08
- and yourself a way to identify patterns
- 48:11
- right
- 48:11
- that you might otherwise miss if you're just
- 48:14
- only solving each bug at a time
- 48:17
- And that's also really the benefit of having
- 48:19
- something like a chief of staff agent is
- 48:22
- it can see the forest
- 48:24
- right, in addition to actually doing the execution
- 48:29
- Fascinating
- 48:30
- so I mean
- 48:31
- my brain is exploding a bit there with
- 48:33
- the sort of chief of staff
- 48:34
- of the software factory. I might have to
- 48:36
- change some of the course that I'm filming
- 48:38
- next week
- 48:41
- All right, let's talk about, let's talk about review
- 48:42
- right? Because this is the reply that you get
- 48:45
- you know, is, did you read
- 48:47
- did you taste all
- 48:48
- 2,500 of those dishes as they swept past
- 48:51
- you?
- 48:52
- And I assume the answer is a variety
- 48:55
- is a version of no
- 48:56
- Yeah, I, I think you, you
- 49:00
- you
- 49:01
- don't want to be in a position where
- 49:02
- you're not tasting your food ever again
- 49:05
- but you also, you know, for scale
- 49:07
- you cannot be tasting every single
- 49:08
- dish that comes out of your kitchen
- 49:10
- especially if you have multiple restaurants
- 49:12
- So it becomes more about Sampling
- 49:15
- right, and thinking about the processes
- 49:17
- In the same way that
- 49:18
- you know, if I guess maybe this is where the
- 49:21
- the, the factory analogy is a bit more apt
- 49:24
- is, you know, as a quality supervisor on a factory
- 49:27
- you, you can't look at every single item
- 49:30
- You sample, right? You take a
- 49:32
- you, you
- 49:32
- you go in there
- 49:34
- every day and you look at the quality
- 49:35
- of the pull requests
- 49:36
- you look at the code that the agents are writing
- 49:39
- and you
- 49:40
- scrutinize it very rigorously and you Think about all the inefficiencies
- 49:46
- the bad patterns that the agent are doing
- 49:48
- and then you think about how to course correct the environment
- 49:52
- right? Not, not that single agent
- 49:55
- because if, maybe if it, if it
- 49:57
- if it was a one-off incident
- 49:59
- it's fine, you know
- 50:00
- that maybe there's nothing to fix there
- 50:03
- But if you actually notice that multiple agents are
- 50:06
- are, are having the same issue
- 50:07
- right? They're taking the same shortcut
- 50:09
- they're, they're propagating the same workaround everywhere
- 50:13
- that's a sign
- 50:14
- that you should go off and think about how to
- 50:17
- amend your kitchen or your factory
- 50:20
- right? Like thinking about your skills
- 50:21
- your constraints, your lint, your type systems, and setting
- 50:27
- or
- 50:28
- adjusting it so that that
- 50:29
- problem doesn't happen again
- 50:31
- And when you do that enough times
- 50:33
- then you get to a place where the code basis
- 50:35
- again, like the environment is so constrained
- 50:38
- and so
- 50:39
- it guides you so well
- 50:41
- that you can just You can just step away
- 50:44
- right? That's the dream. And I'll
- 50:45
- I'll definitely say it
- 50:47
- it's very hard to get to this point
- 50:50
- I don't want to sell this as like
- 50:51
- you know
- 50:52
- something that you can just do easily by using PSAC
- 50:55
- Like
- 50:55
- it takes a lot of time and effort
- 50:57
- to think about your code
- 50:59
- and where you see your agents failing and thinking very thoughtfully
- 51:04
- intentionally
- 51:05
- and
- 51:05
- setting up guardrails and constraints
- 51:08
- so that they
- 51:09
- do the right thing by default
- 51:12
- And you're not, like, if
- 51:15
- to get back to the software factory analogy
- 51:17
- this isn't a dark factory
- 51:18
- right? This, the lights are on
- 51:20
- It kind of is
- 51:21
- yeah, actually
- 51:22
- Is it?
- 51:22
- Yeah, well, it's dark in the sense that
- 51:24
- so, it's dark in the sense that
- 51:27
- well
- 51:27
- I think if my agents are merging their own pull requests
- 51:30
- it's sort of become dark where I go to sleep
- 51:34
- my agents now work 24/7
- 51:37
- I have
- 51:38
- I have the equivalent of like more than
- 51:39
- ten chief of staff
- 51:41
- right, each working on a different area
- 51:43
- Like, for example
- 51:44
- I have one that's working on performance of
- 51:46
- the GraphBot desktop app
- 51:48
- I have one that's working on
- 51:50
- fixing bugs that users report
- 51:52
- I have one that's exploring
- 51:55
- rewriting it in a different language just for fun
- 51:57
- You know, like what if, what if, you know, just reimagining what
- 52:00
- what it would be if it was like
- 52:01
- a native app
- 52:02
- It's just a toy
- 52:05
- but the idea is like
- 52:05
- yeah, I, I
- 52:07
- when you spend the time setting up your environment
- 52:10
- I've gotten to a point where I review
- 52:11
- the pull request after it's landed
- 52:14
- I, I, I'll tell my agents full autopilot is
- 52:17
- is something that you can do in
- 52:18
- in PSAC
- 52:20
- and that will
- 52:21
- trigger off this very intense rigorous verification loop
- 52:24
- where it'll spawn a bunch
- 52:26
- of verifier agents for every pull request
- 52:28
- and it will fuzz
- 52:29
- right?
- 52:30
- Fuzzing, meaning that it will actually run the application
- 52:33
- it's going to click around and try to
- 52:35
- use it like a real human
- 52:36
- look for regressions, look for bugs in your implementation
- 52:40
- and, it will try to find issues with the thing
- 52:44
- and then it will fix it itself
- 52:46
- it'll do that again
- 52:47
- and eventually get
- 52:48
- the PR to a state where it can
- 52:49
- land
- 52:52
- so it does, it does
- 52:53
- it is quite token intensive
- 52:55
- You can tune this
- 52:56
- of course
- 52:58
- so, you know, instead of like ten verifier agents
- 53:00
- you might do like one
- 53:02
- right? Or
- 53:03
- you can just tell the agent to verify
- 53:04
- it doesn't work
- 53:05
- But yeah
- 53:06
- the key thing is
- 53:07
- the verification part
- 53:09
- is really the key piece
- 53:10
- that gives me a lot of confidence
- 53:13
- that I guess verification plus the environment
- 53:15
- right? It's these
- 53:16
- the combination of these two things that allow
- 53:18
- me to step away and say
- 53:19
- agents, go off and merge your thing
- 53:20
- I'll review it in the morning by looking
- 53:23
- at my commit history
- 53:25
- And if I see problems
- 53:26
- I go in course correct
- 53:28
- right? And I'll go and revert or modify
- 53:31
- add new lint rules and whatever
- 53:35
- so it does
- 53:35
- it does take time to get to that
- 53:37
- point
- 53:37
- But once you get it
- 53:38
- oh, it's so, it feels so magical
- 53:42
- I, I, I tell people like
- 53:43
- I'm sleeping so much better now because
- 53:46
- you know, it took, it
- 53:47
- the very first day I turned on the
- 53:49
- sort of dark factory was very scary because I was like
- 53:52
- what if I cause a sever
- 53:53
- right? What if I break something overnight?
- 53:56
- and it took a lot of
- 53:58
- it took a lot of
- 54:00
- bravery, I think, to do that
- 54:01
- but
- 54:02
- somehow I did it and
- 54:03
- now I'm in a place where my agents are
- 54:06
- are merging their own code while I sleep
- 54:09
- So I think it's dark in that sense
- 54:11
- Yes, it's dark sometimes, right?
- 54:12
- You do sometimes. That's true
- 54:14
- that's true. Because I think of a dark factory as like
- 54:17
- almost like
- 54:18
- If you take the original definition of Karpathy's vibe coding
- 54:21
- right, which is the code almost doesn't exist
- 54:23
- you forget the code might be a thing
- 54:25
- I think your approach is totally different from that
- 54:28
- which is that
- 54:29
- code and the environment is essential
- 54:32
- And if the code and the environment are bad
- 54:34
- then you will get bad outputs
- 54:35
- garbage in, garbage out
- 54:36
- So I, I, I think this is a
- 54:40
- this is a different thing
- 54:40
- It's like, you know
- 54:43
- the, I don't know, maybe there's a dimmer switch or something
- 54:46
- right? Like, you know, some parts of the factory are dark
- 54:48
- some parts of it are light
- 54:49
- This is why the
- 54:49
- maybe the, the kitchen is a better analogy
- 54:53
- the restaurant, because, you know, even as a, as a, as a As a restaurateur
- 55:00
- you
- 55:00
- still might go to a restaurant every now
- 55:02
- and then to take a
- 55:03
- take a peek in
- 55:04
- taste the food, right?
- 55:05
- Yeah. The idea of sampling instead of blocking
- 55:08
- I think is really important
- 55:09
- I think what would you say to people who are in
- 55:12
- I guess you're obviously in a pretty security
- 55:15
- conscious environment where you're working
- 55:16
- very security conscious
- 55:20
- maybe there are folks working in like
- 55:23
- medical applications or law or finance or something
- 55:27
- I think of the
- 55:28
- like, some PRs are kind of like two-way doors
- 55:31
- which is you can merge it and then revert it
- 55:33
- right? It's cheap to go back through
- 55:35
- But there are some PRs that are one-way doors
- 55:37
- right? That will cause data loss of some
- 55:39
- kind that will
- 55:41
- do something that can't be easily walked back
- 55:44
- How do you deal with situations where most of your PRs
- 55:48
- let's say, are one-way doors?
- 55:50
- Like, is this something you just wouldn't recommend or like
- 55:52
- what do you think?
- 55:53
- Yeah, I think that's a really good question
- 55:55
- I think that
- 55:57
- it all comes back to me to the
- 55:59
- quality of the verification that you're able to
- 56:04
- get out of your agents
- 56:06
- And I think for domains where the work is verifiable
- 56:11
- this is easier, right? And the
- 56:13
- the
- 56:14
- one-way doors become two-way doors in a sense
- 56:18
- But I guess, I don't know
- 56:19
- if you're working on something that is like extremely
- 56:22
- is very hard to verify programmatically
- 56:26
- then I think, yeah
- 56:26
- you're definitely in a position where it's very
- 56:28
- hard to get to that point
- 56:31
- so I do think like
- 56:32
- yeah
- 56:33
- verifiability of the domain is an important
- 56:36
- aspect to be able to do this
- 56:38
- and software engineering is just one of those
- 56:40
- things where it's quite verifiable in
- 56:43
- in a lot of cases
- 56:44
- maybe not totally
- 56:46
- you know, like other domains like mathematics
- 56:48
- I think, are another example of Not all of it
- 56:51
- of course
- 56:52
- but
- 56:53
- some aspects of mathematics can be verifiable
- 56:57
- if you write a proof
- 56:58
- for example
- 57:00
- And so, yeah
- 57:02
- I think it's a great question
- 57:03
- that I don't really have the answer to
- 57:05
- and I think that this is something
- 57:06
- the industry and us as engineers will have to figure out
- 57:10
- is, you know
- 57:11
- my sort of
- 57:13
- hope and prediction for the future is that
- 57:15
- we'll see more and more interesting new
- 57:18
- agent-oriented programming languages
- 57:21
- And one of the most fascinating ones that
- 57:23
- I've seen so far is this one called Bend
- 57:25
- B-E-N-D, Bend
- 57:27
- And that language is
- 57:30
- one where it kind of
- 57:32
- marries programming with proofs
- 57:35
- Right
- 57:36
- There used to be a time
- 57:38
- you know
- 57:38
- where you actually had to write your proofs
- 57:40
- in a different language
- 57:42
- And proofs, by the way, for those, who, who aren't familiar, is this idea of
- 57:47
- that you can sort of formally
- 57:49
- verify that some code is correct mathematically
- 57:53
- right? Especially if you've written your code in
- 57:55
- a very functional programming way
- 58:00
- but for the longest time
- 58:01
- you had to do that in a separate
- 58:02
- language like Lean or TLA+ or
- 58:06
- I'm blanking on some of the other
- 58:08
- other examples, but, yeah, like languages like that
- 58:10
- where you would construct the mathematical proof and
- 58:13
- then use a solver essentially to
- 58:16
- to check, right, that, that you've covered all the cases
- 58:19
- you don't have like a race condition or
- 58:21
- or whatever
- 58:23
- So, yeah, I think
- 58:27
- Trying to sum up the question
- 58:28
- I think, yeah
- 58:29
- if you are in a position where you
- 58:30
- can figure out how
- 58:31
- your agents can truly verify the work in
- 58:34
- a way that gives you confidence
- 58:36
- you can actually, you know
- 58:39
- have the PRs merge. Because if it compiles
- 58:42
- right, if it, if the proofs show you that it's correct
- 58:45
- then why wouldn't you just merge it?
- 58:49
- but of course, yeah, not all domains are verifiable
- 58:53
- Yeah, it's a tough one
- 58:56
- okay
- 58:57
- I think
- 58:58
- we've got to
- 58:58
- think about wrapping up because we are nearly
- 59:00
- on the hour. Have you
- 59:01
- have you got something after this?
- 59:02
- I mean, I've got something after this
- 59:03
- I've got to get my son dinner
- 59:04
- but
- 59:06
- I, I can go a bit longer after you
- 59:07
- Okay, let's, let's go five minutes longer then
- 59:10
- I think I just want to have one more question
- 59:12
- which is
- 59:14
- I think I want to ask how
- 59:17
- you see P stack and how you see skills in general
- 59:22
- like in terms of Okay
- 59:24
- we talked about this before we went on air
- 59:26
- which is like, people think of lit
- 59:28
- it's like my skills versus your skills
- 59:31
- and how do you combine frameworks together?
- 59:34
- How do you use PStack with my stuff?
- 59:37
- Like, what should you take from each one?
- 59:39
- And
- 59:39
- because I think
- 59:40
- I see skills as sort of just derived from process
- 59:45
- basically. Like, they're just processes turned into words
- 59:49
- and I would love to know
- 59:52
- how you recommend people take PStack and take
- 59:54
- my stuff as well
- 59:55
- and turn it into their own processes
- 60:00
- I think you shared a tip
- 60:01
- actually, to tell you that I thought was actually very relevant
- 60:04
- which is this idea that you go off
- 60:05
- and look at your previous transcripts
- 60:08
- right, and you sort of mine for information of
- 60:11
- you know, your own Look through your own
- 60:14
- your own prompts, right, to the agents where you correct them
- 60:17
- where you have to constantly intervene and
- 60:21
- you know
- 60:22
- take that higher level learning and turn that into a
- 60:24
- a reusable skill, right
- 60:25
- so that agents stop repeating that mistake
- 60:29
- I think that, PSAC and your skills are very complementary
- 60:34
- I, I totally agree with you that they're
- 60:36
- like, a skill is really much just process
- 60:38
- I mean, it's just, at the end of the day
- 60:40
- a skill is just English or
- 60:41
- or language. It's just Markdown
- 60:45
- and I think you can
- 60:46
- you can definitely weave them
- 60:48
- combine them in a way that makes sense
- 60:50
- to you
- 60:51
- But I do think that
- 60:53
- everyone should have their own set of knives
- 60:57
- right? I, I keep going back to the
- 60:59
- the, the cooking analogy, but it's so apt because like
- 61:02
- you know, every chef When they go to a different job
- 61:05
- right, when they go to a different restaurant
- 61:07
- they carry, they bring their knives with them
- 61:09
- the tools go with them
- 61:11
- right? And so trust to me is really
- 61:12
- about trust in your own tools
- 61:14
- And
- 61:15
- when you spend the time sharpening them and understanding them really
- 61:18
- really well, you can do great things
- 61:19
- And
- 61:20
- everybody's skills and tool set is going to
- 61:22
- look different
- 61:23
- You know, someone might find a lot of success combining
- 61:27
- you know, like your grill me with dogs
- 61:30
- or wayfinder skill with some of the execution skills in PSAC
- 61:33
- as an example
- 61:34
- Some people might use more of your skills
- 61:36
- some people might use more of my skills
- 61:39
- I think at, at, at the end of the day
- 61:40
- it really just comes back to how much do you trust
- 61:43
- you know, me and Matt, right?
- 61:45
- Like, if you, if you trust us both
- 61:46
- of course, use our skills, but I also encourage you to
- 61:49
- you know, look at your own transcripts
- 61:53
- Tell your agent to look through
- 61:55
- you know, some of the
- 61:56
- all of the patterns that you've used
- 61:58
- the, the times you've had to intervene
- 62:01
- you know, suggest turning them into lit rules or new skills
- 62:05
- right?
- 62:06
- The, the past chats I
- 62:07
- I often say is like a
- 62:09
- a treasure trove of context because the
- 62:11
- you know, it's
- 62:13
- it's like, the process materialized, right?
- 62:16
- Like it's the real process
- 62:18
- not an abstract idea in your head
- 62:20
- It's the actual thing
- 62:22
- right? And you can actually see how it
- 62:23
- happened in practice and extract so much information
- 62:26
- from that
- 62:27
- And there's so much show that I actually turn
- 62:28
- I have a skill in PSAT called recall
- 62:30
- which is exactly that
- 62:33
- where this was a pattern where You know
- 62:36
- I was working in
- 62:37
- I was working on a
- 62:38
- a similar problem. So specifically
- 62:40
- I was working on virtualization for the cursor application
- 62:43
- and there were a lot of bugs
- 62:45
- And so, you know, every time I started a new chat
- 62:47
- I was like, oh, there's
- 62:48
- there's so much good context from the last one
- 62:49
- though
- 62:50
- You know
- 62:50
- I want to bring it over to the
- 62:51
- new chat. How do I do that?
- 62:52
- And that's where the transcript came
- 62:55
- you know, looking at the past transcript came about
- 62:57
- and then recall was just a way for me to collapse
- 63:00
- collapse and compress that workflow into a skill
- 63:03
- so that I didn't have to just say
- 63:06
- I didn't have to write a long essay every time
- 63:07
- go look at all these chats
- 63:08
- right? And, you know, the blah, blah, blah
- 63:10
- So I, I largely think of skills
- 63:12
- especially as agents get more capable
- 63:15
- as really encoding workflows
- 63:17
- You know, skills from last year were really more about
- 63:20
- like, almost like implementation details
- 63:22
- like, here are the exact script
- 63:24
- Commands, you know, you should use, right?
- 63:27
- I think with the latest models
- 63:29
- you can just delete those parts
- 63:31
- and just really focus on the workflow
- 63:32
- right?
- 63:33
- And tell it, it's more
- 63:34
- the skill becomes more like a series of steps
- 63:38
- a series of, a real process
- 63:41
- and I think over time
- 63:42
- we'll see that skills get smaller and smaller
- 63:45
- you know, more compact
- 63:47
- and yeah, they're very, very compatible
- 63:50
- Or you can, you know
- 63:51
- if you want, why not read our skills
- 63:54
- right? And come, and combine them in your
- 63:56
- of your own
- 63:57
- right? Combine
- 63:58
- Wayfinder with potato mode and make your own custom mode
- 64:01
- right? Like, like skills are, the
- 64:03
- the thing I love about skills is that
- 64:05
- is, are that they're so malleable
- 64:07
- You can do anything you want
- 64:08
- It's just language
- 64:10
- Absolutely. There's nothing magical in them
- 64:11
- right? They're just words. Yes
- 64:13
- exactly. If there, if there is any magic in them
- 64:15
- it's just the
- 64:16
- words chosen and the phrases used and
- 64:20
- The thinking that's been done to turn those
- 64:23
- like
- 64:24
- take abstract process and turn them into language
- 64:27
- And once that thinking has been done
- 64:29
- then it's just there. It's available
- 64:31
- It's on the surface
- 64:31
- and you just nick it
- 64:34
- Lauren, thank you so much
- 64:35
- This has been glorious
- 64:37
- Yeah, this has been super fun
- 64:38
- I really, really enjoyed talking to you
- 64:40
- Hopefully, we can do it again
- 64:42
- I'd love to do it again
- 64:44
- I'd love to do it again
- 64:44
- absolutely
- 64:46
- yeah, we'll check in in
- 64:47
- I don't know, three, yeah, Monday
- 64:50
- Let's do it. Let's do it
- 64:53
- Bye
- 64:53
- Well
- 64:53
- thanks so much. I'm going to close the
- 64:54
- stream here
- 64:55
- Lauren and I will
- 64:56
- chat a little bit and stay here
- 64:58
- but thank you guys so much for watching
- 64:59
- It was glorious
Watch these videos on your TV
XTTV brings video content from X to your living room. Download now and start watching on the big screen.