← Back to videos

Lauren Tan, the engineer at SpaceXAI, put 2,500 PRs into production last month and broke down the se

Chrome

Chrome

@0xchromium

November 18, 2026· 65:00

Lauren Tan, the engineer at SpaceXAI, put 2,500 PRs into production last month and broke down the secret that lets her bots merge their own code while she sleeps the timestamps that matter: > 18:53 - agents kept rewriting the same check, so she put it in the skill > 40:08 - 30 bug reports arrive at once and one coordinator splits them up > 45:16 - a bot that logs bad patterns to a doc instead of fixing them > 48:33 - agents work all night, she checks the commits by morning > 59:07 - a skill that carries an old chat's context into a new one the work that survives the night is the work that got written down before the next agent picked it up if your own bots keep losing the thread between steps, the note each one should leave for the next waits in the article below

Transcript

Thank you so much for joining
0:02
Thanks for having me. Yeah
0:03
very excited to be here
0:04
Big fan of yours
0:06
And a huge fan of yours
0:07
I think people have been talking about this like
0:09
it's like the meeting of the skill minds
0:12
Yes. The skill Mount Olympus or something
0:14
Because both of us have very popular skill libraries
0:16
I'm not, as I was saying before we started
0:19
I've not used a ton of yours and like
0:22
I
0:22
want to get all of the juice out
0:24
of your brain so that I can go
0:25
and use it properly and use it better
0:27
And I think
0:28
where I want to start with this is
0:30
you gave a talk
0:31
pretty recently, like
0:33
about 10 days ago and posted it on X
0:35
which went absolutely nuts
0:37
as about how I shipped 2,500 PRs last
0:39
month to production
0:41
Got about
0:41
3 million views or something on X
0:44
And
0:45
I
0:45
watched it and I loved it and I
0:46
recommended it
0:48
And
0:48
I kind of want to run this as
0:49
almost like a Q&A of that talk
0:51
basically, of giving you Cause I just
0:54
I just
0:55
had tons of questions about it and I
0:56
wanted to dive into it
0:58
And I think where I want to start is
1:00
you talk about a trust ladder with agents
1:04
where you, as you trust agents more
1:06
you can get them to do
1:08
better and better things
1:10
and or scale them
1:11
up to use more and more agents
1:13
So
1:14
what is your story of how you climbed
1:16
the trust ladder and
1:18
how did that work when like you got
1:20
SpaceX and started climbing more and more?
1:23
So I think this
1:24
the, the journey sort of began even before I joined Cursor
1:28
which is now SpaceX AI
1:31
so the story is
1:34
after Meta, so I
1:36
I used to work
1:37
at Meta on the React team
1:39
I took a month off
1:40
cause I was feeling kind of burnt out
1:42
And of course, when, what
1:44
what do you do when you're burnt out?
1:45
You go and start a new side project
1:48
and so I started a side project
1:49
you know, I was, of course using AI to
1:52
to write code
1:54
but then I started to realize
1:56
you know
1:56
I was spending like so many hours just micromanaging one agent
2:00
right? And, you know, at the time
2:02
this was back in February
2:04
maybe February, very early February or January
2:08
you know
2:08
people were really obsessed with this idea of
2:09
like orchestration. This was like
2:11
you know, before, you know, things like cursor, you know, like the agents window was
2:15
had become popular
2:17
So people were still in like
2:18
like 2E land, you know, in their terminal
2:21
and they were all talking about
2:22
oh, here, you know, I built a custom orchestrator
2:24
right?
2:25
And so, of course, I had
2:26
I was a bit nerd sniped by that
2:28
And, you know, as I was building my toy project
2:32
I got nerd sniped by
2:33
oh
2:33
how do I make my AI coding setup
2:35
more efficient?
2:37
And so, you know, I
2:38
I kind of started the journey there where I just
2:42
you know, took a step back and realized
2:44
you know
2:44
I was spending all this time micromanaging a single agent
2:47
you know, I was creating skills
2:49
and I was, like
2:50
finding it quite difficult to measure the output or the
2:53
the results
2:54
the impact of the skill as well
2:57
So I was kind of flying blind
2:58
but I was, you know, iterating really fast, and
3:04
so that project eventually sort of became the basis of PSAC
3:09
even though I didn't know it at the
3:10
time
3:11
and a lot of
3:13
some tricks I had learned
3:14
like building that early set of skills
3:17
actually it's still open source if you wanna
3:19
if anybody wants to take a look
3:20
it's on my GitHub
3:21
like potato slash, noodle
3:25
n o o d l e
3:26
and in there you will see some skills
3:28
and a brain directory
3:30
And so I was really interested in this
3:31
idea of how do I You know
3:34
extract my own ability
3:37
if that makes sense
3:38
and give it to the agent
3:39
right? Because I was
3:40
I, I realized that, you know
3:42
all I was trying to do was trying
3:43
to teach the agent to write code more like me
3:45
you know, do, do, you do workflows more like me
3:49
So, you know
3:49
the skills were like
3:51
an entry point to doing that
3:54
and then, you know, after I joined Cursor
3:57
I was starting to work on the agent's window
3:59
and, it had a lot of performance issues
4:03
it was, it was, it was pretty laggy
4:05
and so since I had experience working on React
4:08
I was asked like
4:09
hey, do you want to come and help out
4:11
with the agent's window?
4:13
and so the, the
4:15
the
4:15
beginning of the Cursor journey was very manual
4:20
I was
4:21
deep
4:21
in like looking at like flame graphs
4:23
and heap snapshots and trying to see like
4:26
why exactly is the app so slow?
4:29
but then coming back to the same realization that
4:31
you know, I was sort of the bottleneck
4:33
I was doing everything manually
4:35
I was sort of the meat proxy
4:36
in a way
4:37
I was the meat proxy between my agent
4:39
and Chrome DevTools
4:41
and I was like really annoyed by that
4:43
And what month of the year is that
4:45
let's say?
4:46
Where are we in the timeline?
4:48
so I joined Cursor in March
4:50
So this was like early
4:52
early April, probably
4:53
Early April is when
4:54
you know
4:56
I joined and I didn't have any skills
4:57
I had, I
4:58
I sort of abandoned my
5:00
personal skills cause I didn't think they'd be
5:01
relevant anymore
5:03
but then working on the agents window
5:06
and now working on Grokbot
5:09
I sort of realized that a lot of
5:10
the lessons I had learned from those skill
5:13
time building the, the initial set of skills were very relevant
5:17
especially around things like verification
5:21
you know, being very rigorous in your work
5:25
because I think agents from my experience
5:28
even the
5:29
the frontier ones tend to
5:32
Tend to take shortcuts
5:34
they tend to do the easy thing
5:38
so
5:39
a lot of the skills that I've built
5:40
have been around
5:42
how do I make the
5:43
easy thing the right thing?
5:44
You know
5:45
how do I make that the best thing?
5:47
The idea of sort of distilling
5:49
your expertise and turning what you do every day into processes
5:55
that's something that feels super familiar to me
5:57
That's exactly what I've been doing with the
5:59
skills
6:00
And I suppose
6:01
there's something in that which is
6:03
a lot of people think domain expertise is
6:05
getting less useful now as people
6:08
start to rely more on AI
6:10
Where, what do you think about that?
6:12
Just as a sort of
6:13
vibe check before we start talking about it
6:16
I actually feel like domain expertise is
6:18
is more important than ever
6:20
You know
6:22
I think I wrote this on my ex at some point
6:24
but You know, at times I sometimes think of
6:28
you know, AI as this, like
6:30
especially as the models get smarter and smarter and more capable
6:33
and the frontier models are just getting so good
6:36
Like, I love Opus 5.5
6:37
by the way
6:40
you know, as the models get really
6:42
really, really good
6:43
it almost becomes like the bottleneck is no longer the agent
6:47
right? It becomes
6:48
your ability to express your intent and your
6:51
goals in a clear way
6:53
that the agent can understand and
6:57
actually carry out
6:58
And that's where I think
6:59
you know, like
6:59
people with a lot of domain expertise are extremely Have a
7:04
have a huge advantage in my opinion
7:06
especially if you're a little bit like
7:07
you know, tech, techno curious, you know, so I, I think of people like
7:11
you know, like, like a doctor or a lawyer or
7:15
you know, someone who
7:16
who has a deep expertise in a particular non-engineering domain
7:21
and if they're actually just a little bit
7:23
tech savvy and they can figure out how to use agents
7:26
they can actually build really
7:27
really great products, right? If they
7:29
if they have a clear enough
7:31
vision in their head
7:33
and they can articulate it in a way
7:34
that the agent can build it
7:36
you know, I think that that is really the
7:40
the bottleneck these days is
7:42
is like the transfer of your intent
7:45
right, and your vision to the agent
7:48
Yeah, I've been obsessed with language basically since agents
7:52
dropped. I just obsessed with thinking constantly about the
7:56
the composition of words
7:58
how I can make things sharper
8:00
what
8:01
what might be hidden in the phrases that
8:03
I'm using
8:04
And it's, and finding
8:06
what I love is when you
8:07
find a word
8:08
that the agent then hooks onto and then goes
8:11
okay, I'm going to reinforce that word
8:13
I'm going to reuse that in my thinking traces
8:15
You know, I found that with
8:17
TDD was an early example of that
8:19
A lot of chat about TDD recently of like
8:21
you know, people say
8:22
should you use TDD with agents?
8:23
It doesn't matter. What you're doing is you're
8:25
getting the agent to think about TDD
8:27
getting it to write tests
8:28
getting it to prioritize things in a different
8:30
way than it did before
8:32
And that's why sort of grilling
8:33
I think, works effectively
8:35
Because grilling is a
8:36
yeah, yeah. It draws those words out of you
8:39
right? Or
8:40
or at least it helps the agent
8:42
Understand your
8:43
thinking so that it can propose those words
8:45
to you and you can pick up and say
8:47
yes, exactly that
8:49
I've actually copied some of the
8:50
the tips that you've shared as well
8:51
where, you know
8:52
one of my favorite ones that you've shared recently or
8:54
or not
8:55
or like maybe in the past couple of weeks is about
8:58
reducing or eliminating tautological tests
9:02
Like one of my pet peeves of agents
9:04
is like all of the useless tests that
9:05
they write
9:06
And so, you know, that was one thing where
9:09
you know, the word tautology, right, is, is
9:11
is I guess, you know, not many people necessarily know that
9:14
if, if, especially if English isn't your first language
9:17
but there's a lot of meaning to that word and it
9:20
it's like It's almost like compressed
9:22
right? Like you compress a lot of
9:24
intent and meaning into words
9:27
And so I, I
9:28
I totally agree with you
9:29
Like I think language
9:30
I've always been interested in language actually
9:33
like programming languages
9:35
natural human languages and how they came to
9:37
be
9:38
And it's so interesting that now with agents
9:40
it's sort of like this meeting of
9:43
natural language with programming language
9:45
but it's all
9:46
it's all language out of the hood
9:47
It's all communication
9:48
Totally. I did a drama degree
9:50
right? So, you know
9:50
I've been
9:51
thinking about language and Shakespeare and stuff for
9:53
a long time. And so this
9:54
all feels very familiar. Yeah
9:57
so okay, there's sort of, before we get into like
10:01
cause I think the thing I want from
10:02
you is like software factory stuff
10:05
right? Software factory is the big buzzword
10:07
Software factory is the thing that I'm thinking
10:09
about too. I'm sort of releasing a course
10:10
in that direction too
10:12
And
10:13
It's this sort of scaling yourself up to
10:17
unrealistic numbers of PRs basically
10:19
or
10:19
PR numbers that sound ridiculous to people who
10:22
don't understand how this works
10:24
So where I want to get to
10:25
is sort of from people who are doing
10:27
kind of like one to five agents a day up to
10:30
you know
10:31
hundreds of agents running at once and how
10:32
that sort of functions
10:34
And so
10:35
I'd love to hear about your metaphor
10:37
of the Michelin kitchen instead of the software factory
10:40
because I think that says a bit about
10:41
the way you think about this stuff
10:43
Yes
10:45
Yeah, I, I, I've
10:46
I've never really liked the term software factory
10:49
not because, you know, it's not accurate, but I think
10:53
I think the, a lot of people when they think factory
10:55
right, they don't necessarily equate that with quality or craft
11:00
right? Things which are very important to me
11:02
and a lot of people and technologists who work
11:06
you know, building products, we care about the user experience
11:09
we care about the things we're building
11:12
So while
11:12
so while I do think software factory is an apt term
11:16
it also, I guess, maybe conjures up negative
11:19
you know, maybe sometimes negative connotations
11:22
So
11:22
Michelin kitchen is the thing that I've sort of landed on
11:25
where it's much more
11:27
I feel like it's much more aspirational and
11:30
I like the metaphor a lot because
11:31
you know, I like food
11:32
I'm called potato, of course
11:34
and I like cooking
11:36
And I see a lot of parallels
11:37
right?
11:38
Like with food, right, when you're cooking a meal for yourself
11:41
for example, it's both utilitarian
11:44
like you're trying to just feed yourself
11:45
right, and, and survive
11:48
but it can actually be transformed into art
11:50
right? And that's what
11:51
what a Michelin
11:53
starred chef or even just a chef or
11:55
a cook can do with food
11:57
is take something
11:58
very ordinary and turn it into a delicious meal that
12:01
you know
12:02
takes you back to your childhood days or
12:04
something like that
12:07
And so it almost like mirrors that trust
12:09
ladder that I talk about
12:10
where
12:12
you can sort of imagine your own journey
12:13
as a home cook
12:15
right?
12:16
as a home cook
12:16
you are doing all of the food
12:18
the cooking yourself
12:19
You cut all the vegetables
12:21
you do all the prep work
12:22
you do all the cleanup
12:24
you know, you are the one man or one woman show
12:28
really
12:31
and it's an interesting thought experiment
12:33
like, okay, if you were to cook a meal
12:36
and then you add people
12:37
right? Your, your, your partner joining to you
12:40
your brother, your sister
12:41
and now suddenly you have your whole family
12:43
in the kitchen. I think most people would
12:45
get very stressed by that
12:46
right? The thought of
12:47
oh
12:47
so many people are just mucking around in
12:49
my kitchen
12:50
They have no
12:50
no idea where all the utensils are
12:52
They're mucking around. Yeah
12:54
exactly
12:56
So I feel like that
12:57
that's really out because
12:59
When you ask yourself that question of how
13:00
do I go from being a solo cook
13:02
right, to having an army or even not even
13:06
not even an army
13:06
but a few sous chefs
13:09
right, that
13:09
that are helping me in the kitchen
13:10
How do I
13:11
think about
13:12
dividing the work in a way that makes sense?
13:15
You know
13:15
I'm not dividing work just for the sake of it
13:18
but in a way that actually makes the sum the total
13:21
the, the better than, you know
13:22
the total of its parts
13:25
And so the Michelin kitchen metaphor to me
13:27
like, works really well in that regard because
13:30
you know, as a chef, you're
13:33
you know, if you become a chef
13:34
you're in a position where
13:36
you're not necessarily cooking all the food yourself
13:38
anymore
13:39
But you are thinking
13:41
you're almost like the tech lead
13:42
right, for the kitchen, where, you know
13:45
chefs have to think about
13:46
you know, not just cooking
13:47
but they have to basically organize the whole kitchen
13:50
and they're like the CEO of the kitchen
13:52
They have to think about when to order ingredients
13:55
how do you store them
13:56
how do you prepare them
13:57
when do they have to be prepared
13:59
You know, it's a whole, it's a whole job
14:00
right, that's not just cooking
14:03
and I think that
14:03
again, it mirrors so much of how engineers write code today
14:08
where you are not writing the code yourself anymore
14:11
you have agents, right? But you, as the human
14:14
are still responsible for the final outcome
14:16
right? Your name still is associated with the
14:19
work that you do
14:20
your reputation, and, you know
14:22
so how you set up your kitchen
14:25
right, and how you set up your skills
14:27
your environment, your code base
14:29
I think are ultimately the new ingredients that go into
14:35
building product
14:37
Yeah
14:38
I think what I love about your approach
14:40
is the amount of focus that you put
14:41
into the environment that the agent operates in
14:44
right? Because I think a lot of people
14:45
they think, right, the agent is good
14:48
I'm
14:49
probably not going to be able to make
14:50
it better
14:51
Let's just trust what these
14:53
magic model people have put
14:54
into the harness and the model combination
14:57
There's nothing I can really do
14:59
Like I can't mess about with Claude Code
15:00
internals or something or whatever you're using
15:04
But what I love about your approach
15:07
and it's something I advocate for too
15:08
is that you can change the environment the agent operates in
15:11
right?
15:12
You can
15:13
make changes in the code base
15:14
and also give it tools for verification
15:17
as well
15:18
and allow it to verify its own work
15:20
So the thing I
15:22
I loved about watching that talk is
15:23
the amount of focus you put in verification
15:26
and like that is the lever that you
15:28
can start to generate trust
15:30
Can you talk about that and what that
15:32
concretely looks like? Let's start like looking at
15:34
practical ways that people can improve their own processes
15:37
their own kitchens
15:39
Yeah, I've, I've, I've said this a lot actually that
15:42
you know, even if you don't use PSAC or
15:45
you know, your skills
15:46
I think that
15:47
the single most important skill
15:49
that should be in your toolkit is verification
15:53
Because without verification, and for, for
15:56
by the way, for those watching, don't know what that means
15:59
it's this idea that you can give
16:01
you can sort of give your agent hands and eyes
16:05
in a way, that's the, the analogy I like
16:08
where the agent is able to run the code
16:11
right, and actually
16:13
interact with it like a normal human user would
16:17
and also do things like
16:18
you know, debug it, you know, take traces and snapshots
16:23
and, funnily enough, like
16:25
that was actually the first skill I built
16:26
when I joined Cursor
16:28
that gave me a lot of
16:29
that was
16:29
that was the thing that actually started to
16:31
let me
16:32
ascend the trust ladder a little bit in
16:34
a way that
16:34
some of the other skills I had looked
16:36
at
16:37
or built had not really let me do
16:40
Because no matter how good
16:42
you know, some of the other skills were
16:43
like the how skill
16:45
the why skill, the unslop skill were
16:48
I was still relying on me
16:50
right, as the proxy between my agent and the output
16:54
so that, you know
16:55
if the agent can actually see the result of its work
16:57
there's no way it can actually iterate
17:00
Right
17:00
and so this is where people start to talk about loops
17:02
this idea of a loop
17:04
And really, I think the term loop
17:06
you know, seems kind of, almost abstract
17:10
Like people are like
17:11
what, what, what is a loop?
17:12
What is an agent loop?
17:13
But really, to me, like
17:15
the most important part of a loop
17:16
that allows it to be a loop is the verification part
17:20
because the agent is able to
17:23
to verify its own work
17:25
And, you know
17:26
that takes you out of the equation
17:28
where now I can actually do something like
17:31
So the very
17:31
one of the very first use cases I
17:32
had for verification was
17:34
you know
17:34
like the performance work that I was doing
17:36
on Cursor's agent window
17:38
And I wanted
17:39
I wanted to get to a point where
17:40
I could do something called hill climbing
17:42
which is a term that I
17:44
I think the labs
17:45
talk about a lot
17:47
which is this idea that
17:48
you know
17:48
you have some kind of
17:49
rubric or a way to judge or score something
17:53
And now, because you have a loop
17:55
you can have an agent
17:56
continually try to make improvements to that
18:00
I think Carpathy, Andre Carpathy also famously released
18:04
something called AutoResearch that has a lot of
18:06
these ideas
18:09
but yeah, verification, I would say
18:10
is probably the most important skill in PSAC
18:15
and many other, you know, tool sets
18:18
and I think it's
18:20
the most important thing to focus on
18:22
So a lot of the
18:24
a lot of, I spent a lot of time actually
18:26
you know, tuning the verification, the creative verification skill
18:31
and also internally the
18:33
the, we have so many verification skills now
18:36
like every app that Cursor has or SpaceX AI has
18:39
has a, verification skill that is auto-maintained as well
18:45
and it's become critical infrastructure for our team
18:47
because everybody uses it
18:49
And you went pretty far with that too
18:51
right? Like you had a
18:53
in your talk
18:53
I saw that you actually built a custom
18:55
CLI for that too
18:56
So what does that CLI do?
18:57
Like how does it execute things and why
18:59
did you I mean
19:00
that's proof of how deep you're going
19:02
right? Of how much you're pushing that
19:04
Yeah
19:05
so this is actually a tip I learned early on where
19:12
I guess, you know, back in January or
19:14
or, or late last year
19:16
the thing that people were concerned about was context window
19:19
right? That was the big
19:20
the big topic at the time was
19:22
how do I, you know, manage the context window?
19:24
Because, you know, compaction
19:26
summarization wasn't really that good yet
19:29
And people were always
19:30
people had this
19:31
there was almost this meme in the community that
19:34
you know, once your agent summarized or compacted once
19:38
it would become sort of stupid
19:40
right? For the rest of your session
19:42
So there was a lot of thinking around
19:44
like, you know
19:46
being very efficient with your context usage
19:48
And so
19:49
that was actually the inspiration for
19:51
some of the
19:53
the CLI work inside of the verification skills
19:57
I guess now it's less so about context because
20:01
you know
20:01
agents are
20:03
much better or harnesses have gotten a lot
20:05
better with summarization
20:09
I still think there's some benefits to
20:10
you know, having a clean context window
20:15
so the CLI is really just more of
20:16
a way for me to
20:18
take the
20:19
deterministic parts of what the skill does
20:22
and encode that into a script or a CLI to reduce
20:25
to kind of take away the judgment
20:29
that would otherwise unnecessarily be used
20:32
Because with judgment, so in, I also think of
20:35
you know, agents and skills in sort of like
20:38
it's like a gradient
20:40
You have some parts of the work
20:57
Refactor some code, right, from one pattern to another
21:00
that's very mechanical, right? You don't
21:02
you don't need an agent to
21:04
think about it and come up with it
21:06
in a novel way each time
21:09
right? And so that
21:09
that was really the inspiration for the CLI
21:12
and you'll see this in a lot of
21:13
the other skills that I've built
21:14
is like I try to extract
21:16
out the deterministic parts and turn that into code
21:20
and just leave only the parts that actually
21:22
require judgment to the agent
21:25
So in a way
21:25
I think of the skill as kind of like a wrapper
21:28
right? It's a wrapper with some light instructions
21:30
on how to use these custom tools that
21:32
are inside of the skill
21:35
Oh yeah
21:36
I don't think the CLI is really that
21:37
interesting in its own
21:38
really. It's not like a novel piece of
21:40
software
21:41
It's just something that interacts with
21:42
like
21:43
Playwright and the Chrome DevTools protocol and
21:46
calls a bunch of APIs
21:47
It's like, it's just a bunch of glue
21:49
No, it's fascinating
21:50
because it's a way of hiding information from the skill
21:52
right? It's a way of conserving the skill
21:54
keeping the skill quite small
21:56
I imagine
21:56
and then you're able to delegate more of
21:58
the complicated deterministic stuff
22:00
into a script within the skill
22:02
So it's almost
22:03
you're compressing
22:04
information and making the agent do
22:07
more consistent things more consistently
22:10
Yeah. It's fascinating
22:12
Yeah, and it also helps
22:14
I guess, if you care about context window
22:16
it does help, because now the agent doesn't need to
22:18
you know
22:20
reinvent things. Because
22:23
one thing I had noticed
22:24
early on when we didn't have a CLI
22:27
was that while the
22:29
the agent would try to verify
22:31
it would basically rebuild the world each time
22:34
and then every agent did it differently
22:36
And I was starting to notice
22:37
like, that's very inefficient, right?
22:38
I was wasting, it
22:40
it was actually not just about context usage
22:42
but also speed, right? Like
22:43
cause now an agent had to actually go
22:44
off and write the scripts or the CLI and test it
22:47
and, you know, and it doesn't work
22:49
And the last agent did it
22:51
and it worked, but it discarded it
22:52
So it was just very obvious at that point
22:54
like
22:55
I should just turn this into a CLI
22:56
and put that instead of the skill
22:59
so that every agent that uses it now
23:02
benefits from that same piece
23:05
but I
23:06
I also think, like, you know
23:07
it's a good push
23:08
for people to think about
23:10
is
23:10
how much of your skills and rules could
23:12
actually be deterministic
23:16
that's like another core thing or
23:18
or
23:18
one of my core principles that I like
23:20
to think about is
23:22
yeah
23:22
how do I
23:24
make very efficient use of determinism and nondeterminism and
23:30
you know, let agents shine at the nondeterministic parts
23:33
right, because that's what they're trained to do
23:36
and the other parts which are much more mechanical or
23:40
you know, straightforward can be just pure determinism
23:44
and you'll see this as well for things like doing migrations
23:48
which is another big thing that I've
23:50
I've talked about is
23:52
you know, going from one technology to another
23:56
especially one that is better for agents
23:58
right? And a
23:59
a lot of how you can do that migration is
24:01
I think, through things like scripts and CLIs
24:04
like the deterministic parts
24:05
like code mods, you know, like crawling the AST
24:09
the abstract syntax tree
24:10
and, and transforming code literally mechanically
24:13
right? Like a script does it for you
24:14
instead of the agent
24:17
Totally makes sense
24:18
I, I mean, I think what, there's another thing there
24:22
which is
24:24
you're taking stuff away from the agent
24:26
and you're
24:27
kind of putting it in the environment too a little bit
24:30
which is, let's say you have a
24:32
a thing that you notice the agent always gets wrong
24:35
you want to make that
24:37
just impossible within the environment
24:40
And that sort of comes down to
24:42
code quality as well. I mean
24:45
I talk about a lot like having a
24:47
what a good code base means
24:49
right? What is a good code base?
24:51
And there's a definition I like
24:52
which is
24:53
a good code base is a code base
24:54
that's easy to make changes in
24:56
right?
24:57
Easy to change stuff without things screwing up
25:01
And that means that you have a lot of guardrails
25:03
that you have a lot of
25:05
the
25:06
agent or the human is constrained to very
25:08
narrow paths
25:09
And that's, again
25:09
something you talk about in your talk
25:12
And
25:12
you talk about this not only
25:13
on the kind of Sort of automated checks side of things
25:17
so linting and time checking
25:18
blah, blah, blah
25:19
But also in the way you design abstractions
25:22
and you guys even
25:23
I think
25:23
built a framework
25:24
for your agent to work into
25:27
I think what I'd love to hear is
25:29
you obviously think of that as very important
25:31
right? And that's
25:33
how important is that compared to other things
25:35
you could be doing
25:36
like building features or shipping work?
25:39
Yeah, I think that's a
25:41
I almost feel like the new job of
25:43
the engineer is really to
25:45
to spend time on the environment
25:48
I almost actually wrote a tweet about this yesterday
25:49
but I, but I didn't
25:51
But I think that
25:55
I think that, you know, if you
25:56
if you haven't really spent time
25:58
you know
25:58
building trust in your agents and building skills and tools
26:01
you can get stuck in this mode where
26:03
you're very low on that trust ladder
26:06
right? You don't have a lot of trust
26:07
in your agent's work
26:08
And so The only way to cope in that
26:12
when you're in that situation
26:13
is just to kind of lock in and micromanage your agents
26:17
and that's very time consuming
26:18
And when you're stuck in that mode
26:19
you don't really have the luxury to think about
26:23
you know, higher level things, like, like
26:27
like making yourself more productive
26:30
In the same way that
26:33
I guess analogy would be like
26:34
if you've never taken the time to learn
26:36
like, your tools, or as a developer
26:38
when you were writing code yourself
26:40
and, you know, you, you have
26:42
you've never heard of VS Code
26:43
you've never heard of Vim
26:44
you only knew about Notepad
26:47
and you hadn't, hadn't even heard about Git
26:49
That's sort of the analogy
26:50
It's like, you
26:50
you haven't spent the time sharpening your own knives
26:54
right? And so, of course, if you have a dull knife
26:57
then everything can take a long time
27:00
and you are not like
27:01
but you're, you're just going to be
27:03
and, and especially if, you know, deadlines are looming, then you don't have the
27:07
now you're stuck in this rut
27:08
right? Where, where you, you, you don't have sharp knives
27:12
you don't have good tools
27:13
but you're under all this pressure to ship
27:15
right? And so all you can do is
27:16
just focus on that
27:18
But I do think that
27:20
you know
27:20
if you can find yourself
27:21
the time to actually spend time thinking about your setup
27:24
it's, again, going back to the cooking
27:27
you know, analogy
27:29
it's like, you know, if you, for example, if, if cutting, cutting
27:34
cutting the garlic is like super slow
27:36
right? There are garlic mashers
27:37
right? You can
27:38
you can buy it and you put it
27:39
in the thing and you like squeeze it out
27:41
right? It's super fast
27:43
machines and tools are invented for a reason
27:46
right?
27:47
And so if you're operating a Michelin kitchen and your
27:50
your, your cooks had no tools
27:52
then of course everything is going to be extremely inefficient
27:55
very, very, you know, every, every
27:57
every cook is going to make something up
27:59
of their own
28:00
So I think the tools and the determinism to me are
28:04
you know, taking that part away
28:06
And, and just like you said about constraints as well
28:09
it's, the constraints are, are to me as well like
28:13
actually a slight tangent on that is
28:17
I think we should talk about TypeScript
28:19
Cause like we
28:20
we actually both share like a background in TypeScript where
28:22
you know
28:23
you obviously have done a lot of work
28:24
with TypeScript and total TypeScript and
28:26
you know, you're a leader in that space
28:28
And I
28:29
had adopted
28:30
TypeScript pretty early
28:32
and I had given like a talk or two at TypeScriptConf
28:35
many years ago
28:37
And so one of the
28:38
the talk that I did actually was about
28:39
type systems and constraining the
28:43
constraining types. Like
28:44
one of my most favorite
28:46
things about type theory is actually type narrowing
28:48
right? This idea that you go from a very broad type
28:51
right, that could be anything
28:53
and then you, through type guards and
28:55
you know, type narrowing and, you know, runtime checks
28:59
you can actually narrow the space and say like
29:01
oh, this isn't just a string
29:03
this is a very special type of string
29:05
It's a constant, right? Like, I, I
29:07
I determine that through the type system
29:10
And in a way
29:10
it's like, there's a lot of parallels
29:12
I think, to that with constraints in your code base
29:16
where is it you're
29:17
you're constraining the space
29:20
right? If you, if you think about category theory as well
29:23
you know, you're constraining the, the number of possible types
29:27
right, that can, can exist
29:30
And you're saying there's only one type
29:32
right? And for, for us
29:34
like that framework that I'm building called Dune
29:38
it's not an open source framework
29:39
It's the, the way I describe it to people
29:42
it's
29:42
it's kind of like
29:43
our internal next.js for our electron apps
29:47
but it comes with a lot of really
29:49
really restrictive lint rules
29:51
and the code base is designed in a
29:53
way that
29:54
there's really only one way to do something
29:56
So we make use of
29:57
we make use of a lot of conventional
30:00
patterns
30:01
So like features all go into a specific directory
30:05
Well, every feature has its own directory
30:07
As an example
30:08
you know, there's like a
30:09
a thing that discovers features like through a
30:11
registry
30:12
and like crawling the code base and stuff
30:14
like that
30:15
But this
30:17
conventional pattern
30:18
and the lint rules
30:20
make for an environment where it's actually very
30:23
hard to write bad code
30:26
And that sort of frees up the It
30:29
it both frees up your own mental
30:32
you know
30:34
capacity as well as the agent
30:36
sort of doesn't have to think about that anymore
30:39
where it's just like
30:40
oh, there's only, there's, I should just
30:41
if I want to add a new feature
30:42
it just goes in the feature
30:44
the new feature directory
30:45
and all the code goes in there
30:46
and I'm not going to append to a God file
30:50
right? That was really actually the inspiration for
30:52
those feature directories is
30:54
the very first couple of versions of Grokbot
30:57
were composed of like eight God files
31:01
which were like at least 10,000 lines long
31:03
if not longer, and so I
31:05
I kind of had to break it up
31:06
into smaller pieces
31:09
but it was just observing
31:10
you know
31:11
I think that's another important part is observing how agents fail
31:15
and then every time you see a mistake
31:17
every time you see something that could be done better
31:20
you think, you step back and think
31:22
how do I turn this into a lint
31:23
rule? How do I make it so that
31:25
the code base makes this impossible
31:27
right?
31:28
And it comes back to me for my
31:29
you know, my background learning TypeScript and types
31:33
type systems is how do I constrain the space so that
31:39
you know, I know precisely what I'm working with
31:41
and I think, yeah
31:41
there's a lot of parallels there
31:43
Totally makes sense
31:44
And don't, I mean
31:46
it's funny that you mentioned TypeScript and God
31:48
files in the same sentence
31:49
because TypeScript famously has a 25,000 line type
31:52
Yeah, I realize that already
31:53
Although I
31:55
don't know if they've rewritten that in Go
31:57
They probably have, right?
32:00
Okay, so the environment is important
32:03
You should watch your agent like a hawk
32:04
to make sure that any mistakes it makes
32:07
you turn them into things in the environment
32:10
And the benefit of the environment is you're
32:11
not overloading your agent
32:13
right? In terms of rules
32:14
in terms of things it has to remember
32:16
it's just in the environment
32:17
And so it stumbles into the rules and
32:21
you know
32:21
bounces off them and hits them at the right moment
32:24
So, okay
32:25
we still haven't talked about the 2,500 PRs
32:27
Where do those come from?
32:28
Like, how do you, you've built your trust ladder
32:31
you've worked on your environment
32:33
and you understand, okay
32:35
I now want to scale up
32:37
So what are the mechanics of that scaling?
32:40
Are you, initiating 2,500, like, chats per month?
32:46
That can't be right. So there must be
32:48
are there any kind of automated triggers
32:50
that trigger stuff in your repo?
32:53
Like
32:53
how do you get the software factory kind
32:55
of triggering work by itself?
32:57
Right
33:00
I'll definitely say that the prerequisite to
33:03
you know, something like a, a very high volume of
33:06
of pull requests, is the environment, you know, the, the stuff we just talked about
33:12
where I definitely would not have been able
33:14
to do this if I had not spent the time
33:17
you know, thinking about the kitchen
33:18
right
33:18
and the knives and the tools for my agents
33:21
And so, in a way, I
33:23
I think of this as
33:25
I've spent a time building one kitchen
33:27
and one restaurant
33:29
And now I
33:30
am in a position where
33:31
I don't actually have to be there anymore because the environment
33:36
you know, the same analogy, right?
33:38
It works really well
33:39
Yeah, I know, that's not
33:42
Yeah, exactly
33:42
Yeah, exactly
33:43
It's like you're Gordon Ramsay and
33:45
you know, you've taught your executive chef
33:48
like, all the tricks of coming up with great menu
33:52
and, like, the kitchen is set up really well
33:54
everything's just perfect
33:56
and you're now in a position where you
33:57
can open your second
33:58
your third restaurant
34:00
And
34:01
I guess I sort of see each
34:03
project that I work on
34:04
like, each big chat is sort of like a restaurant
34:07
right? And
34:09
I have multiple of them operating at the same time
34:12
and I'm sort of
34:13
like, helicoptering between them, sometimes some more than others
34:18
depending on how in the loop I am
34:20
But yeah
34:20
definitely I think
34:22
there are external triggers and context that those projects don't have
34:29
that for a long time
34:32
I was the proxy for that
34:33
So, the best example I have is like
34:35
you know, you have a project that's working on a feature
34:38
or you're trying to fix a bug
34:40
and you're getting bug reports
34:42
but the bug reports are going to things
34:43
like Slack or Linear or X
34:47
right?
34:48
And these are
34:49
external systems that aren't connected to your inner loop
34:53
So I
34:53
I like to talk about this
34:54
outer loop and the inner loop
34:57
I don't know if I'm using the definition correctly
34:59
but to me, my inner loop is like basically my engineers
35:03
my agent engineers working on the code to an
35:07
building towards an intent
35:10
Or snapshot of my intent
35:12
right?
35:12
And
35:13
the thing about that is that the snapshot can go stale
35:16
right? New information comes to light
35:18
that I then have to be the proxy of and
35:21
you know, transfer that context to my agent
35:23
So, you know, if you
35:24
if you don't have
35:25
these triggers pulling information back into the inner loop
35:29
then you sort of have to play that role where you're
35:33
you're off, you know, in Slack or X or
35:36
or whatever, and you're gathering context
35:38
right?
35:39
You're getting context about bug reports
35:41
about feature requests, about, you know, something someone said about
35:45
you know, our backend infrastructure has some limitation
35:48
you know, all that information
35:50
you have to ferry that across
35:52
to your agent
35:53
So that's where I think like
35:55
tools like Grokbot
35:56
are really good because they help you
35:59
automate the outer loop as well
36:01
And when you connect those two loops
36:03
it's very
36:04
very powerful because now all of a sudden
36:06
your agents have the ability to get context for their
36:10
for themselves, right?
36:11
If for example, you know, either through
36:15
just as a simple example
36:17
like maybe you have the Slack MCP
36:19
right? Or you have
36:21
your own harness
36:22
right? That you've built
36:23
a Slack subscription into for a particular Slack
36:26
channel
36:27
Now all of a sudden you can tell your agents
36:29
okay, subscribe to the Slack channel
36:32
Every time there's a
36:34
you know, bug report about something
36:37
go off and triage that thing
36:39
right? Go reproduce the issue
36:41
right? Using the verification skills that we've already
36:43
spent time building
36:45
and all of those other skills that we've
36:46
set up
36:47
So that I have a lot of trust
36:49
right? I have a lot of trust that
36:50
these agents can actually go off and understand the bug
36:54
you know, verify that the bug actually still exists on main
36:58
and it wasn't something about
37:01
you know, maybe the user's setup or their data
37:04
or maybe, I don't know
37:06
they didn't install a dependency or something like that
37:08
Like, basically, I think
37:12
creating that, yeah
37:14
creating those two loops and connecting them
37:16
is really a very important part of the job these days
37:20
especially if you are thinking about how to scale yourself
37:24
So, a big theme here is really just
37:26
like, always thinking about, like, what
37:29
where am I the bottleneck in this process?
37:32
Why do my agents need me
37:33
you know, to answer this question?
37:35
I, I always like to think about that
37:37
and so
37:37
I try to think about how do I
37:38
actually get the agent to answer its own question
37:41
right? But not by hallucinating
37:43
not by guessing, but actually real data
37:46
And, you know
37:47
a lot of people talk about this idea
37:48
of a company brain
37:49
right, or a context graph
37:52
I feel like those terms aren't necessarily complex
37:57
or even abstract
37:58
To me, it's just about
38:00
how do I
38:01
take information that my agent needs that
38:04
I would otherwise have to go and
38:06
pass it myself and just teach it how to do it
38:09
right? And that removes me from the equation
38:12
And so how I arrive at 2000
38:14
or however many PRs
38:16
is the fact that I have all these loops set up
38:20
right? And so, it allows me to open chain restaurants
38:25
right? I can, I can really parallelize myself
38:28
So yeah, I'm not sitting there creating 2500 chats
38:31
right, of course
38:32
It's really like these projects are
38:35
actually Cursor has a new feature called projects
38:37
which are these like coordinator agents
38:42
and so the coordinator
38:43
coordinator agents are really good at
38:45
Sort of delegating and not doing work of their own
38:48
but they manage and supervise like almost a
38:50
list of tasks
38:51
and they sponsor agents to go and do
38:53
them
38:54
And so I'm just constantly
38:55
feeding context or teaching the agents how to
38:57
get their own context and then they're going
38:59
off and doing the work for me
39:02
and really the big
39:03
the last thing I'll say to this is
39:04
like the big unlock
39:05
for me for getting to 2000 PRs
39:08
is
39:09
starting from the question
39:10
and working backwards of
39:12
how do I
39:13
get to the point where my agent can
39:15
merge its own code?
39:16
Because the obvious thing people ask me when
39:20
when they, when I tell them
39:21
oh, I shipped 2000 and 2500 pull requests last month
39:24
they'll be like, how did you review that?
39:27
Right? That
39:27
that's just a lot of PRs to review
39:29
Like your team must hate you
39:30
Do you
39:31
do you mind if we go there in a second?
39:32
Yes, absolutely. Good question about that
39:34
Yeah, yeah, yeah. I want to
39:36
like, this analogy is great
39:37
I want to like deepen it a bit
39:38
which is Before, if you're like manually initiating all those chats
39:43
it's like you're bringing the orders to your chefs manually
39:46
right?
39:47
Whereas if you've got an agent
39:49
sort of like doing the expo
39:50
then you're able to sort of run it yourself
39:53
What does
39:54
what does that concretely look like then?
39:56
You've got these sort of Grok bots that
39:57
are subscribing to channels
39:59
pulling in Slack messages
40:00
and you, it sounds like
40:02
have a couple of coordinator agents or like
40:04
chief of staff agents that
40:07
like monitor that or something
40:08
Like when you look at your computer to manage your agents
40:12
what does it look like?
40:15
Yeah, so, so this is, I guess, somewhat confusing, but we're working on
40:19
you know, simplifying and unifying
40:21
But so there's Grokbot
40:24
which, or, you know, you can use other tools
40:26
of course, as well
40:27
But I
40:28
I largely think of these tools as like
40:29
your outer loop. These are tools like
40:32
you know, Grokbot that have connectors
40:34
These are connectors, I guess they
40:36
a lot of people call them personal agents
40:39
but they are connectors to things like your email
40:41
your calendar, Slack, Plaid, I don't know
40:46
like all these different services
40:48
And they are a great source of pulling
40:49
context in to your work
40:53
So the same way that a human
40:55
like, you know, if I were
40:56
if I was a manager
40:58
and I was leading a team of engineers
41:02
you know, like when I used to work in Netflix
41:04
one of the biggest things that managers would
41:06
talk about was this idea of context
41:08
not control, which funnily enough, you know, has so much
41:13
has so much, carry over to the agent's world
41:17
of, you know, you, you know, you
41:19
you of course can drive to an outcome
41:21
you want by control
41:22
right? Like by micromanaging
41:24
but what you want is to provide context instead
41:27
right? Like teach the agent
41:28
teach your engineers how to be self-sufficient
41:31
and then you don't have to micromanage them
41:35
and so I see a lot of parallels there
41:37
but yeah, GrokBot, so concretely
41:39
I have some GrokBots that look at my Slack channels
41:43
look at my X
41:45
or my emails, or, or linear, and they're just constantly
41:49
they have routines that subscribe
41:51
so they're constantly watching
41:53
and I have, I'll tell them things like
41:55
you know, I'll watch for issues with
41:58
bugs in the GraphBot. One of the really
42:07
cool things about GraphBot is it connects to
42:09
Cursor
42:10
So Cursor has
42:12
like I, I just mentioned, this new feature called Projects
42:15
and a Project is really a
42:18
again, like a
42:18
you get a coordinator agent that's in the cloud
42:21
it has its own computer
42:23
and all it really does is
42:24
like, it's a manager of agents
42:26
It's like your executive chef
42:27
right, your, your chief of staff
42:29
It doesn't do the work itself
42:31
It delegates, right
42:33
and orchestrates and manages the work of other sub-agents to
42:37
you know, that report to your chief
42:41
your chief, of staff
42:43
And it
42:44
basically is responsible for driving the work forward
42:47
and managing things and
42:50
passing context to them
42:52
So if you get a sudden burst of issues
42:53
let's say you get 30 issues at once
42:55
in one payload or something
42:56
or very quickly
42:57
the coordinator agent can figure it out and delegate
42:59
Yeah, exactly. It gets like
43:01
you know, 30, the
43:02
the 30 or so payloads and
43:03
spawns a sub-agent or a single coordinator agent
43:06
It can actually do a
43:07
a bunch of different topologies of agents
43:10
and it will sort of figure out the best way to
43:12
you know
43:13
efficiently distribute the tasks to your
43:17
team of agents
43:21
so I use pro
43:22
cursor projects a lot
43:24
and I also use GraphBot a lot
43:27
and
43:27
cursor projects are my inner loop and GraphBot
43:29
is my outer loop
43:31
GraphBot takes all the context
43:33
external context
43:35
Gives it to the projects
43:36
because it can actually just send messages to those projects
43:39
right? You don't even have to open cursor
43:41
You can just tell your Grok
43:42
okay, create a project, right
43:43
for these series of tasks
43:46
They're all related, right? Maybe as an example
43:49
you know, you've had a
43:52
a big burst of issues that are all about performance
43:54
right? Your app is slow
43:57
and they're all connected
43:59
right? Maybe some of them even have a similar fix
44:02
right? But
44:03
and you can certainly go off and just
44:04
spawn one agent per task
44:07
but then you've lost that sort of thread between them
44:10
right? And, and you may duplicate work
44:12
or you may not
44:12
really think about the higher level problem
44:14
You know, sometimes when you, you solve bugs, you know
44:17
it helps to have multiple bug reports that are
44:20
are slightly different, because it helps you really
44:22
you know, zoom out and see
44:23
actually, you know, the problem
44:25
when I looked at this one report
44:26
I thought the bug was here
44:27
but actually when, when I see the other multitude of bugs
44:30
it's actually up here
44:32
Yeah
44:33
got you. So
44:33
that's why you have so many agents in that loop then
44:36
right? Because it's not just you have
44:38
like you have a bug report comes in
44:40
you spawn a single agent to look at
44:41
that bug report. That
44:43
that single agent will be duplicating work with other
44:46
other agents
44:47
right? Because if there are multiple bug reports
44:49
coming in through the same thing
44:50
that can be duplicated work
44:51
Fascinating
44:53
That's really fascinating
44:55
Okay, and so this, just this endless series of triggers
44:58
coming from real users reporting real reports
45:02
builds up this sort of
45:04
and accelerates the factory
45:05
sort of adds more orders in
45:08
Other than bug reports
45:09
are there any other sources that you use for
45:11
like, accelerating for pushing these PRs?
45:17
well, funnily enough, it's, some of it comes from
45:20
reading the code too
45:22
So I guess I have sort of
45:25
well, I, so To clarify that
45:27
you know, the 2,500 PRs, they're not obviously like 2,500 features
45:32
right? They are
45:33
a lot of the work actually is spent on gardening
45:37
like another term that I really love
45:41
where
45:42
so
45:43
I guess this is more important
45:44
when you have a big team of engineers
45:46
human engineers that you work with
45:48
where
45:50
and it also just goes back a little
45:51
bit to what I was talking about with the environment
45:53
you know
45:53
setting up a really good environment doesn't just
45:55
help you and your agents
45:56
but everybody on your team
45:57
right?
45:58
Think of a new hire who doesn't have
45:59
a lot of context
46:00
on all of your engineering practices joining your
46:03
your team
46:04
And if you have a really good environment
46:06
they can be productive from day one
46:08
right? They can, they don't have to like
46:09
you know, make, open a bunch of low quality PRs
46:12
they can start, you know
46:14
they can start just churning out really good code
46:19
and, yeah, I think I, I
46:22
I sort of lost my train of thought
46:23
I've got a, I've got a follow-up
46:25
which is what, what are the mechanics of like triggering
46:29
like how, when, when do you trigger a
46:31
a, a, a agent to go and look at the code
46:34
right? Because some people might say
46:36
well, let's just do that every hour or something
46:39
or like on a cron job
46:40
or what's. Oh, yeah, yeah, yeah
46:43
I
46:43
yeah. I saw some of your recent tweets
46:45
as well about like
46:45
you know, the, some of the tweets you've been doing
46:47
which are great for setting up these routines
46:52
I have some routines like that as well
46:55
so, one of them is
47:00
like looking through, just another simple example is
47:05
you know, React has a lot of foot guns
47:08
so, as, as I'm sure you're aware
47:10
And so I have an agent that's just
47:12
constantly looking for bad patterns
47:14
And
47:15
The interesting thing about that one is that
47:17
I don't actually tell it to fix the
47:18
issue first. I tell it to append it
47:20
to a document
47:22
And then every
47:23
couple of days I look at it and I say
47:24
actually, these are all the same thing
47:26
you know?
47:27
And so that gives me
47:28
you know, you almost want like a buffer
47:30
a queue
47:31
Sometimes that's actually more effective than
47:34
just spawning off a couple of
47:35
like
47:35
a lot of sub-agents to fix every single
47:37
thing
47:38
Because when you are in
47:40
kind of pure execution mode and just trying to like
47:44
you know
47:45
execute on the orders that are coming in very fast
47:48
you sometimes miss the big picture
47:50
So sometimes having a buffer
47:52
forces you to think about the big picture because you
47:54
you
47:55
you have these
47:56
Artifacts and things that you can look at as a human
48:00
and sort of use your own human judgment to
48:03
or
48:04
I guess you can use an agent to
48:05
do that as well
48:06
but you give the agent
48:08
and yourself a way to identify patterns
48:11
right
48:11
that you might otherwise miss if you're just
48:14
only solving each bug at a time
48:17
And that's also really the benefit of having
48:19
something like a chief of staff agent is
48:22
it can see the forest
48:24
right, in addition to actually doing the execution
48:29
Fascinating
48:30
so I mean
48:31
my brain is exploding a bit there with
48:33
the sort of chief of staff
48:34
of the software factory. I might have to
48:36
change some of the course that I'm filming
48:38
next week
48:41
All right, let's talk about, let's talk about review
48:42
right? Because this is the reply that you get
48:45
you know, is, did you read
48:47
did you taste all
48:48
2,500 of those dishes as they swept past
48:51
you?
48:52
And I assume the answer is a variety
48:55
is a version of no
48:56
Yeah, I, I think you, you
49:00
you
49:01
don't want to be in a position where
49:02
you're not tasting your food ever again
49:05
but you also, you know, for scale
49:07
you cannot be tasting every single
49:08
dish that comes out of your kitchen
49:10
especially if you have multiple restaurants
49:12
So it becomes more about Sampling
49:15
right, and thinking about the processes
49:17
In the same way that
49:18
you know, if I guess maybe this is where the
49:21
the, the factory analogy is a bit more apt
49:24
is, you know, as a quality supervisor on a factory
49:27
you, you can't look at every single item
49:30
You sample, right? You take a
49:32
you, you
49:32
you go in there
49:34
every day and you look at the quality
49:35
of the pull requests
49:36
you look at the code that the agents are writing
49:39
and you
49:40
scrutinize it very rigorously and you Think about all the inefficiencies
49:46
the bad patterns that the agent are doing
49:48
and then you think about how to course correct the environment
49:52
right? Not, not that single agent
49:55
because if, maybe if it, if it
49:57
if it was a one-off incident
49:59
it's fine, you know
50:00
that maybe there's nothing to fix there
50:03
But if you actually notice that multiple agents are
50:06
are, are having the same issue
50:07
right? They're taking the same shortcut
50:09
they're, they're propagating the same workaround everywhere
50:13
that's a sign
50:14
that you should go off and think about how to
50:17
amend your kitchen or your factory
50:20
right? Like thinking about your skills
50:21
your constraints, your lint, your type systems, and setting
50:27
or
50:28
adjusting it so that that
50:29
problem doesn't happen again
50:31
And when you do that enough times
50:33
then you get to a place where the code basis
50:35
again, like the environment is so constrained
50:38
and so
50:39
it guides you so well
50:41
that you can just You can just step away
50:44
right? That's the dream. And I'll
50:45
I'll definitely say it
50:47
it's very hard to get to this point
50:50
I don't want to sell this as like
50:51
you know
50:52
something that you can just do easily by using PSAC
50:55
Like
50:55
it takes a lot of time and effort
50:57
to think about your code
50:59
and where you see your agents failing and thinking very thoughtfully
51:04
intentionally
51:05
and
51:05
setting up guardrails and constraints
51:08
so that they
51:09
do the right thing by default
51:12
And you're not, like, if
51:15
to get back to the software factory analogy
51:17
this isn't a dark factory
51:18
right? This, the lights are on
51:20
It kind of is
51:21
yeah, actually
51:22
Is it?
51:22
Yeah, well, it's dark in the sense that
51:24
so, it's dark in the sense that
51:27
well
51:27
I think if my agents are merging their own pull requests
51:30
it's sort of become dark where I go to sleep
51:34
my agents now work 24/7
51:37
I have
51:38
I have the equivalent of like more than
51:39
ten chief of staff
51:41
right, each working on a different area
51:43
Like, for example
51:44
I have one that's working on performance of
51:46
the GraphBot desktop app
51:48
I have one that's working on
51:50
fixing bugs that users report
51:52
I have one that's exploring
51:55
rewriting it in a different language just for fun
51:57
You know, like what if, what if, you know, just reimagining what
52:00
what it would be if it was like
52:01
a native app
52:02
It's just a toy
52:05
but the idea is like
52:05
yeah, I, I
52:07
when you spend the time setting up your environment
52:10
I've gotten to a point where I review
52:11
the pull request after it's landed
52:14
I, I, I'll tell my agents full autopilot is
52:17
is something that you can do in
52:18
in PSAC
52:20
and that will
52:21
trigger off this very intense rigorous verification loop
52:24
where it'll spawn a bunch
52:26
of verifier agents for every pull request
52:28
and it will fuzz
52:29
right?
52:30
Fuzzing, meaning that it will actually run the application
52:33
it's going to click around and try to
52:35
use it like a real human
52:36
look for regressions, look for bugs in your implementation
52:40
and, it will try to find issues with the thing
52:44
and then it will fix it itself
52:46
it'll do that again
52:47
and eventually get
52:48
the PR to a state where it can
52:49
land
52:52
so it does, it does
52:53
it is quite token intensive
52:55
You can tune this
52:56
of course
52:58
so, you know, instead of like ten verifier agents
53:00
you might do like one
53:02
right? Or
53:03
you can just tell the agent to verify
53:04
it doesn't work
53:05
But yeah
53:06
the key thing is
53:07
the verification part
53:09
is really the key piece
53:10
that gives me a lot of confidence
53:13
that I guess verification plus the environment
53:15
right? It's these
53:16
the combination of these two things that allow
53:18
me to step away and say
53:19
agents, go off and merge your thing
53:20
I'll review it in the morning by looking
53:23
at my commit history
53:25
And if I see problems
53:26
I go in course correct
53:28
right? And I'll go and revert or modify
53:31
add new lint rules and whatever
53:35
so it does
53:35
it does take time to get to that
53:37
point
53:37
But once you get it
53:38
oh, it's so, it feels so magical
53:42
I, I, I tell people like
53:43
I'm sleeping so much better now because
53:46
you know, it took, it
53:47
the very first day I turned on the
53:49
sort of dark factory was very scary because I was like
53:52
what if I cause a sever
53:53
right? What if I break something overnight?
53:56
and it took a lot of
53:58
it took a lot of
54:00
bravery, I think, to do that
54:01
but
54:02
somehow I did it and
54:03
now I'm in a place where my agents are
54:06
are merging their own code while I sleep
54:09
So I think it's dark in that sense
54:11
Yes, it's dark sometimes, right?
54:12
You do sometimes. That's true
54:14
that's true. Because I think of a dark factory as like
54:17
almost like
54:18
If you take the original definition of Karpathy's vibe coding
54:21
right, which is the code almost doesn't exist
54:23
you forget the code might be a thing
54:25
I think your approach is totally different from that
54:28
which is that
54:29
code and the environment is essential
54:32
And if the code and the environment are bad
54:34
then you will get bad outputs
54:35
garbage in, garbage out
54:36
So I, I, I think this is a
54:40
this is a different thing
54:40
It's like, you know
54:43
the, I don't know, maybe there's a dimmer switch or something
54:46
right? Like, you know, some parts of the factory are dark
54:48
some parts of it are light
54:49
This is why the
54:49
maybe the, the kitchen is a better analogy
54:53
the restaurant, because, you know, even as a, as a, as a As a restaurateur
55:00
you
55:00
still might go to a restaurant every now
55:02
and then to take a
55:03
take a peek in
55:04
taste the food, right?
55:05
Yeah. The idea of sampling instead of blocking
55:08
I think is really important
55:09
I think what would you say to people who are in
55:12
I guess you're obviously in a pretty security
55:15
conscious environment where you're working
55:16
very security conscious
55:20
maybe there are folks working in like
55:23
medical applications or law or finance or something
55:27
I think of the
55:28
like, some PRs are kind of like two-way doors
55:31
which is you can merge it and then revert it
55:33
right? It's cheap to go back through
55:35
But there are some PRs that are one-way doors
55:37
right? That will cause data loss of some
55:39
kind that will
55:41
do something that can't be easily walked back
55:44
How do you deal with situations where most of your PRs
55:48
let's say, are one-way doors?
55:50
Like, is this something you just wouldn't recommend or like
55:52
what do you think?
55:53
Yeah, I think that's a really good question
55:55
I think that
55:57
it all comes back to me to the
55:59
quality of the verification that you're able to
56:04
get out of your agents
56:06
And I think for domains where the work is verifiable
56:11
this is easier, right? And the
56:13
the
56:14
one-way doors become two-way doors in a sense
56:18
But I guess, I don't know
56:19
if you're working on something that is like extremely
56:22
is very hard to verify programmatically
56:26
then I think, yeah
56:26
you're definitely in a position where it's very
56:28
hard to get to that point
56:31
so I do think like
56:32
yeah
56:33
verifiability of the domain is an important
56:36
aspect to be able to do this
56:38
and software engineering is just one of those
56:40
things where it's quite verifiable in
56:43
in a lot of cases
56:44
maybe not totally
56:46
you know, like other domains like mathematics
56:48
I think, are another example of Not all of it
56:51
of course
56:52
but
56:53
some aspects of mathematics can be verifiable
56:57
if you write a proof
56:58
for example
57:00
And so, yeah
57:02
I think it's a great question
57:03
that I don't really have the answer to
57:05
and I think that this is something
57:06
the industry and us as engineers will have to figure out
57:10
is, you know
57:11
my sort of
57:13
hope and prediction for the future is that
57:15
we'll see more and more interesting new
57:18
agent-oriented programming languages
57:21
And one of the most fascinating ones that
57:23
I've seen so far is this one called Bend
57:25
B-E-N-D, Bend
57:27
And that language is
57:30
one where it kind of
57:32
marries programming with proofs
57:35
Right
57:36
There used to be a time
57:38
you know
57:38
where you actually had to write your proofs
57:40
in a different language
57:42
And proofs, by the way, for those, who, who aren't familiar, is this idea of
57:47
that you can sort of formally
57:49
verify that some code is correct mathematically
57:53
right? Especially if you've written your code in
57:55
a very functional programming way
58:00
but for the longest time
58:01
you had to do that in a separate
58:02
language like Lean or TLA+ or
58:06
I'm blanking on some of the other
58:08
other examples, but, yeah, like languages like that
58:10
where you would construct the mathematical proof and
58:13
then use a solver essentially to
58:16
to check, right, that, that you've covered all the cases
58:19
you don't have like a race condition or
58:21
or whatever
58:23
So, yeah, I think
58:27
Trying to sum up the question
58:28
I think, yeah
58:29
if you are in a position where you
58:30
can figure out how
58:31
your agents can truly verify the work in
58:34
a way that gives you confidence
58:36
you can actually, you know
58:39
have the PRs merge. Because if it compiles
58:42
right, if it, if the proofs show you that it's correct
58:45
then why wouldn't you just merge it?
58:49
but of course, yeah, not all domains are verifiable
58:53
Yeah, it's a tough one
58:56
okay
58:57
I think
58:58
we've got to
58:58
think about wrapping up because we are nearly
59:00
on the hour. Have you
59:01
have you got something after this?
59:02
I mean, I've got something after this
59:03
I've got to get my son dinner
59:04
but
59:06
I, I can go a bit longer after you
59:07
Okay, let's, let's go five minutes longer then
59:10
I think I just want to have one more question
59:12
which is
59:14
I think I want to ask how
59:17
you see P stack and how you see skills in general
59:22
like in terms of Okay
59:24
we talked about this before we went on air
59:26
which is like, people think of lit
59:28
it's like my skills versus your skills
59:31
and how do you combine frameworks together?
59:34
How do you use PStack with my stuff?
59:37
Like, what should you take from each one?
59:39
And
59:39
because I think
59:40
I see skills as sort of just derived from process
59:45
basically. Like, they're just processes turned into words
59:49
and I would love to know
59:52
how you recommend people take PStack and take
59:54
my stuff as well
59:55
and turn it into their own processes
60:00
I think you shared a tip
60:01
actually, to tell you that I thought was actually very relevant
60:04
which is this idea that you go off
60:05
and look at your previous transcripts
60:08
right, and you sort of mine for information of
60:11
you know, your own Look through your own
60:14
your own prompts, right, to the agents where you correct them
60:17
where you have to constantly intervene and
60:21
you know
60:22
take that higher level learning and turn that into a
60:24
a reusable skill, right
60:25
so that agents stop repeating that mistake
60:29
I think that, PSAC and your skills are very complementary
60:34
I, I totally agree with you that they're
60:36
like, a skill is really much just process
60:38
I mean, it's just, at the end of the day
60:40
a skill is just English or
60:41
or language. It's just Markdown
60:45
and I think you can
60:46
you can definitely weave them
60:48
combine them in a way that makes sense
60:50
to you
60:51
But I do think that
60:53
everyone should have their own set of knives
60:57
right? I, I keep going back to the
60:59
the, the cooking analogy, but it's so apt because like
61:02
you know, every chef When they go to a different job
61:05
right, when they go to a different restaurant
61:07
they carry, they bring their knives with them
61:09
the tools go with them
61:11
right? And so trust to me is really
61:12
about trust in your own tools
61:14
And
61:15
when you spend the time sharpening them and understanding them really
61:18
really well, you can do great things
61:19
And
61:20
everybody's skills and tool set is going to
61:22
look different
61:23
You know, someone might find a lot of success combining
61:27
you know, like your grill me with dogs
61:30
or wayfinder skill with some of the execution skills in PSAC
61:33
as an example
61:34
Some people might use more of your skills
61:36
some people might use more of my skills
61:39
I think at, at, at the end of the day
61:40
it really just comes back to how much do you trust
61:43
you know, me and Matt, right?
61:45
Like, if you, if you trust us both
61:46
of course, use our skills, but I also encourage you to
61:49
you know, look at your own transcripts
61:53
Tell your agent to look through
61:55
you know, some of the
61:56
all of the patterns that you've used
61:58
the, the times you've had to intervene
62:01
you know, suggest turning them into lit rules or new skills
62:05
right?
62:06
The, the past chats I
62:07
I often say is like a
62:09
a treasure trove of context because the
62:11
you know, it's
62:13
it's like, the process materialized, right?
62:16
Like it's the real process
62:18
not an abstract idea in your head
62:20
It's the actual thing
62:22
right? And you can actually see how it
62:23
happened in practice and extract so much information
62:26
from that
62:27
And there's so much show that I actually turn
62:28
I have a skill in PSAT called recall
62:30
which is exactly that
62:33
where this was a pattern where You know
62:36
I was working in
62:37
I was working on a
62:38
a similar problem. So specifically
62:40
I was working on virtualization for the cursor application
62:43
and there were a lot of bugs
62:45
And so, you know, every time I started a new chat
62:47
I was like, oh, there's
62:48
there's so much good context from the last one
62:49
though
62:50
You know
62:50
I want to bring it over to the
62:51
new chat. How do I do that?
62:52
And that's where the transcript came
62:55
you know, looking at the past transcript came about
62:57
and then recall was just a way for me to collapse
63:00
collapse and compress that workflow into a skill
63:03
so that I didn't have to just say
63:06
I didn't have to write a long essay every time
63:07
go look at all these chats
63:08
right? And, you know, the blah, blah, blah
63:10
So I, I largely think of skills
63:12
especially as agents get more capable
63:15
as really encoding workflows
63:17
You know, skills from last year were really more about
63:20
like, almost like implementation details
63:22
like, here are the exact script
63:24
Commands, you know, you should use, right?
63:27
I think with the latest models
63:29
you can just delete those parts
63:31
and just really focus on the workflow
63:32
right?
63:33
And tell it, it's more
63:34
the skill becomes more like a series of steps
63:38
a series of, a real process
63:41
and I think over time
63:42
we'll see that skills get smaller and smaller
63:45
you know, more compact
63:47
and yeah, they're very, very compatible
63:50
Or you can, you know
63:51
if you want, why not read our skills
63:54
right? And come, and combine them in your
63:56
of your own
63:57
right? Combine
63:58
Wayfinder with potato mode and make your own custom mode
64:01
right? Like, like skills are, the
64:03
the thing I love about skills is that
64:05
is, are that they're so malleable
64:07
You can do anything you want
64:08
It's just language
64:10
Absolutely. There's nothing magical in them
64:11
right? They're just words. Yes
64:13
exactly. If there, if there is any magic in them
64:15
it's just the
64:16
words chosen and the phrases used and
64:20
The thinking that's been done to turn those
64:23
like
64:24
take abstract process and turn them into language
64:27
And once that thinking has been done
64:29
then it's just there. It's available
64:31
It's on the surface
64:31
and you just nick it
64:34
Lauren, thank you so much
64:35
This has been glorious
64:37
Yeah, this has been super fun
64:38
I really, really enjoyed talking to you
64:40
Hopefully, we can do it again
64:42
I'd love to do it again
64:44
I'd love to do it again
64:44
absolutely
64:46
yeah, we'll check in in
64:47
I don't know, three, yeah, Monday
64:50
Let's do it. Let's do it
64:53
Bye
64:53
Well
64:53
thanks so much. I'm going to close the
64:54
stream here
64:55
Lauren and I will
64:56
chat a little bit and stay here
64:58
but thank you guys so much for watching
64:59
It was glorious

Watch these videos on your TV

XTTV brings video content from X to your living room. Download now and start watching on the big screen.