Full Walkthrough: Workflow for AI Coding — Matt Pocock

0:07[music]

0:15>> Yeah, we're good.

0:17Okay, folks.

0:18We're at capacity.

0:20Let's kick off. I don't want you waiting

0:22here for 25 more minutes before we some

0:24arbitrary deadline.

0:26So,

0:27welcome.

0:28My name's Matt,

0:30I'm a teacher, and I suppose now I teach

0:32AI.

0:33Um

0:35We have a link up here, if you've not

0:37already been to this, which is has the

0:38exercises for the um stuff we're going

0:41to do today.

0:41This is going to be around 2 hours, so

0:43we might just sort of kick off 2 hours

0:45from now. Is that all right, Mike?

0:47Yeah, perfect.

0:49Um and

0:51the theory behind this talk, or at least

0:52the thesis under which I've been

0:53operating for the last kind of 6 months

0:55or so, is that

0:59we all think that AI is a new paradigm,

1:01right? AI is obviously changing a lot of

1:03things. You guys are obviously

1:04interested in this, and that's why

1:05you've come to this talk.

1:07And

1:09I feel that

1:12when we talk about AI being a new

1:14paradigm, we forget that actually

1:17software engineering fundamentals, the

1:19stuff that's really crucial to working

1:21with humans, also works super well with

1:24AI.

1:25And this is what my keynote is on

1:27tomorrow, really. I'm going to sort of

1:28be fleshing that out a lot more.

1:30And in this workshop, I'm hopefully

1:32going to be able to direct your

1:33attention to those things, and

1:35uh hopefully show you

1:38that I'm right. But we'll see.

1:40Um can I get a quick heads-up first? How

1:43many of you guys um are coding have ever

1:46coded with AI? Raise your hand if you've

1:48ever coded with AI. Perfect. Okay. Uh

1:51keep your hand raised.

1:53Uh

1:54let's all uh share those armpits with

1:56the world. Um

1:58how many of you code every day with AI?

2:01Cool. Okay. Uh right, keep your hand

2:04raised if you've ever been frustrated

2:05with AI.

2:07Okay, very good.

2:09You can put your hands down.

2:11Thank you for that show of obedience. I

2:12really appreciate that. And we are also

2:14being live-streamed to the Gilgood room

2:16as well. I've not

2:17uh

2:18Did we send someone up to the Gilgood

2:19room to just check they're okay?

2:21Don't know.

2:22But I see you,

2:24and there is a way that you can

2:25participate, which is we have the um a

2:28Q&A. We're going to be doing kind of

2:30have a sort of hatred of Q&As cuz

2:31they're not very democratic. They're

2:33mostly the sort of

2:34um most talkative people get to um

2:37get to participate and share. And so,

2:40we're going to be going through this um

2:42Q&A here. So, why do we have to wait

2:43till 3:45? The room is packed, the doors

2:45are closed. 100% agree.

2:47And so, if you want to uh ask a

2:49question, we're going to be I would like

2:50you to pile into this async, and then we

2:53can vote on each other's questions, and

2:54hopefully get the best questions

2:56surfaced so the for the entire room to

2:58enjoy.

3:00So, I want to talk about first the kind

3:02of weird constraints that LLMs have.

3:06And

3:07those weird constraints are sort of what

3:09we have to base a lot of our work

3:11around.

3:12Now,

3:14there's a guy called Dex Hardy who runs

3:16a company called Human Layer, and he

3:17came up with this idea, which is that

3:21when you're working with LLMs, they have

3:23a smart zone

3:25and a dumb zone.

3:27When you're first kind of like

3:29working with an LLM, and it's like

3:31you've just started a new conversation,

3:33you start from nothing, that's when the

3:35LLM is going to do its best work.

3:37Because in that situation, the attention

3:38relationships are the least strained.

3:40Every time you add a token to an LLM,

3:43it's kind of like you're adding a team

3:44to a football league. You think of the

3:46number of matches that get added every

3:49time you add a team to a football

3:50league, it just goes

3:52it scales quadratically. And that's

3:54because you have attention relationships

3:55going from essentially each token to the

3:57other that are positional and the sort

4:00of meaning of the individual token.

4:02And so, this means that by around sort

4:04of 40% or around I would say around 100K

4:07is kind of my new marker for this. Cuz

4:09it doesn't matter whether you're using 1

4:11million

4:12uh context window or 200K,

4:15it's always going to be about this.

4:17It starts to just get dumber.

4:20So, as you continually keep adding stuff

4:22to the same context window, it just gets

4:24dumber and dumber until it's making kind

4:26of stupid decisions. Raise your hand if

4:27that feels familiar to you.

4:30Yeah, cool.

4:31So, this means that we kind of want to

4:33size our tasks in a way that sticks

4:37within the smart zone.

4:38Right? We don't want the AI to bite off

4:41more than it can chew. This goes back to

4:43old advice like Martin Fowler in

4:45refactoring. Uh like uh the pragmatic

4:48programmer talks about this. Don't bite

4:49off more than you can chew. Keep your

4:51tasks small so that you as a developer,

4:54a human developer, don't freak out and

4:56don't start acting and going into the

4:58dumb zone.

5:01But

5:02how do you tackle big tasks? How do you

5:04take a large task like I don't know,

5:07cloning a company or something, or just

5:09doing something crazy,

5:11and how do you break it into small tasks

5:13so they all fit into the dumb zone?

5:16One way, of course, you could do is I

5:17mean, kind of what the AI companies

5:19maybe want you to do, or the natural way

5:21of doing it is just keep going and going

5:22and going, you end up in the dumb zone,

5:24charging you tons of tokens per request.

5:26You then compact back down.

5:28We'll talk about compacting properly in

5:29a minute. And you keep going, keep

5:31going, keep going, compact back down,

5:33keep going, keep going, keep going.

5:35And I think that's doesn't really work

5:37very well because the more sediment I

5:39we'll talk about that in a minute.

5:41So, the theory here is then, and this is

5:43what I was doing for a while,

5:45is I would use these kind of

5:47um multi-phase plans.

5:49Where I would say, "Okay, we have this

5:51sort of number four thing here, this

5:53large large task. Let's break it down

5:55into small sections so that we can then

5:57kind of chunk it up and do each little

6:00bit of work in the smart zone." Raise

6:02your hand if you've ever used a

6:03multi-phase plan before.

6:05Yeah, really common practice, right?

6:07This is kind of how we've been doing it.

6:09Certainly, this is how I was doing it up

6:11until December last year, really.

6:14And any developer worth their salt will

6:16look at this and go, "This is a loop."

6:19Right? This is a loop. We've just got

6:21phase one, phase two, phase three, phase

6:23four. Why don't we just have phase N?

6:27Right?

6:29Phase N. Where we essentially just say,

6:31"Okay,

6:32we have, let's say, a plan operating in

6:34the background, and then we just loop

6:35over the top of it, and we go through

6:37until it's complete."

6:38And this is where um

6:40Raise your hand if you've heard of Ralph

6:41Wiggum as a software practice.

6:44Okay, cool. Raise your hand if you've

6:45not heard of Ralph Wiggum as a software

6:46practice, actually. That's more like it.

6:48Okay. So, there's this idea called Ralph

6:49Wiggum, uh which is kind of um

6:52sort of based on this,

6:54which is essentially

6:56all you need to do is sort of specify

6:58the end of the journey,

7:00where you just say, "Okay, we create a

7:01PRD, a product requirements document, to

7:03say, 'Whoa, okay, let's describe where

7:05we're going.'" And then we just say to

7:07the AI, "Just make a small change. Make

7:10a small change that gets us closer and

7:11closer to that."

7:13And

7:14Ralph works okay, but I prefer a little

7:15bit more structure.

7:17So, that's kind of where we got to in

7:19terms of thinking about the smart zone,

7:21and that's

7:22kind of where I want you to first start

7:25thinking about here.

7:27Another weird constraint of LLMs is LLMs

7:29are kind of like the guy from Memento,

7:31right? They just continually forget.

7:32They could just keep resetting back to

7:34the base state.

7:36Let me pull up this diagram.

7:38I sort of I

7:39I I really should use slides, but I just

7:41prefer just like randomly scrolling

7:43around a

7:44uh infinite uh TL draw canvas. Thank

7:46you, Steve.

7:48Um

7:49So, let's say another concept I want you

7:52to have is that every session with an

7:53LLM kind of goes through the same

7:55stages.

7:56You have, first of all, the system

7:57prompt here. This gray box here is

8:00essentially the stuff that's always in

8:02your context. You want this to be as

8:04small as possible. Cuz if you have a ton

8:07of stuff in here, if you have 250K

8:09tokens, like I have seen people put in

8:11there, then that you're just going to go

8:13straight into the dumb zone without even

8:15being able to do anything.

8:17So, you want this to be tiny.

8:19>> [snorts]

8:19>> You then go into a kind of exploratory

8:21phase. This blue sort of where the

8:23coding agent is going out and exploring

8:25the code base.

8:26Then you go into implementation.

8:28And then you go into testing.

8:30And sort of making sure that it works,

8:32running your feedback loops and things

8:33like this.

8:34Raise your hand if that feels familiar

8:36based on what you've done. Yeah. Sort of

8:38the like the the main cornerstones of

8:40any session.

8:42And when you clear the context, you go

8:44right back to the system prompt.

8:46Oof, you go right back there. So, you

8:48delete everything that's come before.

8:51And

8:53raise your hand if you've heard of

8:54compacting, as well.

8:56Yeah, okay. There are some people who've

8:57not heard of compacting. So, let's just

8:58quickly show what that means.

9:00For instance,

9:02I've just been having a little chat with

9:03my LLM.

9:06Uh

9:07I want to make sure we sort of, you

9:09know, just cover the basics so we're all

9:10sort of on the same wavelength here.

9:12I've just been having a chat with my

9:13LLM.

9:14I've been talking about a thing that I

9:16want to build. How's the font size?

9:17Should I bump it up?

9:19Folks in the back?

9:20Bump. Bump.

9:22Bump. Bump. Bump. Oh.

9:24I'm using Claude Code for this session,

9:25but you don't need to use Claude Code.

9:27Uh

9:28in fact, it's often nice not to use

9:29Claude Code.

9:30Um

9:32so, I've been having a chat with the

9:33LLM, just sort of planning out what I'm

9:34going to do next. It's asking me a bunch

9:35of questions, and I can

9:38I highly recommend you do this.

9:40There's this tiny little status line

9:42here that tells me how many tokens I'm

9:44using, the exact number of tokens I'm

9:46using. Um I have a article on my website

9:49AI Hero if you want to copy this. This

9:52is

9:53Oh, wow, that is that shakes, doesn't

9:54it? Um

9:56this is essential information on every

9:59coding session cuz you need to know

10:00exactly how many tokens you're using so

10:02that you know how close you are to the

10:03dumb zone.

10:05Absolutely essential.

10:06And so let's watch it.

10:08So I've got two options. I can either

10:09clear

10:11wrong and go back to nothing or I can

10:14compact.

10:15And when I compact then it's going to

10:18squeeze all of that conversation, which

10:19admittedly isn't very much, into a much

10:22smaller space.

10:24And this in diagram terms kind of looks

10:26like this.

10:27Where you take all of the information

10:28from the session and you essentially

10:30create a history out of it, a written

10:32record of what happened.

10:36And devs love compacting for some

10:37reason, but I hate it.

10:40I much prefer my AI to behave like

10:43uh the guy from Memento because this

10:45state

10:46is always the same. Always the same

10:48every time you do it. You clear and you

10:50go back to the beginning. And so if

10:51you're able to do that and you're able

10:53to optimize for that then you're in a

10:54great spot.

10:56So that's kind of the two things I want

10:58you to think about with LLMs, the two

10:59constraints that we're working with.

11:01They have a smart zone and a dumb zone

11:03and they're like the guy from Memento.

11:06So let's take a look at the first

11:08exercise.

11:09And I'm while I'm doing this, the way I

11:11want this to work is I'm going to sort

11:13of show you how um I'm going to be sort

11:15of walking through it up here and I want

11:17you folks to be kind of like tapping

11:19away and doing things as well. So that

11:21was just a little lecture bit. Let's now

11:23actually get and do some coding.

11:25For anyone who arrived late or anyone in

11:27the Gilgud room uh go to this link

11:32this link up here

11:35to see the exercises and clone the repo.

11:38You absolutely do not have to, you can

11:39just watch me do it if you fancy it.

11:41But let's go there myself and let's see

11:42what exercises await us.

11:45So essentially I've built a um this is

11:47from my course.

11:49This is a uh a course management

11:52platform essentially, a kind of CMS for

11:55instructors, for students, and this is

11:56what we're going to be building a

11:57feature in. So I'm going to take you

12:00from essentially the idea for the

12:02feature all the way up to building a PRD

12:04for the feature, all the way up to

12:06implementing the feature.

12:08And hopefully you can take inspiration

12:09from this process and use it in your own

12:11work.

12:12So

12:14uh let's kick off. So

12:17we're going to start by using a a skill

12:19which is very close to my heart.

12:21It's the grill me skill.

12:23And this grill me skill is wonderfully

12:27small wonderfully tiny and it helps

12:30prevent one of I think the main issues

12:32when you're working with an AI, which is

12:34misalignments.

12:37The uh

12:39the sort of silent idea that I'm talking

12:41against here, that I'm arguing against,

12:43is the specs to code movement. Has

12:45anyone heard of the specs to code

12:46movement? Raise your hand. It's not

12:48really a movement I suppose, it's just

12:49sort of people saying specs to code.

12:51Um

12:53what it is is people say, "Okay, you can

12:55write a program or you want to build an

12:57app the best way to build that app is to

13:00take some specifications

13:02so to write some sort of like document

13:05and then turn that document into code."

13:09So they just turn it into code. How do

13:10you do that? You pass it to AI. If

13:12there's something wrong with the

13:13resulting code, you don't look at the

13:15code, you look back at the specs. You

13:17change the specs and you sort of just

13:19keep going like this. This is kind of

13:21like vibe coding by another name where

13:22you're essentially ignoring the code.

13:25You don't need to worry about the code.

13:27You just sort of keep editing the specs

13:28and eventually you just keep going. And

13:30I tried this. I really tried it. And it

13:32sucks. It doesn't work.

13:34Because you need to keep a handle on the

13:36code. You need to understand what's in

13:38it. You need to shape it because the

13:40code is your battleground. And so

13:44this is again is where we're going.

13:45Let's let's get some exercises.

13:47So

13:48what I'd like you to do is go to this

13:49page, the the grill me skill.

13:51And inside the repo here

13:54we have a slack message

13:56from our pal. Uh where is it? It's in

13:59the root of the repo and it's under

14:03bur bur bur bur

14:04Oh, where is it?

14:06Mhm mhm client brief.md.

14:09It's a slack message from Sarah Chen.

14:11For some reason the Claude always

14:12chooses Sarah Chen as the name. I don't

14:13know why.

14:14Um it's saying that in cadence, our um

14:18course platform, our retention numbers

14:20are not great. Students sign up to a few

14:22lessons then they drop off. I'd love to

14:24add some gamification to the platform.

14:26And so when you're presented with an

14:28idea like this, you need to find some

14:30way of turning it into reality. Let's

14:31say Sarah Chen is your client, you're on

14:33a tight budget, you need to get this

14:34done fast. How do you go and do it?

14:37Um

14:38raise your hand if you would um

14:40enter plan mode when you're doing this.

14:43Anyone a big user of plan mode? Yep.

14:45Um let's actually shout out quickly any

14:47other ideas about what you would do with

14:49this or any Raise your hand if you

14:51what what would be your first port of

14:52call?

14:54Yep. Ask for more info.

14:55Sorry? Ask for more

14:57info to verify what is the purpose and

14:59where our current standing is. Yes,

15:00exactly. Let's imagine that Sarah Chen's

15:02gone on holiday, you have no idea,

15:03right? Uh she's just posted this thing,

15:05you need to action it before you go.

15:07Well, my first port of call is I go for

15:10this particular skill. I'm going to

15:11clear my context.

15:15I'm going to

15:16uh get rid of

15:18you, you don't need to be there.

15:20And I'm going to say

15:22um I'm going to invoke a skill

15:25which is the grill me skill. Let's

15:27quickly check.

15:28Raise your hands if you don't know what

15:29this is.

15:31Cool.

15:32Oh, sorry sorry. Let me be more

15:33specific. Raise your hands if you don't

15:36know what I'm doing here when I

15:38uh do a forward slash and then type

15:40something.

15:41Anyone Everyone kind of understand what

15:43that is?

15:44I'm invoking a skill. I'm invoking the

15:45grill me skill.

15:47And what I'm going to do is I'm going to

15:49say grill me and I'm going to pass in

15:51the client brief.

15:54So now

15:55the LLM really has only a couple of

15:58things here. It just has the skill and

16:00it has the description of what I want to

16:01do.

16:04And this is virtually how I start every

16:06piece of work with AI.

16:08And while it's exploring the code base

16:11I'm just going to show you what the

16:12grill me skill does.

16:14So this is inside the repo so you can

16:15check it out.

16:17It's extremely short.

16:19"Interview me relentlessly about every

16:21aspect of this plan until we reach a

16:22shared understanding. Walk down each

16:24branch of the decision tree resolving

16:26dependencies one by one. For each

16:28question provide your recommended

16:29answer.

16:30Ask the questions one at a time uh blah

16:33blah blah."

16:34What this does and what I noticed when I

16:36was working with AI, especially in plan

16:38mode actually

16:40is it would

16:42really eagerly try to produce a plan for

16:44me.

16:45It would say, "Okay, I think I've got

16:46enough. I'm just going to poof plan

16:48plan."

16:49And what I found was that

16:53I was really trying to find the words

16:55for this, for for what I wanted instead

16:57of that.

16:58And Frederick P. Brooks in The Design of

17:01Design, he has a great quote uh talking

17:03about the design concept.

17:05When you're working on something new

17:07with someone

17:08when you're uh all trying to build

17:10something together

17:12then there's this shared idea that's

17:14shared between all participants and that

17:16is the design concept. And that's what I

17:18realized I needed with Claude. I needed

17:22I needed to reach a shared

17:24understanding. need an asset, I didn't

17:26need a plan, I needed to be on the same

17:28wavelength as the AI, as my agent. And

17:31this is an extremely effective way of

17:33doing it. So hopefully

17:35Here we go. Nice. It has done its

17:37exploration first of all.

17:39It's invoked a sub agent which spent

17:4297 93.7k tokens

17:45on Opus.

17:47Um

17:48and it's asked me the first question.

17:50Cool.

17:51We can see that even though the sub

17:53agent burned a a ton of tokens I haven't

17:55actually um

17:57uh increased my token usage that much.

17:59Raise your hand if you don't know what

18:01sub agents are. It's important question.

18:04Everyone kind of clear what sub agents

18:05are? Okay, I'll give a brief definition.

18:07Which is that this this sub agents thing

18:10here, this explore sub agent it has

18:12essentially gone and called another LLM

18:14which has an isolated context window.

18:18And then that LLM has reported a summary

18:20back. So a sub agent is kind of like a

18:22delegation. You're delegating a task to

18:24a sub agent. It goes eagerly does all

18:26the thing, explores a ton of stuff and

18:28then just drip feeds the important stuff

18:30back up to the orchestrator agent.

18:33To the parent agent. So okay. So

18:35hopefully you guys have seen the same

18:36thing. It's done an explore.

18:38And we now have our first question.

18:41Points economy. What actions earn points

18:43and how much? Ooh, okay.

18:45At this point you can ask it by the way

18:47questions to um deepen your

18:49understanding of the repo. I obviously

18:50know this repo really well cuz I wrote

18:52it, but you might not um

18:54know what's going on.

18:55So let's say my recommendation, keep it

18:58simple, two point sources to start.

19:00What's so nice about this is that not

19:02only does it give us a question that

19:04kind of aligns us here, we get a

19:06recommendation too. And often what I'll

19:08find is the AI's recommendations are

19:09really good.

19:11And so I'll just say

19:12skip video watch events, they're noisy

19:14and gameable. I agree.

19:16Sarah's asked we'll keep the lessons in

19:17the bread and butter.

19:20Yeah.

19:21Looks good, pal.

19:24>> [snorts]

19:25>> Now what I usually do is I usually

19:26dictate to the AI. I'm usually actually

19:28chatting to the AI instead of uh typing

19:31here, but uh this is a relatively new

19:33laptop and I couldn't get my dictation

19:35software working on it um because

19:37Windows is crap. Um

19:40So, should points be retroactive? There

19:43are existing lesson progress records

19:45with completion at timestamps. This is a

19:47really nasty question, right? Should we

19:49actually go back and backfill all of the

19:51lesson progress events? This is a kind

19:53of question that you need to be aligned

19:55on if you're going to fulfill the

19:57feature properly. This is not something

19:58I considered and Sarah Chen certainly

19:59didn't consider.

20:01Do I want it to be retroactive? Hmm.

20:04Let's actually do a vote inside here.

20:07Should we go back and backfill all the

20:08records? Raise your hand if you think we

20:09should backfill all the records.

20:13Raise your hand if you think we

20:14shouldn't backfill all the records.

20:17There are a lot of fence-sitters in the

20:19room. I'm going to say

20:22you know, this is the kind of discussion

20:23you're sort of having with the AI.

20:24You're getting further aligned. Yes, I'm

20:25just going to go with his recommendation

20:27cuz I'm lazy.

20:31Notice too how I'm able to keep in the

20:33loop here with AI. I'm not you know,

20:35it's it's pinging me these questions

20:36pretty quickly.

20:39I'm not having to go off and check

20:40Twitter or something.

20:42Levels. What's the progression curve?

20:44Yeah, that looks about right. For

20:46instance, yes, okay.

20:47So hopefully you should be able to go

20:49and um

20:50kind of work through this with the AI.

20:52>> [clears throat]

20:52>> And essentially

20:54try to reach an alignment. And this

20:56grill me skill, this can last a long

20:58time. This can I've had it ask me 40

21:00questions. I've had it ask me 80

21:02questions. I've had some people that

21:03asks 100 questions too. Literally you're

21:06sat there for an hour chatting to the

21:08AI.

21:09And what you end up with is essentially

21:11this conversation history

21:13that works really nicely and works

21:15really nicely as an asset of the design

21:17concept that you're creating.

21:19This can also function like this. You

21:21can

21:22have a meeting with someone who's a

21:24maybe a domain expert. Maybe I have a

21:25meeting with Sarah. I feed that meeting

21:28transcript into

21:30I don't know, Gemini meetings or

21:32whatever you guys are using. You take

21:34that, you feed it into a grilling

21:36session and you grill through the

21:37assumptions that you didn't have.

21:39So this ends up being a really nice kind

21:41of

21:41um

21:43a really nice way of just taking inputs

21:45from the world and then just turning and

21:47validating them.

21:49So okay.

21:51Let's see. I really want to get to the

21:53end of this, but I also don't want to

21:54just like be sat here talking to the AI

21:56in front of you for uh

21:58a thousand days. So I'm just going to

21:59say yes.

22:03Let's see what happens.

22:05So I'll tell you what, um while you guys

22:07sort of have a little fiddle with this

22:08locally, let's start a little Q&A

22:10session now.

22:11And

22:13let's see. How's this going to work?

22:15Can we keep the door closed or turn up

22:16the microphone? It's quite noisy.

22:19Uh

22:20let's see. Mike, can we uh

22:22door closed. Oh it has been closed. Mark

22:24has answered. Beautiful.

22:26So what I'd like you to do

22:28is there any air con? Yeah, there is

22:30some air con, I think.

22:32There is some air con.

22:34You guys aren't being lit here. I'm

22:35being fro I'm being fried alive here.

22:38Uh so what I'd like you to do is go on

22:40to the Slido, which you can join here.

22:42Have a if if you're not taking the

22:44exercise, go on to the Slido, have a

22:46little fiddle and vote on some good

22:47questions. I'm just going to chat to the

22:49AI for a second

22:51uh until we reach a stopping point. So

22:53do streaks earn points?

22:56Um

22:57streaks are standalone.

23:06Let's see what else it comes up with.

23:13Where does gamification UI live?

23:15Let's have it in the dashboard.

23:19I'm just going to scan these and blast

23:20through them basically.

23:21So how are we doing with our Slido?

23:24Okay.

23:26Have I tried Spec Kit, Open Spec or

23:28Taskmaster instead of the Grill Me

23:30skill? Do I find them more verbose or a

23:32structured alternative? This is a great

23:33question. So there are a ton of

23:35different frameworks out there that

23:36allow you to um sort of build up this

23:39planning process for you. I personally

23:42believe you at at this stage, when

23:44there's no clear winner, when there's no

23:46kind of like one true way and when

23:48things are changing all the time, you

23:50need to own as much of your planning

23:52stack as you possibly can.

23:54What I've noticed and a lot of my

23:56students

23:57is

23:59they tend to overuse a certain stack.

24:03They get into trouble

24:05and they because they don't own the

24:06stack and they don't have observability

24:08over the whole thing, they just go

24:10this isn't working. This sucks. Whereas

24:13if

24:14um

24:14if you have control over the whole

24:16thing, then at least you know how to fix

24:19it or potentially know how to fix it.

24:21So I'm even though I'm sort of giving

24:24you uh a stack basically, I believe in

24:28inversion of control and you should be

24:29in control of the stack.

24:32So bur bur bur.

24:33Can I press zero, please?

24:38Sorry?

24:40Sorry, that was a lot of sort of

24:41mumbling. Can I

24:48Thank you.

24:50I'm so sorry.

24:50>> [laughter]

24:51>> What you didn't want to give Claude good

24:53feedback? What is what is wrong with

24:54you?

24:57Uh okay, cool.

24:59Uh many of the questions asked by the

25:01Grill Me skill are not necessarily

25:02appropriate for a developer, rather a

25:03PO. In larger teams, who should use it?

25:05Yeah.

25:06Um

25:07Raise your hand if um

25:10you've ever done pair programming.

25:12Anyone ever done pair programming?

25:13Right. I keep Put your hands down and

25:16raise your hand again if you've ever

25:17done a pair programming session with an

25:18AI.

25:20Right.

25:21How did it go? Was it good? You enjoy

25:23it? I think pair programming sessions

25:25with AI is a great idea because you've

25:27got a third person in the room who will

25:28relentlessly quiz you and ask you

25:30questions. It should If you don't know

25:32the answer, it should be you, the domain

25:33expert and the AI in the same room. If

25:36you're have a question about

25:37implementation, it should be you, a

25:39fellow developer and the AI in the same

25:41room, you know. You can be sort of

25:42working through these questions in your

25:44team. And I think actually

25:47we're going to look at implementation in

25:48a bit and we're going to see how you can

25:50make implementation so much faster.

25:52And but I think the really crucial

25:54decisions, the ones you need humans for

25:57you actually need a lot of humans and it

25:59doesn't really matter how many humans

26:00are in there. You can actually throw a

26:02bunch like a kind of like mob

26:04programming with AI essentially.

26:07Uh what's my favorite meta prompting

26:08tool? I think I kind of answered that.

26:10Uh there's no air con. Let's just live

26:12with it. Uh

26:14how do I use the conversation as an

26:15asset after the Grill Me session? Well,

26:18we're going to get there.

26:20Um okay, so I really want to

26:24I want to speed this up sort of

26:25artificially.

26:28Just what

26:29I This is the thing. So someone just

26:31said okay, Ralph loop this. But this is

26:33crucial because I can't loop over this,

26:36right? I can't um

26:39I think of there is being two types of

26:41tasks in the AI age.

26:43Where you have human in the loop tasks,

26:46where a human needs to sit there and do

26:48it.

26:49Which is this.

26:50We are the human in the loop, with

26:51multiple humans in the loop. And there

26:53are AFK tasks. There are tasks where the

26:55human can be away from the keyboard and

26:57it doesn't matter. Implementation, as

26:59we'll see, can be turned into an AFK

27:01task. But planning, this alignment

27:04phase, has to be human in the loop. Has

27:07to be.

27:09So I've got to do it, unfortunately.

27:11Um

27:12I don't know.

27:13Uh

27:14give me a long list of all your

27:18recommendations.

27:20I'm running a workshop right now.

27:24So I artificially

27:26need you to

27:28pull more weight.

27:31So let's see what it does.

27:33Uh let's answer a couple more questions

27:34while it's doing its thing.

27:37What is my opinion on PMs or other

27:39non-dev roles vibe coding task?

27:42Hmm.

27:45Um I'm going to return to this later, I

27:48think. I'm going to leave this

27:48unanswered.

27:51A bit of mystery.

27:53I notice I'm not using the ask user

27:55questions UI for Grill Me. Why? Um

27:57there's a specific uh

27:59UI that you can bring up in Claude Code.

28:01I'll answer this just quickly.

28:03Uh ask me a question using the ask user

28:08question tool.

28:10>> [snorts]

28:10>> And this UI um is just sort of broken in

28:13Claude and I really hate it.

28:17You notice I'm using Claude, but I don't

28:19like Claude very much. Like you you

28:20really are free with this method to

28:22choose any um system you like. And this

28:24is what the UI looks like.

28:26It's very pleasing when you first

28:27encounter it, but then you realize it is

28:28actually broken in a ton of different

28:29ways.

28:32All right, what did it come back with?

28:33Oh blimey.

28:35Oh no.

28:37So

28:40while this is doing its thing, let me do

28:41some teaching in the meantime.

28:43The plan here is that we take our Grill

28:46Me skill

28:47and we need to essentially find some way

28:49of turning it into

28:51a destination.

28:53We need to go down to the

28:56uh

28:57We essentially need to

28:58we're figuring out the shape of this.

29:01That's what we're doing. We're figuring

29:02out the shape of the tasks during the

29:03grilling session.

29:05And in order to

29:08turn it into a bunch of actionable

29:10actions for the AI

29:12we essentially need to figure out the

29:13destination. We need to know where we're

29:15going. We need to know the shape of this

29:16entire thing.

29:18So I think of there is being two

29:20essential documents that we need.

29:22We need a document that

29:24documents the destination.

29:27Oh no.

29:29It's so not bright enough. There we go.

29:33Still not brighter. There we go.

29:35We need something to document the

29:36destination.

29:38And we need something to document the

29:39journey.

29:41In other words, we need something a

29:42document that's going to

29:44figure out what this even looks like in

29:46all of its user stories and figure out a

29:48definition of done

29:50and then we need to figure out what the

29:51split looks like.

29:53So, that's where we're going to go to

29:54next.

29:55So, once we finish with the grilling

29:57session,

29:59yeah, it looks great. Fantastic. I love

30:01it. It answered

30:02it answered 22 of its own questions.

30:04There you go. That's quite

30:05representative of what a grilling

30:06session looks like.

30:09So, at this point now,

30:12I have used 25k tokens and all of that

30:16or loads of that stuff is gold. I want

30:18to keep that around. I've I've got 25k

30:22great tokens there.

30:24And what I want to do is kind of

30:25summarize it in some kind of destination

30:27documents.

30:28So, this is um the next exercise

30:31where we're going to

30:35uh we're going to write a product

30:37requirements document.

30:39And the the product requirements

30:40documents or the PRD

30:43is essentially

30:44that's its function. It's the

30:46destination documents. And it's sort of

30:48doesn't matter what shape it is. I've

30:51got a shape that I prefer and I quite

30:53like.

30:54But, you can just choose your own shape

30:56or whatever your company uses.

31:00And all we're really doing is I'm not

31:03too worried about that.

31:05All we're really doing is summarizing

31:07the design concept that we have so far.

31:10And

31:12the So, let let's try this.

31:15So, I'm going to initiate this. I'm

31:16going to say

31:17zoom all the way to the bottom.

31:19All I'm going to do is just say write a

31:20PRD.

31:23And we can take a look at that skill

31:24now.

31:26Write a PRD.

31:29So, this skill

31:31it does a few things.

31:34It first asks the user for a long

31:35detailed description of the problem. You

31:36can use write a PRD without grilling

31:38first, but I just like to grill first

31:40and then write the PRD afterwards.

31:42Then you can um get it to install the

31:45repo which we've kind of already done.

31:47Then we get it to

31:49interview the user relentlessly so we

31:50have a kind of grilling session again

31:52and then we start um putting together a

31:55PRD template. So, this is available in

31:57the repo if you want to check it out.

31:59And essentially this is what it looks

32:00like. We've got some problem statements,

32:02the problem the user is facing, the

32:04solution to the problem and a set of

32:06user stories. And these user stories

32:08sort of define what this is. You know,

32:10as

32:11you you guys have probably seen things

32:12like this if you've been a developer at

32:13all. Um you know, there are cucumber is

32:16a language you can use to write these in

32:17or we just sort of

32:18um

32:20uh write them ourselves essentially.

32:22Then we have a list of implementation

32:23decisions that were made and list of

32:25crucially testing decisions, too.

32:28So,

32:31I'm going to run this. Okay. And so,

32:33it's finished its thing.

32:35Ah!

32:37Windows, let me close the thing. Thank

32:39you.

32:40I don't know why I bought a Windows

32:41laptop. I think I just

32:43I like the challenge. Um

32:46>> [clears throat]

32:46>> So, the first thing that it's going to

32:47give me

32:49are a set of proposed modules it wants

32:51to modify.

32:54Now, there's a deep reason why I'm

32:55thinking about this. So, this is

32:58at this stage

33:00we have an idea, we have sort of specked

33:02out the idea, we've reached a sort of

33:05understanding of what we're trying to do

33:07and then we need to start thinking about

33:09the code

33:10because at this point we need to

33:13this is not specs to code. This is not

33:15where we're ignoring the code. We

33:17actually keep the code in mind

33:18throughout the whole process.

33:20And

33:21the way I like to do this is I like to

33:23just sort of think about a set of

33:24proposed modules to modify. We're going

33:26to return to this this idea of

33:28continually designing your system and

33:31keeping your system in mind.

33:33So, it's it's saying recommend tests for

33:34the gamification service is the only

33:36deep module with meaningful logic. These

33:38modules look right. Yeah.

33:41Looks good.

33:44And it's going to hang out a PRD.

33:48Now, for ease of setup

33:50I've got it so that it creates a set of

33:52issues locally.

33:54So, it's just going to create

33:55essentially a PRD inside this issues

33:57directory.

33:59But, the way I usually do it

34:01and you can check this out yourself is

34:04you can go to my um essentially what I

34:05consider my work repo

34:07which is GitHub um dot com forward slash

34:10Matt Pocock forward slash course video

34:13manager up here.

34:15And in here, this is essentially a app

34:17that I create um that I use all the time

34:20to record my videos and things like

34:21this. I think I've recorded like

34:24I pulled out the stats. I think I've

34:25recorded like a thousand videos in here

34:27or something nuts.

34:28Um and you can see here that it's got

34:30744 closed issues.

34:32And this is essentially all of the uh

34:35PRDs and all of the implementation

34:37issues that I've put into here. So, this

34:39is how I usually like to do it.

34:40>> [clears throat]

34:42>> So, that's what I'm doing with the There

34:45we go. Yeah, I'm just going to say yes

34:47and uh

34:49and get that issue out.

34:51Let's see. It is inside here.

34:53So, we've got the problem statements.

34:55People signing up for courses.

34:57Uh the solution, the user stories, uh 18

35:00user stories looks nice, some

35:02implementation decisions, level

35:03thresholds, etc. This is enough

35:05information. We've kind of clarified

35:07where we're going and what we're doing.

35:09So, that's what we do. We essentially

35:11have a grilling session and we've

35:12created an asset out of it. Now, raise

35:14your hand.

35:16Should I be reviewing this document?

35:19Raise your hand if you think I should be

35:20reviewing the documents.

35:23Yeah, I don't I don't look at these.

35:24I don't look at these.

35:26The reason I don't look at these is

35:27because what am I testing at this point?

35:30What am I Like when I read it,

35:33what am I testing? What am I What are

35:34the failure modes I'm trying to test

35:35for?

35:36I know that LLMs are great at

35:37summarization

35:39cuz they are. They're really good at

35:40summarization.

35:41I have reached the same wavelength as

35:44the LLM, right? Using the grill me

35:45skill, we have a shared design concept.

35:48So, if I have a shared design concept,

35:49all I'm doing

35:51is I'm just essentially checking the

35:53LLM's ability to summarize.

35:56So, I don't tend to read these.

36:00Let's have Let's have a Q&A cuz I can

36:02feel you guys are itching for it. And I

36:03think we might have like

36:05I don't know, just a 5-minute comfort

36:07break just to uh rest my voice and so

36:08you can catch up with the exercises for

36:09a minute if that's all right. So, let's

36:11have a little Q&A sesh.

36:14Uh

36:15If I don't like Claude Code, which one

36:16do I actually like? Um

36:19uh

36:20Have you ever heard the phrase um

36:23uh democracy is the worst way to run a

36:24country apart from all the other ways?

36:27That's how I feel about Claude Code.

36:30Uh we've answered that one.

36:33Uh

36:34What's your thoughts on developers

36:36needing to very deeply understand

36:37TypeScript now that fix the TS make no

36:40mistakes exist? I don't understand the

36:42phrasing of this,

36:43but I think I understand meaning,

36:46which is that

36:48I believe that code is very important

36:50and this is kind of going to feed

36:52through the whole session and that bad

36:54code bases make bad agents. If you have

36:57a garbage code base, you're going to get

36:59garbage out of the agent that's working

37:01in that code base. We'll talk more about

37:02that in a bit.

37:03And so, I think understanding these

37:05tools very deeply, understanding code

37:07deeply is going to make you a much much

37:10better developer and get more out of AI.

37:14Uh and that answers that question, too.

37:16Sweet.

37:19Uh

37:20Get out of there. There you are.

37:24Now that we have 1 million tokens

37:25available, do we ever actually want to

37:27take advantage of that?

37:30I've noticed that the dumb zone has

37:31become less dumb lately. Okay, great

37:33question. This goes back to our kind of

37:35initial idea on the dumb zone.

37:41Uh

37:43I am I recorded my Claude Code course

37:46using a 200k context window and on the

37:48day that I launched the course they

37:50announced the 1 million context window.

37:53My take on this is that what Claude Code

37:54did is they essentially just did this.

37:56Wee!

37:58They shipped a lot more dumb zone to you

38:01essentially. Now, this is good for tasks

38:03where you want to retrieve things from a

38:05large context window. If you want to

38:07pass five copies of War and Peace or

38:09something to it and you want to find out

38:11all the things that uh

38:14uh I can't remember a character from War

38:15and Peace. Uh

38:17Why did I start with that?

38:18It's good for retrieval.

38:19It's less good for coding.

38:21So, I consider that it is about 100k at

38:26the moment is the smart zone. The smart

38:28zone will get bigger and that will be a

38:31really nice improvement.

38:33So, folks, we're going to take it like a

38:345-minute comfort break if that's all

38:36right just for my voice and to maybe you

38:38can have a little move around or

38:39something or grab a drink. I can just

38:41notice some sleepy eyes and I want to

38:42make sure that we're awake for the next

38:44bit if that's all right. So, we'll take

38:455 minutes and I will see you back here

38:49then. All right?

38:51So, we have

38:53our PRD

38:55which I'm not going to read, our kind of

38:56destination document. Let's quickly scan

38:58for any good questions before we zoom

39:00ahead.

39:02And

39:05Rediscovering the role of software

39:06engineering today's world, top three

39:08disciplines you recommend.

39:10Um

39:11Taekwondo is good, I've heard. I've no

39:13I've no idea how to answer this

39:14question. Um

39:16thank you for asking it though. Um Top

39:18three disciplines I recommend.

39:20I mean

39:21Sorry? Plumbing. Plumbing is a good one.

39:23Yeah, yeah, yeah. I don't know if that's

39:25a discipline. I the plumbers I've hired

39:26are not usually very disciplined.

39:28Um

39:30Right.

39:32So, okay. We now have our destination,

39:34okay? Um

39:37Perfect.

39:38So, how do we actually get to our

39:40destination? How do we We have a sort of

39:42vague PRD? How do we split it so that we

39:46don't put things into the dumb zone?

39:48In other words, we have our number four,

39:50how do we split it into this kind of

39:52multi-phase plan? Well, probably what

39:54you would do at this point is you would

39:55say, "Okay, Claude, give me a

39:57multi-phase plan that gets me to this

39:59destination, right?" That sort of makes

40:00sense. This is what we've been doing

40:01before.

40:03But I have um

40:04a sort of better way of doing it now,

40:05which is that

40:08I like

40:10creating a Kanban board out of this.

40:13Raise your hand if you don't know what a

40:15Kanban board is.

40:17Mm, cool. Okay. A Kanban board is

40:19essentially just a set of tickets that

40:21you put on the wall that have blocking

40:23relationships to each other. So, we're

40:25going to see what it kind of looks like

40:26here. This is how we've worked um

40:29as developers for a long time, really

40:31since Agile came around. And what it

40:34does, we can see it here,

40:36it has proposed that we split this setup

40:39into

40:41um five different tasks here.

40:43We have the first one, which is the

40:44schema and the gamification service.

40:47Yeah, well, that looks pretty good. This

40:48is blocked by nothing.

40:50And we can even see here that it's a

40:52it's given it a type of AFK, too. You

40:54remember I talked about human in the

40:55loop and AFK earlier? This is an AFK

40:57task. This is something we can just pass

40:59off to an agent to do its thing.

41:01Streak tracking, okay, that looks good.

41:04Uh

41:05then wire points and streaks into

41:07lessons quiz completion. This is blocked

41:08by one and two.

41:10Retroactive backfill. This is blocked

41:11only by one.

41:13And then this one here is blocked by all

41:15of the tasks. Cool.

41:19Hmm.

41:20Now, I consider this you could say, "Why

41:23don't we just make this sort of

41:24generation of the issues, why don't we

41:26just hand that over to the AI? Why do I

41:28need to be involved here, right?" Cuz

41:30it's given us quite a good selection of

41:31tools here. Why do I need to review this

41:34and sort of

41:35figure out what's next?

41:37Now, my take here is that this is really

41:39cheap to do, like very quick to do once

41:42I've done the PR, and I can immediately

41:43see some issues here.

41:47There's a really, really important

41:49technique when you're kind of figuring

41:51out what the shape of this journey

41:53should look like.

41:55And

41:57it sort of comes to this very classic

42:00idea, uh which comes from the Pragmatic

42:02Programmer called traceable bullets or

42:04vertical slices.

42:07And traceable bullets really transformed

42:09the way I think about actually

42:11getting AI to pick its own tasks.

42:14Systems have layers, right?

42:17There are layers in your system.

42:19These might be different deployable

42:20units. You might have a database that

42:22lives somewhere. You might have an API

42:23that lives maybe close to the database

42:25but in a separate bit. You might have a

42:27front end that lives somewhere totally

42:28different like a CDN.

42:30Or within these deployable units, you

42:32might have different layers within

42:34those. In for instance, the code base

42:36that we're working in, we have a ton of

42:38different services. Service. We have a

42:41quiz service, a team service, a user

42:43service, coupon service, core service.

42:45And these services have dependencies on

42:47each other. So, they're kind of like

42:48individual layers.

42:50Well,

42:51what I noticed is that AI loves to code

42:55horizontally.

42:57So, it loves to code layer by layer.

43:00So, in other words, in phase one, it

43:01will do all of the database stuff, all

43:03of the schema, all of the you know, all

43:05the stuff related to that unit. Then it

43:08will go into phase two and do all of the

43:10API stuff. Then it will add the front

43:12end on top of that.

43:14Does Can anyone tell me what's wrong

43:16with that picture? Why is that not a

43:18good thing to do? Raise your hand if you

43:20have an answer.

43:21Yeah.

43:21>> have that whole feedback loop.

43:23Exactly. You don't get feedback on your

43:26work until you've

43:28really started or completed phase three.

43:32So,

43:33what you really need to do is you you're

43:34not until you get to phase three, you're

43:36not actually testing that all the layers

43:38work together.

43:41You haven't got an integrated system

43:42that you can test against.

43:44And so,

43:45instead you need to think about vertical

43:47layers. You need to think about thin

43:49slices of functionality that cross all

43:52of the layers that you need to.

43:54And this is a much better way to work,

43:57much better way for the AI to work, too,

43:59because it means at the end of phase one

44:00or during phase one it can get feedback

44:02on its entire flow.

44:04So, what this means to me

44:07is inside the PRD to issues skill up

44:11here,

44:12I have got break a PRD into

44:15independently grabbable issues using

44:17vertical slices traceable bullets

44:18written as local markdown files.

44:19[snorts]

44:21We first locate the PRD.

44:23Uh again, explore the code base if this

44:25is a fresh session. We draft vertical

44:27slices.

44:28So, we break the PRD into traceable

44:30issues. A traceable bullet, by the way,

44:32is uh

44:34essentially when you're like an

44:35anti-aircraft gunner. It's quite a

44:37violent idea, actually. Uh

44:39and you're looking up in the sky and

44:40it's night. If you're just shooting

44:42normal bullets, you have no idea what

44:44you're firing at, right? You could just

44:45be you know, you you see the plane but

44:47you don't see where your bullets are

44:48going.

44:48Traceable bullets is they attach a tiny

44:50bit of phosphorescence or phosphor or

44:52something to make it glow as it goes.

44:55So, this means that every sixth bullet

44:57or something you actually see a line in

44:58the sky. So, you have feedback on where

45:01you're aiming. So, this is what this is

45:03the idea here is that we increase our

45:05level of feedback and we get near

45:07instant feedback on what we're building.

45:09Cuz without that the AI is kind of

45:11coding blind until it reaches the later

45:12phases.

45:14We got some vertical slice rules. We

45:15quiz the user.

45:17And then we create the issue files. So,

45:20what I see here

45:21is that even though

45:23I've I've told it to do vertical slices,

45:26it's proposing to

45:29create the gamification service

45:32first on its own. That's just one slice

45:34there. And that to me feels like a

45:36horizontal slice. What I want to see in

45:38the first vertical slice especially is I

45:40want to see the schema changes or some

45:42schema changes. I want to see some new

45:45service being created and I want a

45:46minimal representation of that on the

45:48front end. So, I want it to go through

45:50the vertical slices, not just the

45:52horizontal. Does that make sense?

45:54Okay. So, I'm going to give the AI

45:57a rollicking.

45:58Uh bad boy. No, I'm not.

46:01I'm not going to waste tokens just being

46:04just naming. Um

46:06So, the first slice is too horizontal.

46:10I'll just start with that and see if it

46:11picks it up.

46:12Does that make sense as a concept?

46:14And I think having that um

46:17what I really like about going back to

46:18those old books is that we're really

46:21trying to in this day and age like get

46:24uh

46:25verbalize best software practices in

46:27English.

46:29And these books, 20-year-old books, have

46:31already done that. And it's an absolute

46:33gold mine if you want to throw that into

46:34prompts. But even with that, it's not

46:36going to um not going to do a perfect

46:38job each time.

46:39So,

46:40award points for lesson completion

46:42visible on dashboard. Yes, that's a

46:44beautiful vertical slice because it's

46:47definitely a big chunk of stuff. It's

46:48doing a lot of stories there, but we're

46:51going to see something visible at the

46:52end and the AI will then just be able to

46:54add to that. You see why that's

46:56preferable to the first one. Cool.

46:58Uh looks great.

47:01So, we're getting closer now. Anyone

47:03following at home as well, you know, not

47:05at home but you get the idea.

47:06Um will hopefully see the same thing,

47:09too, and start developing the same

47:10instincts.

47:11Let's open up for questions just while

47:13I'm still creating these GitHub issues.

47:16Uh ba ba ba ba Oh, not GitHub issues. Uh

47:18local issues.

47:20When will I stop using Windows? Never.

47:22What is your Okay, we'll get to that

47:24later.

47:25How does AI um decide when to stop

47:27grilling? Cuz AI can ask incessantly,

47:30can we have a smarter way to decide the

47:31stop point? Yeah, it does tend to really

47:34um

47:34those grilling sessions can be super

47:35intense. And the thing about these

47:37skills is you can tune them if you want

47:39to. If you feel like the AI is just

47:41absolutely hammering you, hammering you,

47:42hammering you, then you can just

47:44tell it to just pull back a little bit

47:46or get it to do, you know, stop points

47:48and that kind of thing. So, if that's a

47:49failure mode that you run into a lot,

47:51then you just, you know, change the

47:52skill.

47:55Uh do I still use uh be extremely

47:57concise, sacrifice grammar for the sake

47:58of concision? Um there was a tip that I

48:00gave folks um

48:035 months ago, which is that

48:05to basically increase the readability of

48:07your plans. So, when you're using plan

48:09mode,

48:10then you can put it in your Claude.md

48:13and you can say, "Okay, yeah, approve

48:15that."

48:17Let's open up Claude.md.

48:21Uh do I have a Claude.md? Maybe I don't.

48:23I I really don't use Claude.md very

48:24much. I'm just going to put a dummy

48:26inside here.

48:28Um when

48:30No.

48:31When talking to me,

48:33uh sacrifice grammar for the sake of

48:34concision.

48:40And this um prompt was uh really useful

48:43to me when I was reading the plans

48:45because it meant that the plans would

48:46come out and they would be very concise,

48:48really nice, easy to read, often very

48:50concise. But I've

48:53since dropped this idea in preference to

48:56a grilling session because what I

48:57noticed with it just I didn't want to

48:59read the plans. I wanted to get on the

49:01same wavelength as the LLM. I wanted it

49:03to ask aggressive questions to me. And

49:04when I stopped reading the plans, I

49:06stopped needing them to be concise.

49:08So, I think of the plans really in the

49:09destination document as uh the end

49:12state. And I don't need that end state

49:13to be concise.

49:15Hopefully that answers your question.

49:19Uh

49:20What do I think will be the outcome of

49:22the Mexican standoff of future roles of

49:23PMs and other roles converging? Uh I've

49:25no idea. I'm not a pundit. I've no idea.

49:29Uh okay.

49:31So, we should

49:33uh after a couple of approvals,

49:37uh end up with a set of issues.

49:39Now,

49:40these issues that we're creating,

49:42they're designed to be independently

49:44grabbable,

49:45which means that this Kanban board ends

49:48up looking kind of like this.

49:51Where you have

49:53essentially a set of tickets with a

49:55whole load of independent relationships.

49:57So, this one needs to be done before

49:58this one. This one needs to be done

50:00before this one.

50:01And this one, let's say we got another

50:03one over here.

50:05This one needs to be done before this

50:05one.

50:06This means that you can start to

50:09parallelize.

50:10You can start to get agents working at

50:13the same time on these tasks. Because

50:15yeah, this one needs to be done first.

50:18And then

50:19these two

50:21can be grabbed at the same time by

50:24independent agents.

50:26Raise your hand if you've done any kind

50:27of parallelization work with agents.

50:30Okay, cool. So, this allows you

50:33um to turn those plans into to optimally

50:35kind of like into a directed acyclic

50:38graphs essentially, where you just are

50:40able to um

50:42essentially have three phases here.

50:45Where you have

50:46phase one.

50:48Uh let me grab move that.

50:51Uh

50:52above this line here,

50:55you do this one.

50:56Then phase two, you do the two below it.

50:58And then phase three, you do this third

51:00one and add it onto that.

51:02And when you think about there could be

51:04This could This is a relatively simple

51:06plan, but you could have many different

51:08plans operating all at once. It means

51:10that you can do really nice

51:11parallelization. And we'll talk more

51:12about that in a bit. But that's why I

51:14prefer a Kanban board set up like this

51:18to a sequential plan. Because a

51:20sequential plan can really only be

51:21picked up by one agent.

51:24So, this

51:26Where did it go? Over here.

51:29Yeah, this plan here

51:31This is really only one loop, right?

51:33Only one agent can work on these because

51:36we have numbered phases and they're not

51:38parallelizable. Does that make sense?

51:40Cool.

51:42So, we've got our issues. Ah, come on.

51:44Stop asking me for I know it's creating

51:46them on GitHub. I really don't want

51:47that.

51:49Oh, no.

51:51You fool.

51:53Create them in issues instead.

51:57No.

51:58That's not precise enough.

52:00Uh you fool.

52:01Create them in local markdown files

52:05instead, referencing the local version.

52:11Sorry about this.

52:15So, once we get to this point,

52:17we [clears throat] have a bunch of

52:18issues locally

52:20that we can start um looping over and

52:24implementing. And it's at this point

52:26that the human leaves the loop.

52:28So, so far

52:31Let me pull up a a proper overview of

52:33this kind of flow that we're exploring

52:35here.

52:37So far

52:40we have taken an idea.

52:43I'll zoom this in a bit for the folks at

52:44the back.

52:46And we've grilled ourselves about the

52:49idea.

52:51We can skip over research and prototype,

52:52but we turn that into a PRD, into a

52:54destination document.

52:56We then turn that PRD into a Kanban

52:59board. And all of those steps

53:01are human reviewed.

53:03And now

53:05the implementation stage, we step back.

53:08And we let an agent um work through that

53:10Kanban board or multiple agents work

53:12through the Kanban board.

53:15Now, what this means is that yeah, we

53:17spent a lot of time planning here, but

53:19it means that we've queued up a lot of

53:20work for the agent. We can think of this

53:23as kind of like the day shift and the

53:24night shift. This is the day shift for

53:26the human, right? Planning everything,

53:28getting all the all the stuff ready. And

53:30then once we kick it over to the night

53:32shift, the AI can just work AFK. But

53:35what does that look like?

53:37Well,

53:39so I'm just going to Oh, yeah. Just

53:40allow it. It's perfect.

53:42So, this looks like

53:44if we head to the next exercise,

53:47which is

53:51uh in fact, the last exercise here,

53:52running your AFK agent.

53:55Now,

53:57I've called this uh Ralph really cuz it

53:59is a it is essentially a Ralph loop.

54:02And this prompt here, I want to walk

54:04through this really closely.

54:06The first thing it's doing here is we're

54:08essentially going to run Claude

54:10and we're going to basically try to

54:11encourage it to work um

54:14completely AFK.

54:16I'll show you what the sort of script

54:17for this looks like in a minute.

54:19But you say, "Okay, local issue files

54:21from issues are provided at the start of

54:22context."

54:24The way we do that is if you look inside

54:26once.sh here inside the repo,

54:29we have

54:31uh it's essentially just a bash script,

54:34where we grab all of the issues,

54:36um [clears throat] which are inside

54:38markdown files, and we cat them into a

54:40local variable. So, that issues variable

54:42contains all of the issues that are in

54:45our entire backlog.

54:47Then we grab the last five commits. I'll

54:50explain why in a minute.

54:52And then we grab the prompt and we just

54:54run Claude code with permission mode

54:56accept edits.

54:57And then just essentially just pass it

55:00all of the information.

55:02This is what the implementer looks like.

55:04So, that's what a very very simple

55:05version of this sort of loop looks like.

55:08And of course, this is not a loop. This

55:09is just running it once.

55:12The loop

55:13is in the AFK version up here,

55:15which is uh a fair bit more complicated.

55:18And the crucial part here is we're

55:20running it in Docker sandbox as well.

55:22So, I I don't want you to install Docker

55:25on your laptops because we're just going

55:26to be like, "You need to download a

55:28special image and we're going to tank

55:29the conference Wi-Fi if we do that." So,

55:31I'm I am going to demo this to you, but

55:33you um

55:34won't need to run this yourself, but

55:35I'll talk through this in a minute. But

55:37essentially, this once loop here,

55:41and ba ba ba ba boom.

55:44We're just essentially running one

55:46version of the thing that we're going to

55:48loop again and again and again. So, this

55:50is kind of like the human in the loop

55:51version. And this is essential. Running

55:54this again and again is essential

55:55because you're going to see what the

55:56agent does and see how it ends up

55:58working. And any tuning that you need to

56:01add to the prompt, then you can do that.

56:03Let's go to the prompt.

56:06Um

56:09So, local issue files are being passed

56:11in.

56:12You're going to work on the AFK issues

56:13only. That makes sense.

56:15If all AFK tasks are complete, output

56:17this no more tasks thing.

56:19And then the next thing, pick the next

56:21task.

56:23So,

56:26what we're doing here is we're

56:27essentially running a backlog or

56:30curating a backlog that our AFK agent is

56:32going to pick up. That's the purpose of

56:34all of these um setups in the beginning.

56:38In this uh

56:39all the way to this Kanban board here,

56:41we're just essentially creating a

56:43backlog of tasks for the night shift to

56:45pick up.

56:46And the night shift, this sort of Ralph

56:49prompt here, it's got its own idea about

56:52what a good task looks like to next pick

56:54up.

56:56I'm I did talk about parallelization. I

56:58will show you this later, but this is

56:59essentially a sequential loop here.

57:01We're just going to run one coding agent

57:03at a time. This is a good way to just

57:04sort of um get your feet wet

57:06essentially.

57:08So, it's prioritizing critical bug

57:10fixes, development infrastructure, then

57:12trace bullets,

57:14then polishing quick wins and refactors.

57:17And then we just have a very simple kind

57:19of instruction on how to complete the

57:20task.

57:21So, we explore the repo.

57:23Use TDD to complete the task. I'll get

57:25to that later.

57:27And

57:28we then run some feedback loops. So,

57:30let's let's just try this and let's just

57:31see what happens.

57:33So, good. It's created the issue files.

57:34We should be good to go. I'm going to

57:36cancel out of this.

57:38I'll clear and I'm going to run

57:40uh

57:41Where is it? Ralph

57:43once.sh. And you can feel free if you're

57:45following along to do the same thing.

57:48So, we can see it's just running Claude

57:50inside here

57:51with the prompt and with all of the

57:53issues that have been passed in.

57:56And while it's doing its thing,

57:59you probably have some questions about

58:01this setup and about the decisions that

58:03I've made to essentially

58:05delegate all of my coding to AI, right?

58:08So, let's let's do a quick Q&A while

58:10it's getting its feet under it.

58:14Uh okay. Ba ba ba ba ba.

58:17I'm going to just

58:19remove those.

58:23How do you retain negative decisions,

58:25things that you decided against, and

58:26rationales when persisting the results

58:28from the grill me session? Uh great

58:30question.

58:31There's a very simple answer, which is

58:33the in the PRD uh write a PRD section,

58:37there is a stuff at the bottom, a

58:39section of the things that are out of

58:40scope. So, the things we're not going to

58:42tackle in this PRD, which is very

58:44important for giving a definition of

58:45done.

58:47Feel free to ping on the Slido if you've

58:48got any more questions.

58:51Uh what's my front end workflow? Okay,

58:53it's a great question. I'm going to I'm

58:55going to answer that in a minute, I

58:56think.

58:58How to deal with agents producing more

59:00code that we can review? How to properly

59:02parallelize and use multiple agents

59:05separate way. Okay, that's That's two

59:06questions there.

59:08Um

59:09Raise your hand

59:10if you feel like you're doing more code

59:12review now than you used to.

59:16Yeah, definitely.

59:18Um

59:18I don't think there's a way to avoid

59:20this.

59:22If we delegate all of our coding to

59:25agents,

59:27you notice that the implementation here

59:29is really the only AFK bit. We then also

59:32need to QA the work and code review the

59:34work, right?

59:36And if we are

59:38running these loops where it's

59:39essentially going to implement four

59:40issues in one,

59:42it's hard to pair that with the dictum

59:45that you should keep pull requests small

59:47and self-contained, right? Like small

59:49self-contained pull requests means

59:52you're needing to do fewer loops or

59:55shorter loops or something.

59:57Or maybe you do like a big stack of PRs,

59:58but that seems horrible as well. That's

1:00:00still just more separated code to

1:00:02review. I don't honestly know what the

1:00:04answer to this yet.

1:00:06I think we just need to be ready to be

1:00:07doing more code review, essentially.

1:00:10Which is not fun. That's not fun thing

1:00:11to say. That's not like I don't know. I

1:00:13don't feel good saying that, but I do

1:00:15think it's probably the

1:00:17the way things are going.

1:00:18It's a great question.

1:00:21Uh

1:00:23Can we grab a couple of questions from

1:00:25the room as well? Let's not We won't do

1:00:27the mic, but uh raise your hand if

1:00:28you've got a question for me

1:00:29immediately.

1:00:31Yeah.

1:00:32So, the approach is very linear from an

1:00:34idea to uh QA code review. Of course,

1:00:38the real world is a lot more messy. So,

1:00:40you have all these ideas that are in

1:00:42parallel and

1:00:43nobody has the full picture. And

1:00:46uh while you're working on something,

1:00:47something else comes in as

1:00:49a bug. Yeah. How do you deal with the

1:00:50messiness? How do you tighten that

1:00:52feedback loop? Great question. So, the

1:00:54question was

1:00:55if this all looks great if you're a solo

1:00:57developer, but actually how do you

1:00:58implement this in a team? How do you

1:01:00gather team feedback on this?

1:01:02And my answer to that is that if you

1:01:04have an idea up there

1:01:06and

1:01:07essentially the sort of journey from the

1:01:10idea to the destination

1:01:12is something you need to figure out with

1:01:13the team, right? So, all of this stuff

1:01:16up here, this is kind of like team

1:01:17stuff, you know what I mean? This So, if

1:01:20you have an idea and you do a grilling

1:01:22session on it and you have a question

1:01:23that you don't know how to answer, then

1:01:25you need to loop in your team as we

1:01:27described before. Then you might need to

1:01:29go, "Okay, like we just need to build a

1:01:30prototype of this. We need to actually

1:01:32hash this out. We need something that

1:01:33the domain experts can fiddle with."

1:01:36Or okay, we might need to integrate a a

1:01:38third-party library into this. We might

1:01:39need to do some research. We might need

1:01:41to actually kind of like um

1:01:44ping this back and forth and find a

1:01:45third-party service that we can get the

1:01:46most out of. We might need to go back

1:01:49with the information that we gathered

1:01:50there to the idea phase. So, all the way

1:01:53up to the sort of PRD in the journey,

1:01:55that's something you need to involve

1:01:56your team with. That's something where

1:01:58these assets are going to be shared over

1:02:01and you're going to have requests for

1:02:02comments on them and that that loop is

1:02:05going to just keep grinding and grinding

1:02:07until you figure out where you're going.

1:02:09Once you figure out where you're going,

1:02:11then you can start doing the Kanban

1:02:12board implementation. But this is

1:02:14essentially super arguable and the

1:02:16you'll be bouncing back and forth

1:02:17between the phases. Does that make

1:02:18sense? Yeah.

1:02:20Would you not need a

1:02:21PRD for your prototype?

1:02:23Say again, sorry. Would you not want to

1:02:24have a PRD for your prototype? The

1:02:26question was, do you want to go through

1:02:27this whole session just to sort of

1:02:29create a prototype? You don't need a PRD

1:02:31for your prototype as well. Let's just

1:02:33quickly talk about prototypes for a

1:02:34second.

1:02:35Um there was a question about how do you

1:02:36make this work for front end?

1:02:39Like how do you cuz front end is like

1:02:41really sensitive to human eyes. You need

1:02:43human eyes looking at the front end all

1:02:45the time to make sure that it looks

1:02:47good.

1:02:48AI doesn't really have any eyes. It can

1:02:51look at code,

1:02:52but it front end is multimodal.

1:02:55And so my experiences with trying to

1:02:58plug AI into um let's say agent browser

1:03:02or Playwright MCP to give it

1:03:04You can give it tools to allow it to

1:03:06look through a front end and sort of

1:03:07look at images, but in my experience the

1:03:10um it's not very good at that yet and it

1:03:12can't create a nice front end in a

1:03:15mature code base. It can sort of spit

1:03:17one out. But what it can do is you say,

1:03:20"Okay, uh I want some ideas on how uh

1:03:22this front end might look. Give me three

1:03:24prototypes um that I can click between

1:03:27in a throwaway uh

1:03:29throwaway route that I can decide which

1:03:31one looks best." And you take the asset

1:03:33of that prototype and you then feed it

1:03:35back into the grilling session or you

1:03:37get feedback on it, blah blah blah blah

1:03:38blah.

1:03:39Answer your question kind of thing?

1:03:41The prototype is just, you know, it's

1:03:42messy. It's supposed to give you

1:03:44feedback earlier on the process.

1:03:46So, that's a great way of working with

1:03:47front end code, great way of looking at

1:03:48software architecture in general. Let's

1:03:50go one more question here. Yes.

1:03:52>> [clears throat]

1:03:52>> In your system, how do you integrate

1:03:54respecting an architecture and design

1:03:57with API contracts and fitting with your

1:03:59larger system?

1:04:01Uh security constraints, all kinds of

1:04:03constraints like that.

1:04:04Yeah.

1:04:05There's a lot in that question. The

1:04:07question was, how do you conform with

1:04:08existing architecture? How do you do um

1:04:12how do you make it conform to the code

1:04:13standards

1:04:14like of your code base or Yeah, the

1:04:17architecture design APIs, Yeah. security

1:04:19rules that constrain your design. Yeah.

1:04:23I'm going to answer that in a bit.

1:04:25That's okay.

1:04:26So, hopefully we have started to get

1:04:28some stuff cook cooking. Uh it's just

1:04:32pinging on the explore phase here.

1:04:36Hmm, tempted to just start running it

1:04:38AFK.

1:04:40Maybe I will, maybe I won't.

1:04:43Um

1:04:44What it's essentially doing is it's

1:04:45exploring the repo. It's going to then

1:04:47start implementing based on what we

1:04:48wanted.

1:04:49Let's actually have one more question

1:04:50just while it's running. Yeah.

1:04:52Why not AI

1:04:54QA everything

1:04:58Yeah.

1:04:59So, the question was, why do you not get

1:05:02AI to QA?

1:05:05AI to QA.

1:05:06I just got uh jargon overload for a

1:05:08second. Um why do you not get AI to uh

1:05:11test its own code? Now, of course, you

1:05:13absolutely can. And I think while it's

1:05:16doing while it's cooking here,

1:05:18okay, it's got a clear picture of the

1:05:19code base. It's assessing the issues.

1:05:22It's doing issue 02 as the next task.

1:05:24I'm again going to show you that in a

1:05:25bit, I think. The sort of uh cuz you

1:05:28definitely should do an automated review

1:05:31step as part of implementation.

1:05:33So, you have your implementation, you

1:05:35should then, because tokens are pretty

1:05:37cheap and AI is actually really good at

1:05:38reviewing stuff, you should get it to

1:05:40review its own code before you then QA

1:05:42it.

1:05:43I found that that catches a ton of

1:05:44different bugs

1:05:46and

1:05:47the way that works is I will just do a

1:05:50little diagram is if you have, let's

1:05:52say, an implementation that sort of like

1:05:54used up a bunch of tokens in the smart

1:05:56zone,

1:05:57if you get it to sort of try to

1:06:00do its reviewing, it's going to be doing

1:06:01the reviewing in the dumb zone.

1:06:05And so, the reviewer will be dumber than

1:06:06the thing that actually implemented it.

1:06:08If we imagine this is the

1:06:11uh let's be consistent. That's the

1:06:12review.

1:06:13That's the implementation.

1:06:15Whereas if you clear the context,

1:06:19then

1:06:21you're essentially going to be able to

1:06:22just review in the smart zone, which is

1:06:24where you want to be.

1:06:27Let's see how our implementation is

1:06:28doing.

1:06:29Okay, good. It's generating a migration.

1:06:31That looks pretty nice.

1:06:32We're getting some code spitting out.

1:06:37And

1:06:38while I'm sort of like Aha, here we go.

1:06:42TDD.

1:06:43Let's talk about TDD and then I think

1:06:45we'll have a little another little

1:06:46break.

1:06:48TDD I found is absolutely essential for

1:06:51getting the most out of agents. Uh raise

1:06:53your hand if uh you know what TDD is.

1:06:56Cool. Okay. TDD is test-driven

1:06:58development. What it's essentially doing

1:07:00is it's doing a something called red

1:07:03green refactor. And if you look in the

1:07:05code base, you'll be able to find a um a

1:07:07skill which really describes how to do

1:07:10red green refactor and teaches the AI

1:07:12how to do it.

1:07:13So, what it's doing is it's writing a

1:07:15failing test first. So, it's saying,

1:07:18"Okay, I've broken down the idea of what

1:07:20I'm doing and I'm just going to write a

1:07:22single test that fails and then I need

1:07:25to make the implementation pass."

1:07:27I have found that

1:07:30first of all, this adds tests to the

1:07:31code base and these this tends to add

1:07:33good tests to the code base. And so,

1:07:35we've got this kind of gamification

1:07:37service.

1:07:38It looks like it's

1:07:39using some existing stuff to create a

1:07:41test database. Test fails because the

1:07:43module doesn't exist yet. Okay, we've

1:07:45confirmed red. And then it goes and

1:07:48hopefully runs it and it passes.

1:07:51I found that uh raise your hand if

1:07:54you've ever had AI write bad tests.

1:07:58Yeah.

1:07:59It tends to try to cheat at the tests

1:08:01because it's sort of doing it in layers.

1:08:03It will do the entire implementation and

1:08:05then it will do the entire test layer

1:08:07just below it.

1:08:08Uh

1:08:09I'm just going to say yes, you're

1:08:10allowed to use NPX V test.

1:08:12And using this technique, it generally

1:08:15is a lot harder to

1:08:18cheat because it's

1:08:20sort of instrumenting the code before

1:08:22it's then writing the code. So, I find

1:08:24that TDD is so so good for places where

1:08:28you can pull it off. In fact, it's so

1:08:29good that I sort of warped my whole uh

1:08:32technique around getting TDD to work

1:08:34better.

1:08:35I can see some dripping eyes. It is so

1:08:37hot in here.

1:08:38You can't imagine how hot it is up here.

1:08:40Let's take another 5-minute comfort

1:08:41break. Let's come back at quarter to, I

1:08:45think. Have a nice generous one.

1:08:47And we'll be back in about 6 7 minutes

1:08:50and I'll talk about how

1:08:52uh I think about modules, think about

1:08:54constructing a code base to make this

1:08:55possible.

1:08:57I've just been sort of fiddling with the

1:08:58AI here and we have ended up with some

1:09:00with a commit.

1:09:02So, we have something to test. Issue

1:09:04number two is complete. Here's what was

1:09:06done.

1:09:07This is kind of what it looks like when

1:09:09a Ralph loop completes is you end up

1:09:10with a little summary.

1:09:12Um and we have now something we can QA.

1:09:15Because we did the feedback loops

1:09:17because we did the trace bullets because

1:09:19we were uh said, "Okay, give us

1:09:21something reviewable at the end of

1:09:22this." We can immediately go and QA it.

1:09:24Now, there's nothing uh less exciting

1:09:26than watching someone else QA something.

1:09:29But, hopefully we can have a little

1:09:30play.

1:09:31Let's just check that it uh works at

1:09:33all.

1:09:34In fact, before I go there, I just want

1:09:36to sort of work through what just

1:09:38happened.

1:09:39Which is we see that it's created some

1:09:42stuff on the dashboard.

1:09:45And it then ran the feedback loops. So,

1:09:47it then ran the tests and the types.

1:09:51Now, TDD is obviously really important.

1:09:53And it's really important because these

1:09:55feedback loops are essential to AI,

1:09:58essential to get AI to produce anything

1:10:01reasonable.

1:10:02Because without this, AI is totally

1:10:04coding blind, right?

1:10:06You have to have to um

1:10:09If if your code base doesn't have

1:10:10feedback loops, you're never ever ever

1:10:13going to get decent AI decent output out

1:10:15of AI. And often what you'll find is

1:10:18that the quality of your feedback loops

1:10:21influences how good your AI can code,

1:10:24essentially. That is the ceiling. So, if

1:10:26you're getting bad outputs from your AI,

1:10:28you often need to increase the quality

1:10:30of your feedback loops.

1:10:32We'll talk about how to do that in a

1:10:33minute.

1:10:35Now, so it ran NPM run test, NPM run

1:10:39type check. It got one type error, and

1:10:41it needed to fix it with a nice bit of

1:10:43TypeScript magic. Very good. Yeah, type

1:10:45of level threshold number. Okay.

1:10:48Uh you see why I stopped teaching

1:10:50TypeScript cuz just AI knows everything

1:10:51now.

1:10:52Um

1:10:54So, and it ran the tests, and it passed,

1:10:57and it's looking good. So, we now end up

1:10:58with 284 tests in this repo. Pretty

1:11:01good.

1:11:03I I do find uh front end really hard to

1:11:06test here. We're essentially just

1:11:07testing the service. So, we've created a

1:11:09gamification service, if we look up

1:11:11here.

1:11:13And then we have a test for that

1:11:14service. You can see that the service

1:11:16and the test itself.

1:11:17Now, if I was doing code review here, I

1:11:19would then go to I would first go to

1:11:21review the tests, make sure the tests

1:11:23were testing reasonable things,

1:11:25and then go and kind of review the code

1:11:28itself just to make sure that it's it's

1:11:30not doing anything too crazy, right?

1:11:32The essential thing is I need to

1:11:33actually um look at the dashboard.

1:11:36I'm going to log in as a student.

1:11:40Oh, if it'll let me. Maybe it won't let

1:11:42me.

1:11:43Come on, son. There we go.

1:11:45Let's log in as Emma Wilson.

1:11:47Head into courses.

1:11:49Uh let's say I've got an introduction to

1:11:50TypeScript.

1:11:52Continue learning.

1:11:54Uh yes, I completed this lesson.

1:11:57And something went wrong. I imagine it's

1:11:59because I don't have

1:12:02Uh SQLite error. I don't have the right

1:12:05table. So, I need a table point events.

1:12:08Point events is a strange table name.

1:12:09I'm not sure quite what it was thinking

1:12:10there.

1:12:11Uh let's suspend. Let's run uh NPM DB

1:12:15migrate.

1:12:17Push, I think.

1:12:19I can't remember which one it was.

1:12:21But, you kind of get the idea, right? I

1:12:23I'm not going to subject you to uh

1:12:24watching me do QA because it's so dull.

1:12:27Um but at this point, I would

1:12:29essentially go back in. I would um

1:12:31Let me open the project back up.

1:12:35Uh and I would

1:12:36This This is a crucial moment, um and

1:12:39it's so important to um

1:12:41QA it manually here because QA Oh, dear,

1:12:45oh dear. What's going wrong? There we

1:12:46go.

1:12:47QA is how I then um impose my

1:12:51uh

1:12:52opinions back onto the code base, how I

1:12:54impose my taste.

1:12:56What you'll often find is that um there

1:12:58are teams out there who are trying to

1:12:59automate everything, like every part of

1:13:02this process. And they will tend to

1:13:06uh if you try to like automate the sort

1:13:08of creation of the idea, automate

1:13:11uh the QA, automate the research,

1:13:12automate the prototype, you end up with

1:13:15uh apps that I feel just lack taste

1:13:19and are bad.

1:13:21Maybe they just don't work, or they they

1:13:23don't even work as intended, or there's

1:13:25just no

1:13:26You need a human touch when you're

1:13:28building this stuff because without

1:13:29that, you just end up with slop.

1:13:32And we are not producing slop here.

1:13:33We're trying to produce high-quality

1:13:34stuff, and so that's what the QA is for.

1:13:37Mhm.

1:13:39So, I'm going to do two things in this

1:13:41final section.

1:13:43Which is I'm going to first tell you how

1:13:45to

1:13:46There's probably a question in your mind

1:13:48here, which is let's say I have a code

1:13:50base that I'm working on.

1:13:52And it's a bad code base. It's a code

1:13:54base that's like really complicated, uh

1:13:57that AI just never does good work in,

1:13:59and maybe actually most humans that go

1:14:01into that code base don't do good work.

1:14:03How what How do I improve that code

1:14:05base?

1:14:06And the second thing is I'll show you my

1:14:07setup for parallelization.

1:14:10So, let's go with um

1:14:12bad code first.

1:14:14Now,

1:14:16where is it? Where's the diagram? Here

1:14:17it is.

1:14:19In his book, um The Philosophy of

1:14:21Software Design,

1:14:23John Ousterhout talks about

1:14:25the ideal type of module.

1:14:28And let's imagine that you have a code

1:14:30base that looks like this. Each of these

1:14:32uh blocks here are individual files.

1:14:35And these files

1:14:36export things from them. You know, they

1:14:38have um things that you pull from the

1:14:40files that you then use in other things.

1:14:42And so, you might have these weird

1:14:43dependencies where this file over here

1:14:45might rely on this file, or might rely

1:14:47on that file, for instance.

1:14:49Now, if these files are small and they

1:14:51don't kind of ex- like

1:14:54export many things, then John Ousterhout

1:14:56would call these shallow modules,

1:14:58essentially. Where they're not very um

1:15:02They kind of look like uh this, if I No,

1:15:05actually no. I can't can't make a good

1:15:06diagram of it.

1:15:07They're essentially lots and lots of

1:15:09small chunks. Now, this is hard for the

1:15:11AI to navigate

1:15:13cuz it doesn't really understand the

1:15:14dependencies between everything. It

1:15:15can't work out where everything is. You

1:15:17know, it has to sort of manually track

1:15:19through the entire graph and go, "Okay,

1:15:20this relies on this. This one relies on

1:15:22this one. This one relies on this one."

1:15:26And it's then also hard to test this, as

1:15:28well, because where do you draw your

1:15:29test boundaries here?

1:15:31Do you test each module individually?

1:15:35Like just literally draw a test boundary

1:15:36No, don't do that.

1:15:38Around this one?

1:15:40And then maybe another test boundary

1:15:41around the next one, and then the next

1:15:43one?

1:15:45Or should you sort of do big groups of

1:15:48it? Should you say, "Okay, we're going

1:15:49to test all of these related modules

1:15:51together, and just sort of, you know,

1:15:53hope and pray that they work."

1:15:57Now,

1:15:58>> [sighs]

1:15:58>> this means that if I think that bad

1:16:00tests mostly look like that, where the

1:16:04AI essentially tries to sort of wrap

1:16:06every tiny function in its own test

1:16:08boundary, and then just sort of test

1:16:10that those individually work. But, what

1:16:12that does is it means that when, let's

1:16:15say, this module over here calls those

1:16:17two,

1:16:19so it depends on both of these, then

1:16:21this module might miss order the

1:16:23functions, or there might be sort of

1:16:24stuff inside that poor module that's

1:16:27worth testing on its own. And if you

1:16:29then wrap this in a test boundary, what

1:16:31do you do? Do you mock the other two

1:16:32modules? How does that work?

1:16:36So, actually figuring out how to um

1:16:40build a code base that is easy to test

1:16:43is essential here. Because if our code

1:16:46base is easy to test, then our code our

1:16:48feedback loops are going to be better,

1:16:50and the AI is going to do better work in

1:16:52our code base. Does that make sense?

1:16:54So, what does a good code base looks

1:16:55like look like?

1:16:57Well, not like that.

1:17:00It looks like this.

1:17:02Where you have

1:17:05what John Ousterhout calls deep modules.

1:17:07Modules that have a little interface on

1:17:09there that expose a small, simple

1:17:11interface that have a lot of

1:17:13functionality inside them.

1:17:16Now,

1:17:18what this means is that these are easy

1:17:20to test cuz you just Let's say that

1:17:22there's a dependency between this one

1:17:23and this one.

1:17:25My arrow working? Yeah, there we go.

1:17:28Then,

1:17:30what you do is you just wrap a big test

1:17:32boundary around that one module, around

1:17:34this one up here,

1:17:35and you're going to catch a lot of good

1:17:37stuff.

1:17:40Because there's lots of functionality

1:17:41that you're testing, and really the

1:17:43caller, the person calling the module,

1:17:45is going to have a simple interface to

1:17:47work from. So, it's not not too tricky.

1:17:50That makes sense? Deep modules versus

1:17:51shallow modules. This is good.

1:17:54This shallow version is bad. And what I

1:17:56find is that unaided

1:17:59um or if you don't

1:18:02uh

1:18:04if you don't watch AI carefully, it's

1:18:05going to produce a code base that looks

1:18:07like this.

1:18:08So, you need to be really, really

1:18:09careful when you're directing it.

1:18:11And that's why, too,

1:18:13is that if we look inside the PRD,

1:18:16uh where is the PRD gone? It's inside

1:18:18the issues. It's inside the gamification

1:18:20system.

1:18:21Uh not found. Of course, it's not. Here

1:18:23it is.

1:18:25Then I have

1:18:27uh inside here

1:18:29data model the modules.

1:18:31So, it's specifically saying, "Okay,

1:18:33this gamification service is a new deep

1:18:36module, which we're going to test

1:18:37around.

1:18:38It's going to have this particular

1:18:40interface.

1:18:42And it's going to have um Okay, we're

1:18:44modifying the progress service, too.

1:18:46We're modifying the lesson route. We're

1:18:47modifying the dashboard route, etc. So,

1:18:50it's I'm being really specific about the

1:18:51modules that I'm editing, and I'm making

1:18:53sure that I keep that module map in my

1:18:56mind at all times, throughout the

1:18:57planning, and then throughout the

1:18:59implementation. Does that make sense?

1:19:01Very, very useful.

1:19:03It's useful for one other reason, too.

1:19:04Not only does it make your app more

1:19:05testable,

1:19:07but you get to do a little mental trick.

1:19:11And I'm going to refill my water while

1:19:13you wait for what that is.

1:19:17Uh let me

1:19:20Let me get a question from you guys. So,

1:19:21raise your hands if you feel like

1:19:26Uh if you feel like you're working

1:19:28harder than ever before with AI.

1:19:32Yeah.

1:19:33Uh raise your hands if you feel like you

1:19:36know your code base less well

1:19:38than you used to.

1:19:40Yeah.

1:19:43This is a real thing. Um

1:19:45because we're moving fast, because we're

1:19:46delegating more things, we end up losing

1:19:49a sense of our code base. And if we lose

1:19:52the sense of our code base, we're not

1:19:54going to be able to improve it, and

1:19:56we're essentially delegating the shape

1:19:57of it to AI.

1:19:59I [snorts] don't think that's good. But

1:20:00then how do we

1:20:03how do we make it so that we can move

1:20:04fast while still keeping enough space in

1:20:06our brains?

1:20:08I think that this is a way to do it.

1:20:10Because what you're doing here is not

1:20:12only are you thinking about creating big

1:20:15shapes in your code base, big services.

1:20:19What I think you should do is

1:20:21design the interface for these modules,

1:20:24but then delegate the implementation.

1:20:27In other words, these modules can become

1:20:28like gray boxes, where you just need to

1:20:31know the shape of them, you need to know

1:20:33what they do, and it's sort of how they

1:20:34behave, but you can delegate the

1:20:36implementation of those modules. I found

1:20:38this is really nice. I don't necessarily

1:20:40need to code review everything inside

1:20:42that module. I don't necessarily need to

1:20:43know everything of what it's doing. I

1:20:45just need to know that it behaves a

1:20:47certain way under certain conditions,

1:20:49and that it does its thing. So, it's

1:20:50kind of like

1:20:52okay, I've got a big overview of my code

1:20:54base, and I understand kind of the

1:20:55shapes inside it, understand what the

1:20:57interfaces all do, but

1:20:59I can delegate what's inside.

1:21:01I found that has been a really nice way

1:21:03to retain my sense of the code base

1:21:06while preserving my sanity.

1:21:08Make sense?

1:21:12And so, you might ask, how do I take a

1:21:14code base

1:21:16that looks like this

1:21:17and then turn it into a code base that

1:21:19looks like this? How do I deepen the

1:21:21modules?

1:21:23Well, we have Hopefully, it's in here.

1:21:25Pretty sure it is. We have a skill.

1:21:28And that skill is called improve code

1:21:30base architecture.

1:21:32Nice and direct.

1:21:35Uh let's run it.

1:21:37What this skill is going to do is it's

1:21:38essentially just going to do it a scan

1:21:40of our code base and looking for what's

1:21:42available here. And feel free to run

1:21:43this yourself if you're um

1:21:45uh

1:21:46running the exercises.

1:21:48And it's exploring the architecture,

1:21:50exploring um

1:21:51essentially how to work within this code

1:21:53base, and it's going to attempt to

1:21:57uh find places to deepen the modules.

1:22:00Pretty simple. One really cool um thing

1:22:04that it found here is part of my uh part

1:22:07of my course video manager app is a

1:22:09video editor. A video editor built in

1:22:11the browser, which is really hardcore.

1:22:13Uh it's a decent bit of engineering. And

1:22:16I wanted a way that I could wrap the

1:22:18entire front end all the way to the back

1:22:21end in like a single big module, so that

1:22:23I could test the fact that I press

1:22:24something on the front end and it goes

1:22:26all the way to the back end. And so, I

1:22:28found a way essentially by using a kind

1:22:30of discriminated union between the two

1:22:32types here by sort of I was able to use

1:22:35this uh skill to essentially have a huge

1:22:39great big module that just tested from

1:22:41the outside, it was testable from the

1:22:43outside, this video editor

1:22:44infrastructure. And it meant that AI

1:22:46could see the entire flow, could act on

1:22:49the entire flow, and test on the entire

1:22:50flow. And honestly, it was just night

1:22:53and day in terms of the uh ability of AI

1:22:56to actually make changes, cuz AI working

1:22:58on a video editor is pretty brutal if

1:23:00you don't give it good tests. So, that

1:23:02is

1:23:03Honestly, I

1:23:04If you take one thing away from today,

1:23:05just try running this skill

1:23:07on your repo and see what happens.

1:23:09Let's go to Slido. Let's ask a

1:23:11check a couple of questions as well this

1:23:13is running.

1:23:15So, let's see. Have you tried Claude's

1:23:17auto mode with Claude enable auto mode?

1:23:19That way you can avoid many of the

1:23:20obvious permission checks. We'll talk

1:23:21about permission checks in a second.

1:23:23Do I keep the markdown plans and issues

1:23:26for later reference?

1:23:28Okay.

1:23:29This is a great question.

1:23:31So,

1:23:34let's say

1:23:35that you uh have a great idea, you turn

1:23:38it into a PRD,

1:23:40raise and you then implement that PRD,

1:23:43and the PRD is essentially done.

1:23:45Raise your hand if you keep that

1:23:47information in the repo, so you turn it

1:23:49into a markdown file. Raise your hand if

1:23:50you want to keep that around.

1:23:53Cool. Okay. And raise your hand if you

1:23:55if you don't want to keep it around. If

1:23:57you want to get rid of it as soon as

1:23:58possible. Yeah, this is I think an

1:24:02a question that doesn't have a clear

1:24:03answer.

1:24:05What I'm really scared of

1:24:08with any documentation decision is that

1:24:11let's say that we have a PRD for this

1:24:13gamification system, we keep it in the

1:24:14repo.

1:24:15We go on, go on, go on. Let's say a

1:24:17month later, we want some edits to the

1:24:19gamification system.

1:24:21And we go in with Claude, and it finds

1:24:23this old PRD and says, yes, I found the

1:24:25original documentation for the PRD

1:24:27system.

1:24:28Well, it turns out that the actual code

1:24:29has changed so much from the original

1:24:31PRD that it's almost unrecognizable. The

1:24:33names of things have changed, the um

1:24:35file structure has changed, even the

1:24:37requirements may have changed. We might

1:24:38have actually tested it with users. This

1:24:40is doc rot, where the documentation for

1:24:43something is rotting away in your repo

1:24:46and influencing Claude badly. Or Claude,

1:24:49agents badly.

1:24:50So, I tend to not keep it around. I tend

1:24:53to get rid of it. And for me, because my

1:24:56setup uses GitHub issues, I just mark it

1:24:58as closed. It can fetch it if it wants

1:25:00to, but it's got a visual indicator that

1:25:02it's done. So, I tend to prefer

1:25:05ditching these.

1:25:07Thoughts on the BEADS framework from

1:25:08Steve. Uh I've not tested it, but it

1:25:10seems like sort of um another way to

1:25:13manage Kanban boards and issues. Seems

1:25:15uh very good, but I've not tried it.

1:25:18Um

1:25:20>> [clears throat]

1:25:22>> Uh let me just quickly check the uh

1:25:24setup here.

1:25:26Let's take a couple of questions from

1:25:27the room. Anybody got any questions at

1:25:29this point about anything that we've

1:25:30covered so far, especially this last

1:25:32bit? Yes.

1:25:33I thought it was

1:25:35interesting your answer about like the

1:25:36markdown files that you delete because

1:25:38they

1:25:39create like doc rot.

1:25:41How about migrations? Like with

1:25:43migration files, would you also squash

1:25:45them after that?

1:25:47Like database migrations? Yeah.

1:25:51I don't know.

1:25:53I hope that answers your question. I'm

1:25:54so sorry. No, no. I think database

1:25:56migrations are a different thing because

1:25:57you have a sort of running record of

1:25:59exactly what changed, and it's more

1:26:00deterministic. And I think

1:26:04Yeah, it's an interesting analogy. I'm

1:26:06not sure. Let's talk about it

1:26:07afterwards.

1:26:08That's a good way of saying I've no

1:26:10idea.

1:26:11Yeah. Yeah. So, you mentioned that you

1:26:12don't delete the PRD. You mentioned you

1:26:14don't review the PRD once it's done.

1:26:16Sorry, guys. Um I'm just trying to

1:26:17listen to this guy's question. Have you

1:26:18considered

1:26:19uh using a deep think like ChatGPT or

1:26:21something

1:26:25to tell it, "Look at this PRD and tell

1:26:26me if it

1:26:29It takes about an hour.

1:26:30Yeah, the question

1:26:32The question here is um

1:26:35should I um in the sort of early

1:26:37planning stage be trying to optimize the

1:26:39plan?

1:26:40This is something I actually see a lot

1:26:41of people doing, and it's a really good

1:26:43um

1:26:44idea. So, when you

1:26:49Let's go back to the phases.

1:26:51So, let's say that you have all of these

1:26:52phases here.

1:26:55And you

1:26:56uh you get to the point where you've

1:26:58sort of figured out everything with the

1:26:59LLM, you understand where you're going,

1:27:01you've created this sort of uh journey

1:27:03destination documents here. How do you

1:27:05then

1:27:06uh

1:27:08Like should you then try to optimize and

1:27:10optimize and optimize that PRD until

1:27:12it's the perfect PRD you can possibly

1:27:13imagine?

1:27:14I don't think there's a lot of value in

1:27:16that.

1:27:17Because I think the journey is really

1:27:20just sort of a hint of where you want to

1:27:21go, and the place that you need to be

1:27:24putting the work is in QA.

1:27:26And you can sort of do that AFK, I

1:27:28suppose, but in my experience, you're

1:27:29not going to get a lot of juice out of

1:27:31it. Like it's the

1:27:33The thing that really matters is getting

1:27:34alignment with the AI, which is you do

1:27:37in the grilling session initially.

1:27:40Let's have one more question. Anyone got

1:27:41any more? Yeah. How do you get in in

1:27:43your workflow to get it to code the way

1:27:46you want it to code it so by the time

1:27:48you get to code review, it's at least

1:27:49familiar, it uses the libraries you

1:27:51wanted to use, Yeah. Um we had this

1:27:53question before, actually, which was

1:27:54like uh how do you uh enforce your

1:27:57coding standards on the agents,

1:27:59essentially? How do you get it to code

1:28:01how you want it to code?

1:28:02Now, there's essentially two different

1:28:04ways of doing it.

1:28:05Um you've got

1:28:08I don't know. Come on. Push.

1:28:11And you've got pull.

1:28:14What do I mean mean by push and pull?

1:28:17Um

1:28:18Push is where you push instructions to

1:28:20the LLM.

1:28:22So, you say, okay, if you put something