Full transcript
0:07[music]
0:14>> Hello everyone. Having a good conference
0:16so far? Yeah. Are you having a good
0:18conference so far? Yeah. Good.
0:20Wonderful.
0:22I have a message for you that I hope
0:24will be um a comforting message for
0:27folks who believe that uh
0:29their skill set is no longer worth
0:31anything in this new age, which is I
0:33believe that software fundamentals
0:35matter now more than they actually ever
0:37have.
0:39And
0:40I'm a teacher,
0:42and I've been recently teaching a course
0:44called Clojure Code for Real Engineers.
0:47Nice and provocative. And
0:49in the process of kind of working on
0:50this course, I had to come up with a
0:52curriculum about
0:54AI coding, which is a bit of a nightmare
0:57because things are changing all the
0:59time, right? AI is a whole new paradigm.
1:02We need to chuck out all of the old
1:03rules, surely, so that we can bring in
1:05the new stuff.
1:08And there's
1:09a kind of movement that has come up
1:12around this, which is the specs-to-code
1:14movement. And the specs-to-code movement
1:16says that, "Okay, you can write a
1:18specification about how an application
1:19is supposed to work. Then you can use AI
1:21to turn it into code. If there's a
1:23problem with the application, you then
1:26go back to the spec. You don't really
1:27look at the code. You just change the
1:29spec, you run the compiler again, and
1:32you end up with more code." Raise your
1:34hand if you've heard of that.
1:37Keep your hand raised if you've tried
1:38it.
1:39Okay, I've tried it, too. You can put
1:41your hands down.
1:42And what I noticed was I would run it,
1:45and I would try not to look at the code,
1:48but I would look at the code, and I
1:50realized I would get code out, first of
1:51all, and then I would run it, I would
1:53get worse code. And I did it again, I
1:55got even worse code. I got it again, I
1:58kept running the compiler, kept running
1:59the compiler, and I would just end up
2:00with garbage.
2:03You know, raise your hand if that's
2:04happened to you.
2:06Yes. I don't think this works. The idea
2:09that we can just ignore the code and
2:10just have the code let it manage itself
2:13is just sort of vibe coding by another
2:14name.
2:16And I didn't believe that back then. I
2:18thought, "Okay, how do I fix the
2:20compiler? How do I make it so that it
2:22doesn't produce bad code each time, or
2:23worse code?"
2:25And so I thought, "Okay, I need to
2:26explain to the LLM in English what a
2:30good code base looks like." Let me dig
2:32out one of my old favorite books, which
2:34is a Philosophy of Software Design by
2:36John Osterhout.
2:37Go on Amazon, get it.
2:39Um
2:40and he has a definition for what bad
2:43code looks like.
2:45He calls it complex code. Complexity is
2:47anything related to the structure of a
2:48software system that makes it hard to
2:50understand and modify the system, right?
2:53So a a bad code base is a code base
2:55that's hard to change. If you can't
2:58change a code base without causing bugs,
3:00then it's a bad code base. Good code
3:01bases are easy to change.
3:04So I thought, "Ooh, that was good.
3:05Let's try another book. Let's try The
3:06Pragmatic Programmer."
3:08Go on Amazon, get it.
3:10They have a whole chapter on something
3:12called software entropy. And this is
3:14exactly what I was seeing. Entropy is
3:16the idea that things tend towards um
3:19disaster and uh floating away from each
3:21other and collapse. And this is exactly
3:23how most software systems behave, too,
3:25is that every time you make a change to
3:27a code base, if you're only thinking
3:28about that change and not thinking about
3:30the design of the whole system, your
3:32code base is going to get worse and
3:34worse and worse. And that's what I was
3:35seeing.
3:36Everything inside the specs-to-code idea
3:39that you just run the compiler again and
3:40again was making worse code.
3:43Now, there's an idea that sort of drives
3:46the specs-to-code movement,
3:47which is that code is cheap. Raise your
3:50hand if you've heard that phrase before,
3:51that code is cheap. Yeah.
3:55Well, I don't think this is right.
3:57I think code is not cheap. In fact, bad
4:00code is the most expensive it's ever
4:02been.
4:03Because if you have a code base that's
4:04hard to change, you're not able to take
4:07all of the bounty that AI can offer, cuz
4:10AI in a good code base actually does
4:12really, really well.
4:15And this means good code bases matter
4:16more than ever, which means software
4:18fundamentals matter more than ever.
4:20That's the thesis of this talk.
4:22So let's actually get into practical
4:23stuff.
4:25I'm going to talk about different
4:26failure modes that you may have
4:27experienced, or you may not have
4:28experienced yet with AI, and how you can
4:30avoid them by just going back to old
4:32books and looking at good software
4:34practices. Sound good?
4:36So the first one is that the AI didn't
4:38do what I wanted.
4:40You know, I I thought I had a good idea
4:42in my head, and the AI just did
4:43something totally different, or it did
4:45some uh like specs that I, you know, it
4:47just made something I didn't want. Raise
4:49your hand if you've hit this mode.
4:51Cool. Okay.
4:53Well,
4:54this is what they say in The Pragmatic
4:55Programmer, is that no one knows exactly
4:57what they want. It's that you and the
5:00AI, there is a communication barrier
5:02there, right?
5:04And so when you're talking to the AI,
5:05that's kind of like the AI doing its
5:07requirements gathering. It's basically
5:09working out from you what it is that you
5:11need.
5:12And
5:14I realized that there was another book,
5:16Frederick P. Brooks' The Design of
5:17Design,
5:19and it talks about this idea called the
5:20design concept.
5:22It's that when you have more than one
5:23person designing something together, you
5:25have this idea sort of floating between
5:28you, this ephemeral idea of the thing
5:30that you're building. And that thing
5:32that you're building, or the idea of it,
5:34is called the design concept. It's not
5:36an asset, it's not something you can put
5:37in a markdown file, it is the invisible
5:40sort of
5:41theory of what you're building.
5:44And so I thought, "Okay, that's what's
5:46going on. Me and the AI don't share a
5:48design concept." So I came up with a
5:50skill.
5:51The skill is very, very simple. It's
5:53called Grill Me,
5:54and it looks like this.
5:57"Interview me relentlessly about every
5:58aspect of this plan until we reach a
6:01shared understanding. Walk down each
6:03branch of the design tree, which is
6:05another thing from Frederick P. Brooks,
6:07resolving dependencies between decisions
6:09one by one."
6:10This skill is like uh the repo
6:12containing this skill has like 13,000
6:14stars or something. Like, it just went
6:15nuts, went viral. People love this
6:17thing. It These couple of lines means
6:20the AI asks you like 40 questions, 60
6:23questions. I've had it ask uh people 100
6:25questions before it's satisfied they've
6:27reached a shared understanding. And it
6:29means it turns the AI into a kind of
6:32adversary, where it's just continually
6:34pinging you ideas and trying to reach a
6:36shared understanding.
6:38And that means that the conversation
6:39that you then generate, you can take
6:41that and turn it into a product
6:43requirements document or something. Or
6:45if it's a small change, you can just uh
6:48do
6:49uh turn it directly into issues.
6:52And then your AFK agent will then pick
6:53it up.
6:54And
6:55don't at me on this, but I personally
6:57believe this is better than the default
6:59plan mode in uh the
7:02tool that I use, which is Clojure Code.
7:04Plan mode is extremely eager to create
7:07an asset. It really wants to uh just
7:09create a plan and start working.
7:12Whereas I think it's a lot nicer to
7:15reach a shared design concept first.
7:18So that's tip number one.
7:21Now, failure mode number two is that the
7:22AI is just way too verbose.
7:25It's like you're almost talking at
7:27cross-purposes with the AI. Raise your
7:29hand if you uh feel this. If you've ever
7:31experienced that failure mode. Yeah.
7:33It's kind of like the AI is like talking
7:34just using too many words to try to
7:36communicate what it's doing. It's not
7:38like you're talking uh using the same
7:40language.
7:41And this to me felt very, very familiar,
7:44right? If you've ever been a developer
7:46for a long time, and you've worked with,
7:48let's say, domain experts, someone
7:49building an application, um let's say
7:52the domain expert wants you to build
7:53something on uh I don't know,
7:54microchips. You have no idea what
7:55microchips are.
7:57You need to establish some kind of
7:58shared language, right? Cuz otherwise,
8:00they're going to be using terms you
8:01don't understand. You're going to be
8:02translating that into code that maybe
8:04you don't even understand, and certainly
8:06the domain expert won't.
8:07And so there's this kind of language
8:11gap between you and the domain I went
8:14back to domain-driven design, DDD.
8:17This is something I'm still kind of on
8:19the edge of exploring, but everything
8:20I'm reading about DDD is just music to
8:23my ears. I freaking love it.
8:25And DDD has a concept of a ubiquitous
8:27language.
8:30With a ubiquitous language,
8:32conversations among developers, and
8:34expressions of the code, and
8:35conversations with domain experts are
8:37all derived from the same domain model.
8:39It's essentially a markdown file full of
8:41a list of terms that you and the AI have
8:43in common. And you really focus on those
8:46terms, and you really make sure that
8:47they're aligned with what it actually
8:49means, and you use them all the time in
8:51the code, when you're talking about the
8:52code, when you're talking to domain
8:54experts, or in our case, when you're
8:55talking with AI.
8:57So I made a skill.
8:59This skill is the ubiquitous language
9:01skill. Basically just scans your code
9:03base, looks for terminology, and then um
9:07creates a markdown file. Creates the
9:09ubiquitous language markdown file, a
9:11bunch of markdown tables with all of the
9:13terminology.
9:14And this, then I pass it to the AI,
9:17and I'm able to read it, too. And I
9:19actually have it open all the time when
9:21I'm grilling with the AI and planning
9:22and that. What I noticed by reading the
9:24thinking traces of the AI, it not only
9:26improves the planning, but it allows the
9:29AI to think in a less verbose way, and
9:32actually means that the implementation
9:33is more aligned with what you actually
9:36planned. So this has absolutely been a
9:38powerhouse. It's been unbelievably good.
9:41So that's tip number two. Create a
9:42shared language with the AI.
9:45So okay, let's imagine that you've
9:47aligned with the AI. You know what it is
9:49you're supposed to be building. The AI
9:51has built the right thing,
9:53but it doesn't work.
9:55Raise your hands if that's happened to
9:56you.
9:57Yeah, just doesn't work.
9:59Well, there's an obvious thing that we
10:01can do to make that better, which is we
10:03can use feedback loops. We can use um
10:06static types, you know, if you're not
10:07using TypeScript, uh
10:09that's crazy. Uh if you're not using uh
10:12if you're building a front-end app and
10:13you're not giving it the LLM access to
10:15the browser so it can look around,
10:17absolutely needs that.
10:19And you obviously also need automated
10:21tests.
10:23And one sort of
10:26thing I notice here is that even with
10:28these feedback loops, the LLM doesn't
10:30use them very well. It doesn't kind of
10:32like get the most out of its feedback
10:34loops in the way that a veteran
10:35developer would. And so it does what it
10:38tends to do is just does way too much at
10:40once. It will produce like a huge
10:42amounts of code and then think, "Oh, I
10:44should probably type check that
10:45actually." Or I should uh you know,
10:47maybe check a test on that or maybe do
10:48something like that.
10:50And this in the Pragmatic Programmer
10:52they describe as outrunning your
10:53headlights. It's essentially driving too
10:56fast because
10:58the rate of feedback is your speed
11:00limit.
11:02The rate of feedback is your speed
11:03limit, which means that you should be
11:05testing as you go, taking small
11:07deliberate steps. And the AI by default
11:09is really not very good at that.
11:11So, skill number three is TDD.
11:14You should be using test-driven
11:16development
11:17because TDD forces the LLM to
11:21really take small steps. You create a
11:24test first, you make that test pass, and
11:27then you refactor the code to make it
11:29nicer and consider the design.
11:32The issue here
11:33is that testing is really hard.
11:36Testing has always been hard.
11:38And the reason for that
11:41is there are a ton
11:43of different decisions you need to make
11:44when you write a test.
11:46You need to figure out how big a unit do
11:48you want to test?
11:50You need to figure out what to mock. You
11:52need to figure out what behaviors do you
11:54even want to test in the first place?
11:55And all of these decisions are
11:56dependent. So, if you are testing a
11:58really big unit like an entire uh
12:00massive application, then it might be
12:03quite flaky. You might not want to test
12:04that many behaviors. You know, if you
12:06only test this unit, you need to mock
12:08this unit, you know. It's all
12:09interlinked. And I've been thinking
12:11about this for years, for my entire
12:12development career.
12:15And what we notice is that good
12:17codebases are easy codebases to test.
12:20Right? So, here we're starting to get
12:22back to the idea of code being
12:24important. It's that the better your
12:26codebase is, the better your feedback
12:27loops are because you're able to um
12:31give better feedback to the LLM, it
12:33produces better code.
12:35And so I thought, what does a good
12:37codebase, what does a testable codebase
12:38look like? Again, we go to John
12:41Ousterhout. [clears throat]
12:42He talks about having deep modules in
12:45your codebase. Not shallow modules, not
12:47lots of modules that expose type kind of
12:50um lots of functions.
12:52They should be relatively few large deep
12:54modules with simple interfaces.
12:57Let's compare them quickly.
12:59Deep modules, lots of functionality
13:01hidden behind a simple interface. Hiding
13:04the complexity.
13:05You can look inside the deep module if
13:06you want to, but you don't need to. You
13:08can just use the interface. Shallow
13:10modules, not much functionality, complex
13:12interface.
13:13And
13:15I'll just wait for you to take the
13:16photos.
13:18Shallow modules in a codebase kind of
13:19look like this, where you have a ton of
13:22different tiny little blobs that the AI
13:24has to walk through and navigate. And
13:26this is really hard for the AI to
13:29explore actually.
13:30And so often what you'll see is if you
13:32have a codebase like this, which AI is
13:33really good at creating codebases like
13:35this,
13:36is that you'll have a situation where AI
13:38doesn't understand what your code is
13:40doing. It will attempt to explore the
13:42code, but because it's poorly laid out,
13:45filled with shallow modules, it doesn't
13:47maybe get to the right module in time or
13:49doesn't understand all the dependencies,
13:50all that stuff. It doesn't understand
13:52your code.
13:53And so what does a
13:54codebase full of deep modules look like?
13:57Well, it looks like this.
14:00Where it's the same code, but it's just
14:02structured inside boundaries, where you
14:04have these interfaces on the top.
14:08And these interfaces, you should
14:10probably have a lot of control over them
14:12and design them really well. Otherwise,
14:14you know, AI might mess up the design.
14:17But the implementation, you can kind of
14:18leave that to the AI bit.
14:20So, how do you turn a codebase that
14:23looks like this into a codebase that
14:26looks like that?
14:29Well, I've got a skill for that. Improve
14:31codebase architecture. Turns out this is
14:33not It's it's quite complicated to do
14:35this, but it's a
14:36like a set of steps that you can
14:38reusably do again and again. You just
14:40sort of explore the codebase, look for
14:42opportunities where there's code that's
14:44kind of look um
14:45related, and wrap all of that in a deep
14:47module.
14:50And this is a testable codebase because
14:52the boundaries around this code are so
14:54so simple. You test at the interface,
14:56you verify using that interface,
14:59and you're good to go. And so this is a
15:00codebase that rewards TDD.
15:04But how about failure mode number six?
15:06Which is your Okay, let's say your
15:07feedback loops are working. Let's say
15:09that things are kicking into gear.
15:11You're able to ship more code than you
15:12ever have before, but your brain can't
15:14keep up.
15:16Right? Uh raise your hand if you've felt
15:18more tired than you have ever before in
15:20your development career.
15:22Yeah, me too. It's knackering.
15:25And I think that this is a codebase that
15:27actually makes it harder for your brain
15:30because you, as well as the AI, need to
15:32keep all of that information in your
15:33head.
15:34Whereas this, not only is it simpler
15:38for you to read and understand, it also
15:40means you can kind of treat these
15:42modules, or these deep modules, as gray
15:45boxes.
15:47You can kind of say,
15:49"Okay,
15:50I'm going to just design the interface,
15:52but I'm not going to worry too much or
15:53not review the implementation too much."
15:57You can do this obviously with uh things
15:58that are less critical in your
15:59application. Can't do this with uh you
16:01know, various things like finance or
16:03whatever, but in many many modules in
16:05your app, you don't need to think about
16:07the implementation too much as long as
16:09you have a testable boundary outside the
16:11module, and as long as you understand
16:13its purpose and can design it from the
16:14outside. I have found this has really
16:17saved my brain because I can just go,
16:19"Okay, the AI, I'll let you handle
16:21what's inside the big blob. I'm just
16:23going to test from the outside and
16:24verify it."
16:26So, that's tip number five. Design the
16:27interface, delegate the implementation.
16:32But this means that whenever we're
16:34touching the code, whenever we're
16:35planning stuff, we need to think about
16:37and be aware of the modules in our
16:39application. We need to know that map
16:41really well. It needs to be part of our
16:43ubiquitous language. We need to build it
16:45into our planning skills as well. So, my
16:47write a PRD, inside the PRD I'm specific
16:50about the module changes and the
16:52interfaces inside those modules, how
16:54they're being modified. I'm thinking
16:55about them all the time. And this comes
16:57from Kent Beck.
16:58Invest in the design of the system every
17:01day.
17:02And this is the core of it, right?
17:03Because specs to code, we are not
17:06investing in the design of the system.
17:08We are divesting from it. We're getting
17:10rid of that.
17:12Whereas this, I think, is absolutely
17:13key.
17:16And so
17:18code is not cheap. That's the message I
17:19want you to take away. Code is
17:21important.
17:23And if we think about AI as a really
17:25great on-the-ground programmer,
17:27a kind of tactical programmer, a
17:29sergeant on the ground making the code
17:32changes, you need someone above that.
17:35You need someone thinking on the
17:36strategic level. And that's you.
17:39And that requires software fundamental
17:41skills that we've been using for 20
17:43years, for longer.
17:46Now, if you were interested in any of
17:48the skills I put up here, it's in the
17:49GitHub repo macpocockskills.
17:52And if you're interested in the training
17:53that I do or any free stuff, I'm on
17:55YouTube, I'm on Twitter, but I'm also at
17:57aihero.dev, where I have a newsletter
17:59you can check out.
18:01Thank you so much. I hope that this
18:03gives you confidence in this new AI age
18:05that you can actually make a good
18:06impact.
18:07Thank you.
18:09>> [music]
18:10[applause]
18:15[music]
18:21[music]