Full transcript
0:01[music]
0:06Welcome to the Scaling Tech podcast, the
0:08podcast for growing engineering teams.
0:10As always, this episode is brought to
0:12you by agilityfeat.com,
0:14providers of technical [music] staffing
0:16and software development services in
0:18Latin America for clients globally.
0:22We all know that AI can write code, but
0:24it's not great at architecting a system,
0:26making product decisions, or iterating
0:28based on what your customers actually
0:30need. Builder Pods by Agility Feet
0:33combine [music] the best of both worlds.
0:35Senior human engineering experience and
0:38cutting edge [music] AI tooling. This
0:40combination of experienced teams and AI
0:43native workflows [music] is built to
0:44ship highquality and valuable software
0:47faster than ever before [music] at a
0:49reasonable price and using talent in the
0:51North American time zones. It's time to
0:54[music] meet your builder pod. Each of
0:57our builder pods are a small senior AI
0:59powered nearshore team that builds
1:01[music] fast without the bloat. Reserve
1:03a dedicated pod for your next product
1:05launch, skunk works project, or [music]
1:07proof of concept. Each pod has
1:10particular expertise and experience, but
1:12they can all work across many tech
1:14stacks and [music] industries. For a
1:16fixed low price, you get two core team
1:18members dedicated to your success. And
1:21in addition to AI tooling, that core
1:23team can tap into the [music] rest of
1:24Agility Feed's experienced human talent
1:27for roles like UX, design, DevOps, and
1:29QA.
1:31Because these teams are built to ship
1:33fast, we only require a two-eek initial
1:36commitment and you can keep extending
1:37the team as long as you need. This
1:39allows you to go from interest to launch
1:41in days with no lengthy procurement
1:44process [music]
1:44and lots of opportunities to iterate on
1:46customer feedback. Software development
1:49has changed and at Agility [music] Feet,
1:51we have adapted. Yes, it's true that AI
1:53tools make coding faster, but the hard
1:56parts haven't gone away. building
1:58cohesive and beautiful applications that
2:00your users will love and which [music]
2:02you can scale your business around. And
2:05that's exactly where builderpods shine.
2:08Learn more by visiting agilityfeat.com.
2:16Welcome. Um, we talk about AI assisted
2:19development as though everyone is doing
2:21the same thing, but no two engineering
2:23teams are exactly alike. The culture is
2:26different, the processes can vary, and
2:28the constraints are not the same.
2:30Therefore, the amount of freedom an
2:31individual engineer has can change
2:33dramatically. Using AI on a team where
2:36two engineers own nearly everything is
2:39very different from using AI when
2:40seemingly simple decisions touch
2:42everything, back-end services, mobile
2:44application, downstream products, or QA.
2:48So today I've invited three senior
2:50software engineers working in very
2:51different environments to compare what
2:53AI assisted engineering actually looks
2:55like for them. Not theoretically, not
2:57what demos tell us AI can do, but what
3:00you're comfortable letting it do. And
3:01ultimately, I want to answer a simple
3:03question. How can you move faster with
3:05AI without simply getting better at
3:07running in the wrong direction? Before
3:09we talk about agents, autonomy, context,
3:12or guardrails, I want I want our
3:14listeners to understand how different
3:16your working environments are. So tell
3:18me what does your engineering
3:20environment look like and where has AI
3:22changed the way you work personally?
3:25>> My name is Taher and I'm currently
3:28working in an environment like uh we
3:30have 13 or 14 people and uh we have QA
3:35uh mobile two mobile applications
3:38actually one iOS and one Android we have
3:41the backend team and we have DevOps team
3:44uh on top of that. Uh so we are kind of
3:48a big team and we are ready to we are
3:50preparing to go to a production. Uh
3:54right now uh it's been a couple years
3:57that uh we are doing this development
4:00and it was at the beginning a research
4:04project uh and then when it gots matured
4:08and matured uh we are now preparing to
4:11release it to the production.
4:14Uh and that's my setup at the moment.
4:19About the AI changing the way I
4:22personally work the most is uh probably
4:25I'm more moved into an architect role
4:29rather than being a developer that's uh
4:32struggling with the code and uh doing
4:35things like I don't know gather met set
4:38of methods the rest APIs and we were
4:41doing it by hand all the time. and now
4:45we have AI to do it. So, uh
4:48also I got some experience throughout
4:50the years and I got into more of an
4:53architectural role.
4:55>> Okay. For me,
4:57well, now I work at a small team inside
5:01of very large organization.
5:03It's very strict. It's healthcare
5:06regulated.
5:07Architecture and the frameworks and
5:09everything was there before I got here.
5:12So every that part it's already done.
5:18The biggest change for me with AI and I
5:22agree with here is it's not the code
5:25itself anymore but the the decisions you
5:28know making sure that the decision that
5:30where the code is going is the right
5:32path
5:33>> for me. Um yeah, I work currently a
5:36couple across a couple different
5:37projects, but uh for the most part uh
5:40work mainly on a project where it's me
5:42and one other developer working across
5:45the stack. Uh I've been on this project
5:47uh since inception and even pre-ai. Um
5:51so it has really come a long way and
5:52it's been quite interesting to kind of
5:54see how things grown and evolved. Um but
5:57also since I've worked on this stock for
5:58so long, I know it quite intimately. So
6:01um I essentially kind of tailored my
6:03workflow a little bit to create more
6:06augmented workflow uh environment. um
6:09you know building a lot of really custom
6:11skills to do things like you know from
6:13code review to ensuring that we maintain
6:15the standards that we had before that
6:17the Aentic code is following the same
6:19design patterns and so on to even
6:21creating skills to do um production
6:24debugging and incident investigation you
6:27know combing logs looking at different
6:29things whereas that's you know looking
6:30at Kubernetes servers and so on um so
6:33being able to do this you know I've
6:35always had like small scripts on the
6:37side but with a IM can kind of do this a
6:39little bit more readily and faster and
6:42uh trial out scripts a lot more quickly,
6:44keep what works and throw away what
6:46doesn't work. Um, so that's been really
6:49beneficial that way. I've been able to
6:50kind of be a lot more optimal and
6:52efficient uh recently.
6:54>> Thank you. What is one thing
6:56particularly that you think you can do
6:58in your environment with AI that might
7:00be a terrible idea somewhere else and
7:03why? actually so
7:06I don't have a quite clear answer to
7:08this because as a team we are quite
7:11concerned about AI doing stuff on its
7:15own and we review uh very very carefully
7:19uh and we never accept the code that's
7:21not reviewed or uh that's been
7:26fully generated by AI. uh so I I cannot
7:30say one thing that would be a terrible
7:32idea probably and we also I personally
7:37divide the work and whatever I provide
7:40to AI as a context is very very very
7:42small like I I don't let AI to take
7:45control over the bigger things and
7:48decisions. So
7:50>> uh that's one thing that that doesn't
7:52mean we don't use AI or uh against it.
7:55Of course, we are using AI heavily in
7:58the project. Uh but we are very strict
8:00about the autonomy that we are providing
8:03to AI.
8:04I agree with that here giving an giving
8:08it access to the whole codebase like
8:13that would or it could cause problems
8:16not having tests not have not not having
8:18CI you know if you don't have those
8:21guard rails then don't give it access to
8:25all of the context of the project right
8:28like the seams like where one repo
8:30starts don't let it cross over you
8:34having those rules separate physical
8:37boundaries that what of what it can
8:39touch that's that's what I wouldn't do
8:43you know in another project because the
8:45guard rails are there
8:47>> right
8:48>> yeah I'm I'm quite similar there too um
8:50yeah I rely a lot on guard rails and
8:53depending on what I'm doing um you know
8:55if it's just writing code uh I I will
8:57let the agent often run in auto mode but
9:00if it's doing things like investigation
9:02production incident and so on, then
9:04that's something that um either I'll get
9:06it to run existing scripts or help me
9:09write scripts that I'll reuse. Um or,
9:12you know, let it I'll have to manually
9:14approve everything that runs. But, um
9:16yeah, heavily reliant on the guards uh
9:19like to hear and Andre said um and then
9:22like I said too, these are quite
9:24specific workflows that I have set up.
9:25So, obviously I can't remove and drop
9:27and replace them into other projects.
9:30Um, but you know, if I do work on a
9:33different project, I do like to take
9:34away things that I have, uh, you know,
9:36things like scripts that are specific to
9:38AWS. I can kind of port them over to,
9:41you know, the other projects, and that
9:42really works pretty well. But, um, yeah,
9:45for the most part, I I have a really
9:46good time with very specific workflows.
9:49>> So, Justin, you've reached the point
9:51where there's specific workflows or
9:53tasks that you can give an agent um, and
9:56walk away. Uh
9:59what amount of time is that that an
10:01agent can sort of work on its own?
10:05>> Yeah. Well, I mean I I do try to remain
10:07as close to the code and implementation
10:09for many of the things I work on. Um you
10:12know for things like new implementations
10:13and designs I try to do it more
10:15iteratively with a lot of review in
10:17between. Um you know for work on like a
10:19well- definfined codebase as I mentioned
10:21that we've had uh for at least five
10:23years. um you know that work is quite
10:26well defined in itself and in those
10:28cases you know if it's like a feature or
10:29a bug fix or something like that I can
10:31actually provide the agent more autonomy
10:33since it has more context and the guard
10:35rails as well as the existing like the
10:37design patterns and tests and things
10:40like that uh actually do steer the agent
10:42quite well and also in those cases too
10:44it's quite easy to spot when the agent
10:46goes astray or if it's not following the
10:48existing pattern so um those types of
10:51situations I actually can give it bigger
10:53unattended stretch touches. Um, but
10:55those do tend to be on the order of like
10:56up to tens of minutes if most nothing
10:58too crazy more than that.
11:01>> Awesome. And what kinds of work do you
11:04do you have that still make you say
11:06like, "Nope, staying right here, not
11:08going to give an inch."
11:10>> Yeah, for sure. I mean mostly if it's um
11:13like you know like as I mentioned
11:14working with AWS services um if it's
11:18like spinning up a server or um you know
11:21something like that I prefer to have uh
11:22it maybe write a script rather than call
11:24the command line tools itself. Uh I'll
11:26have it generate the script for me and
11:28I'll review it. Um and that's how I
11:30would kind of control it that way. Um or
11:33uh I can also do things like give it a
11:35readonly account to do investigations on
11:37its own which I've done a couple times
11:38and that's worked pretty well. Um, so
11:41yeah, it really depends on on the
11:42situation as you're saying and um, yeah,
11:46very situational.
11:48>> I can understand that nobody wants an
11:50AWS bill with a lot of surprise costs
11:53added in. So I appreciate that. Um, so
11:57to hear um, you said you give the agent
12:01very limited context. How do you prevent
12:04the agent from working on one component
12:07and making assumptions?
12:09uh about another component it doesn't
12:12fully understand.
12:14>> Yeah. uh so as I mentioned we are I am
12:17giving it's a very very very limited
12:20context and uh very small thing uh
12:22because also giving a huge context uh
12:26can actually make it overfitit to the
12:28context you give and they don't have a
12:31mechanism to you know uh weight the
12:34context you you you have given like you
12:36you give a lot of documents and they
12:39took an average of it uh mostly so in
12:43that sense
12:45uh by making it smaller, by making the
12:48tasks smaller, it's actually my job to
12:53kind of make it uh in a level that AI
12:58can understand and not cross any other
13:01boundaries
13:03and uh I also always tell them to like
13:07discuss it with me actually like a human
13:10[clears throat]
13:10>> and ask me questions if they don't
13:13understand and uh any important
13:16decisions or uh I clear the boundaries
13:20of like saying do not involve with this,
13:24do not get involved with this. Uh and it
13:27helps a lot uh actually to let them know
13:30if they have some questions or uh any
13:33confusion to ask you.
13:37>> Um how do you balance a little bit that
13:39coordination with AI agents be be
13:43not for that not becoming a bottleneck
13:46uh instead of just going in and coding.
13:49>> It's not a big issue to you know because
13:53when you don't use a lot of AI agents
13:55like I I see people using eight agents
13:59working on a single project like
14:02discussing with each other and they give
14:04characters to the agents and uh they do
14:07a lot of autonomous work which we
14:10completely avoid. So uh it's not
14:13something when when you have this
14:15mindset it's it's not a hard thing to
14:18balance or uh let it stay in the context
14:22uh or boundaries that you are drawing to
14:26the AI.
14:27>> Got it. So so you keep it simple and you
14:30keep the amount of agents to a
14:31reasonable amount and that way you can
14:34make sure that they're working on what
14:35they're supposed to instead of running
14:37in all sorts of directions. Um, have you
14:41ever seen AI your agents or somebody
14:44else's on your team change something it
14:47wasn't absolutely supposed to touch?
14:50>> Oh, I seen it quite a lot actually. I
14:53have ever seen uh one case that the AI
14:56just deleted the staging branch.
14:59>> Oh, no.
15:00>> Yeah. So, it happens a lot. Uh, and you
15:04you need to have some guard rails like
15:07Andrea and Justin said. uh and that you
15:11should be very careful with that. So AI
15:14still does a lot of weird thing uh even
15:17the best models out there.
15:20>> Got it. I assume you you changed some of
15:22the permissions after that incident.
15:25>> Exactly.
15:26>> Um so thank you to hear. So talking a
15:29little bit about organizational guard
15:31rails. Andre um you mentioned that your
15:33organization has significant guard
15:36rails. Um, do the existing frameworks
15:38make it easier to trust your agents in
15:41doing autonomous work?
15:43>> Not necessarily trust it. Um,
15:47but yeah, I can relax a little bit more
15:50maybe because I it's it's it's cheaper
15:53to find where it went wrong. You know,
15:56all of the alarms are there because
15:57somebody else put them there.
16:00So it's may maybe easier to see where
16:05it's it's not supposed to be going. Um
16:10so it's not necessarily trust. It's just
16:12you can relax a little bit more, I
16:15think.
16:17>> You know,
16:18>> does that make you feel like AI can work
16:20alone for a little bit longer?
16:24>> Um yeah. Well, the architecture doesn't
16:28make the agent smarter per se. Uh, but
16:32it, as I said before, it if it if it
16:35bounds what it can reach, then yeah, it
16:39could potentially work longer.
16:42>> Do you ever hit walls and what things
16:44that you would like to do, but the guard
16:46rails don't let you do that in within
16:48the enterprise framework?
16:50Yes, it does limit what the agents can
16:53do, but the limits are there for a
16:56reason and they're to protect an entire
16:59organization or, you know, code base.
17:01So, errors or bugs or things that
17:05shouldn't be there don't spread all
17:06over. Um, so yeah, it does limit, but
17:11they are there for a reason.
17:14>> Nice. So, I've I've heard a little bit
17:16from all of you that it it does matter
17:18how much context you give your agents,
17:20keeping them in within compartments,
17:23making sure that you give the right
17:26guard rails, and that agents aren't
17:29running rampant across your code, that
17:32they can't add costs essentially. So, I
17:35I think that those are all interested uh
17:38limits on how much you can work with
17:40your agents. Could you think of things
17:43that would actually improve for you to
17:45let your agent work twice as long or
17:48work on other parts of the code? I mean
17:52for me uh it should be AGI I guess
17:56because
17:57the amount of development and the pace
18:00of it is great actually like two three
18:04years ago it couldn't any model couldn't
18:07generate a regular PDF like a good
18:10looking PDF and now uh they can do a lot
18:13of stuff also in the logic they can
18:17establish it's it's pretty good uh but
18:20still uh I mean there is a huge huge uh
18:26chance of making an error for the models
18:28still uh so
18:31I need consensus maybe I need all of the
18:35most of the software engineers uh go
18:38together and then say that yeah now
18:41everybody can use it without checking it
18:43or something like that you so I need a
18:46lot of proof actually
18:48>> we have the code reviewers We have
18:50everything, but I don't know. For me, we
18:52still need a human in the loop that
18:54approves the decision that it's getting
18:56done. So,
18:58I'm not quite there yet as to just
19:01letting them run.
19:03>> Yeah. And for me, kind of like in that
19:04similar vein, you know, some some way
19:06of, you know, verifying guardrails. I
19:08think a lot of the code harnesses like
19:10cloud code and so on have a lot of
19:12different things within it to verify if
19:14it's making uh you know, a CLI command.
19:16it'll use a a separate model to you know
19:20see if that command is safe to run. Um
19:22but sometimes I don't know if it's
19:24always safe to trust the guardrails 100%
19:27of the time. It might be fine like 98%
19:29of the time but you know very few times
19:30something uh could come out. The other
19:32thing too is sandboxing. Um you know if
19:35you are able to set up a really nice
19:37sandbox uh that could be something that
19:39works really well. Um but you know
19:41sandboxing is quite tedious and often
19:44limiting. So depending on what you're
19:45doing, if it's writing a lot of code,
19:46that might work well, but then if it
19:48needs to do more things to get the job
19:50done, sometimes it might be too limited
19:53that there's almost not too much point
19:55to let it run for too long. So um yeah,
19:58it's it does feel like it's making
20:00progress in that direction and things do
20:02get better, but there's always that
20:03little fear of the the one to 2% of the
20:06time where it doesn't follow the
20:07guardrails properly.
20:09>> Yeah. And um Justin, you mentioned like
20:12not letting it push code to production
20:14or not letting it um spin up resources
20:17that could end up costing you money to
20:19hear. You mentioned your team change
20:21some of its practices so that it doesn't
20:23have permissions to delete branches. Um
20:26what kind of checkpoints would you never
20:29remove even if the models became
20:32dramatically better tomorrow? AGI so to
20:35speak.
20:35>> I I I wouldn't remove the automated
20:38tests at all. uh and I would always uh
20:41look after AI that yeah at least I will
20:45review the test codes. I mean even even
20:48if it's a perfect uh developer the AI is
20:52I will still review the test codes about
20:54the logic issues and stuff like that. Uh
20:58the integration tests also uh very
21:01important for me. uh and the QA also I
21:06wouldn't uh take it out of the equation
21:08because the product that we are building
21:12is not just depending on my code and uh
21:15there is a lot of uh pieces
21:18>> and I guess um for me uh yeah in the sim
21:21vein like automated tests and anything
21:23automated that's actually we've been
21:25using before AI still you know works
21:28really well maybe is even more important
21:30these days um Yeah, and you know things
21:33like having ensuring your context is
21:35correct and up to date, maybe concise so
21:38that it's not uh too wordy to lead the
21:40agent astray and have a higher chance of
21:42doing random things. Um, and then you
21:45can do things like running like a a code
21:47review skill that goes alongside your
21:49own manual review process can work
21:52pretty well. Um, so yeah, those are sort
21:54of the checkpoints that I would kind of
21:55make sure they're always in place. Yeah,
21:57I think you mentioned something very
21:59interesting just in in terms of like
22:01context and I know to hear you said you
22:03give it as little context as you think
22:05it might need. Um, for me it's it's
22:09really interesting because you know an
22:11agent can be perfectly confident but it
22:13could still produce a wrong answer
22:15because somebody gave it an incomplete
22:16assumption, an outdated document um or
22:20you know it picked up something that was
22:22decided 18 months ago and it decided
22:24that that was the most important thing.
22:25So, I'd love to hear how um you guys
22:29handle context and if Andre in
22:31particular, your enterprise has a way of
22:34documenting um how you provide that
22:36context to your to your agents.
22:39>> Um they have the the agents um document
22:45but not necessarily context per se. Erh,
22:50I I do use, you know, I make sure that
22:54the whenever I'm coding that the agent
22:56knows what it's doing and what it's
22:58supposed to do. It goes to go through
23:01the code, make sure that we have all the
23:03answers that we need, that we understand
23:05perfectly what we're doing. And we say
23:07we because we're a team. Erh, and then
23:11yeah, then we code and and run it
23:14through the whole pipeline. But no, no,
23:16there is no documentation per se.
23:20>> Um,
23:22which I'm glad because of that that you
23:24said documentation can grow stale in a
23:26minute and and agents will read it and
23:29they will think that that's active in
23:32the code until you catch it and you say,
23:35"No, but we did that five months ago. Go
23:38check." And they go and they're like,
23:40"Ah, yes, you're right. This was updated
23:42in last week or whatever." And yeah, I
23:46not fond of, you know, documentation
23:49inside the project because of that
23:52reason.
23:53>> Yeah.
23:54>> Yeah. I definitely think that there's a
23:57a big optimization that you can do in
24:00what context you give it. And and to
24:02hear in terms of that, I know that you
24:04said this was a project that was very
24:06research intensive and it also has other
24:08vendors and other teams. How do you know
24:10how little enough is enough? Um, and how
24:13do you make that switch?
24:15>> This project is that I'm working on is
24:18actually having [snorts] some pieces
24:20that doesn't even have any documentation
24:23or uh information in the internet yet. I
24:26mean, I always say that to AI that don't
24:30do any assumptions at all uh and uh ask
24:34me about everything. M
24:37>> uh so because of this situation also I
24:41find myself in a position that I'm
24:44writing a lot of things to AI because of
24:47the custom development
24:49uh that's done for the satellite
24:52communication I mean I just manually do
24:55that to be honest like
24:58I go and uh provide a lot of information
25:00I copy paste from the what what we have
25:05basically
25:07uh and it's working like that. Uh yeah,
25:11it's hard in that sense to provide
25:13context
25:15when you are working on something that's
25:17not common or widely used.
25:22>> Yeah. So, it sounds like you have a
25:24little bit of a Q&A going with your
25:25agents typically.
25:27>> Yeah. Yeah. Yeah. We discuss a lot
25:29actually.
25:30>> Nice. Um, and uh, Justin, in terms of
25:34like you've been working on this
25:36platform for a really long time, it's in
25:39production. Um, how do you handle
25:41context that might become stale or that
25:45might not be relevant anymore?
25:47>> For sure. Depending on which part of the
25:49stock I'm working on that might be more
25:51or less up to date depending on when
25:53last time we worked on it. Um, so yeah,
25:56very similar to do a lot of Q&A. um
25:58always say don't make assumptions which
26:01um more recently these days does quite a
26:03good job at I find um especially with
26:06like something like cloud code it'll not
26:08make assumptions and try to ask you um
26:10and then another thing I like to do is
26:12you know we do have like a fair amount
26:14of context files and documentation but u
26:16my another thing I like to do is tell it
26:19to use the code as a source of truth
26:20rather than documentation um that really
26:24is the best way uh to ensure that
26:26everything is accurate because that's
26:27actually what's running. Um, and then
26:30trying to update the documentation as
26:32you go, keeping everything up to date
26:34does help in the future. Um, but again,
26:36using the code as a source of truth is
26:39another nice way to kind of save on
26:40tokens as well instead of duplicating
26:42information in documentation. Just I I
26:45like to have a lot of my docs just point
26:46to the code. So, it's really just, you
26:48know, various indexes pointing to
26:50different parts of the codebase to for
26:52different reasons.
26:53>> Nice. So, so we've I' I've sort of
26:56learned that you you guys don't trust
26:58documentation as much as as um one would
27:01think. It's more about what's in your
27:03heads, what's in the code, and what was
27:06the what was the initial ask that you
27:08gave the agents. Um so that's how you
27:11sort of give it direction within the
27:14guard rails of course. Um so I want to
27:17talk a little bit about when the agent's
27:18work is done, how can you believe that
27:21it did what it was supposed to do? And I
27:23think to hear you talked a little bit
27:24about AI tests and unit tests, but I'd
27:28love to know how do you know AI is is
27:30working in the right direction based on
27:31what you you asked it to do.
27:34>> Actually, the unit tests and integration
27:36tests are uh quite important for me
27:38because that's that's where I check the
27:41first. I mean, I also tell AI to do
27:44testdriven development. Uh it's kind of
27:47old right now, but uh I still do that.
27:51uh and then when I check the testing
27:53logic like I I I find some problems uh
27:58sometimes actually the recent models are
28:02doing a quite good job uh in that too.
28:07So when I when I make sure it's testing
28:10the right thing, then I know that it's
28:14going to write the uh good code. And
28:17then I use reviewers like ponytail uh to
28:23check if it if the code can be smaller
28:26uh or more optimized or uh done like
28:29that. Then at the end when all of this
28:32done I just go and check the whole code
28:36myself too. Uh
28:39without a doubt I mean I always do that.
28:42Uh that's how I
28:45how I am satisfied with the AI work and
28:48then I can push it to the repo.
28:50>> And how about you? Um for me it's the
28:55same uh the tests everything I do run it
29:00by
29:02one at least uh code reviewer before
29:04pushing the PR you know another model
29:07another session just to make sure that
29:09everything is okay and I do manual
29:12checking as well whenever it's possible
29:15sometimes it's not because it's you're
29:18building a little piece of a big feature
29:20But whenever you can go through the
29:24whole flow.
29:25>> How about
29:25>> Yeah, for me um yeah I would say very
29:28similar vein in terms of review. Um
29:30obviously the more automation you have
29:32the better. So even things like using
29:34tight code that really goes a long way
29:36having llinters ensuring uh your coding
29:38standards and practices are followed. Um
29:42you know practices that we had before AI
29:44as I said continue to do really good.
29:47Um, and then another thing too is I
29:48being more iterative is really good way
29:51to go. So I mean even if you're
29:52reviewing the code, if it's generated a
29:54lot of code, it's really hard to review
29:56it effectively. So smaller targeted code
29:59changes um are really helpful that way
30:02uh to keep your context in your head uh
30:04really small and to not get too like
30:06fatigued with reviewing code because
30:09you'll probably make more mistakes that
30:10way.
30:12>> Yes, I like I like a lot of the things
30:14that you've said. So iterative, smaller
30:17changes, smaller PRs to manage. Um we
30:20mentioned testing, unit testing,
30:22integration testing, and your um
30:24suggestion of like making sure the end
30:26toend flow works whenever possible also
30:29makes a lot of sense. The manual review
30:31of code um but to hear you mentioned
30:33something really interesting and I'd
30:35like to dig into that. Um you mentioned
30:37ponytail. Is that um like an AI judging
30:40AI's work? Um how does that work? Tell
30:43me a little bit more about that. Yeah,
30:45it's actually a plug-in uh that that's
30:48doing some uh reviews to for an
30:52optimized coding. I mean uh you can you
30:56just like if you are using cloud for
30:58example, you can install it from the
30:59marketplace and then it's free by the
31:01way open source. Uh you can use it like
31:05the with the command uh the slash and
31:08then ponytail review. It has some very
31:11specific uh instructions to review the
31:14code and then tells you like this can be
31:18done with smaller lines of code and this
31:21can be done with smaller lines of code.
31:22So uh it's working like that.
31:25>> Andrea Justin, do you guys use um AI
31:29judging AIS?
31:32>> I do pony as well. I I use ponytail as
31:36well. Not necessarily only for code
31:39reviews, but yeah, I use it to mainly
31:43for the code reviews.
31:45One or two [laughter] at least.
31:48>> One or two different agents reviewing
31:50the
31:51>> Yeah. When when when they have different
31:54directions or something, I just run it
31:56by a third, read it, and guide them.
32:00>> Oh. Oh, that's interesting. So you
32:02you'll have two different judges and
32:04then you'll have a tiebreaker when they
32:06have um interesting opinions. Oh, that's
32:09>> I like that.
32:10>> Yeah.
32:11>> And Justin, how about you?
32:13>> Um for me, I haven't tried ponytail,
32:14although I think it looks really
32:15interesting, but I do take a lot of the
32:17concepts uh from it. So, you know, one
32:20of the things that uh I think ponytail
32:22says is ensure you don't have code that
32:24you're not going to need, so you aren't
32:25going to need it type of thing. So,
32:27remove those. Um, another thing that
32:29I've done and it seems to work actually
32:31quite well, which is kind of funny, but
32:32if I get cloud code to do an
32:34implementation, I go to codeex and be
32:35like, uh, this is what cloud code did.
32:37Please review it. So, almost letting the
32:39agent know at least sets that context,
32:42you know, it's probably more superficial
32:44than anything else. But, um, setting the
32:46context to say this is something that
32:48maybe shouldn't really be trusted too
32:50well kind of informs the agent and kind
32:53of gets it to perhaps review it a little
32:55bit more strictly. Um what what do you
32:58guys think makes a judge agent
33:02independent
33:04from the agent that produced the code
33:05and why is that relevant too?
33:08>> Uh yeah by the way I also do it like
33:12ponytail is just one one line of
33:14working. So then I get the output of
33:16ponytail and then ask it to cloud or ch
33:20or any other agents if this holds or
33:23not. uh so
33:26I think when you
33:29use the models in between like if if you
33:32ask what one model's job to another
33:35model then it gives you pretty good
33:38insight to be honest I don't know what's
33:40the technical explanation about behind
33:43this but it really gives you pretty good
33:46insight about the output and also like
33:50yesterday I saw something that if you
33:53say that one model done better job than
33:56you to a model then it goes and reviews
34:00all of the structure it made and does a
34:04better job actually
34:05>> nice some of that teamwork uh that we
34:08all learn in school right having
34:10multiple eyes making makes better
34:13product and competition
34:15makes you a better better coder
34:18>> you say oh codex told me to tell you
34:21this and it'll go like oh maybe codex
34:24That's right.
34:26>> Like I love that.
34:27>> Um Andre, you've told me that you've
34:29experimented with using multiple agents.
34:32Are more agents better? How many agents
34:36too many agents?
34:37>> It depends on the task. Um like I for
34:41this like I don't use multiple agents
34:44for client work. I do babysitting and go
34:47with one agent at a time. erh for like
34:50my pet projects like I do let them
34:55work on their own sometimes and I do the
34:58tasks and I have you know my board and I
35:01have a coordinator or a PM that works
35:04with the dev, a reviewer and a test
35:06architect. So they will talk to each
35:09other and the the PM or no the reviewer
35:13will actually be the one merging. I do
35:16four like to check every stage. Um again
35:22for my work for my pet projects I made a
35:26mistake yesterday like I wanted to check
35:29I don't know I wanted a team of
35:30marketing that could you know tell me
35:33where a project was going. It was a
35:36mess. Like it ate up half of the tokens
35:40in a 10 minute window and I was like
35:44what is it doing? So yeah, there's
35:45there's places where maybe it too much
35:49is not the way to go.
35:52>> So So they might all have started from
35:54the wrong assumption and just ate all of
35:58all your tokens.
35:59>> Yeah. I don't know. It was a mess. Ended
36:01up just throwing everything away and
36:03starting from scratch with one.
36:07>> Got it. Got it. So, so it it really does
36:09matter like if you have the problem
36:11defined correctly before you split it
36:14into many.
36:15>> Yeah.
36:16>> Because more agents doing the wrong
36:18thing is definitely going to get you
36:21faster to the wrong place or use up all
36:24your tokens [laughter]
36:25>> or Yeah. in a 10-minute window. Like
36:28boom.
36:30>> It was and I was I was even checking. I
36:32was like, "What model is this using?" It
36:34was just going up and up and up and up.
36:36Yeah,
36:37>> that brings me sort of to the idea that,
36:40you know, AI might be making
36:42implementation easier, but that doesn't
36:44mean engineering itself is becoming
36:46easier. Um, it might just mean that the
36:49difficult part has moved somewhere else.
36:51First of all, does everybody agree with
36:54the premise that coding has become
36:55easier?
36:56>> Yeah, I mean,
36:57>> yeah.
36:58>> So, I' I'd love to do like a little bit
37:00of a fill in the blanks um exercise. So
37:04AI is making blank easier but it is
37:07making blank more important. So what
37:11would those blanks be for you?
37:13>> Yeah. Uh for me I mean AI is actually
37:16making the fun part easier which is the
37:19coding itself and the worst part which
37:23is the review uh is a lot more important
37:26at the moment. Uh and like half of my
37:31day, not even half, I mean almost full
37:34day I'm reviewing code now. Uh instead
37:38of you know going to find stuff and uh
37:43solve problems as the instead of that
37:46I'm just a review right now.
37:47>> For me it's the same the code review
37:50it's it's yeah it's a lot worse. It's
37:53maybe the bottleneck now because the
37:56code is just gets added and added and we
37:59need the human in order to to review it.
38:03Uh not necessarily
38:06like we did before like hunting for the
38:09bugs. I I trust that part more but
38:13reviewing that the decision again that
38:15was made is the right path.
38:17>> Yeah, I would say in a similar vein the
38:19the testing one is really interesting.
38:21um the for some side projects, what I
38:24like to do is I'll write the code and
38:25get the agent to write the tests and or
38:28maybe see where I'm missing a test. And
38:29I actually found it does a pretty good
38:31job of identifying areas where it should
38:34write some automated tests and uh do
38:36things like from unit test to end to end
38:38tests. So that's been really nice
38:40because I don't always like writing
38:41tests. So if the agent can do that,
38:43that's that's pretty great. But again,
38:45you do need to review the test because
38:47sometimes it'll take shortcuts and write
38:48tests that don't actually do too much.
38:50It's it sort of sounds like um AI has
38:53just made teamwork um more important and
38:57all of the skills that you've learned in
38:58like how to give somebody context, how
39:00to tell them what you want to be done,
39:02how to evaluate their work has become
39:04even more important. Could you think of
39:06things that AI agents have exposed in
39:08ways your teams had previously worked?
39:10Um and you can answer for past projects
39:13or side projects as well. Yeah, I'd say
39:16for me um one thing that it actually has
39:20identified which uh [snorts]
39:22have to be careful how I say it, but uh
39:24in terms of identifying tech debt or you
39:26know really problematic issues in the
39:28code, it does quite a good job of you
39:30know maybe bringing up things that uh
39:32somebody else wouldn't bring up in the
39:34past like oh we've committed this uh
39:36secret to the repo and you know every
39:38time you make a code changes saying by
39:40the way this secret is here we really
39:41need to remove it and rotate the secret.
39:43So, you know, big problems like that, it
39:45does bring up um and it kind of makes
39:47you feel like, okay, yeah, this probably
39:49is a problem that we should uh try to
39:51resolve right now. So, there is some of
39:53that, which is nice and annoying, I
39:55guess.
39:56>> Well, that's nice. You can't just put it
39:58on a story and you're put it to the side
40:01and forget about it and you kept getting
40:03reminded of it. Nice.
40:05>> Exactly.
40:06So if um somebody listening to this
40:09podcast wants to improve how their team
40:11works with AI um I want to do a little
40:14exercise of keep kill start. So um keep
40:20what existing software engineering
40:22practice becomes even more important in
40:24an AI heavy team?
40:26>> Yes.
40:28[laughter]
40:29>> Yeah. Yeah. I think I think we can all
40:31agree that that's been an underlying
40:33theme of the episode is um make sure you
40:37do unit tests, integration tests, end to
40:39end testing, manual tests, and automated
40:42tests. Um all al all sound like very
40:45sound practicing. Um let's talk about
40:49kill. What engineering habit or process
40:52makes less sense now that AI is part of
40:54the workflow?
40:56I think the llinters maybe going away
41:00because
41:02the AI can write like llinter code
41:04already. So
41:07>> yeah, I might be redundant now.
41:08>> Mhm.
41:10>> Yeah. For me, I'd say um kind of guess
41:12similar to the point I made just a
41:15second ago, which is, you know, stop
41:17deferring cleaning up tech debt. So a
41:18lot of times when you work on a feature,
41:20you want to get the feature out. Um, but
41:22then you know oftentimes you'll have
41:24issues identified in the PR and you'll
41:25be like, "Okay, well this is actually
41:27the scope creep. So let's uh move it
41:29over to a follow-up ticket that never
41:31gets done." But uh with AI, like I said,
41:34they're pretty good at identifying these
41:35types of problems and gaps in the code
41:38um and like when conventions and things
41:40are not met. So like those fairly
41:42tedious to do and you wouldn't want to
41:43do them right away, but you know now
41:45doing that sort of stuff and clean up is
41:47is almost enjoyable because now you
41:49don't have to like go in manually
41:50tediously clean up those things and you
41:52get the agent to do it. So I'd say now
41:54there's like less of a reason to defer
41:56that type of work the next time and try
41:58to like shure everything up and make
42:00sure things are good and then now you
42:01actually save time in the future.
42:03>> Love it. Um a million project managers
42:06are going like yes we get to tackle
42:07touch that [laughter]
42:10amazing. Um so um what is one practice
42:13engineering team should start adopting
42:15now in terms of starting something?
42:18Yeah. Uh I I may be controversial on
42:22this with Andrea and Justin, but we are
42:24keeping documentation on the project and
42:27uh I think the contracts schemas and
42:30decisions and reason behind them uh
42:32should be documented but it should be
42:35done in a very very very precise and
42:38controlled way. So you need to polish
42:40the documentation
42:42uh quite hard and I completely agree on
42:45that sense because the documentation can
42:47be stol like uh very easily. So uh
42:51whenever we get some PRs like also our
42:55project manager is very very cautious
42:58about it uh Lucas and uh we we do the
43:02review about the documentation and we
43:05have some very strict guidelines about
43:08what to write and how to how many words
43:10you need to use how much effort we do it
43:14and we do not accept generally if it's
43:17completely from AI if we smell that we
43:20we do not accept it. We have have a rule
43:24about documentation to be done by the uh
43:27PR sender uh which when you index it
43:32actually when you are providing the
43:33context to AI it's kind of helps in my
43:37experience uh to the AI like if you can
43:40index it well uh then it can go and read
43:44but you need to keep it very short and
43:46concise.
43:48>> Yeah. Maybe on that note,
43:51there could be two ways. Like what if we
43:54had an agent that after the PR is
43:57merged, it just goes looks into the
43:59code, looks into the documentation and
44:00updates it. So you don't forget, there's
44:03no way of forgetting. It's an automatic
44:04job. Like that could work. Um, another
44:08one would be,
44:10and I don't know how you guys feel about
44:12it, is like commenting on the code. like
44:15at the end it's what the agent's going
44:17to read and it might need to know why a
44:21line was changed and since you it's
44:24already in GitHub and it's been pushed
44:26you'll know who did it and if you have
44:28to ask you could go to that person
44:30specifically or when in time this
44:33decision was made like it could get to
44:36be too much I don't know what how you
44:38guys feel
44:38>> yeah I would agree with that I think the
44:40comments are definitely a tough one I
44:42think agents sometimes really like to
44:43write a lot of comments and as you're
44:44think kind of dirties up the code, adds
44:46extra context, maybe gets out of date,
44:48confuses an agent, but then again,
44:50sometimes uh like you said, it's it's
44:53helpful to identify certain problems and
44:55if something is done a certain way so
44:57the agents don't make that uh mistake in
44:59the future, that could be fairly
45:01annoying to deal with. So it sounds like
45:04some structured documentation whether
45:07that is in a separate way or in comments
45:10but but with a lot of human oversight to
45:12not bloat it in terms of like what AI
45:15can add into the mix could be something
45:19that teams should start doing. So, one
45:21final question, um, and it's more for
45:24like product or engineering managers
45:27listening to this episode. Um, if if
45:30they change one thing about how they ask
45:32you to use AI, what would you want that
45:35change to be? I had one thing I thought
45:38of and kind of similar I think to
45:39something I mentioned a little bit
45:40earlier which is to not necessarily
45:42focus on how did AI get you there faster
45:45but how did AI help you verify the work
45:48that you did uh was correct and is
45:50working well kind of as you know we were
45:52saying a little bit earlier getting it
45:54to write tests uh it's a lot less
45:56tedious so you know saying something
45:58like did AI actually help you verify
46:00this and even like bringing it to your
46:02team instead of you know saying did AI
46:04help you get this done quicker you say
46:06that AI actually help you uh evaluate
46:08that the feature was done correctly and
46:10having your team oriented to uh
46:13verifying the work that was done kind of
46:15helps a lot in a lot of different ways
46:17in the future, right? So, you know,
46:18you're going to save time helpfully
46:20avoiding bugs, maybe catching different
46:22edge cases you might not have thought
46:23of. Um so, kind of think treating that
46:26as more of like a measurement rather
46:28than uh like just a productivity tool.
46:30um that that's a really nice way to kind
46:32of frame it and then kind of having
46:34everybody in that sort of mindset
46:36organizing their workspace and their
46:38workflows around verifying. So whether
46:40that's you know you do some code and the
46:42clip a for instance cloud will spin a
46:44playright so it'll actually bring up a
46:46browser and test things for you. Um
46:48that's a really nice process I found
46:49works well for me.
46:51>> Nice. Nice. I really like what you're
46:53saying because to hear you mention like
46:55two or three models ago, you know, AI
46:58started producing real value and and
47:01Justin, what you're saying is the
47:03conversation should shift into let's
47:06definitely not think about how fast
47:08we're going, but more in terms of like
47:10verifying accuracy, verifying quality of
47:13the things that we're producing. And
47:14Andre, like you mentioned, it's not
47:16about token usage at all. You can use up
47:18tokens by spinning multiple agents. It's
47:22it's it's more about um are we are we
47:25producing the code that is going to make
47:27things easier in the future that is
47:30going to make it maintainable and um
47:32that is actually solving the problems
47:34that we want to solve. So I think that
47:36those are all excellent um things to
47:38consider if you're putting together a
47:40team and you're uh trying to build
47:42something meaningful with AI. Um but
47:46making sure that that your intent is
47:48correct. It's not about moving faster.
47:50It's about moving better. So amazing.
47:53Thank you guys. Thanks for being on this
47:55podcast.
47:56>> Thank you.
47:57>> Thanks for [music] listening to the
47:58Scaling Tech podcast. In addition to
48:01bringing the Scaling Tech podcast to you
48:03at Agility Feet, we help companies to
48:05build [music] tech teams across Latin
48:06America that are truly theirs. We can
48:09help you scale with confidence with
48:11everything from staff augmentation to
48:13establishing your own technical center
48:15[music] of excellence through our build,
48:16operate, transfer model. We can do that
48:18for you in Colombia, Panama, Mexico, or
48:21elsewhere in Latin America. So check us
48:23out at agilityfeat.com to learn more.
48:26[music] And if you haven't already,
48:27please subscribe to the Scaling Tech
48:29podcast on YouTube, Apple Podcast,
48:31Spotify, or wherever you're enjoying
48:34this episode. We'll continue to bring
48:36you insightful [music] conversations
48:37about engineering, leadership, and
48:39building technical teams. To see our
48:41past episodes, check out scaling
48:43techchpod.com.
48:47>> [music]