Full transcript
0:00Hi everyone, and welcome to our talk
0:02today about bending a public MCP server
0:05without breaking it.
0:06But today we may have just broken it
0:08because our MCP server seems to have
0:10caught on fire. We're glad you're here.
0:12We need all the help we can get.
0:14Let's go through our talk and see how we
0:16can improve whatever is going on right
0:19here. I'm Hauser. I work at Buzz. We've
0:22been building AI powered code reviewers
0:23for the past few years now, as well as a
0:26bunch of other features, anything that
0:27can can make the lives of people at the
0:29R&D easier and better, whether they're
0:31devs, PMs, anything else. If it's
0:34Agentech, we're probably tinkering with
0:35it.
0:36But let's jump right in and start
0:38looking at what's going on with our MCP
0:40server. I suspect it's the tools.
0:44Uh we're going to talk about third-party
0:45tools and why they might blow our
0:47applications.
0:49First of all, it's me.
0:52As I said, Nimrod Hauser, a founding
0:54engineer at Buzz. I've been with the
0:55company since it was founded in 2023.
0:59I've been at back end data for the past
1:0120 years or so. Had a In my career, had
1:03a brief stint in Salesforce, and ever
1:05since, mostly startups,
1:07um
1:07cyber, crypto, and now developer tools.
1:10Nowadays, I mostly want to talk to you
1:12guys about Agentech tools.
1:17All right.
1:18Agentech tools, and specifically
1:21third-party tools.
1:22They can be a great force, a great
1:25addition to our application, but they
1:27don't always work out of the box. We
1:29expect them to make our application
1:31better. Sometimes, we'll see
1:33degradation. And we'll try to understand
1:35why that happens.
1:37After that, we'll explore a framework of
1:39five best practices that we can follow
1:42in order to turn this around and make
1:44our application kick ass. Along the way,
1:48hopefully, we'll fix the busted MCP
1:50server that we just saw, put out that
1:51fire, make it work, and make our agents
1:54behave the way we want them to.
1:58Yep, this is looking kind of bad.
2:01Think we should dive in.
2:04So,
2:05we're going to talk about Agentech
2:06tools.
2:07When we use MCP servers, we get tools
2:11from coming from the MCP server. So, as
2:14long as we're talking about third-party
2:15tools, I don't care if they're coming
2:17from an MCP server, from a library,
2:19maybe we copy and pasted them from
2:21somewhere else. If they're Agentech
2:23tools, they were written by a different
2:24team, they're relevant for this
2:26discussion.
2:27So, what are these tools?
2:29Essentially, tools are just callable
2:31functions wrapped with a nice
2:33description. The description is
2:35important because it lets the agents
2:36know when to use the code and how to use
2:40the code. And we'll dive deep into these
2:43aspects of the description. But again,
2:46it's kind of like glorified integration
2:48code written by a third party. In
2:51today's talk, we're going to take
2:53Playwright's MCP server as an example.
2:55So, essentially, we're looking at
2:57integration code written by the good
2:58people at the team of Playwright wrapped
3:01with their descriptions.
3:02And we'll see how we can make these
3:05tools
3:06work better, kind of tailor tailor them
3:08for our use case.
3:12Yeah, so
3:13third-party tools have their challenges.
3:16Uh first and foremost, they might cause
3:19our agents to behave unexpectedly. You
3:21know, agents are already
3:22non-deterministic, unpredictable things.
3:24You give them tools, and you get
3:26unpredictability at scale.
3:28But also, they can just degrade
3:30performance. You might want the agent to
3:32do a certain thing, and you get subpar
3:34results, wrong results, or maybe it just
3:37does it, but in a way that's not
3:40optimal.
3:41And through today's best practices,
3:43hopefully, we can see how we can make
3:46the implementation of third-party tools
3:48in our Agentech workflows much, much
3:50better. Last and foremost, these bad
3:52performances, that unexpected behavior,
3:55that can also mean full-blown security
3:57issues. I mean, just imagine a scenario,
4:01pretty classic scenario, uh like a
4:04multi-tenant architecture, and your
4:06agent might not know all there is to
4:08know about your architecture and the
4:09division into folders or databases and
4:12schemas. Uh it just does doesn't have
4:14the proper guardrails, and it might leak
4:17client data to another client to another
4:19client, things of that nature. You
4:21really want to guardrail your agents,
4:23and and this is becomes even more
4:26important when dealing with third-party
4:28tools who are not aware of your
4:29architecture. So, we'll cover that as
4:32well.
4:35All right. I think
4:38I think we're going to we're about ready
4:41to look at a use case.
4:43To look at some code, actually. But
4:45we'll need a use case, and with your
4:46permission, I'll we'll choose one of
4:48ours.
4:50So, today our use case will be Buzz's
4:54spec reviewer. So, what is a spec
4:55reviewer? It's one of our products,
4:58which is essentially an Agentech
4:59reviewer that knows how to compare
5:01requirements with implementation. So, as
5:05a first step, it needs to kind of
5:08collect requirements. It will go to your
5:10ticketing systems like your Jiras or
5:12Linear or anything of that nature, and
5:15read a ticket.
5:16And it can also go to Figma and look at
5:19visual designs in kind of a multimodal
5:23way of operation. It will actually see
5:25the design that is intended, and that
5:28part is the requirements. What's Once it
5:30understands what a developer was tasked
5:32with, that's when it will spin up
5:35Playwright's MCP server to actually open
5:38up a browser, go into your system, check
5:41the branch, see the implementation, and
5:44it will need to assess whether the
5:46implementation meets the requirement. It
5:49will give us kind of a verdict. It will
5:51take a snapshot as evidence whether this
5:53was fulfilled or wasn't fulfilled, and
5:56it does all this automatically and can
5:58save uh people, mostly PMs, a lot and
6:02lot of time doing menial validation
6:05work. So, we've built a toy example of
6:08our spec reviewer, and we're going to
6:10see how we handle the tools to get the
6:12most of it.
6:14I hope this makes sense.
6:17At a high level, I think it's time to
6:19look at some code, and hopefully,
6:21everything will be much, much clearer.
6:25All right. So,
6:27we have a toy example of our spec
6:29reviewer. We'll go through it kind of
6:31quickly.
6:32We don't need to dive into every aspect
6:34of it. It's It's a pretty small project.
6:37And we'll see what's going on and
6:40focus on the parts that we care about.
6:42So, we start here with our main
6:44function.
6:46And um
6:48we have a some a directory where we want
6:52to save snapshots. We'll get to that
6:54later. But right off the bat, we have
6:56our MCP server configuration. We have
6:58just the one. We're using only
7:00Playwright's MCP server. This is
7:03pretty standard.
7:04So, we have the one MCP.
7:07As we go into our main function,
7:10you can see that we're
7:12uh defining our MCP client.
7:15And we're going to use it in just a
7:17little bit. We'll put it in our agent,
7:19but I want to focus on this. This is
7:21where the magic of this talk happens. We
7:23have built a base class
7:28for getting the tools. And all it does,
7:31it has one function called get tools. As
7:34we will go through the talk, we will
7:37go in increasing complexity and improve
7:41the way we handle the tools that are
7:43coming from our third-party MCP server.
7:47So, here it is.
7:50We're starting with a baseline. We'll
7:51look at it in just a second. And as we
7:54start our session, this is
7:57this is where this inheritance is going
7:59to take place. Every time we run this,
8:02we will uh the get tools will do
8:04something a little bit more advanced.
8:08So,
8:09we start we we want to start the our
8:12flow. We have this function called login
8:14to Buzz, because for this talk, our
8:16example is going to be logging into our
8:18system, and we will talk about why we
8:21need this towards the end. There's
8:23actually an interesting point here.
8:25We will define an LLM. We'll create an
8:28agent.
8:29Uh we will uh give it a system message
8:33and a human message to start to kick to
8:35kick it off. These are the messages it's
8:37going to get. And we will
8:40invoke it.
8:42Um I'm you're probably kind of wondering
8:44maybe you want to see a little bit more
8:46under the hood, maybe look at the
8:47prompts.
8:49So, this should be
8:51uh relatively um straightforward, you
8:53know.
8:54System prompt, this is mostly AI
8:56generated, saying things like you are a
8:57meticulous QA agent. You need to review
9:01requirements from the ticket, as well as
9:02visual verification. Everything we
9:04talked about at high level is right
9:07here. Some guidelines, first read the
9:09ticket, understand it, navigate through
9:11the system, uh and then at the end, like
9:14we said, it needs to give us a pass or
9:15fail verdict, um
9:17specific observations, and reference
9:20everything with a screenshot for
9:22evidence. Uh human prompt is very
9:25similar. It does have a multimodal
9:27aspect to it, where we take images and
9:29we embed them in the human prompt. But
9:32these days, it's very straightforward,
9:34and any coding agent can just whip that
9:37out for you if you need it. Speaking of
9:40images, we have two images here.
9:44We have a ticket that we took a snapshot
9:46of. Our real product doesn't take
9:49tickets as snapshots. We were just lazy.
9:51But um the agent can definitely read
9:54this, understand the requirement. There
9:57is a an accompanying design,
10:00which is this one. So,
10:03the ticket states that we want to have a
10:05configuration drawer for our spec
10:07reviewer in our system in Buzz. It
10:11explains how it should look,
10:12and a design is given. So, the agent
10:16should understand that it's looking for
10:17a drawer
10:19inside our agents tab for spec reviewer,
10:22and it should look roughly like this.
10:25Amazing.
10:28I think we it's about time we just fire
10:30this up, and hopefully it will make
10:33everything so much clearer.
10:35We have a breakpoint here right after we
10:37get the tools. Almost forgot. Our first
10:39run is going to be with this V0, the
10:43benchmark. What is our benchmark? If we
10:46go to our get tools, we see that what we
10:49do for V0 is classic out of the box. We
10:53just use LangChain's load MCP tools
10:57uh method. That is it. For the first
11:01round, we're not tinkering with tools at
11:03all. Let's see how it behaves vanilla.
11:08All right.
11:15Okay. So, this is starting up, and we
11:18have our tools.
11:20Let's see what we have here. So, right
11:22off the bat, the good people at the
11:24Playwright have given us 21 tools, and
11:28everything that has to do with
11:30manipulating the browser, browser close,
11:32browser resize, console messages, handle
11:35dialogue, file upload, fill form,
11:37install all the press key. And then we
11:39can look at the descriptions. What is
11:41the description for a tool called press
11:43key? Press a key on the keyboard. What
11:45is the description for something like
11:47resize? Resize the browser window.
11:49Browser close? Close the page. These
11:52seem very shallow and very generic, but
11:56we don't blame them. The people at
11:57Playwright don't know what our specific
11:59use case is. This MCP server will need
12:02to cater to
12:04I don't know how many different use
12:06cases. It has to be generic. But, for
12:09us, using this, we and we'll see this
12:12going forward, we might want to put in
12:15our own descriptions that really are
12:17tailored to our use case, but we're not
12:19there yet. We're still at the baseline.
12:22So, let's just continue. And we will see
12:25that this is running.
12:30Okay.
12:31So,
12:32Playwright is running. It's spinning up
12:35a browser.
12:37And now it's going to log in.
12:41And once it's logged in, the agent is
12:42going to
12:44take over and start
12:46running according to the prompt.
12:51And there it's off to the races. It's
12:54opening uh it's it's logged in. This is
12:56our home page, which is the changes
12:58screen. And now it's going to need to
13:00find the relevant um page, which is the
13:05agents tab.
13:06So, it's going to need to explore the
13:08system a little bit.
13:10And it might work, it might not work.
13:13Remember, the tools are not optimized at
13:16this point. And it's done. Let's see how
13:18it did. So, looking at the results, it
13:22tells me that the requirement is not
13:25implemented, the status is
13:27it's a failed verdict. It gives me an
13:30observation, and it tells me that the
13:32require requirement is not met because
13:34it couldn't navigate to a seemingly
13:37made-up page called buzz.co
13:40/spec-reviewer. This might be a
13:42hallucination, a lapse in judgment on
13:44the agent's part, a bunch of other
13:46things. And it gives an evidence of a
13:49404 screenshot, which probably took and
13:51we can probably check out in our
13:52screenshots folder. It didn't even
13:55manage to take the screenshot properly.
13:57So, a lot of things went wrong, and this
13:59is actually a great outcome for the
14:02beginning of a talk whose whole concept
14:04is optimizing our use of agentic tools.
14:08So, let's see what we can do to improve
14:10our tools, and we'll run this again and
14:12see if we can turn this upside down.
14:17All right. Cool.
14:19Our MCP server is already starting to
14:21look a little bit better. The fire is
14:22put out. It's just this spark now. And
14:25this is probably because we've gone
14:27through some code. We're starting to
14:28understand the problem, but we still
14:29need to start to actually implementing
14:31our improvements and see what can be
14:33done to really make the system
14:36better.
14:37So, time to introduce our five concepts
14:39that we're going to go over. We're going
14:41to look at how we can curate third-party
14:43tools,
14:44wrap third-party tools with our own
14:46descriptions and perhaps some additional
14:49things,
14:50adding deterministic guardrails whenever
14:53we feel it's necessary, and we'll give
14:54an example, creating new tools out of
14:58the existing tools, actually using the
14:59existing tools as building blocks. And
15:02lastly, there's always the option to
15:04treat tools as simple functions, just
15:06calling them, using them as that
15:09integration code we spoke about written
15:11to us by the good people at the team of
15:12Playwright.
15:14Um you know, taking some parts of the
15:15workflow outside of the agentic flow
15:18whenever we feel it's necessary. We'll
15:20talk about this towards the end.
15:23So, it's also a tool in our arsenal.
15:25I did kind of split these into two
15:27buckets. One is more in the realm of
15:28context engineering, the other
15:30deterministic guardrails. It doesn't
15:31really matter. At the end of the day,
15:33whatever gets our application um to work
15:36as we want it, that's what we need to
15:38use.
15:40So, now we'll go over them one by one,
15:42looking at code, see how we can improve
15:44our toy example that we just saw.
15:47Starting with our first point, curating
15:50third-party tools.
15:52All right. Let's see how this one looks.
15:55All right, we're back here at our
15:56familiar project. And through the magic
16:00of video editing, we have now imported
16:02V1. It used to be V0 original. It's now
16:06V1 curated.
16:07The only difference, like we we've seen,
16:10is that now we have this as V1, and
16:13this, as we said, it's that class that
16:15inherits from the base class. It used to
16:17have just get tools vanilla using
16:20LangChain's function. Now, we can see
16:23what we have implemented here.
16:25So, we go in, and we used to return
16:28this, right? But now, we have this big
16:32list of all the tool names that we get
16:34from our Playwright MCP.
16:36And this small list, this is pretty, you
16:39know, standard stuff in Python.
16:41Um
16:42list comprehension. So, we just created
16:44this list of tools that we want to
16:45exclude.
16:46We just went over them, and we we know
16:48the tools. We've been using the this MCP
16:50server for a while, and we decided that
16:53for our use case, we might not need
16:55resizing the browser. We don't want our
16:57agent to drag things. We don't want to
17:00run code inside the browser on its own.
17:03These are just not things that our spec
17:05reviewer needs to do as part of its
17:07operations. Maybe for your use case,
17:09this is needed, but for ours, not so
17:12much. So, all we do is we get all the
17:14tools, and instead of just returning
17:16them, we simply exclude the ones that we
17:18don't want. So, there are a bunch here,
17:20six here, that we're going to simply not
17:23use. We fire this up.
17:26We have our breakpoint, and instead of
17:2821 tools, which we used to have, I
17:31expect to see less. And so, we have 16.
17:36Amazing. Um so,
17:39this means our context window already
17:41has less tools in it. Our agent has less
17:44to choose from. So, everything might
17:46become simpler.
17:48We'll see that not all the guidelines
17:50that we're going to go through will
17:52necessarily um reduce stuff from context
17:55window. Some will actually add to it,
17:58but this is all part of this trade-off,
18:01this juggling act that we're going to
18:02talk about.
18:06Moving on to our next point,
18:10the practice of wrapping third-party
18:13tools. This is amazing. We talked about
18:16how the descriptions, specifically
18:18coming from the Playwright MCP, are
18:21super shallow and very very generic, and
18:23that it's totally understandable because
18:25they need to cater to every possible use
18:28case in the world that might want to use
18:29the browser. But, if you really want to
18:31optimize, you might want to start
18:33tailoring stuff for your own use case.
18:36Let's see how this happens.
18:39Okay, this is becoming familiar
18:40territory by now. And as always, through
18:43the magic of video editing, we have V2
18:46imported, wrapped. So, we're wrapping
18:49tools this time. Going down, we see that
18:51we're calling the V2
18:54class, which will implement get tools,
18:57and we'll see what's going on here. If I
18:59go to V2 wrapped, I see that we, as
19:02before, we get all the tools, but now we
19:04have this new class called tool wrapper,
19:06which has a method that we're calling
19:08wrap Playwright tools. Let's see what's
19:10going on here.
19:12As before, we still have this list of
19:13all the tool names. We'll do the
19:15filtering a little bit further, but
19:17instead of just the tool names, we also
19:19have all these descriptions. And so, for
19:22every tool, we we want to specify what
19:26needs to happen. And from experience, we
19:30have our own kind of little emphasis
19:33that we want to give our agent. We might
19:35tell it, you know, before calling the
19:37browser tool, first call this other
19:39tool. This tool we found to be
19:41especially helpful. It's uh it has kind
19:44of a misleading name. It's called the
19:45snapshot tool. It's actually not a
19:47visual snapshot. It's the accessibility
19:49snapshot
19:50that [snorts]
19:51kind of shows you all the different
19:53buttons and all the different menu items
19:55in text. And we feel that the agent the
19:58agent really gets a good understanding
20:01of what is in a page when it calls that
20:03tool. So, we tell it for a bunch of
20:06tools, you know, instead before calling
20:08hover, before calling click, please use
20:10this tool before. So, we can kind of
20:12really affect its behavior. We can make
20:15it more eager to choose one tool over
20:17the other. We can do a bunch of things.
20:19Um
20:21for instance, this is the tool I just
20:22talked about, the accessibility
20:23snapshot. We will tell it always prefer
20:26this over taking an actual snapshot,
20:29which is this tool. So, you can really
20:31give a lot of gu- guidance from your own
20:34experience for your own particular use
20:37case. And this is very very powerful.
20:41In here, we have this dictionary, which
20:43just maps tool names with their new
20:46enhanced descriptions. Still, we have
20:48our tools to filter. At the end, we have
20:50the function that we called called wrap
20:53Playwright tools, and it just goes
20:55through all the tools that we get from
20:57Playwright out of the box.
20:59We filter what needs to be filtered. And
21:02for for other tools, we
21:05get our enhanced description based on
21:07the tool name, and we create this tool
21:10and we append it to the list of wrapped
21:13tools. So, we get enhanced tools.
21:16What is this method that creates an
21:18enhanced tool? Well,
21:20it's a method that gets the original
21:22tool and the enhanced description,
21:24creates a new tool,
21:27and returns it.
21:29And so, what does this amazing new tool
21:32what what does it do?
21:33Exactly what the old tool did. It just
21:36invokes the original tool.
21:39It just has an enhanced description.
21:41So, if we run this,
21:44going back to main,
21:45and we run this,
21:48and we still have our breakpoint,
21:52we can see that we still have less tools
21:56like we wanted to from before, even
21:58less, we filtered a bunch more. But when
22:00we look at the descriptions,
22:02you see that they're much longer, and
22:04they are they are the ones that we
22:06wanted. For example, here is the tool we
22:08spoke about, browser snapshot, capture
22:10an accessibility snapshot of the current
22:12page, yada yada yada, all the things we
22:15said. If we look at another one, browser
22:16click, here's our guideline for first
22:19call the other tool, and then call this
22:21one. Now, our agent knows how we want it
22:24to behave.
22:29All right, on to the next one.
22:33First of all, our MCP server, I don't
22:35know if you can notice, but things are
22:37looking even better. Some of the
22:39interfaces seem to work, lights
22:41blinking, things firing, but we're still
22:44far from the home stretch.
22:47We'll move on to point number three and
22:49keep making this better.
22:53Now, we're moving into the realm of
22:55deterministic guardrails. And this is
22:58putting in deterministic guardrails,
23:00taking control of sensitive or
23:01mission-critical aspects of our tasks
23:04with deterministic logic that is not up
23:07to agentic decision-making.
23:09Sometimes, there are
23:12aspects of your tasks that are just too
23:15sensitive to leave at the hands of the
23:16agents. We talked before about scenarios
23:19like multi-tenant architecture and may-
23:22and scenarios where the agent might not
23:24be fully aware of your architecture,
23:26things of that nature. And of course,
23:27you need to specify everything you can
23:29in the tool descriptions and the
23:30prompts, but sometimes you really want
23:32to enforce that it is not doing anything
23:35funky. You know, agents are
23:37non-deterministic things, and sometimes
23:39they will ignore you. We know of all
23:41these phenomenons such as needle in the
23:43haystack and lost in the middle and a
23:46lot of instances where agents will just
23:48not work as you intend them intended
23:51them to.
23:53This is where you want to put some
23:55deterministic enforcement. We did this
23:58around the tool that takes actual visual
24:01snapshots. Not the accessibility
24:02snapshot we talked about before, but the
24:03actual visual snapshots.
24:05We had a folder that we defined and we
24:08said, "This is the output folder. This
24:09is where we want you to put images."
24:11But there is a possibility that the
24:13agents will go rogue and just store
24:16images in other places. So, that's where
24:18we want to draw the line and make sure
24:21this never happens.
24:25Okay.
24:26So, as always, we have V3 now, which is
24:30the one we want to look at. So, going
24:33back to main, we see we have this here,
24:36V3 guardrails. We dive in and we see
24:40uh that we have again our wrapped
24:43Playwright tools. Obviously, this time
24:44it's going to do something a little bit
24:46different as we increment every time.
24:49So, we still have the names, we still
24:51have the descriptions, and uh going down
24:54down down down. By the way, we can
24:55already see that apart from the V3 that
24:58we al- always have, which looks like
25:00this, and the tool wrapper, which we had
25:02before, we now have another class called
25:04path validation. Let's see where we use
25:06it. So, we're going down down down.
25:08We're going past the dictionary, past
25:10the tools to filter. We're in the um the
25:13method that method that we're always
25:15importing, wrap Playwright tools. And
25:17wrap Playwright tools, as before, it
25:19goes over all the original tools we got
25:22from our Playwright MCP,
25:24filters what needs to be filtered, uh
25:26gives the enhanced description for each
25:28of the tools if we find it, and then as
25:31before, we have the same helper function
25:32create Playwright tool wrapper, that
25:35function that takes a tool, gives it the
25:37enhanced description, and creates a new
25:39tool out of it. Same functionality, new
25:42description. Let's see what's changed
25:44now.
25:45So, when we want to create the new tool,
25:47right? So, this is the tool we're
25:49creating,
25:50as we said, a tool is just a a callable
25:53function with some description. We give
25:55the new description,
25:57and before we had this part because we
26:00said, "What does the new tool do?
26:02Exactly what the old one did." But we
26:04added this part. We're saying, "If the
26:07tool that is now
26:09being activated, if it is the take
26:11screenshot tool, and we've kind of
26:13researched the tool and we know that it
26:15uses under the hood it will use either
26:17the path or file name as keywords, at
26:19least the relevant keywords for us.
26:22So, if if you're trying to invoke the
26:24tool and it is this tool, then
26:28find the- these keywords
26:30and validate them. And we have some
26:33helper functions. This path validation,
26:35these are just helper functions. We
26:37don't need to go too much too much into
26:39them, but they're just deterministic
26:41logic where we take the path that the
26:43tool chose, and we take our path where
26:46we want to enforce things being stored.
26:49We call it the screenshots root.
26:51And we just use this method.
26:55We want to know if that path is relative
26:57to the scree- the chosen path is
26:59relative to the screenshot path. So, it
27:01is a deterministic way once we
27:03understand where the tool intends of
27:05sto- to store the image, we know if it
27:09is
27:10inside the right folder.
27:12So, going back to our helper helper
27:14function, every time the tool is
27:17invoked, before it actually gets
27:19invoked, we we kind of extract the path
27:23it intends to save to to invoke it with.
27:27>> [snorts]
27:27>> We try to validate, and if it is not a
27:30valid path, it will not reach
27:32invocation. It will raise an error.
27:36So, this is a deterministic way where we
27:38can stop the tool right in its tracks
27:40with a deterministic guardrail. Now,
27:42another nice thing to note is that we
27:44might raise an excep- exception, we
27:45don't return an exception. We don't want
27:47the whole agentic process to fail.
27:49Instead, we handle this nicely by
27:52creating a very nice agent-facing
27:54explanation. And and what the error what
27:56the agent will get back is this nice
27:58match message saying, "Listen, access is
28:01denied. You can't save it there. You
28:03need to save it here. Please provide a
28:05proper file name and proper path." An
28:08agent that gets this message is most
28:10likely to just try again,
28:12but aligned. This is our way of aligning
28:15it, and it will try again, give a
28:17correct path, uh manage to traverse
28:21this, get here,
28:23and save the image. So, that's exactly
28:25what we want to happen in these very
28:28mission-sensitive
28:29um
28:30security-related scenarios. So, that is
28:33amazing.
28:35Okay.
28:36Personally, I love this point. I think
28:39it is so common and a a tool everybody
28:41must have in their arsenal. But it is
28:43time to go forward. We have two more.
28:47This might be a little bit more niche,
28:48more in the advanced side, but it is
28:50worth kind of checking out, knowing that
28:53it is a possibility. This talks about
28:55composing new tools from existing tools.
28:58And the place where we did that
29:00was interesting. It was actually it also
29:02had to do with the same take snapshot
29:04tool.
29:05We felt that sometimes we want to take
29:08snapshots in general. It's kind of a
29:11generic thing.
29:13But when we take screenshots
29:16uh in the context of evidence at the end
29:19of the flow,
29:20we felt that maybe having a separate
29:23tool for that was
29:25uh
29:26was in order. Why?
29:28We can create a new tool
29:30whose functionality is essentially a
29:34pretty much the same as take regular
29:36snapshot, but because it will have
29:38separate descriptions,
29:40the agent can choose either this or
29:42that. And we can also tell it to behave
29:45slightly differently in both cases. And
29:48we can also give some additional
29:51additional actions to do, maybe even
29:53deterministic actions just before
29:56invoking the original tool.
29:58So, we created a new agent called the
30:00evidence tool. And let's take a look
30:01what we did there.
30:08All right. This time around, we have V4
30:11imported as we would expect by this
30:14point. So, we can take a look at V4, see
30:16how it implements get tools. We go in,
30:20this opens up. This looks very similar
30:22to before. Actually, everything looks
30:24similar. We have tool wrapper, we have
30:26path validation, very very similar.
30:28What's the difference? Well, we inject a
30:30new tool.
30:32So, we have our, you know, inside tool
30:34wrapper, what we would have come to
30:35expect by now. We have the the tool
30:37names, the tool descriptions, the
30:39dictionary.
30:41We have our tools to filter.
30:42Everything's the same. And then we have
30:44the same old, same old wrap play rights
30:46tool, which iterates over all the tools,
30:49filters what needs to be filtered, and
30:51creates any tools that we need to create
30:53with enhanced descriptions. But, so this
30:57is what we used to have. We finished the
30:59loop, and we have this. We have this
31:01part injecting a brand new tool. We
31:04chose, and you don't have to go the same
31:06route, but we said, because this is
31:08building on the screenshot tool, then
31:10only create this new tool if the
31:12screenshot tool has not been filtered
31:13out. Totally optional, just what we
31:15chose to do in this particular use case.
31:18And so, we have our new
31:20description for this new tool.
31:23And we create a new tool. Let's take a
31:25look at the at the screenshot.
31:28And sorry, the description.
31:30So,
31:32it's a description as you might expect
31:35from a description of this kind of tool.
31:37Take a screenshot specifically for the
31:39purposes of evidence. And because the
31:41prompt, when it describes the flow,
31:44tells it that this ends with taking
31:46snapshots for evidence, the agent will
31:48probably know to choose this tool over
31:51the regular screenshot taking tool.
31:53Uh we tell it to use this only when
31:54capturing things for evidence. And we
31:57also specify how to go about doing
32:00evidence. We say, for example, that when
32:02it creates an image and it wants to
32:04store it, we want it to identify the
32:07relevant ticket and put include the
32:10ticket number in the file name being
32:13saved. And so, you can see example of
32:15how we're going to have two tools. The
32:17agent will know to differentiate when to
32:19use this one or whether to use the other
32:20one, specifically in the context of
32:22evidence taking. And when it chooses
32:25that tool in the context of evidence
32:26taking, it will This will cause it to
32:30have different considerations when
32:31choosing the image name, for example.
32:33And you can do a bunch of other things.
32:35You can add guardrails or deterministic
32:37actions that are specific to this tool,
32:39for example. So, really, the sky's the
32:42limit, the world is your oyster, and
32:44knock yourselves out.
32:47So, this was this.
32:49Moving on to our final fifth and final
32:52point.
32:53Look at this. Our server all the fires
32:56out. It The system is booting. Looks
33:00And still, we have one final point,
33:02which we actually touched touched on in
33:05the beginning of the talk. This is
33:08treating tools as deterministic
33:09functions or callable functions.
33:11Sometimes, we get all these wonderful
33:14functions all this code from the people
33:17at these third-party teams. In our case,
33:19the team of Playwright, they gave us all
33:21this really nice code, and we can just
33:23call this code
33:25outside of the agentic flow. Just just
33:28call it. Just
33:30Just disregard the description and just
33:32use the function they gave us. Where do
33:34we use that? I don't know if you
33:36remember, but when we first looked at
33:37the code, we had this function called
33:39login to Buzz, and we said we'll get
33:41back to that. This is us
33:43uh kind of closing that loop.
33:46Let's take a look.
33:49Okay.
33:50Back to the top of our main pie file. We
33:54go down down down down down down down
33:56down down. Here is the imported V4, but
33:59today we're interested in something But
34:00now we're interested in something else,
34:02which is this part. When we just When we
34:05just start out, we kind of define the
34:08MCP client, we define this
34:11a class that implements get tools, we
34:13get our tools, and the first thing we do
34:16is we have this deterministic function
34:19called login to Buzz. Only after we log
34:22in to Buzz, do we actually create an
34:24agent, and then and then we give it all
34:26the messages it needs, and we invoke it.
34:29So, why is the logging deterministic
34:32here? Well,
34:33it seems that logging is kind of tricky,
34:37because this is a toy example again, but
34:40in a real product, we need to log in to
34:42our system, we need to log in to client
34:43systems.
34:45So, and each client might have a
34:47different logging mechanism. And they
34:49can be tricky, and they can have
34:50secrets, and they can can A lot of
34:53things can make this kind of
34:54complicated. So, on the one hand, it's
34:56complicated, and on the other hand, it
34:58is a an action that we will always want
35:02to take. There is no agentic flow for
35:04spec reviewer that does not begin with a
35:07login.
35:08And so, we And so, we do use tools we
35:12got from MCP server
35:14in order to achieve this. We're just not
35:17letting the agent try to do it, because
35:19it's We saw it gave us subpar results.
35:23So, for these very specific niche
35:27use cases, sometimes you might want to
35:30take matters into your own hand.
35:32Here, if we're going to the function,
35:34not much going on. I
35:37Again, toy example, the real product
35:39behaves very differently, but here I
35:41just hid some JWT tokens as environment
35:44variables, and I, you know, this
35:47function accepts all the tools from the
35:49MCP server. We kind of pluck the the
35:52ones we want.
35:54And we
35:56And the gist of it is that we are going
35:59to inject these JWT tokens into the
36:02browser's local storage,
36:05and then when we click click the login
36:07button, we will just log in like magic.
36:09So, we just do this deterministically.
36:11Once we have logged in, we take the
36:14reins and give it to the agent. We say,
36:16"You're logged in, you're off to the
36:18races." And that's how we just unburden
36:20the agent from this
36:23somewhat clunky action it needs to take,
36:26which we can just take off its hands,
36:28not bother it it, and its context with
36:30it.
36:31Um
36:32Cool.
36:33So, that simplifies things, at least for
36:36us.
36:38Oh, I almost forgot. I still owe you one
36:41last thing.
36:42I think we should fire this thing up now
36:44with all the improvements,
36:46and see where we land. We failed on the
36:48previous run, didn't we?
36:50Okay. Let's Let's see how it works.
36:53So, going back to the familiar code
36:56base,
36:57our main,
36:58Nope. And we run this.
37:02We no longer need the the breakpoint. I
37:04think I took it out. Yep. And this is
37:07just running.
37:09Okay. So, this is going to start by
37:11doing all that JWT magic in the back,
37:13which we now know how it works.
37:16And let's see how it goes.
37:19I'll take the time to remind you that we
37:20gave it a ticket
37:23and a
37:25a design, and it needs to see this
37:27drawer. That's what it's actually doing
37:29now. Okay. So, it kind of reloaded. I
37:32think maybe the JWT tokens have kicked
37:34in. It go It's going into our home page
37:37called changes, and it's probably going
37:39to do a little bit of reasoning, and
37:41it's going to need to find the way to
37:43navigate itself through the system to
37:45the agents tab.
37:47It doesn't always show it. It has a lot
37:49of late latency and lag, at least from
37:51our experience, but it does take a
37:54screenshot as evidence. It finished.
37:56Let's see what it did. So, it says that
37:59this passed. It said It claims that the
38:02configuration drawer is present and
38:04includes the sections that are need to
38:07be there. It has a screenshot. The
38:09screenshot has a file name that includes
38:12the ticket, like we said. It has
38:13reasoning. Let's take a look at the
38:15screenshot. So, you see the the image we
38:19gave it was a dark mode, and it It was
38:23navigating in in light mode, and it took
38:25it, and it looks It looks great. It
38:28looks like it's it's done, and it's
38:30correct. Now, I will say that
38:33it did see a configuration drawer. It
38:36did see everything that's supposed to be
38:37in it. It might not be pixel perfect,
38:41but pixel perfect verification is
38:43something that we worked very hard on in
38:45our real product. So,
38:48outside of this toy example, pixel
38:49perfect does work, because it is super
38:52important for front-end validations,
38:54making sure the padding and the margins
38:55and all the
38:56style is just as it needs to be.
39:00Amazing. So, with that done, I think
39:03we're
39:04It's said it It's almost goodbye time.
39:07So, I think it's We're all We're
39:09starting to get ready to wrap up. Look
39:11at our server. Look at it. It's amazing.
39:13It's beautiful. It's working. Green
39:15lights are blinking. Engines roaring.
39:17GPUs churning.
39:20Millions of spec reviewer being being
39:22executed and a great acceptance rate.
39:24Thank you for all your help.
39:27Let's just do this one final summary
39:32before we each go our separate ways. So,
39:35as we said, we looked at agentic tools
39:38today, whether they comes whether they
39:40come from libraries, MCP servers, or any
39:42other place. We saw that sometimes they
39:44will fail out of the box. They might be
39:46very generic. They might be not tailored
39:48enough for our use case. And tailoring
39:51them is what's going to make our agents
39:54pop. Now, sometimes you wanted to curate
39:57the tools and kind of ease the load on
40:00the on the context window. Sometimes we
40:02wanted long descriptions and more
40:04verbose tools. So, there's no
40:06one-size-fits-all. It's mainly a
40:07question of how do I kind of mold the
40:10tools to best fit my use case. Sometimes
40:14they'll be deterministic. Sometimes
40:15they'll be flexible. It depends. And
40:17you're going to need to tinker with it
40:19and make it your own and and and and
40:22strive for the best possible setup that
40:25you can do to achieve your goals. I hope
40:28this has given you a few pointers, uh
40:30things that you can try out. This has
40:32been amazing and great fun. Uh I'm I'm
40:35going to move up here now because I have
40:37a shameless plug
40:39after I thank you for listening. And so,
40:42always feel free to reach out. I will
40:44see you guys in the next one. Cheers.