Full transcript
0:07[music]
0:16>> Hello everyone.
0:17Welcome.
0:19Quiet down. Quiet down. All right.
0:22Awesome.
0:27How is everyone?
0:28Yeah, good?
0:30>> [applause]
0:31>> Ooh, thanks. Um
0:33Want to hear some MCP versus CLI
0:35debates?
0:36Yeah, is that why you all came? Anyway,
0:38um
0:39Hello, my name is Matt. Uh I work on MCP
0:42and agents at Cloudflare. And welcome to
0:44my talk. It's all about how we can make
0:46every API a tool for agents. APIs exist
0:48in the wild. How can we connect them to
0:50agents and make them do things?
0:54So, I I really love my job because every
0:57day I get to decide like
0:58uh if an agent looked like this, would
1:00he do this or would he do this? And I
1:02think it's kind of fun.
1:03Um and we often fluctuate weight between
1:05the two of them. Uh someone does
1:06something slightly you think is slightly
1:08funny and then 6 months later we're all
1:09doing it and claiming it was the best
1:11thing in the world.
1:12So, yeah, it's really good crack. But,
1:15the main part of the the role I guess
1:17and like what I end up doing day-to-day
1:19is like how do we give agents hands? How
1:21do we let them
1:22interact with the outside world? And
1:24you're probably familiar with something
1:25like this.
1:26This is tool calling, function calling.
1:28Uh it's been around for a while now. The
1:29LLM writes a function, you execute the
1:31function.
1:33Bash bash bash, weather in London is
1:3418°. It's not, it's like eight and it's
1:36freezing.
1:37Um
1:38sad times.
1:40And then from there we went from uh
1:41bundled tools to something like shared
1:43tools. Like people made tools in their
1:45agents. And the you probably This is
1:48like all like recent history, so
1:49everyone's probably aware of this, but
1:51before MCP we had like uh people would
1:53bundle all their tools in their agents
1:54and then they would keep them bundled in
1:55their agents and then if I was like
1:57trying to interact with Gmail or
1:59something, I would make loads of tools
2:00for Gmail, bundle them with my agents,
2:02and that would be it. And the next
2:04person would have to do exactly the same
2:05thing. And then we ended up with this
2:06like big explosion of uh MCP and remote
2:09MCP about April last year. And the
2:12service providers were like, "Oh, we can
2:14we can like uh
2:16give everyone MCP tools and then
2:18everyone can use the same standardized
2:19tools and we just make it once and we
2:21provide it as another surface for people
2:23to consume our API. Maybe there's a CLI,
2:26there's an API, maybe there's like I
2:28don't know, GraphQL API,
2:30um and there's now MCP as like another
2:32surface. But, this got a little bit fun
2:35because it was okay with like eight
2:37tools.
2:39But, then what happens if you added like
2:40a few more?
2:41Or a few more?
2:43Or a few more?
2:45Or a few more?
2:47Or a few more? And now you you you're
2:49like, "I want to give an agent access to
2:51our whole API surface."
2:53And
2:54well, that ain't going to that ain't
2:55going to happen. Why is it not going to
2:56happen? You've exploded a context window
2:58of the agent.
2:59You've like completely annihilated it.
3:01This is 1.something million tokens. Uh
3:04And this was the problem that we like
3:06came across a few
3:08uh well, around a year ago now.
3:10We were trying to give access to the
3:11whole of the Cloudflare API to agents.
3:14You put all of the the You try and make
3:16naive tools that have every single API
3:18endpoint and you fully explode a context
3:20window. Uh open API spec is 2.3 million
3:22tokens. Into tools, that's something
3:24like 1.1 million tokens. And that's like
3:27never going to fly even with like the
3:30biggest foundational models.
3:32Uh uh And in that time we were like,
3:35"We know this is not necessarily an MCP
3:38problem, but it's how everyone else is
3:40doing it. So, we're we're we're going to
3:42we're going to adapt. We're going to
3:43adapt. We're going to improvise."
3:45Um and we're going to split up our API
3:47into lots of different product-based MCP
3:49services. So, you've probably seen this
3:50like uh a company that publishes 16 MCP
3:54service potentially and then users have
3:55to uh interact with the one that they
3:57want to use when they want to use it. Um
4:00there's much less context, but the user
4:01has to select. And most of the time
4:03there's kind of incomplete coverage. So,
4:06like
4:07for instance like uh one of our one of
4:09our product suites, we might have like
4:11six tools in our MCP server, but the
4:13total API maybe has like 30 endpoints.
4:16Like you've completely missed some
4:17coverage there.
4:19Uh And this is not like fulfilling the
4:20goal of like how do we make every API a
4:23tool for agents. It's actually kind of
4:25kind of annoying.
4:27Uh so so I think we did all a little bit
4:29wrong.
4:30Um well, in Cloudflare we had 16 servers
4:33very very quickly. We were hovering
4:35around 2 and 1/2 thousand endpoints. I
4:36think we're actually at like 2,600 API
4:39endpoints now. Um but
4:41we basically couldn't split up all of
4:43these into all of our servers and the
4:44users had to pick the ones that we
4:45wanted. What we really needed was
4:47progressive discovery of tools. Who's
4:48heard of progressive discovery? Anyone
4:50heard? Yeah, cool.
4:52And that brings us to the crux of the
4:54debates that everyone has on the on
4:56online. Um and that is like
4:59how do we do progressive discovery? And
5:00and like is MCP dead? Like is Was MCP
5:02like a really bad idea? Um
5:05uh And I'm going to say like I I don't
5:06think it was. MCP's a protocol. All of
5:09these can be exposed over MCP. We just
5:11shouldn't be dumping loads of tools into
5:13context. That's like the main thing. We
5:14shouldn't be dumping tools in context.
5:16Um and all capabilities like in the
5:18future we might have prompts and
5:19resources more. Skills are basically
5:20resources.
5:22Um
5:23And we just shouldn't be like loading
5:24all of those at once. So, there's like
5:25sort of three ways you can get around
5:27that problem. Uh there's a CLI which
5:30uh people really like. Uh there's tool
5:31search. Or there's a third one that
5:33we're going to come to a little bit
5:34later.
5:36Um
5:38But, like how does how would a CLI work
5:40for agents? So, this is a a sandbox in
5:42the background.
5:44And if I use our our CLI
5:46and I do something like uh I just call
5:49Wrangler.
5:50We get a bunch of commands. The agent
5:51can like read these commands, pass these
5:53commands and be like, "Oh, I want to
5:54interact with the database. Let's Let's
5:56do Wrangler D1." And maybe we want to
5:59list
6:01list our databases, whatever. And then
6:03after some period of time
6:07and some interactive process apparently,
6:09uh we get uh we get like the databases I
6:12have on my account. And like an agent
6:13can kind of do this um
6:15mostly.
6:16Uh and it and it can call {dash} {dash}
6:18help to get like uh
6:20introspection on like which parameters
6:21it needs and th- this this mostly works.
6:23It mostly works.
6:25Uh it's used very popular by things like
6:26open claw and like people generally
6:29really like CLIs. But, you need shell
6:30access. This is like the main thing.
6:32This is I guess the crux of it. Like you
6:33have to have shell access and that's
6:35kind of annoying.
6:36So,
6:37for things like Cloud Code, they wanted
6:38a bit more of a structured way of doing
6:40things. So, they have uh like tool
6:41search. They have a uh search tool which
6:43loads the tools that they need when they
6:45need them into context. So, say I want
6:47to like create a worker.
6:50Uh what it would do is you it would take
6:52the user the user question, it would do
6:54some sort of keyword matching, and then
6:56it would add K equals say eight tools to
6:59context. And then at some point the LLM
7:01is going to look at, "Oh, actually
7:02workers create, this is the one we
7:04need." And so we're going to use that
7:05one. But, the rest of them stay in
7:06context. Maybe it's not eight, maybe
7:08it's six. They change it changes, but um
7:10yeah, you end up with like two 2,100
7:12tokens and only 500 of them are being
7:13used. But, like it works. It it it's it
7:16works quite well. Um you only load the
7:18tools that are relevant.
7:20Uh And then this last thing is a blog
7:23post that uh Cloudflare published in the
7:24summer of well, last summer. And it's
7:27like how can we instead of doing like a
7:30um
7:30a static search tool or instead of like
7:33enforcing an agent to need a CLI, how
7:35can we do something where we just let
7:37the agent write code? And we let the
7:40agent write code against our API.
7:42And it turns out that TypeScript is
7:43actually well, types are a very concise
7:46way of representing inputs and outputs
7:50um in in a way that an agent can reason
7:52about. So, say you have all of these
7:54endpoints. Have like a get worker
7:56scripts or a create a worker or
7:58something like that. We generate these
7:59types.
8:01Uh and then we let the model
8:03given these types, write some code
8:05against these types. So, here we're
8:06doing code mode list workers. I hope you
8:07guys can see that. Uh
8:09And we're going to try and list some
8:10workers. So, this might be like a user
8:11request to list workers. The model
8:13generates this code um against a typed
8:15SDK that we generate from our API. You
8:18can generate them from open API specs.
8:20Uh
8:21And then we can run that and we can like
8:22list the workers that we have on our
8:23account.
8:25Um we could deploy a worker.
8:27That would be fun. Hello world.
8:30And we could put it behind one of the
8:31hardest things to do at Cloudflare,
8:33which is so weird because it's such a
8:35powerful product, but we can add access,
8:37which is like our our managed IDP. And
8:39now this worker is secure behind access.
8:42Kind of cool. With like a access policy
8:44to only allow me into it and all of this
8:45sort of good stuff. Super super easy.
8:47And an agent can generate all of this
8:49code given our types.
8:51So, this feels like a step in the right
8:52direction.
8:54We just let the model write code. We
8:55benefit from the model getting better.
8:57We benefit from
8:59I don't know, or like our improving our
9:01open API spec. It's like that should be
9:03the source of truth.
9:06But, we had this like kind of weird
9:08thing where we thought this was awesome
9:10and we were pretty stoked about it. Uh
9:11but the clients didn't implement it.
9:14And then I'm when I mean clients, I've
9:16gotten to like MCP terms now. So, the
9:17client is the agent. So, we'll be
9:19referring to the agent as a client from
9:21now on. Um but it So, the clients didn't
9:23really implement it.
9:25And we're like a little bit confused
9:27about why this is the case. Like this
9:28was sort of eight, nine months ago now
9:31and it's a better way of interacting
9:33with
9:34uh with APIs. Just let the model write
9:36code against the API, but they didn't
9:37implement it. And why not?
9:40Uh And that's because like running
9:41untrusted code is mega mega scary. Like
9:44i- if I had said to you a few years ago,
9:46"Oh, we're just going to let
9:49a language model write some code that
9:51we're going to gen- we're going to
9:52execute for our users without looking at
9:54it, without reading it, without seeing
9:55what it does, that might have
9:57potentially like secrets access.
9:59Ideally, it has some secret access.
10:01You'd be like, "That's crazy. That's a
10:02CV, right?" It's a CV. Like, it's a
10:05vulnerability.
10:06That's a problem.
10:08And now we're proposing you to do this.
10:10So,
10:11it is quite scary. What loads of things
10:13can go wrong. We could It could read a
10:15file system, read some secrets that you
10:16don't want it to read. It could
10:17exfiltrate those secrets into a network
10:19request, run infinite loops, consume all
10:21your resources, do like really scary
10:23stuff, run a crypto miner, you know?
10:24That would be That would be bad.
10:26And in the past, people have tried loads
10:28of things to let people run code-like
10:31solutions. So, if anyone's ever written
10:33a DSL,
10:35some sort of like JSON spec about how to
10:37and to interpret that as code, that is
10:39basically this.
10:40If you ever used one of those
10:41integration softwares where you have to
10:43do that, that is this. They just don't
10:44trust you to write code on their
10:45servers.
10:47VMs also, people spinning up sandboxes
10:49to run code,
10:50big sandboxes, big VMs, that's this. And
10:53also code review.
10:56But,
10:57it's kind of lucky cuz we have like a
10:59pretty cool primitive that solves this.
11:00And there will be other primitives that
11:02solve this. I just think this is the
11:03first and so it's worth like worth
11:06shouting about, really. And this is like
11:08how do you run untrusted code in a way
11:10that's super safe for you and your
11:13infrastructure?
11:15And it's kind of like this. So, we can
11:17execute a worker
11:19from a string. And a worker is just a
11:21like a little is like an isolat
11:23in V8. There's many blogs about how all
11:25this works. I'm not going to go into it
11:26super deeply. I'm just going to show you
11:28what it can do.
11:29So,
11:31for instance, we have this like
11:33this piece of code that was generated
11:35and we're going to run this piece of
11:37code that was generated. And this ran on
11:38the back end. It didn't run in my
11:39browser. It ran in a dynamic worker
11:43that's fully isolated. And how how I
11:44guess how can I prove that to you?
11:46If we do this one,
11:47we are trying to get some secrets here,
11:49process.env.
11:51And if we print them, there are no
11:53secrets.
11:54And we also have this weird Cloudflare
11:56global. Ooh, interesting.
11:58Um
12:00if we turn That was with node compat on.
12:01If we turn node compatibility off, we
12:03don't even have We don't even have
12:04process.env there and it all errors out.
12:07So,
12:09we can like influent We have this like
12:10programmable sandbox. It's not quite a
12:13sandbox. It's like a very lightweight
12:14thing that you can put load code into it
12:17and then run it. And I'll show you some
12:18other options later. Like, it's not just
12:20us that has this, but we have one that
12:21we host for you and goes to like
12:23Cloudflare level scale. If you want to
12:24do billions of requests, knock yourself
12:26out.
12:28And now like here's one where
12:30the agent's written some code that
12:31accesses like an external API. And if we
12:34run this one, it's like this worker is
12:35not permitted to access the internet via
12:37global functions.
12:39Or maybe we want it to access the
12:40internet. And now we can give it access.
12:43So, it's a programmable sandbox with got
12:45with programmable guardrails.
12:47And all we're doing here is like
12:48flicking a boolean in the server. That's
12:51like all that's happening here. But you
12:52can provide like a more in-depth
12:54function to be like, "Only access things
12:56to these domains." And that's what we do
12:58on the Cloudflare MCP.
13:01Um if we go to next.
13:03Oh, speaking of the Cloudflare MCP,
13:05this is where I really hope the demo
13:06works.
13:09So, this is an MCP client in this slide.
13:12And if we
13:14ask it a question, we're going to get
13:16some We're going to get like a auth
13:17screen pop-up.
13:21And then hopefully all this works.
13:28Oh, insane.
13:30So, now we have like complete Well, we
13:32have read-only access to the whole of
13:33the Cloudflare API. All of my Cloudflare
13:35infrastructure, I have read-only access
13:37to.
13:38Um
13:39which is pretty cool. These account IDs,
13:41don't worry about them. They're not
13:42secrets in in Cloudflare world.
13:46Uh cool. So,
13:48so we just listed a worker, but you
13:50could do many more things here. Like,
13:51you can deploy workers from your command
13:53line. You can do what we did earlier and
13:54add access to something. You could
13:57[snorts] inspect your DNS. You could
13:59send emails soon.
14:01You can do loads and loads of other
14:03stuff. Like, it's very very cool what
14:05you can do here because you have access
14:06to the whole of the Cloudflare API, all
14:072,000 and something endpoints.
14:13And I guess like
14:16it kind of brings up the question like
14:17where where are we going with letting
14:19agents access external tools? Like, what
14:21does this look like? Like, you have
14:22people installing CLIs for everything
14:24and running it on their own running it
14:25on their own machine. Maybe running it
14:27on a VM. That's kind of cool.
14:29Um
14:31You have us being like, "Oh, you could
14:32just run untrusted code in this like
14:34other in this other place that's like
14:36really isolated."
14:38You have people doing tool search.
14:40You have people
14:42rendering UI's JSON.
14:44I don't know.
14:45Um
14:47And I guess my main thought is that like
14:50we're going to have so many isolated
14:52environments on the on the web.
14:55And there's going to be loads of
14:56infrastructure primitives that allow you
14:58to run this type of untrusted code on
15:00the web because code is actually a very
15:02compact compact plan. Instead of doing
15:05tool calls, you can have one tool called
15:08code where the model generates the code
15:10of your choice and then you run it. And
15:12that code has so many more degrees of
15:13freedom than like an individual tool
15:15call. So, it makes sense to me that as
15:16the models get smarter,
15:18this is what this is what we will do.
15:20And people will adapt their
15:21infrastructure primitives to do this.
15:22So, there'll be so many more of this.
15:23And you see this starting with like
15:25Pydantic Monty,
15:26Deno also, and like we also have it with
15:29WorkerD, the dynamic workers I showed
15:31earlier. Like, people more people are
15:33going to build these primitives because
15:34they're going to become more and more
15:35useful. So,
15:37just like a little explanation.
15:39Um this is WorkerD
15:42like spawning a dynamic worker in this
15:43sandbox and running some code to get a
15:45fib sequence.
15:47You can do the same thing with Deno
15:49with Deno run with some
15:52questionable checking. I have no idea
15:53what that does.
15:55And then you can also kind of do the
15:56same thing with Pydantic. Monty, their
15:59new their new code interpreter for
16:01running untrusted Python.
16:03Because it's Python, we have to download
16:04Python.
16:06Sucks.
16:09This might never work. I actually have
16:10no idea.
16:11Oh, there we go. Great. Uh
16:14So, maybe you can see like where we're
16:16trying to go with this. That there is
16:18there was a previous time where no one
16:20would ever run untrusted code. That was
16:21a CV. Like, you would just immediately
16:23like you have to like
16:25stop allowing that.
16:26Uh
16:27And then it seems that LLMs it's
16:29actually really good for them to run for
16:31them to write code that you can run.
16:33And so now we're building the primitives
16:34to actually enable us to do that. And it
16:36feels like we missed out on this whole
16:37part of the the
16:39the tech scene that like we've never
16:40tried before. Like, in the 1950s, when
16:42you wanted to run something on a
16:43computer in your local town, you printed
16:45out some punch cards and you stamped
16:47them and you gave them to the guy.
16:49And that was kind of like running
16:50untrusted code, right? Like,
16:52that was kind of it. And then when we
16:54went to the cloud, we got away from
16:55that. And now I think we're going to go
16:57much more back to that where
16:59your users can write code cuz your users
17:02are AI.
17:03And AI is very good at writing code. And
17:05that is how they're going to interact
17:06with your platform, whether through MCP,
17:09whether even through like bash and CLI.
17:11Like, I don't mind.
17:12I think they're just going to write code
17:13against your services. And your services
17:14have to be ready for this. Like, your
17:16APIs have to be ready to take a beating
17:18because they have to have good rate
17:20limiting. Cuz I can run this in a for
17:22loop on multiple sandboxes at once and
17:24just hammer your API. Like, you have to
17:26have some way of protecting against
17:27that. Like, this is the new world that
17:29we're that we're now going to be living
17:31in.
17:33And that's like on the server side on
17:35the on the services side. Now, what's
17:36going to happen
17:38on the client side?
17:40So, I think that's almost even more
17:42interesting because that's the
17:43user-facing side of things. Like, the
17:45user's not going to see the server. The
17:46user doesn't care.
17:47The user just What Why is my agent not
17:49getting my Gmail emails? Or why has it
17:51deleted my whole inbox? They're
17:53not going to like They're not going to
17:54see that.
17:55But on the client side, like there's a
17:57lot of innovation that's going to happen
17:58here. And I think we've stalled a little
17:59bit recently
18:01because actually building an MCP client
18:03in particular got really really hard.
18:06Like, to to to actually build a client
18:08that was performative, that worked, you
18:10needed to manage stateful connections,
18:12you needed to manage resumability
18:13between those connections.
18:15There's
18:16There's plenty of other reasons why
18:17building an MCP client was hard, but
18:19like it was a pain, an absolute pain.
18:21And so people had like the most
18:23stripped-down clients they possibly
18:24could. They mostly offloaded to the MCP
18:27SDKs, which are quite bare-bones.
18:29And
18:31no one was building these like more
18:34unique UI experiences on top of that.
18:36And I think that that is going to come
18:37like very very soon. So,
18:39the most the most obvious thing is we're
18:41going to have programmatic tool calling
18:42in the clients. The the previous slide
18:44we just did showing those showing those
18:46sandboxes with WorkerD, Deno, and
18:48Pydantic, that is like just running
18:49untrusted code in a client. People are
18:51going to do that. If your client is
18:52remote, you're going to do it like that.
18:54If your client is locally, well, just
18:55YOLO it, whatever. Just eval it, you
18:57know? It's
18:58going to be fine. But more people are
19:00going to do this programmatic tool
19:01calling. It's going to happen.
19:04And because you're generating code,
19:06people are going to save this code. And
19:08they're going to save it in these like
19:09mini scripts.
19:10And users might be able to decide, "Oh,
19:13this action that I just did, that the
19:14LLM generated for me, I want to keep
19:16that for later." And then it will be a
19:18much faster. So, you can see things for
19:20things like cron jobs. A user might set
19:23up some web scraping job like without
19:25any knowledge of how web scraping works.
19:28And then it generates a script and that
19:30script is run like every day, every 2
19:32days. And whenever it breaks because web
19:34scraping's like pretty brittle, the
19:36agent will fix it and resave the script.
19:37Like, this stuff is how is going to
19:39happen. And I think like these saved
19:40mini scripts, they only work when you
19:42embrace like programmatic tool calling,
19:44but they really do work. And then and
19:46then the the last thing
19:49is we're probably going to have many
19:50many more clients because they've been
19:52so hard to make up until now, and it is
19:54going to get easier.
19:55There's actually only There's not
19:57There's not a huge amount of really
19:58well-used
20:00MCP clients.
20:02That's going to change. And with that
20:05change, like
20:06more people are going to be able to make
20:07them, more people are going to deploy
20:08agents to the cloud that end up being an
20:10MCP client. And I think more people are
20:12going to try and do this like stateless
20:13agent loop thing. Like it it was fine to
20:16have sandboxes for every agent like
20:18running code code locally if
20:21if there were a million agents in this
20:23world. I think when there are like 100
20:26agents for each person
20:28Oh, hello. Let's not do that.
20:30That's going to be That's going to start
20:31getting really tough. And you're going
20:32to have to like embrace a cloud-native
20:34way of doing things, which means that
20:36state has to be something you can turn
20:37on or off.
20:40And
20:42this is I think we're we're nearing the
20:43end, but this is my last thing. It's
20:44like I work a lot on MCP servers and on
20:47the SDK.
20:48And this is where I think that bit's
20:50going.
20:51I think we're going to see MCP as a
20:53middleware
20:54in an when you build an MCP server. When
20:56you build an API, and you build an API
20:58service, it will be a flag that you can
21:00flag on in your favorite framework.
21:02The
21:04the SDK itself is getting super, super
21:05lightweight. And I think by the end of
21:07this year, we'll be like natively in
21:10every single at least TypeScript big
21:13full stack framework. It will just be
21:14there natively. Because it will be so
21:16small, it will literally just express
21:18the protocol in itself. And it will be
21:20silly for them not to have it. They'll
21:21just have a native integration.
21:23And they'll be able to do MCP is true on
21:25all of your APIs. And because all of the
21:27clients will be doing programmatic tool
21:29calling, you can express like your
21:30thousand APIs from one Next.js app and
21:34just do MCP equals true, and expose it
21:36over at the MS tools over MCP as well.
21:39And I think that I think that will
21:40happen.
21:41I've been thinking that's going to
21:42happen for a while, but I think we're
21:44pretty, really close there. And the last
21:46blocker is like fixing the SDK really,
21:48so that it's capable of doing that. It's
21:50capable of fitting in every single front
21:52every single bundle really.
21:54And that's the plan.
21:56Um
21:58You can find out more the We have a Code
22:00Mode blog post that came out pretty
22:01recently. It's how we gave agents an
22:03entire API in a thousand tokens.
22:06If you have a big API, you should
22:07probably do this.
22:09Any accessibility providers, please just
22:10do this, cuz it's really, really good
22:12for people to access your data.
22:13Um
22:15and thank you.
22:16Try out NPMI agents. Thank you very
22:19much. Woo!
22:20>> [applause]
22:25[applause]
22:34[music]