Full transcript
0:00Awesome.
0:01[Music]
0:17Thanks so much for coming. Um, I wanted
0:20to give a bit of a talk on implementing
0:22MCP clients and talk to more about MCP
0:25at scale within a large organization
0:27like Anthropic. Um, I wanted to give
0:30first a little introduction from me. Uh,
0:33my name is John. I've spent 20 years
0:34building large scale systems and dealing
0:36with the problems that that causes. And
0:38so I've made a lot of mistakes and uh
0:41I'm excited to give maybe some thoughts
0:43on avoiding some of those mistakes. I'm
0:45currently a member of technical staff
0:47here at Anthropic and I've spent the
0:49past few months um focusing on tool
0:52calling and integration and implementing
0:54MCP support for all of our internal like
0:56external integrations within the or
1:00so
1:02looking at tool integration with models
1:04we've kind of hit this timeline where uh
1:06models only really got good at calling
1:08tools
1:09uh like kind of late midl last year and
1:13suddenly
1:15Everyone got very excited because like
1:17your model could go and call your Google
1:19Drive and then it could call your maps
1:20and then it could send a text message to
1:22people. And so there's a huge explosion
1:24with like very little effort you can
1:26make very cool things and so um teams
1:29are all trying to move fast. Everyone's
1:30moving very fast in AI. Custom endpoints
1:32start proliferating for every use case.
1:34there's a lot of like services popping
1:36up with like slashcall tool and slash
1:38like get context and then people um
1:41start to realize there's additional
1:43needs some authentication a b a bunch of
1:45stuff there and this kind of led to some
1:48integration chaos where you're
1:49duplicating a bunch of functionality
1:51around your org nothing really works the
1:54same you have an integration that works
1:56really well in service A but then you
1:58want to use it in service B but you
2:00can't because it's going to take you
2:01three weeks to rewrite it to talk to the
2:03new interface And so we're in this kind
2:05of spot and the place that we came to at
2:10Anthropic is realizing that over time
2:12all of these endpoints started to look a
2:14lot like MCP. Uh you you end up with
2:17some get tools, some get resources, some
2:20elicitation of of details. Um,
2:24and even if you're not using the entire
2:26feature space of MCP uh as a whole
2:29immediately, like you're probably going
2:31to go extend into something that kind of
2:33looks like it over time. And when I'm
2:37talking about MCP here, there's kind of
2:39two sides to MCP that in my mind feel a
2:43bit unrelated. There's this JSON RPC
2:45specification which is really valuable
2:47as engineers. It's like a standard way
2:49of sending messages and communicating
2:51back and forth between uh providers of
2:53context for your models and the code
2:55that's interacting with the models. And
2:57uh getting those messages right is the
3:00topic of huge debate on like the MCP
3:03uh repos. If you're involved with any
3:05standardization process ever, you know
3:07how those conversations end up going.
3:09And then on the other side there's this
3:10global transport standard which is the
3:12stuff around streamable HTTP oath 2.1
3:15session management. And global transport
3:20standard is hard because you're trying
3:21to get everyone to speak the same
3:22language. And so it's really nitty but
3:25there's not a lot of like most of the
3:27juice of MCP is in this the message
3:29specification and the way that the uh
3:31servers are interacting. Um and so we
3:35started asking ourselves like can we
3:36just MCP for everything and we said yes
3:38with the caveat that yes is for
3:42everything involved in presiding model
3:43context to models. Um we have this
3:46format where your client is sending
3:50these messages. Something's responding
3:52with these messages. Um where that
3:54stream is going it really doesn't
3:55matter. It can be on the same process.
3:58It can be another data center. can be
4:00through a giant pile of uh enterprise
4:03networking stuff. Um, it doesn't really
4:07care at the point that your code is
4:08interacting with it. You're just calling
4:09a connect to MCP and you have a a set of
4:12uh a set of tools and methods that you
4:15can call. So, uh,
4:19standardizing on that seemed useful. Um,
4:21standard why standardize on anything
4:22internally? Um, being boring on stuff
4:25like this is good. It's not a
4:27competitive advantage to be really good
4:30at making Google Drive talk to your app.
4:33It's just a thing that you need to do.
4:36It's not your differentiator. Uh having
4:38a single approach to learn as engineers
4:41makes things faster. You can spend your
4:44cycles working on interesting problems
4:45instead of trying to figure out how to
4:47plum uh integration. And uh if you're
4:50using the same thing everywhere, then
4:51like each new integration might clean up
4:52the field a bit for the next person who
4:54comes along. Um it's it's over overall a
4:57good thing in cases like this where
4:59we're we're not really doing anything
5:00interesting. We're plumbing context
5:02between integrations and things that are
5:04consuming the integrations. Uh
5:07what is standardized on MCP internally?
5:09Um I this is where I might make an
5:11argument to everyone that there's
5:13already ecosystem demand. You have to
5:15implement MCP because everyone's
5:16implementing MCP. So why do two things?
5:19Um it's becoming an industry standard.
5:22there's a large coalition of engineers
5:24and organizations that are all involved
5:26in building out the standard. Uh all of
5:28the major AI labs are represented in
5:30that. So you you know that as new model
5:34capabilities start to be developed uh
5:36those patterns will be added to the
5:37protocol because all the labs want you
5:40to use their features. So I think the
5:42standardizing on MCP internally for this
5:44type of context is a is a good bet. And
5:47one of the things you get with MCP is
5:48that it solves problems that you haven't
5:50actually run into yet. like there's a
5:51bunch of stuff in the protocol that
5:54exists because there's a problem and a
5:56need and
5:57having those solutions at hand when you
5:59run into them is really important. So
6:00sampling an example of where this might
6:02be valuable in your company. You might
6:03have four products that have four
6:05different billing models uh for reasons
6:08because you're building fast. Um you
6:10might have a bunch of different token
6:11limits. You might have different ways of
6:12tracking usage. This is really painful
6:15because you want to write one
6:17integration service to connect your
6:19slides and how do you go and like hook
6:23the billing and the tokens up correctly
6:24and MCP has uh already has sampling
6:27primitives so you can build your
6:28integration you can just be like okay
6:31your integration sends a sampling
6:32request over the stream uh the other end
6:34of the pipe fulfills that request you
6:36can go and hook it in everything works
6:38great and so this is a thing that uh a
6:40shape problem that might take you a
6:42bunch of effort
6:44uh internally without this, but you
6:46already have the answer kind of gift
6:47wrapped for you in the protocol.
6:49And so at Anthropic, we're running into
6:51some requirements converging. We're
6:53starting to see external remote MCP
6:55services popping up like mcp.
6:59We wanted to be able to talk to those.
7:01Talking to those is complex because you
7:03need external network connectivity, you
7:05need authentication. Uh
7:08there's a proliferation of internal
7:09agents. people have started building uh
7:13PR review bots and like Slack management
7:16things and just lots of people have lots
7:19of ideas. No one's really sure what's
7:20going to hit. So, we're having a huge
7:22explosion of LLMbacked services
7:24internally. Uh with that explosion,
7:25there's a bunch of security concerns
7:27where uh you don't really want all of
7:31those services to be going and accessing
7:34user credentials uh because that ends up
7:37being being kind of a nightmare. don't
7:39want uh outbound external network
7:41connectivity everywhere. Um auditing
7:44becomes really complex. Uh and so we are
7:48looking at this problem. We wanted to be
7:50able to build our integrations once and
7:51use them anywhere. And so, uh, a model I
7:55was introduced to by a mentor of mine
7:57and a while ago is the pit of success,
7:59which is the idea that, um, if you make
8:03the
8:05right thing to do, the easiest thing to
8:08do, then everyone in your org kind of
8:10falls into it. And so uh we designed a
8:14service which is just a piece of shared
8:16infrastructure called the MCP gateway
8:17that provides a single point of entry
8:19and provided engineers just with a
8:21connect to MCP call that returns a MCP
8:24SDK client session on the end and we're
8:26trying to make that as simple as
8:28possible uh
8:30because
8:32that way people will use it if it's the
8:34easiest thing to do. Um, we used URL
8:36based routing to route to external
8:37servers, internal servers, it doesn't
8:39matter. It's all the same call. Uh, we
8:41handle all the credential management
8:43automatically because you don't want to
8:44be implementing OOTH five times in your
8:46company. Uh, gives you a centralized
8:48place for rate limiting and
8:50observability.
8:51Uh, I have an obligatory diagram here of
8:54a bunch of lines going in and out. But,
8:56uh, here's a a gateway in the middle.
8:58This is kind of the thing. Just one more
9:00box will solve all our problems. Uh, can
9:03I go next?
9:04Uh where is my
9:08yeah ah uh so the uh the code that we
9:12have here we just made some client
9:13libraries where we just MCB gateway
9:15connect to MCP uh we pass in a URL an
9:18org ID account ID this is like a bit
9:20simplified we actually pass a signed
9:22token to authenticate because it's
9:23accessing credentials but this is the
9:26basic idea and then importantly this
9:29call returns an MCP SDK object which
9:31means that when new features get added
9:33to protocol. You just update your MCP
9:35packages internally. You get those
9:37features across the board. Everything
9:39works great. The same code seamlessly
9:41connects to internal external
9:42integrations.
9:44When it comes to transports, uh, and
9:47this is a bit high level and handwavy
9:48because everyone's setup is different.
9:50Um, internally within your network, it
9:53really doesn't matter. You can do
9:54anything you want. We've got the
9:55standardized transport for connecting to
9:57external MCP servers. Um, but really
10:01just picking the best thing for your
10:02org. So we went and picked
10:06uh websockets for our internal
10:08transport. And here's just a quick code
10:12example. It's nothing special. We just
10:13have a websocket uh that's being opened.
10:16We are sending these JSON RPC blobs back
10:18and forth over the websocket. And then
10:20if I can make this scroll down
10:25at the end, we just pipe those uh read
10:28streams and write streams into an MCPS
10:30SDK client session and we're good to go.
10:32We've got MCP going. Um you might want
10:35to do this with gRPC because you want to
10:37wrap these in some multipplexed
10:39transport so you don't have to open one
10:40soocket per connection. That's pretty
10:42simple. Also, uh we have read stream
10:45right stream at the end. Uh starting to
10:47see a pattern here. You can do like Unix
10:49socket transport if you want. You can
10:50just have uh messages be passed that
10:53way. Read, stream, write, stream at the
10:54end. MCU works great. Um I threw in an
10:57enterprisegrade email transport
10:58implementation over IMAP. Um which is
11:01pretty much the same thing. You just go
11:03through here is our server. We're
11:06sending emails back and forth. Uh dear
11:08server, I hope this finds you well. Uh
11:11MCP request start. And then we pipe
11:13those into a client session at the end.
11:14And so it truly doesn't matter like
11:16whatever it takes inside your
11:18organization is great. We set up this
11:21unified authentication model where we're
11:23handling
11:25ooth at the gateway uh which means that
11:27consumers don't have to worry about all
11:29that uh all that complexity in their
11:32apps. Uh we added a get ooth
11:35authorization URL function and a
11:37complete ooth flow because you might
11:39have different endpoints at anthropic.
11:40We have api.anthropic.com and we have
11:42cloud.ai AI and we might want those
11:43redirects to go back to different
11:45places, but uh this is tied on the
11:47gateway. It's really easy to start a new
11:49authentication. Uh a real advantage of
11:51having this put on your gateway is that
11:54the credentials are portable. If you
11:55have a batch job that you're kicking
11:57off, um your users don't have to
11:59reauthenticate to that. You're just
12:01calling the same MCP with your internal
12:03user ID and they get everything added
12:05correctly. You're also internal services
12:07don't have to worry about your tokens.
12:09Um so your request comes in internally
12:12for us. We're hitting a websocket
12:14connection to MCP gateway uh with O
12:18token provided as headers to that. Uh
12:20the gateway receives your stored
12:22credentials. You create an authenticated
12:24SDK client. You just pass in the bearer
12:28token to the O header. Uh and then
12:31you're good to go. The MCP client
12:33receives a readstream and a right
12:34stream. And so you just plum those read
12:36stream and write stream into your
12:37internal transport and you're and you're
12:39you're good to go. Uh one of the things
12:42that this gives for your org that's not
12:44immediately obvious but is really
12:46valuable is a central place for all of
12:50your context that your models are asking
12:52for and all the context that's flowing
12:55into your models for your org. Uh
12:57there's some papers written on MCP
13:00prompt injection attacks. There is a
13:03general risk of uh models going and
13:06having access to Google Drive and
13:07deleting everything in Google Drive.
13:09There's some need of uh enforcing
13:12policy. You might want to be able to
13:13like ban
13:16malicious servers. Um do some content
13:19classification on the request, see
13:21what's coming in, kind of given audit.
13:23And the really nice thing about this is
13:24that because it's MCP, all of your
13:26messages are in a standardized format.
13:28So it's really easy to hook into that
13:29stream and be like, "Okay, here is my
13:31tool execution message processor or here
13:34is my tool definition thing or here's my
13:36resource management." And so the payoff
13:38that you get from this is um adding MCP
13:41support to new services is as simple as
13:44possible. You just go and import a
13:46package. Uh it doesn't matter what
13:47language you're in. We've got multiple
13:49languages internally. They all have
13:51their own kind of packages. Engineers
13:53can focus on building features and not
13:54plumbing. uh you have the operational
13:57simplicity of having a single point of
13:58ingress egress and standardized message
14:00formats and you get future features for
14:02free as the protocol involves you get
14:05all of that work uh naturally
14:08and so just wanted to go through some
14:13takeaways from this that I I want to put
14:15to you is that MCP is really just JSON
14:18streams and how you pipe those streams
14:19around your infrastructure is a small
14:21implementation detail. That's a couple
14:22lines of code hook the stream into the
14:24client SDK that makes the messages. Uh
14:28the
14:30you should standardize on something,
14:32anything. I think MCP is a good idea. If
14:34you don't think it's a good idea, like
14:35just pick something. Um your future self
14:37will thank you. Uh build some pits of
14:39success. You really want to make the
14:41right way to do a thing the easiest way
14:43to do a thing and then everyone just
14:44falls into doing the right thing
14:45naturally. And also centralizing at the
14:48correct layer. So solving some shared
14:50problems off and external connectivity
14:53once allows you to spend your time
14:54working on uh more interesting problems
14:57that are more valuable to you and your
14:58your business.
15:01Thanks. That's all I got for you. Uh
15:03thank you so much for coming out.
15:06[Music]