Full transcript
0:07[music]
0:15>> All right. Hello, London.
0:19>> [applause]
0:21>> And I hope everyone's been enjoying the
0:23AI Engineer Europe so far. For there's
0:26so many amazing speakers. I've been like
0:29watching talks and talking to people for
0:30days now and it's been immense.
0:33I'm Sam. I lead development of GitHub's
0:36MCP server and yeah, I'm here to talk
0:38about mostly challenges we've faced
0:41building and scaling our remote server,
0:44how we've overcome them,
0:46and uh before I will start, I just like
0:48I like messing with people. So, you
0:49know, here quick show of hands,
0:52who's used an MCP server?
0:56Good. Good.
0:57Uh who's used GitHubs?
0:59Who has a hot take on GitHub?
1:02>> [laughter]
1:03>> And uh yeah, does anyone build a server
1:05or client?
1:08Oh, nice. Quite a few. Um and yeah, has
1:10anyone contributed to the specification?
1:14Oh. Oh, yeah. I got I got one. That's
1:16actually the first one, I think, other
1:18than the MCP dev summit. There was quite
1:20a lot of them. Uh
1:22but um [snorts]
1:23yeah, anyway, it's really awesome to see
1:25so many hands. So, I'm glad that I've
1:26actually come to the right place. But
1:28yes, for GitHub, you know, our MCP
1:30journey started well, at least in public
1:32in April last year.
1:35And uh we actually open-sourced our
1:38local MCP in April last year.
1:42And we've just turned 1 year's old. So,
1:44I'm super stoked by that. But um yeah,
1:46back then, right, there was a tremendous
1:48buzz.
1:49Uh we were the most starred repo on
1:51GitHub of of the particular week. And uh
1:54like the exposure meant we got a high
1:56volume of public contributions,
1:59uh rapidly filling gaps in plat-
2:01platform coverage that people kind of
2:03wanted to add tools and things. And you
2:05know, not everything was perfect, right?
2:07After a month or so of new features,
2:09agents in some ways were getting worse
2:11at using GitHub and context windows were
2:14getting blown out quicker.
2:16And uh you know, we picked, I think,
2:18over 100 tools. And certainly at the
2:20time, that was just too many.
2:24Uh LangChain had already produced
2:26research
2:28uh they published in February that year,
2:29you know, of the exact kind of problems
2:31we were seeing.
2:33More tools don't make better agents, you
2:35know, they get confused and forgetful.
2:37Well, I say more tools like more
2:40context and more tools shoved directly
2:42into the context to be precise. But uh
2:45yeah, GitHub's a really expansive
2:46platform. And we provided tools, you
2:48know, for repos, issues, PRs, actions,
2:51projects, like even more things.
2:54Uh but the hard part of solving this was
2:56like we didn't want to prevent users
2:57from having the tools individually that
2:59they needed and they used. Uh and
3:02suffice to say our user base is pretty
3:04diverse.
3:05And probably even like on GitHub
3:07platform at the moment, there might be
3:08like one or two clauses as well. Uh
3:11[snorts] and for the record,
3:13uh there's a team of us who work on it.
3:15It's not just me.
3:17>> [snorts]
3:17>> And my team is awesome.
3:19But uh
3:20yeah, so
3:21to try and fix some of this, you know, I
3:22quick- quickly added this thing tool
3:24sets, which was, you know, a kind of
3:25grouping concept of related product
3:27tools. And users could just pick which
3:29ones they wanted and configure it. Uh I
3:32also like added a dynamic tool selection
3:35thing uh where agents could discover
3:38sets of tools and then turn on in
3:40chunks. And uh we never released it, but
3:43I made a kind of rag version of the
3:44same. Um you know, for kind of semantic
3:47tool search and discovery. But it
3:50Uh
3:51like what what do you think happened
3:54even in spite of all this stuff?
3:56>> [snorts]
3:59[laughter]
4:01>> Everyone used the default settings. It
4:03was really annoying because like in a
4:05way we had all these elegant solutions.
4:08Uh all they did was require users to
4:09actually, you know, configure the JSON a
4:11little bit. And most users just don't.
4:15Uh
4:15maybe it's even a partially a spec
4:17problem cuz
4:19uh you know, for like every proposal so
4:21far for grouping to the MCP
4:23specification for various reasons has
4:24been rejected. And there have been
4:26several attempts. Uh
4:28and like in a sense, like every mode or
4:31configuration we add, you you know, one
4:34could argue is papering over potential
4:36gaps. Uh like or gaps in client
4:39implementations. So, like as an example,
4:42like we have a read-only mode and
4:45uh roughly 17% of our users use it, but
4:47it maps one-to-one to the read-only uh
4:50sorry, yeah, the read-only hint
4:52annotation.
4:53But like no client exposes that as a
4:55method of filtering servers. I think
4:57some gateways now do, but anyway, it's a
5:00it's an interesting easy win for more
5:03enterprise use cases where people often
5:05only want that.
5:06>> [snorts]
5:06>> But uh yeah, we needed to find better
5:08solutions to context reduction. And uh
5:10you don't need to worry too much about
5:12the specifics. This is dated now. But uh
5:15like we started trying to optimize and
5:17we looked at the use the usage patterns
5:20on our remote server. And initially, you
5:22know, we cut the
5:24amount of context used by
5:28focusing the tools more specifically to
5:29the general case. And based on usage to
5:32like about 49% reduction of the initial
5:36load. And then we subsequently also
5:37grouped CRUD tools and brought that down
5:40even more. And I think like I think you
5:42get about 40 tools if you use the
5:43default configuration. And then you can
5:45kind of expand or contract that based on
5:47your own preference. But uh yeah,
5:51like it's easy to customize. And uh
5:53we've also like recently had a massive
5:56push to, you know, reduce output tokens
5:58of a lot of tools as well.
6:00And um in this example,
6:03you know, just by tailoring exactly what
6:06comes with the list pull requests, it's
6:08like actually lost more than 75% of the
6:11tokens used in the output. So,
6:13you know, in terms of how token hungry
6:15GitHub server is, like it's it's it's a
6:17moving target. We're constantly changing
6:19things that improve it. And uh if you
6:21haven't used it in a while, like it's
6:22likely very different from a few months
6:25ago even. And um
6:27yeah, anyway, like and we haven't ruled
6:29out more advanced approaches like code
6:31mode. And we're always experimenting
6:33internally. But uh
6:35on the heels of this, we also dug into
6:37our data and we found some more
6:38opportunities.
6:40>> [snorts]
6:43>> So, yeah, like uh we made a big push to
6:44reduce tool failures as well. And the
6:47success rate is roughly, I think, over
6:4995% at this point. But uh
6:52like not all failure is preventable cuz
6:54agents don't necessarily know which
6:56repos they have write permission on.
6:59They still hallucinate. But uh
7:01we've been able to identify significant
7:03numbers of areas that could be overcome,
7:06mostly by encoding a sort of agent
7:08intent into our tool surface.
7:11And you know, you might have to make
7:13five API calls to make it more robust.
7:16But you know, in that case, we do that
7:18in the server side to reduce round trips
7:21cuz that, you know, saves context, saves
7:23time, and usually um [snorts]
7:26makes uh massively better experience,
7:28you know, makes the agents more
7:30successful.
7:32And yeah, we also started to run evals
7:34last year. Um
7:36I'm not going to go into detail. The
7:38that link takes you to a blog article
7:41that my colleague senior wrote about
7:43doing it. But uh
7:45one of the gists is instead of
7:46micro-optimizing
7:48individual tool descriptions, you know,
7:49you try to test them against each other
7:52to try and make sure that they're called
7:54at the right times and not called at the
7:55wrong times. So that in the pool of each
7:57other, they don't fight for like you
8:01know, you like the perfect tool
8:02description that makes the agent call it
8:04all the time is terrible as is the
8:05reverse of that. So, you need to try and
8:08get that as tight as possible. Um
8:11But yeah, this could be a whole other
8:12talk.
8:13Security, on the other hand, is
8:15something that's like a kind of constant
8:17menace in all of this. I've seen lots of
8:19people talking about this.
8:22Um
8:22and it's a real problem in some ways for
8:24us because, you know, we've a lot of
8:26people using plain text access tokens
8:29for MCP in the wild.
8:31And uh usually they're stored somewhere
8:33the agent can access.
8:35They're frequently long-lived. They're
8:37often over-privileged. And they're kind
8:39of sat there just waiting to be abused.
8:41Uh
8:42end users, like I I don't think they're
8:44choosing this, you know, like it's it's
8:46actually hard to make configuration easy
8:49and secure at the same time. And clients
8:52have to make use of system keyrings or
8:54encrypted storage. And like VS Code
8:56does. Uh but uh you know, the MCP spec
9:00also provided a better way with remote
9:03HTTP, which, you know, is all the way
9:05back to April last year as well.
9:08Um [snorts]
9:08and we embraced this, of course. Um
9:11And we wanted to make secure connection
9:13path of least resistance. Uh we didn't
9:15want users to have to download a local
9:17runtime.
9:18And you know, our remote server supports
9:20OAuth 2.1. And my team even helped add
9:23the proof key for code exchange support,
9:26which is commonly known as PKCE, to
9:28GitHub's authorization server to improve
9:30the security posture for client apps. Um
9:33but as I said, we hoped OAuth would be
9:36the path of least resistance. And again,
9:38perhaps some of you might know what
9:39happened.
9:44Everyone expected us to support the
9:46dynamic client registration. And for us,
9:49like it created more problems than it
9:52solved because like
9:54if you implement it kind of properly,
9:55it's hard not to have unbounded growth
9:57of app databases and challenges of how
10:00you would bucket them for rate limits
10:02and there isn't a reliable app identity.
10:04So, we just considered it and rejected
10:07it and
10:09like we feel like it's a
10:10well-intentioned mistake and we're you
10:12know, we're not the only authorization
10:14server to not support this.
10:16And um
10:19even um
10:21like MCP itself, right? It decided that
10:25client ID metadata
10:27is probably the way to go and I can't
10:30promise that we're going to support it,
10:31but I promise that I am trying to get us
10:33to support it and that should make
10:35logging in like massively easier.
10:38But um yeah, more on that in the future.
10:41And also, speaking of security, some of
10:43you may have seen this.
10:45Um
10:46this was a fun day.
10:48>> [laughter]
10:48>> But
10:50like you know, Invariant Labs published
10:52this and you know, like it's a correct
10:54sort of correctly done prompt injection
10:56exfil attack for getting private data
10:59out of GitHub and
11:01um the thing is you know, they call it
11:03specifically GitHub's MCP server out and
11:06I think that
11:08we you know, we do provide the tools
11:10that can enable that if you just kind of
11:12enable them all, but uh
11:15it applies to almost every agent setup
11:18whether they use MCP or not or whether
11:20they use GitHub MCP, you know, like the
11:21lethal trifecta stuff which I'm not
11:23going to rehash now cuz I think many of
11:25you have probably seen it or you can
11:26look it up like Simon's
11:28Simon Wilson's blog post on that's
11:29excellent, but
11:31you know,
11:32the utility of agents is in conflict
11:35direct conflict with kind of protecting
11:36this stuff and it's like it's an active
11:39space trying to work out how to prevent
11:40these problems, but uh it's not solved
11:42and it's very much not unique to GitHub
11:45and we have users with wildly different
11:47risk profiles, you know, like um
11:52we you know, we even have people that
11:54have like air-gapped GitHub Enterprise
11:57server instances
11:59in like much more secure
12:01and then you know, obviously the
12:03collaborators etc. are also
12:06just running straight to GitHub with
12:08like you know, probably full token
12:10access to the agent everything and
12:12that's kind of also interesting, right?
12:13And like I'm
12:15I'm not naysaying any of this. It's just
12:17it's cool to kind of
12:19see what people do and see if we can
12:20actually support the different use cases
12:22and security postures while everyone
12:24experiments with this stuff.
12:26And uh
12:27we also kind of use like lean on off to
12:31uh
12:31manage tools as well. And this is
12:33something I'm pretty happy with. Um
12:36if you log into GitHub MCP with a PAT
12:40token that we just immediately filter
12:43the tools down by the scopes that the
12:45token has. You
12:47uh you don't have to do anything other
12:48than give it the token.
12:50On OAuth, we support step-up OAuth. So,
12:54you know, you can get a we could return
12:56a scope challenge and then it will
12:57interactively ask the user if they want
12:59to allow the scope.
13:01And if you do, then you can
13:04uh like continue the tool call. It
13:05doesn't fail, which I think is also
13:07nice. And then VS Code, for example,
13:09supports that and I initially worked on
13:12this with them just because they already
13:14have a token to use GitHub and what they
13:16wanted was that if their baked-in token
13:18doesn't have permissions to use
13:20everything, that it instead of just
13:22failing, there was a mechanism for
13:24users having a clean install and then an
13:27upscoping later if they need it.
13:29And yeah, lastly, server tokens as well.
13:31Like they didn't have a like on actions
13:33and things, they didn't have a user.
13:36So, user-specific tools are kind of out
13:38there and then by removing those, we're
13:40just removing kind of constant sources
13:42of failure and wasted context at the
13:44same time.
13:46Uh
13:48we run a completely sort of stateless
13:51server setup and um
13:54we have been using Redis for session
13:56storage, you know, it's standard
13:58observability and deep kind of stack.
14:00Like this is not a weird picture, but I
14:03guess one of the weird things for some
14:04people is a lot of people are running a
14:06stateful MCP server process in the
14:09singular and have kind of struggled with
14:11how you get it into this shape.
14:14But um
14:16for us like we did a few things cuz it's
14:18very dynamic, but like one of the fun
14:20things we did is um
14:23we uh
14:24we actually make a brand new in the SDK
14:26sense a brand new server instance on
14:28every single request and we add the
14:31tools to it at the start. So, whatever
14:33your configuration is, it just builds
14:35this and then you get what you've asked
14:38for or what you're allowed to use cuz
14:40some things have policies that impact
14:42whether you've got tools or not. Um
14:44and
14:46yeah, like we've been able to scale to
14:47this point we serve around 7 million
14:50tool calls a week. And we you know, we
14:53don't have session affinity.
14:55Uh the even the sessions we generally
14:57only use them to identify that's the
14:58only way to identify the self-reported
15:00client identity that comes through MCP.
15:03So, it's useful for us to understand
15:05like what clients people are using the
15:06server with. So,
15:08yeah, like we use sessions for that, but
15:10um
15:13yeah, we also have a like wanted to
15:15bring experiments to all of you and
15:17everyone. And um [snorts]
15:19we have this thing that's a Insiders
15:21mode and
15:23all it all it does is it it turns on
15:26certain feature flags and things for
15:28experiments that we're happy to just
15:30ship to anyone who wants to use them.
15:33And uh this just takes you to the
15:34documentation, but um
15:36like
15:37an example of something that we haven't
15:39released generally yet, but is on
15:41Insiders is our MCP apps and like just
15:45you know, I I set up the example before
15:47I came in, but like it's quite nice when
15:49you're talking to the agent to have the
15:51opportunity to kind of edit the
15:53AI-generated
15:55uh issue especially if you're you know,
15:57you're working heavily in professional
15:59open source stuff and you want to make
16:01sure that it's you posting and it's not
16:03going to get closed as a sort of
16:04bot-generated thing. It like this is a
16:07nice human in the loop thing that MCP
16:09enables and I I much you know, I I
16:13wasn't sure how much I would like it at
16:14first, but then I've come to love it
16:16because I kind of care about how
16:19my issues and things are received by
16:21people and this is just a really great
16:23way to make sure that I can I can check
16:25that.
16:26Um
16:29So, yeah, like in terms of where I think
16:31it's going like if something along these
16:33lines,
16:34I think [snorts] a near future, you
16:35know, server discovery will hopefully be
16:37automatic and tool tool use will
16:39probably become more compositional like
16:42bash or piping tools into other tools,
16:44streaming data through them or like you
16:46know, Cloudflare's code mode approach or
16:48Anthropic's tool search tool API which
16:51just landed in Claude Code a couple of
16:53weeks ago.
16:54And OpenAI recently added a similar API
16:57as well. Sorry, OpenAI added a similar
16:59API, too. And uh
17:01I you know, I I fully expect that like
17:03thousands of tools will be normal very
17:05soon. We're trying to iron out all the
17:06problems that prevented it in the first
17:08place and I'll probably reverse many of
17:10the fewer tools decisions. And uh users
17:13hopefully won't even have to know what
17:15MCP is. They'll just convey what it is
17:18they want to do and the
17:20OAuth setup and like you know, the
17:23tool selection things will become truly
17:25autonomous and I don't think we're that
17:27far away from this, but we're we're kind
17:29of in this experimental phase where
17:31we're not really there yet. But um
17:34I think harnesses like Pi are also
17:36interesting because
17:38you can build a weird client that maybe
17:40optimizes this in a really good way
17:41yourself. So, I would encourage people
17:43to experiment with crazy clients. I I
17:45feel like
17:47you never know, you could be like the
17:49next
17:50um
17:51>> [snorts]
17:51>> uh
17:52like well, if you're super lucky, you
17:54could be like the next Claude, right?
17:56You could
17:57publish something that goes so viral it
18:00totally changes the agentic game. Uh
18:02I wanted to end on a high and look at
18:04some numbers.
18:06So,
18:08like GitHub itself, it's actually got
18:10over 11 million Docker downloads of our
18:13standard IO server which is by not like
18:16by far not the most used version of it
18:17either. Um we've got 126 contributors
18:21now
18:22and over 2,300 issues and PRs which it's
18:25been over seven a day like every single
18:28day for over a year now which I do look
18:31at almost every single thing eventually.
18:33So, it's been like
18:35>> [laughter]
18:35>> quite a year. Um I mean I other some
18:38repos have it even worse, but like I
18:40also love it. So, I please keep doing
18:41it. Um
18:43And yeah, we've got almost 4,000 forks
18:45which blows my mind. I kind of want to
18:47know like the weirder things that people
18:48have done that they haven't contributed
18:50back. Uh [snorts]
18:51yeah, nearly 30,000 stars and uh we're
18:54fast approaching 8 million tool calls a
18:56week. And GitHub itself is also facing a
19:00new challenge.
19:02>> [snorts]
19:06>> This is really intense, right? And it
19:08shows no sign of slowing down.
19:10Uh I still wanted you to keep opening
19:12issues and PRs for us. Like we will
19:14cope, but you know, this is new
19:16territory and um
19:19uh you know, everything's like mildly on
19:21fire for everyone I think these days and
19:23it's just exciting and fun.
19:25But uh yeah, thank you so much FOR
19:27HAVING ME.
19:28>> [applause]
19:33[applause]
19:35>> I THINK I GOT LIKE 30 SECONDS. I DON'T
19:37KNOW IF anyone has anything they want to
19:39ask, but
19:42What's
19:43What's your take on piping tool calls?
19:46Um I you know what? I think like things
19:49like trying out MCP CLIs and things like
19:52that is a fun avenue. I don't think it's
19:54entirely ironed out, but like one thing
19:56you can do take the read-only tools from
19:58some MCP wrapped in a CLI and just give
20:01it a proper help and just see how see
20:04how the agent does like stuff like that
20:05is surprisingly effective.
20:07And
20:08you know, I like I said I want people to
20:10mess with this stuff. So I would
20:11encourage you to just try it if you're
20:13interested.
20:14All right, I'm 0 seconds. I will answer
20:17you but in person if that's okay.
20:24>> [music]