MCP or CLI: Why Not Both? - Jan Peer Stöcklmair, Sentry

0:00I hope you're not getting too confused about the title, but at the end of the presentation I promise you understand what this is about.

0:06Alright, so before I get started, let's go back in time.

0:09The reason why we actually are sitting here is because one of the biggest events in history, I think

0:14is because of the ChatGPT launch in November 2022.

0:18I mean we're also here because of Mastra, so thanks for them.

0:22But they were founded actually later in mid-2024, followed by a lot of more hype right after and o1-preview release from ChatGPT, or OpenAI again.

0:33Funny enough, in November 2024, MCP launched.

0:38There was also more hype around it, more agents are coming out.

0:41But after one year of

0:44MCP being in existence, people said it was dead.

0:48Everybody claimed that the CLI is actually better than the MCP because of various reasons.

0:55We at Sentry we have an MCP.

0:57It's called Sentry MCP.

0:58So is it actually dead now?

1:01It is actually not.

1:02So we are improving it every day further and further and users love it and they're adopting it every day.

1:09So it's not really that.

1:10Also, the MCP version 2 comes out soon, so I think end of the month, so it's far away from that

1:17By looking at the chart on how an MCP server actually works, you have an LLM, you have a round trip to the MCP server where you have tool

1:25tool inputs, there's something going on in the MCP server itself, and then you have an output going back to the LLM.

1:31So it was actually designed for

1:33being used by LLMs.

1:35So now if you want to use it as a user or as a code in a program, it's kind of hard to use it.

1:41It works, but it's not it's not actually designed for it.

1:45And this is why the CLI comes into place.

1:47And luckily we at Sentry have a CLI, and it was created in 2015, long before agents were ever in existence.

1:56The features that we had at the CLI were the release management, we had source map uploads, debug information file uploads, and sending events.

2:04Everything you needed in order to have everything running in Sentry

2:09To work.

2:09It was set up once and then you barely touch it because it's just uploading the source maps on build time

2:16Of course, it didn't work with the project setup.

2:20There was no CLI which acts as a second UI.

2:23There was also no agent integration because you know in 2015 there were no agents.

2:28And you couldn't use the API directly with the CLI.

2:31About the project setup, we thought about something.

2:35First we thought about okay putting in the CLI, but well.

2:39So when we think about Python, how it works in Python, when we want to set it up, you have normal code like this, the only thing you have to add is importing it from the Sentry SDK and then initialize it.

2:51So that's everything you have to do in Python.

2:54However, in JavaScript, there are different frameworks like Next.js

2:59React Router, Nuxt.js

3:00And so on.

3:01There are also different runtimes like Node.js

3:04Different browsers, Bun, Deno, and so on.

3:07On top of that, because JavaScript is kind of robust, and it needs to be shipped to the clients on the browsers, there are bundlers which helps you to minify the code.

3:17Of course, it's cumbersome to add all of that, but once you run it and you run into an issue, you upload or the error gets sent to Sentry and it kind of looks like this.

3:28It's not nice to read

3:30But the CLI actually helped you uploading the source maps to make it magically look like this.

3:35So you can read the code within Sentry itself.

3:39Sentry Wizard was actually here to help you with that.

3:42So the wizard was just a CLI which kind of sets up another CLI, like sentry-cli.

3:50That goes from top to bottom and says, okay, this is what you've done, this is what the features you want to have, and you're good to go.

3:56So the project setup is now done with two CLIs.

4:00We had sentry-cli and the Wizard

4:02But we still didn't have all the features we wanted to have.

4:06And sentry-cli was built in 2015 for one reason, uploading the source maps, and it was okay and it was working.

4:14Really good.

4:15It's also working today really nicely.

4:17And if you're looking at the releases on the sentry CLI, you see a lot of files.

4:22Those files are actually binaries because it is written in Rust.

4:29The thing is, if you want to use it as a SDK within your code, you need to have some binary execution.

4:36In this case

4:38It was fine because it was designed for that.

4:40It was only running within CI/CD pipelines or sometimes locally on your code.

4:44So it was always running.

4:46What we always wanted to have and what we strive for was to have a CLI which acts as a second UI.

4:52So if you want to debug something, you have to go to the web browser, click that and that, and then you find something

4:59But it's kind of cumbersome if you want to have it as a CLI.

5:02Also there is GitHub which has the gh CLI, there's also 1Password with op and so on.

5:09We didn't have that.

5:10until now.

5:12We have now the Sentry CLI, which is a different CLI than the Sentry CLI.

5:17It's called sentry.

5:19And we've rewritten everything from Rust with a TypeScript layer on top to TypeScript alone.

5:25And there's a good reason for that, but I come to that later.

5:29So we didn't just rewrite everything, but we also dropped the "-cli" because it's way cleaner, right?

5:36So sentry-cli is gone and we now only have sentry.

5:39Cool.

5:40So what kind of features do we have?

5:42So first we rewrote the entire code base from Rust to TypeScript.

5:46So we already had feature parity on its own, but we wanted to combine the Sentry Wizard into the Sentry CLI.

5:53So now we have the sentry init command, which helps you to initialize your code with help of an LLM, because every code base is kind of different and it helps, so LLM LLMs can help you with that in doing it better.

6:06The way how it works is like that.

6:07You have your code running, your Sentry CLI is also locally, and the Sentry CLI communicates with the Sentry server and exposes features to the LLM.

6:16So we have

6:18A way to detect your platform.

6:20We also have a way to select the features based on the sentry docs, so we are always up to date on this part.

6:25And there's also planning code mods and applying those code mods.

6:28So it's working, I think, a little bit better than the Sentry Wizard because every code base is a little different than the other.

6:36So how it looks like is once you type in sentry init, you come to this dashboard, you press continue, and then you come to this nice dashboard where on the top right you see the tasks and which step you are currently in.

6:46In this case we are in a selecting features

6:49task and on the left side we get prompted for which features you want.

6:53In this case I have sessions, session replay, tracing, logs, profiling and user feedback.

7:00If you would have now a backend service, you might not have session replay or user feedback because it's depending on which platform you are in.

7:09Cool.

7:10So now when we had that, the sentry init goes back at the end, it verifies with, I think, an eval if everything is working correctly.

7:19And gives you all the stats.

7:21So which platform are you on, which programming language, which were the docs, and so on.

7:26So you always know what is happening behind it.

7:30So now we have not just feature parity, but we also combine two CLIs into just one CLI.

7:36But our main goal was actually to have the CLI acting as a second UI.

7:40This was our main motivator to have the Sentry CLI replacing sentry-cli.

7:44But how does it look like?

7:47This was kind of straightforward.

7:48We just packed all the features we had in the product into the Sentry CLI.

7:54We have two colleagues over here who work on the Sentry CLI.

7:58Thank you for putting everything from the product into the CLI

8:02And we have that now.

8:04I think it was not that straightforward, but from the looks it is kind of straightforward.

8:09And nowadays everybody is using LLMs, or mostly

8:15So we also try to focus now on the agent integration.

8:18Yeah, to just have an easier way on integrating with it.

8:24How it looks like is we have the sentry help command.

8:27The sentry help command gives you like any other help command

8:32information of what commands there are inside one command.

8:37So in this case we have sentry help issue which shows you all the commands like list, events, explain and so on.

8:44The thing is it is human readable, but LLMs need more context.

8:48Of course not too much context because you know LLMs could be quite misleading if there's too much context, but just something in between.

8:55So we added a thing called --json to it, which gives you a little bit more addition to it that the LLM has enough context to figure it out.

9:04So if you scroll down

9:05We have now sentry issues events with a full description, which is kinda human-readable, but better for LLMs because they just love text.

9:14Cool.

9:16But there's another thing.

9:17So with sentry setup, which we run conveniently for you when you install it

9:22It is doing of course all the paths but it also inserts a skill for you.

9:27What it does is in your coding harness you just say

9:31What are my last issues and also include the most important traces?

9:34It reasons in the background, it says, okay, I'll use the Sentry CLI to fetch the recent issues and traces.

9:39Let me load the skill first.

9:40It's loading the skill and the skill translates everything from your natural language into the CLI command.

9:47So in this case you don't have to touch the UI ever again, you just type it in your harness what you want to have

9:53and we translate it for you or actually the LLM translates it for you.

9:58So now we actually have an agent integration baked in.

10:02Fun fact we also have human integration because the Sentry CLI was built by humans and agents for humans and agents.

10:10So now the last thing.

10:12How to use the Sentry API.

10:14Because sometimes the product moves fast and fast, but the Sentry CLI might not keep up with the pace.

10:19So it could happen, it's not that it will happen, but it could.

10:23For that we of course have the sentry api command.

10:26With the API command, you just have an endpoint, you make an authenticated API request, and that's it.

10:33The problem now is that LLMs need context about which endpoints it got a call.

10:38So most of the times it's fetching the docs — sorry, it goes online, goes to your product.

10:41But we actually thought about it and have the sentry schema command

10:45The sentry schema command gives you all the resources there are, with the resources, counts and methods, everything what the LLM actually needs.

10:53The LLM might not use this one, but the --json command.

10:58So now again if the product moves faster than the CLI, it could happen that the API, or the schema, is not up to date

11:06It isn't, because we share the code with the Sentry API.

11:09The Sentry API is always up to date in the product, of course, and because if the Sentry API is updating, the CLI is getting updated as well.

11:17So at least on the schema, we are always up to date and you never miss out on the features on the CLI itself.

11:23Nice.

11:25So now that we have it, we are kind of complete, right?

11:28So what if you want to use the CLI in your code programmatically?

11:35Of course that worked before, but as we said before, it was written in Rust and it could not run on every runtime at all.

11:42So if you want to use it in the browser, yeah, bad luck

11:46Actually, because it was written now completely in TypeScript, you can do it something like this.

11:51You have you just import createSentrySDK from sentry, you use it, create a SDK, and use all the commands there are.

12:00What you want to have.

12:01Of course, like with the Sentry API, we wanted to have an escape hatch in case we missed something.

12:07So at the very end, you see an escape hatch.

12:10in order to use all the features in case we miss something.

12:14So in this case we just dash run sdk run and then --version

12:19So you can always use everything at any moment.

12:22And this is kind of important because we want you and LLMs to use everything at any time point in time without you touching the UI or the web UI

12:33So now how what is the difference between an MCP and a CLI?

12:37Should we now kill our MCP or should we just move everything to the CLI?

12:41Well, the difference between the CLI and the MCP is they have different use cases.

12:46Of course, a lot of code is shared in our case between the MCP and the CLI, so we move everything into one code base, but there are different use cases

12:54Just to name a few, there's the reduced context, which might not be true in the version two of the MCP, but it is so true in version one of the MCP, that's why there are so many blog articles out there, why the CLI is better than MCP

13:08But the CLI is also better for using it programmatically.

13:11Of course you can do it with the MCP, but not that nicely.

13:14It can be used by humans, like you just type in Sentry help or whatever.

13:19You can also easier switch between harnesses.

13:21So Peter here, he loves to use the CLI not just because of the context reasons, but he can use he can switch between harnesses at any point in time.

13:31So the CLI is set up once on the on your on your shell, but the MCP needs to be set up on every MCP on its own.

13:39Of course the MCP also has some strong arguments for it.

13:43For example, it works without a binary, so you don't need to install anything.

13:47You can also deploy it easier and you have you have better control at enterprise scale, which is

13:52Super nice.

13:53So we keep both and that's it.

13:55So thank you, and now we have some time for questions, I guess.

14:06- Thank you for the presentation.

14:08My question is the agent integration.

14:10Is it just skills with the agent, or is it any more than that?

14:19- It's just that.

14:20So I'll give the mic to Burak, who's mainly responsible.

14:27- Round of applause for Burak.

14:29- Let's go. I was asking the question just in case.

14:35So the question was is it just the skills or is there anything else

14:39Well, I mean it's mostly the skills that we install.

14:41The skills are auto-generated, so they're never stale.

14:44But they're auto-generated with some human input around them.

14:48There's some interesting stuff in our repo which you can see

14:51And the other stuff we do is the --json and the schema thing.

14:55So those are the agent integration parts.

14:57The schema command is not amazing for humans, but it is built there mostly for

15:02For agents and the inspiration was the Google's Workspace CLI, I guess, like this notorious one.

15:09The guy who wrote that had a nice blog post and we were like, oh those are nice ideas, so we borrowed some of them.

15:14Yeah.

15:16- Do you somehow detect when you run the CLI that this command was called by an agent versus a human?

15:23- Yes, we do.

15:24But we don't change the behavior.

15:25It's only for reporting purposes.

15:27- What about if there are some commands that you want to hide from an agent?

15:33- So far I don't think we had the need for hiding commands.

15:38But yeah, that's an interesting idea.

15:39Okay.

15:40So the few detection layers we have are agent detection plus interactive versus non-interactive.

15:46So if you detect a non-interactive flow, but it doesn't have to be an agent, it can be like a CI flow

15:51we behave differently in certain aspects, like we try to not ask you questions, etc.

15:55But like we don't do any command hiding, what's that like the behavior doesn't change.

15:59That's that's the main point I guess.

16:01- Yes.

16:01Are you able to tell which agent exactly is calling it?

16:05- Most of the time because they emit like some environment variables and stuff.

16:08So it's not like we're doing anything crazy — they actually want to be known.

16:11- Got it.

16:11Thanks.