Full transcript
0:03[Music]
0:15Hey folks. Um, my name is Alex Volov.
0:17I'm an AI evangelist with Weights and
0:19Biases. I'm Benjamin Eckle. I am
0:22co-founder CTO of DIPso. We're creators
0:24of MCP.Run.
0:26All right. And we're here to talk to you
0:27about MCP observability.
0:30Hey Ben, I want to ask you a question.
0:32Somebody who worked at Data Dog before
0:34and somebody who runs multiple MCP
0:36servers and uh clients on production. Uh
0:38something that happened advice that
0:40happened something in my agent uh in
0:43production the other day. Okay. Uh yeah,
0:46I mean we've been running MCP clients
0:48and servers in production since the
0:50beginning. Uh yeah, but wait, aren't you
0:53like working at an observability
0:54company, Weights and Biases? And don't
0:56you work on like what's it called?
0:57Weave. Yep, that's true. I I work about
0:59weave and but since I started adding
1:01some powers to my agent via MCP all that
1:04observability that I'm used to from just
1:06having my own code run end to end has
1:09gone a little bit dark. Gotcha. So this
1:12is what we're here to talk to you guys
1:13about. U the rise of MCP is creating an
1:16observability blind spot. As AI agents
1:18become more uh prevalent, the problem
1:20can compound with more and more tools
1:22via MCPS. The less they the developers
1:25can know about the endto-end happenings
1:27within their agent. Yeah. Um, yeah. So,
1:30on MCP Run, we're running both clients
1:31and servers. And because it's a new
1:33ecosystem, we've had to like cobble
1:35together a lot of our own ways to do
1:37observability. And I've been looking
1:39around. It seems like everyone is sort
1:40of doing this in isolation. They're sort
1:42of solving the same problems. Um, so,
1:46you know, we wanted to bring the
1:47community together on this issue. And
1:48so, today we're going to talk about the
1:50state of observability in the MCP
1:52ecosystem.
1:53Yep. So, why do we care about this? And
1:56why do we think that you guys should
1:57care about this? So if you don't have
1:58the ability to quickly understand why
2:00things went wrong on production, where
2:02they went wrong and how, your ability to
2:04quickly respond is greatly diminished.
2:06And we care deeply about we both build
2:10tools that need MCP observability. And
2:12we support MCP and we both care deeply
2:14about developer experience as well.
2:16Yeah, it's it's really important to me
2:17because enterprise engineering teams
2:20don't ship something to production
2:21unless they know for sure that they're
2:22going to be able to identify security
2:24and reliability problems before their
2:26customers do. Um, and that's why they
2:28invest a ton of money in observability
2:30platforms. And uh, so if you're going to
2:32ship MCP to these production
2:34environments, you must seamlessly
2:36integrate with these observability
2:38platforms. Yep. So, because we care
2:40deeply about uh developer experience at
2:43W&B Weave, uh I'm happy to announce here
2:45on stage that we've supports MCP. Yay.
2:48As long as you're a developer of both
2:49the client and the server, all you need
2:51to do is set this MCP trace list
2:53operation environment variable on your
2:55client and server. And uh we'll show you
2:57the the list tool calls and we'll show
2:59you the the duration of your MCP calls.
3:02This works currently with our Python
3:03based clients. And this is how it looks
3:06super quick. With the red arrows, you
3:08can see the client traces, for example.
3:10And with the blue arrows, you can see
3:13we're pointing to the calculate BMI tool
3:15and and the other tool. And that's it.
3:17Observability solves, right? Let's get
3:18off the stage. We're done. Wait a
3:20second. So, uh what about this like
3:22calculate BMI tool? This uh MCP server.
3:25Why can't I see into that? Um you Yeah,
3:28we're working on this.
3:30Uh yeah, also this seems like this is
3:32specific to Weave, right? Um is there
3:34not like a vendor neutral way to do this
3:36that's standardized? Yeah, that's right.
3:37Uh this is a bespoke integration that we
3:40built into weave into our SDKs in
3:41Python. And while working on this, while
3:43our developers have been building this
3:45like u integration within our MCP
3:48tooling, I was advocating internally and
3:50externally that we should align with the
3:52open nature of MCP as a concept and
3:54created observable. Maybe some of you
3:56have seen this. This is a manifesto to
3:58drive a conversation that this is a
4:00problem that needs solving and uh
4:02between observability providers uh such
4:04as us and other folks that's been on
4:06stage before and going to be on the evol
4:08tomorrow uh to do observability in a
4:10vendor neutral and standardized way. And
4:13so while working on observable tools I
4:15realized I I did some search realized
4:16that a vendor neutral scalable way to
4:18add observability exists uh and there
4:21could be a great way to marry the two
4:22open protocols to work together. Yeah,
4:25exactly. Uh fortunately MCP powered
4:27agents are really just another
4:29distributed system and we've been doing
4:30that for decades. So open telemetry is
4:33just the way that's that we've like
4:35settled on doing that. Um we're going to
4:37talk about OTL a little bit. If you're
4:39not uh familiar with it, we need to
4:41learn about a few primitives first. So
4:43the main primitive that we need to learn
4:45about is the trace. So a trace is kind
4:47of like an atomic operation in your
4:49system. It's made up of a treel like
4:50structure of steps that we call spans.
4:53And a span represents the duration and
4:55some arbitrary metadata for each step.
4:58And what this step is exactly is
5:00completely up to you to define. It can
5:02be as high level as like an HTTP
5:03request. It can be as low level as a
5:05tiny little function call. Um here's an
5:08example of like a checkout experience,
5:10an API for a checkout. The size and
5:12position of each of these spans
5:14correspond to how long it took and where
5:15it sits in the call graph respectively.
5:18And just from this data, you can tell a
5:19lot about a system and how to observe
5:21it.
5:22Um the other primitive you need to be
5:24aware of is syncs. So a sync is kind of
5:27like a centralized database where all
5:29your telemetry goes, but often they come
5:31in the form of this like whole platform
5:33with like a UI and dashboards and
5:34alerting and monitoring and all those
5:36things. So there's a lot of logos here,
5:39Ben. Uh basically a sync is an open
5:41standard way for folks like collectors
5:43to like receive those spans. As long as
5:45the developer instrumented their
5:48application code in a certain standard
5:50spec way, everybody can just receive
5:51those in in the same unified way. Right.
5:53Exactly. Yeah. It's if you squint, it's
5:55just kind of like a bunch of databases
5:56that all support the same schema and
5:58wired protocol. You could switch them
6:00out and in fact they don't have to
6:01change much of their code or even change
6:03the code at all. It could be just
6:04config. Right. Right. Uh by the way
6:06observability tools like W&B weave and
6:08some friends Simon from Lockfire here
6:10before and some other friends all have
6:12switched to support otel as well. Open
6:14telemetry is becoming like this global
6:16standard.
6:17Great. Uh yeah, another great thing
6:19about having a centralized sync uh is
6:20the last concept distributed tracing. So
6:22going back to our checkout endpoint, if
6:25the uh fraud service sends its span to
6:28the same sync, then we can stitch back
6:30the together the traces and show the
6:32whole context. So maybe you're kind of
6:33seeing where the MCP server stuff comes
6:35in here. Yeah. So, hey Ben, if it's
6:38possible via the integration to the open
6:40protocol, um what if I want to use MCP
6:43servers that other people host like
6:45GitHub, like Stripe, like other folks?
6:47Yeah, it's a good question. So, um with
6:49MCP enabled agents or really just any
6:51distributed system, there are kind of
6:53two scenarios. There's when the client
6:55and server are in different domains and
6:58then there's when they're in the same
6:59domain. And by domain here, I don't
7:00necessarily mean the literal definition.
7:03I mean like the administration
7:04administrative domain of control, right?
7:06like do like do you own this MCP server?
7:08Do you own this MCP client or is it a
7:10third party thing? So your GitHub stripe
7:13example is like a great example of like
7:15the different domain scenario. So um
7:18this is a trace of an agent that is
7:20executing the prompt read and summarize
7:23the top article on hacker news. So it's
7:25going to reach out to this like remote
7:27fetch server to read hacker news, but it
7:29appears to us in the trace as a single
7:31service span because it's it runs
7:33outside of our domain of control. So it
7:34appears to black box to us.
7:38Um but suppose we do own the server like
7:40maybe it's running in a different data
7:41center than the client. Um how do we get
7:45actually the whole context? Uh it's
7:47pretty simple. So with distributed
7:48tracing and context propagation, we can
7:51have the remote fetch server send its
7:52spans to the same sync as the client and
7:55the sync will just stitch together the
7:56missing uh parts of the trace back for
7:58us. So in this graphic you can see that
8:00we can now break into that fetch server
8:02and we can see what it's doing. It's
8:04making some HTTP request that's taking
8:06roughly 350 milliseconds and then it's
8:08doing a little uh crunching to to create
8:11some markdown.
8:14Okay, so that that is great in theory
8:16and we went through this. We could have
8:17a whole hour talking about hotel. Not
8:19that we got an hour. Uh but how do we
8:21can actually marry those two protocols
8:23together? Right. Uh is there a standard
8:24way? Did the MCP spec folk deploy a way
8:27for us for observability? Um not quite.
8:30It was it was uh pretty tricky to get to
8:32get working. um it does work today but
8:35uh it required a little bit more work
8:37than it should have. So in order to do
8:38this we need to as I said propagate the
8:41trace context from the client to the
8:43server. So here's a TypeScript example
8:46and when we call a tool in the client um
8:48we're going to extract our current span
8:51and we're going to uh pass it along to
8:54the server. And we achieve this by
8:55basically just shuttling the data
8:57through the protocol's meta payload.
9:02And uh now that we're inside the server,
9:04uh this would be like in the fetch
9:05server, we can pull that trace context
9:07out, inherit it as our current span, and
9:11then when we send our spans off to the
9:12sync, uh it it's as if it came from that
9:16parent span, and they the sync can
9:17stitch it back together. Then this is
9:19awesome. So you basically used an
9:21undocumented kind of property of the
9:24sending the payload together with the
9:25payload between clients and servers um
9:28to pass along the data that hotel needs
9:30to connect those things together, right?
9:31Yeah, sort of. I just kind of had to
9:33abuse the lower level interface reserved
9:35for the protocol, but a higher level way
9:37should be provided through tooling. And
9:39that's something we should talk about a
9:41little bit later in the talk. Yep.
9:44That's all. Oh, yeah. So, by the way,
9:46this is uh this is not just um a
9:48screenshot. This is a working demo. So,
9:51um it's a lot more code than what I
9:53showed in the slide. So, if you want to
9:55actually go see how this works and adapt
9:56this for your needs, uh go check out
9:58this GitHub link. And I think actually
10:00you did that to to get it to work with
10:01weave, right? Yeah. So now that we know
10:04how to pass context after you you you
10:05showed me the way, uh let's see how
10:07amazing the solution actually is in
10:09practice. While Weave MCP, the thing I
10:11showed you guys before was a bespoke
10:13solution baked into our Python SDK for
10:15Weave. The huge benefit of MCP generally
10:18not only observability related is that
10:20servers and clients don't have to run on
10:22the same environment or share the same
10:24code or be from the same programming
10:25language. So while we were working on
10:27the Python SDK, you built an agent in
10:29Typescript and so because Wave WB weave
10:33supports hotel open telemetry and it's
10:35an open protocol uh your TypeScript
10:37agent it took me a few minutes to by
10:39without changing much code to just send
10:41those traces into weave from a
10:43TypeScript agent and not necessarily
10:44from a Python edge. So here uh here you
10:48could see in the green the the client
10:51traces are in the green and then the
10:52server traces actually show what happens
10:54within those calls uh
10:57on kind of the the server side as well.
11:00Yeah, that's really cool. So how did how
11:01did you actually get the traces into
11:02weave? So this is very very simple way
11:05simpler than before. Uh we just define
11:07W&B weave as the OTLP endpoint standard
11:09that you kind of like showed me around.
11:11Uh and then folks can send their traces
11:13into 1b.ai I/O hotel and all you need to
11:16do in addition to this is authorize. So
11:18add authorization headers and specify
11:20which project you want to go into. Cool.
11:23Yep. So while we talk to you about
11:25thoseability, while I was working on
11:27this, I had a magic moment happening
11:28with MCP. I wanted to share this with
11:29everybody and I love you as well. MCP
11:32story. Yeah. So um
11:35I used quadopus 4 that just came out to
11:37weify your agent that you built and to
11:40add this uh MCP observability and W&B
11:43weave is going to get a little meta.
11:45Stay with us. Uh also has an MCP server.
11:47Okay. What what does it do? So we have
11:49an MCP server that lets your agents or
11:50or chats etc talk to your traces and see
11:53the data and summarize the data for you.
11:55Okay. So we have this MCP. it's been
11:57configured in my windsurf uh and and CL
11:59code uh OPUS 4 uh was able to use this
12:03MCP server to kind of work through it.
12:05So here you see an example. Um the agent
12:08basically started working on your code
12:11and then decided okay I'm going to run
12:12the code and then said okay I'm going to
12:14go and actually see if the traces showed
12:15up at at W&BWeave. Then it noticed that
12:19they showed up but they showed up
12:20incorrectly. So some input or output a
12:22specific parameter that it needed to do.
12:24It didn't know how to do it wasn't part
12:25of the documentation. And so uh the next
12:28moment just absolutely blew my mind.
12:31this OPUS 4 discovered that our MCP
12:34server exposes a support bot. So
12:37essentially another agent uh decided to
12:39write a query for it, received the the
12:42right information after a while and
12:44acted upon this information, learned how
12:46to fix the thing that it needed to fix,
12:48fixed it and then went back to notice
12:50whether or not the fix was correct. So
12:53my um my coding agent talked to another
12:56agent via support vcb that it discovered
12:58on its own. I didn't even know that this
13:00ability exists to work on your coding
13:02agent in in things. Things got a little
13:04bit meta and my head was like
13:05absolutely. I was sitting like this
13:06while all this happened. Didn't touch
13:08the keyboard once.
13:10That's awesome. Yeah, it's pretty meta.
13:12Uh yeah, before we go, I also wanted to
13:15have uh take a moment to have an
13:16announcement. So um MCP run will also be
13:19exporting telemetry to hotel compatible
13:22syncs. Um so as I mentioned before, we
13:25run both servers and clients. Uh so for
13:28servers we have this concept called
13:29profiles and these allow you to like
13:32slice and dice multiple MCP servers into
13:34one single virtual server and on on uh
13:38we also have the an MCP client called
13:40task and this is like a single prompt
13:44agent that could be triggered via URL or
13:46a schedule and it also just sort of
13:47marries with the idea of profiles. Um,
13:50but yeah, soon you'll be able to get
13:51Otel out of both of these and hopefully,
13:54you know, we'll uh connect up to weights
13:56and biases and have a little party.
13:57Yeah, you can send those to Wave
13:58Straighter from MCP.Run.
14:01Okay, so uh to recap, um observability
14:04is here at in MCP today, but it's not
14:07evenly distributed. Uh should get you
14:10most of the way there, but the community
14:12needs to come together uh create
14:15creating tooling and conventions to make
14:17it smoother. um you shouldn't need to be
14:20an expert in observability to like get
14:22this stuff working.
14:24So how do you get involved? Well AI
14:26engineers just start thinking about
14:28observability via MCP tooling and
14:30whether or not you're getting uh
14:32observability to the end to end of of
14:34your execution chain. Um for tool
14:37builders and platform providers we
14:39should join and work on higher level
14:41SDKs. So uh arises as open inference for
14:44example is a great start but all of us
14:46should help with instrumentation for our
14:48clients who use bespoke SDKs to work on
14:50conventions also together. Ben can you
14:52explain semantic conventions super
14:53quick? Yeah sure. So as we learned
14:55earlier um spans they carry userdefined
14:59attributes right so if they're
15:01userdefined how does the sync know that
15:03a span is actually say an HTTP request
15:06with a 200 status code or how does it
15:08know that it's an MCP tool call that has
15:11an error. Um that's where semantic
15:15conventions come in. Um and you can be a
15:17part of defining what the conventions
15:18are for agents that all observability
15:20platforms agree on. And if you're
15:22interested in this, I would suggest
15:24going to check out the uh Genai semantic
15:26conventions effort by the hotel team.
15:30And um yeah, lastly, for platform
15:31builders such as MCP Run, um you know,
15:34go add hotel support, help review RFC's.
15:38And finally, yeah, just come like talk
15:39to us about ideas because we're just
15:42everything's just kind of coming
15:43together. Everything's so new and fresh
15:44and we don't really know exactly what to
15:46do. There's an additional track here at
15:48at uh AI engineer. this called the
15:50hallway track and I've learned more
15:52about the stuff that we were talking
15:53about uh out there by actually talking
15:55to people who implement this than I
15:57learned while preparing uh before the
15:58talk. It's quite incredible. So, um
16:02yeah, sure. Um yeah, so again, I'm Ben.
16:05Uh my call to action here just be go
16:08check out MCP Run. You can get a free
16:10account, try it out. Uh yeah, that's it.
16:13And I'm Alex. Uh uh check out W&BWeave
16:16MCPOP to learn how to trace MCP with
16:20hotel. Uh I'm also I did the observable
16:22tools initiative. I would love for you
16:24to check out the manifesto to see if
16:25this resonates with you to join forces
16:27to talk about observability and uh we
16:30yeah please visit us at the booth. We
16:32have some very interesting surprises for
16:34you. We have a robotic dog right here uh
16:36that's observable. I also run the
16:37Thursday I podcast. I want to send Swix
16:40a huge huge shout out for uh having uh
16:43giving me the support to show up here
16:44and give if you guys are interested in
16:46AI news, we're going to record an
16:48episode tomorrow. That's it. Thank you
16:49so much.
16:55[Music]