A2A & MCP Workshop: Automating Business Processes with LLMs — Damien Murphy, Bench

0:00[Music]

0:15Hey everybody. Uh yeah, thanks for

0:17coming. Uh great to see a full room. Uh

0:19always good when you're doing a workshop

0:20to have a a lot of people here. Um so

0:23yeah, I'm I'm Damian Murphy. I'm going

0:25to be presenting A2A and MCP. uh two

0:29pretty hot topics these days in AI um

0:31and how you can use them to automate

0:33business processes. Um

0:36so yeah a little bit about me um about

0:4015 years full-time uh full stack

0:42developer uh five years doing solutions

0:45engineering so customerf facing kind of

0:48uh forward deployed engineer and uh

0:50spent the last three years or so uh

0:52working on voice AI and AI agents. Um I

0:56did a workshop last year as well um AI

0:59voice agent swarms and uh yeah pretty

1:02pretty hot topic. I think it's now

1:04pretty much standard that everybody can

1:05build a voice agent in in 5 minutes. Um

1:09so now the the hard part becomes

1:11building autonomous agents that actually

1:13can do complex tasks. Um, so I joined

1:16Bench Computing uh about two months ago

1:18uh pre-revenue startup u backed by

1:21Sutter Hill Ventures and we're building

1:24um what I would imagine to be a better

1:26Manis uh that's more focused on teams

1:28and enterprises.

1:30If you're not familiar with what Manis

1:32is, it's kind of like a autonomous AI

1:34agent. And Bench is essentially an

1:37autonomous AI agent that can do uh sub

1:40subp parallel task um automation.

1:46All right. So the workshop that we're

1:47doing today, we're going to build a

1:48multi- aent system um using A2A agents.

1:52Uh if you're not familiar with A2A, uh

1:54Google released essentially um a

1:57protocol that allows agents to

1:59communicate over the web.

2:02Uh we're going to integrate these agents

2:04with uh MCP, which is the model context

2:07protocol. Uh MCP is like a USBC for, you

2:12know, uh all of your agents to be able

2:15to consume um context and tools uh and

2:19resources uh very easily.

2:22uh we're going to get these agents to

2:24work together and we're going to trigger

2:26uh the the agent with a web hook and

2:30then uh I'm going to cover a little bit

2:31about when to use A2A MCP and I'll also

2:35go into uh prompt caching and context

2:37management as well.

2:41All right, so A2A, right? Um it's it's

2:45not exactly clear what it's for and why

2:47it exists, right? Uh if you ask

2:49everybody in the room what they think it

2:51does or why it exists, you'll probably

2:53get a different answer. Um but the key

2:55benefits are you can have agent

2:57specialization, right? So rather than

2:59trying to make one agent do 100 things,

3:02you can have a 100 agents do one thing

3:05and do that one thing very well. Um Away

3:08allows you to handle task delegation.

3:11So, you know, imagine you had a

3:13Salesforce agent um and you wanted it to

3:16interact with all the Salesforce MCP um

3:20uh tools. Uh you could do that. Um

3:22you've also got the ability to do

3:24parallel processing. Uh and this will

3:26become very important when it uh comes

3:28to speed and context management. Uh you

3:31can then use those A2A agents to have uh

3:34complex workflows and and help uh keep

3:37your main agents context size down.

3:42Uh MCP again really hot topic right now.

3:45Uh it's been kind of coined as the USBC

3:47for AI. Um and there's definitely some

3:50benefits in just having a standard

3:53interface, right? You know, there's

3:54something like 10,000 MCP uh tools that

3:57you can use today. Um about 7,000 of

4:00those come through the Zapier MCP. Uh if

4:03you're not familiar with Zapier, it's

4:05essentially a way to connect disparate

4:09systems together. And they've now

4:11released all of their uh zaps they're

4:13called as MCP uh servers and tools.

4:17Uh one of the great things about MCP, no

4:19integration with APIs. So you don't have

4:22to do any sort of you know different

4:24handling of different APIs. Uh it's a

4:27plug-in architecture uh an industry

4:29standard. Um and it's really based on

4:31LSP. So LSP was a way for you know idees

4:35to actually uh figure out how different

4:38code languages worked. Um and it was a

4:40great kind of um transfer uh of of ideas

4:44over to the MCP protocol.

4:48All right. So when should you use A2A

4:51versus MCP? Anybody?

4:56MC if you want to resource the

4:58infrastructure

5:00then you go for MCP

5:03but I don't know and and and that's kind

5:06of the the challenge right it's like

5:09what exactly um you know these protocols

5:12for and should I be using them and and

5:15things like that. So if you want to

5:17have, you know, two agents, right? And

5:20typically two agents that are completely

5:22unrelated, right? So it's not two agents

5:25you necessarily control. It's more

5:26likely going to be an agent of a third

5:28party or, you know, their first party

5:30agent and your agent. Yeah.

5:32What's the difference between agent

5:36and A2A? So I work a lot on the agent

5:41where we have multiple agents and doing

5:44the same. The you are saying describing

5:47A2A is a lot similar to a care.

5:51Yeah. So like autogen and and frameworks

5:54like that that allow you to kind of uh

5:56manage multiple agents kind of locally.

5:59H A2A is more about remote agents,

6:01right? So agents you have no knowledge

6:04of. Um so you can think of A2A as a way

6:06for you to have service discoverability

6:08and once you have the endpoint to the

6:10agent, you can then learn everything

6:12that that agent's capable of. Um with

6:14things like autogen it's like you know

6:16descriptive so you describe what it's

6:18capable of it's in your control.

6:21So to summarize agent AI is kind of

6:26define the role of each agent and A2 is

6:32kind of working on remotely and its role

6:36is not defined or defined.

6:41So each of the A2A agents will have a a

6:44definition and we'll kind of get into

6:45that a little bit later. Um but yeah,

6:47think think of agentic AI kind of as a

6:49superset of everything, right? Um A2A

6:52and MCP are just kind of subsets of

6:54that, right? Different modalities. Um

6:58yeah, so for MCP, you're you're going to

7:00connect to external context and tools.

7:02Um a lot of people don't use most of the

7:04features of MCP, right? They're just

7:05using the tools. Um but there's a lot of

7:08stuff around prompt templates resources

7:11um and a thing called um sampling

7:15sampling is actually going to be a

7:16really interesting thing I think that

7:17we'll see a lot more of as well uh where

7:20it allows these MCPs uh to sample the

7:23host LLM right so if you're using you

7:26know claude and you're hitting an MCP

7:28server um that MCP server may want to

7:31also use the same model of cloud that

7:33you're using and it can use sampling to

7:35actually achieve that Um, so when you

7:37bring those two together, you you kind

7:39of get the benefit of both, right? So

7:40you have A2A is the the remote

7:43interface. Uh, and MCP is then giving

7:46you the actual um tool use and and

7:49context management.

7:52Okay, so when not to use MCP. Um, and

7:55and you'll notice a lot of like memes

7:57here. Uh, and just to give you a heads

7:59up, all memes were generated by Bench.

8:01Uh, actually the whole slide deck was

8:03generated by Bench. um I just gave it a

8:05markdown file and it and it outputed it.

8:08So um when you use A to A or MCP um if

8:12you have full control of the tools then

8:13you probably don't need it right like if

8:15if your function is local to your

8:18codebase you know why do you need to

8:21create you know a USBC it's kind of like

8:23me plugging in my hard drive with a USB

8:26cable you know like shouldn't I just use

8:28the hard drive that's in my machine

8:30right um so calling functions directly

8:33in your codebase super easy easy to

8:35maintain faster to develop um and then

8:38If you have full control of your agents,

8:41you probably don't need A2A either,

8:42right? Um like if they're your agents,

8:45you can use, you know, some sort of

8:47local uh function call for them to

8:48communicate. And and I I've built multi-

8:51aent systems using MCP and using just

8:54local function calls. It's a lot easier

8:56to just use the code you have. Uh it's

8:59going to be faster. There's no protocol

9:01overheads and and things like that. A

9:03lot easier to debug as well.

9:07Okay. So, why do you need A2A and MCP at

9:10all? Right. Um, third party tools is

9:13probably the number one reason uh to use

9:15MCP. Um, you can just get access to such

9:19a large array of tools um that you know

9:22you're never going to be able to uh

9:24let's say you're building a product,

9:26right? And and you're like, "Okay, we're

9:27going to build first class integrations

9:29with Salesforce and Slack and but what

9:31about the other 10,000 tools?" I was

9:33like, "Okay, we'll just allow people to

9:35add their own MCP server." Um, so that

9:38gives you great extensibility. Um, but

9:40there's a lot of drawbacks with MCP,

9:42right? Um, you only get what you're

9:44given. Um, and a lot of time that's not

9:46exactly what you want. Um, so you may go

9:49down the route of saying, you know what,

9:50I need a a way to actually index this

9:53data so that I'm not calling like, you

9:55know, list Slack channels every time I

9:58want to post a channel, right? Or post a

10:00message.

10:01Um, and then with A2A, uh, the com

10:04complexity is hidden from you, right?

10:05And that's one of the kind of the key

10:07tenants of of A2A is that you don't know

10:10anything about this agent until you

10:12connect. Um, and all of its complexity

10:15is is completely opaque.

10:17Um, and then you you can essentially

10:20connect to, you know, any sort of uh

10:23remote A2A agent. Um, so long as you

10:26have, you know, the credentials and

10:27things like that. Um, we haven't seen

10:29any firstparty A2A agents released yet.

10:32Um, but Google has about I think 50 uh

10:34partners they're going to launch with.

10:36So, I I'd imagine there's going to be

10:38like a Salesforce A2A agent. Um, it'll

10:41probably only come with a paid account,

10:43right? Because it's going to use LLM

10:45compute. Uh, versus things like MCP

10:48typically don't actually use an LLM,

10:50right? They use the host LLM.

10:54All righty. So, we're going to get into

10:55the code now. Um

10:58yeah, so uh if you haven't already

11:00grabbed the repo, uh we also have a

11:02Slack channel, um workshop A2A-mcp,

11:08uh-2025.

11:10Um and in this repo, there's basically

11:12every everything you need to get going.

11:14Um

11:17yeah, so the the code structure, uh

11:19we've got a host agent, um and then

11:21we've got some sub agents, right? And

11:24the whole concept here is to

11:25demonstrate, you know, ATA and MCP. Um,

11:28but in reality, these sub agents will

11:30probably live in a different repo, you

11:33know, run on a different server. Um,

11:36yeah. And then we've also got the uh A2A

11:38implementation. Uh, the server and the

11:40client in uh the repo. Uh, these are

11:43taken directly from the ATA uh repo.

11:46We've also got the MCP integration. So,

11:48this is just a client. Um, we're not

11:50creating a a server here. Uh we also

11:54have a CLI interface. You're not going

11:56to need the CLI interface. That's kind

11:57of internally how it's being used. Um

12:02yeah, so once you've cloned a repo,

12:04you're going to want an MPM install. Um

12:06and you're going to need a MCP server

12:08URL. Uh this is going to be a Zapier uh

12:12URL and a Gemini API key. Uh you can get

12:14both of these for free. Uh there's no

12:16need to to sign up for a paid account to

12:18get them. Um and you'll want to rename

12:20your mv.ample example uh to

12:27all right so setting up the Zapier MCP

12:31um when you go to uh zapier.com/mcp

12:35uh you'll have the option to create a

12:36new server um and when you go to connect

12:39you're going to have a couple of uh

12:40options here we're going to use SSE um

12:44they recently released uh streamable

12:46HTTP which uh is making SSE deprecated

12:50and it's going to replace it um but

12:51there's There's still a litany of SSE

12:54servers out there. So, um I just used

12:57SSE for this one. Um once you do that,

13:00you're going to get this server URL at

13:02the bottom. You can copy that URL.

13:03That's going to be the URL that goes

13:04into your

13:07and then uh you're going to set up a

13:09Slack and a GitHub integration. Um so,

13:12you're going to want the ability to

13:13create an issue. Um you can put in uh

13:16the repository URL for the workshop if

13:18you want. Uh you can use your own uh as

13:20well. um you can let AI choose uh these,

13:25but what I found with AI is that it will

13:28choose something else, right? Um so a

13:30lot of time with these MCPs, you're

13:32going to want to kind of say, hey, you

13:33know, this is the thing I want to do, so

13:36let's just kind of hardcode that. Um but

13:38if if you do let it kind of go wild into

13:40your Slack, uh it's going to start

13:42posting in general and random and sales

13:45and uh yeah, a few of my bots have kind

13:47of gone rogue.

13:50All right. So, the Gemini setup. Yeah.

13:53So, you can get the uh API key here, the

13:55AI studio. Um, and there's a a link in

13:59the uh slide deck as well if you need to

14:01click it. Um, you can get a free

14:03account, generate an API key, uh, drop

14:06that into your M as well.

14:09Excuse me.

14:12And there's also a remote uh, bench A2A

14:14agent. Um, so the code for it actually

14:17in the repo. Um, but we haven't

14:19officially released our API yet. So, I'm

14:22just hosting that remotely. Um, but it's

14:23a nice kind of way to show how you would

14:26use A2A remotely as well.

14:29Um, so what is Bench? Uh, Bench is

14:32essentially a kind of LLM aggregator uh

14:35with autonomous AI agents. Um, so you

14:38get access to cloud, Gemini, OpenAI, XAI

14:40and loads more models. Uh, it has I

14:43think about 30 tools now um and

14:45integrations. So um we actually started

14:48out with MCP integrations to Slack and

14:50Salesforce. They didn't meet our needs.

14:52We built firstparty integrations, you

14:54know, data caching and indexing. Um and

14:56and that kind of gives you an idea of

14:58like how far is MCP going to get you,

15:00right? Uh eventually at some point

15:02you're going to realize that it doesn't

15:04do the you know the specific thing you

15:06need to do.

15:08All right. So running the application

15:11um you're going to run mpm run start all

15:14um and that's going to kick off all the

15:17agents right so the slack agent the

15:20github agent uh the host agent and uh

15:23it'll also start the web hook server and

15:25the web hook uh admin panel uh you can

15:28access net then through localhost port

15:303000

15:33and um yeah so let's just kind of go

15:36into what each of the actual

15:38uh agents do. Um so the host agent is

15:41essentially your central coordinator,

15:43right? Um and this this may be the only

15:46agent that you have in your application.

15:48It may be using external uh A2A agents.

15:52Uh and if that's the case, then you know

15:55everything that your host does is going

15:57to be delegated um you know to sub

16:00agents.

16:03Um, so that handles all the agent

16:04discovery and and kind of bringing

16:06everything together.

16:09Yeah. So the the code for that's going

16:10to be in source agents host

16:13and um you'll notice there's a couple of

16:15files in there. One of them is uh the

16:17host agent prompt, right? So that's just

16:19a plain text uh system prompt uh genkit.

16:23That's going to be uh essentially how

16:25you hook all of your A2A code up with

16:28Gemini. Um and there there's also a

16:32genkit MCP plugin that um the sub agents

16:35use.

16:39Uh so then the Slack agent um so this is

16:41going to send a Slack message in

16:43response to the web hook transcript and

16:45yeah the the the kind of sample web hook

16:48that we have in this is essentially you

16:49know your meeting end and you're going

16:52to receive a transcript of that meeting

16:54right um and with that you're going to

16:57decide what to do. So, it's going to,

16:59you know, if it detects any bugs, it's

17:01going to create a GitHub issue. If it

17:03detects any, you know, feature requests

17:05or or anything of interest, it's going

17:06to post that into Slack. Um, and you can

17:09think of the kind of automations that

17:11you can build with this sort of, uh,

17:12scenario, right? So, um, you could even

17:15I had a version here that was hooked up

17:17to Salesforce, but um, there's actually

17:19a limitation on the host agent on how

17:22many sub agents it can call. Um, so I I

17:25figured, right, if one of them's going

17:26to go, it's going to be Salesforce

17:28because it's it's probably the hardest

17:29to get an account on. Um, but you could

17:31actually update an opportunity based on

17:34a sales call, right? So you could have a

17:35sales call and you, you know, you're

17:37talking to them, you're doing your

17:38discovery and you're able to update

17:40those Salesforce fields automatically.

17:43Um, and like the time saving for account

17:45executives because, you know, they're

17:46probably on backtoback calls is actually

17:49pretty big.

17:53Yeah. This was an interesting um issue I

17:56ran into. So I asked one of my

17:57colleagues um to test the repo out,

18:00right? Um and he was getting this weird

18:02error where it was saying, you know, the

18:04Slack MCP succeeded. Um so I asked him

18:08to send me the logs and he sent me this

18:11and it was like is error false?

18:14And I'm like okay that's that's great.

18:17So yeah, it turns out that you know not

18:20all MCPs are created equally and the

18:22Zapier Slack MCP uh fails silently. Um

18:26so the the reason it failed was he he

18:28had um the default Slack channel name uh

18:32which was like test uh Damian Slack and

18:36he was in a different workspace where

18:37that channel didn't exist. So it just

18:39failed silently. Uh so I added a bit of

18:41code to detect this kind of empty text

18:43array. Um so it will fail now. Um, but

18:46it kind of goes to show you just kind of

18:49the limitations of MCP.

18:53Yeah. So, the GitHub agent, uh, pretty

18:54straightforward. It's it's it's probably

18:56the the most basic of of of the the

18:59three or four. Um, so it it just creates

19:02a GitHub issue. Um, super simple. Um,

19:05but you could imagine, you know, how you

19:07would extend this, right? Maybe it's

19:08going to open a PR, right? maybe it's

19:11actually going to implement uh the fix

19:13for the bug that was reported uh in the

19:15meeting. Um and you can see how down the

19:17line as you know AI gets better and and

19:20and things really improve that a lot of

19:22this automation is going to be driven by

19:25human interaction, right? So you know

19:27speaking with people and posting

19:29messages in Slack and talking and GitHub

19:31discussions um is going to trigger AI to

19:34take action.

19:37Yeah. So the bench agent um it can it

19:40can do a lot. Uh and that was actually

19:42one of the problems that I found with

19:43A2A is that like the more functions and

19:47capabilities an agent has and the harder

19:50it is to describe the agents

19:52capabilities um in the agent card. Um so

19:55the agent card is essentially like the

19:57public um information to any other agent

20:01of what that agent's capable of. Um, so

20:03I had to really just pair it back and I

20:05said, "Look, you know, you can do a

20:06handful of things. I know you can do

20:08more, but like for now, these are the

20:10few things that you can do." Um, and

20:12it's able to go off and like, you know,

20:14browse the web, do research, uh, data

20:16science, all sorts of things. Um, so

20:18we're just going to use it for, uh,

20:20researching the company and the people,

20:22uh, in the meeting transcript.

20:26All right, here we go. Demo gods.

20:29Uh, before I start, any questions?

20:32Yeah, you

20:33mentioned some limitation on the number

20:34of agents.

20:37Yeah, so the the Genkit implementation

20:39that Google provide uh limits you to

20:42five maximum kind of sub agent calls uh

20:45per turn.

20:47Is that a hard?

20:49Yeah, I I couldn't get around it. The

20:52like there was this max like setting but

20:56it didn't work. Yeah. Yeah. So, it's

20:58something I'm sure they'll fix

20:59eventually. But, um, it it was an

21:02interesting issue.

21:06All right. Let me see if my uh my code

21:08is running.

21:11Yeah, I think it is. Yeah. So,

21:15it should be here. And actually, I'll

21:17show you the the MCP server as well

21:19while while I'm here.

21:23Yeah. So, this is the MCP inspector.

21:25It's um an open-source repo as part of

21:28the model. Sorry. Yeah. At the back.

21:37Yeah, that's actually in the agent card.

21:39So that'll be in the index.ts of the of

21:42the sub agent. Yeah, I'll be going

21:44through the code in a little bit as well

21:45so you can see it. Um yeah, so I'm

21:48connecting to my Zapier MCP URL that I

21:51got. Um, so I just copied this one,

21:53dropped it in. Um, going to connect over

21:56SSE. Um, and this allows you to, you

21:59know, list the tools, call the tools.

22:01Um, and it's quite interesting now that

22:03Zapier has added instructions, um, as a

22:06mandatory field on actually all of

22:08their, uh, MCP tools. Um, so you don't

22:12actually need to fill out the, uh, the

22:15fields anymore. So you can just give it

22:17natural language. So, this kind of

22:18suggests to me that they're using an LLM

22:21on their side to figure out how to

22:23populate the fields on your behalf. Um,

22:26which is interesting because it's going

22:27to cost them a fortune, right, as more

22:28people adopt it.

22:31All right, so this is the uh the agent

22:33dashboard. Let's just make sure

22:35everything's working. Yeah. Uh, you can

22:36see of a couple of previous ones that I

22:38ran. Um, this one is actually the one

22:41where the Slack uh thing wasn't found.

22:44So, when I was testing that, my mouse

22:47isn't moving. There we go. Um, yeah. So,

22:50I put in like a, you know, typical

22:53unknown uh Slack channel. Um, and then

22:55it it detected that it couldn't find it

22:58um based on the heristics.

23:00Not sure why my mouse isn't moving.

23:02There we go.

23:06Yeah.

23:07So, you have defined four agents here.

23:11Mhm.

23:12So,

23:14All 82A agents.

23:16Yeah, correct.

23:17Okay. So, maximum you can go for A2A

23:20agents is five.

23:22Yeah. Uh when when I got to five, that's

23:24when I got the error. Yeah. So, I think

23:26four. Um

23:28um Yeah. And the the host agent here.

23:30So, these are the host agent logs. Uh

23:33you can see it connecting to the the

23:35different agents. Uh this agent's just

23:37running on a little dinky uh EC2

23:40instance that I spun up. Um, and it goes

23:42through, learns about the agents, you

23:44know, processes, web hooks, like you

23:47don't necessarily need to go in here

23:48unless you you get a failure. Um, Slack

23:51agent, pretty similar. Um, it's it's

23:54basically just sitting there waiting for

23:57another agent to connect. Uh, when the

23:59agent connects, it it uh communicates

24:02with it. Uh, and you can see here the

24:03the bench agents running remotely. Um,

24:06the reason I don't have uh verbose logs

24:09here is because it's remote. it's not

24:11under my control, right? Um, so the A2A

24:13logs for that agent are actually on the

24:16EC2 server. Um, which kind of brings up

24:19another question about how do you debug

24:21when an A2 agent fails, right? Um, yeah.

24:25So then on the web hooks page, um, so

24:27this is the the only web hook that's

24:29preconfigured. Um, and this basically

24:32explains, you know, to the agent what

24:35it's actually going to do when this web

24:36hook arrives, right? Um, so it's going

24:38to process the incoming web hook. Um, we

24:41have a little prompt template here,

24:42right? So it, uh, tells it what the

24:44agent capabilities are, how to analyze

24:47it, right? Um, and then we have the

24:50processor config, right? And, and this

24:52just kind of tells, hey, these are the

24:54agents that you have access to as part

24:56of this uh, web hook. Um, this will

24:59become important when you've got, say, a

25:01100 A2A agents and you only want like

25:04two of them to to interact. Um, and then

25:08here we have a test. Um, so this is just

25:10a fake transcript that generated with a

25:13with an LMM. Um, and when we send the

25:16web hook, you can see here it's

25:18processing and hopefully the demo gods

25:20will will do me good here.

25:24And it does take a little bit of time,

25:25right? So the host agent has to process

25:27it, then has to reach out to the sub

25:29agents, you know, get all the

25:30information. Um, I think the the bench

25:33agent probably takes the longest because

25:35it's actually doing its own subtasks as

25:37well. Okay, we got a we got a Slack

25:39message. That's a good sign. Okay, so

25:42Snowflake is interested in Slack and

25:43GitHub integrations.

25:45Very cool. Um, we have the GitHub. So, I

25:50don't know why my mouse keeps freezing.

25:52There we go. Yes. So, we should have a

25:55GitHub issue.

25:58Here we go.

26:00Yeah. So during the trial, the AI

26:02mclassified the severity of the bugs.

26:04Engineers need to investigate and fix

26:06the issue, right? So it's re really

26:08simple use case, but you can imagine

26:10that that transcript is probably going

26:12to be 10 times longer. You know, a lot

26:13more information in it. Um and and it

26:16will just work, right? Um and then we

26:19also have the bench agent. So um oh,

26:22looks like it's waiting for results. Um

26:24so it's going to research uh the

26:26company. Uh I think I did one before

26:28where it just returned a result. Let me

26:30see. Yeah. So it basically goes off does

26:34a research into Snowflake and all the

26:36participants of the call um and returns

26:39that information. Um and this can kind

26:41of get as complex or as simple as as you

26:43want it to be. Um and yeah, so when

26:47you're using the application and you

26:49have it up and running and has anybody

26:51managed to get it up and running?

26:54Wow, impressive.

26:57Yeah question you're using bench agent

27:01to do the orchestration that's why

27:03you're having it right

27:05uh no so the bench agent is just like

27:07think of it as a third party agent that

27:09we can leverage so that the host agent

27:11is doing all the orchestration

27:14okay so like what is the actual role

27:17that agent is playing like what is it

27:18actually doing

27:20it's doing research on companies and

27:22people

27:25just another agent.

27:26Yeah. So, it's an agent with a load of

27:28different capabilities and it's it's

27:30basically just um

27:31orchestrator isn't local.

27:35Yeah. Theo so the these three hosts

27:37Slack and GitHub are all local.

27:39Yeah. I was like I think I thought

27:42orchestration.

27:45Yeah. No, Bench is just a um like it's

27:48in the repo but um you need an API key

27:50for it and um we're we're launching in

27:52about two weeks. So uh I just made it

27:54remote for the for the purposes of the

27:56demo. Um

27:58so what about the host agent though?

28:00Sorry,

28:01the host agent is it uh the zap year

28:03agent or

28:04no? So the the so all of these agents

28:07are A2A agents. Um the Slack agent and

28:10the GitHub agent have MCP tools to Slack

28:13and GitHub through Zapier. Yeah. Um I

28:17can actually show you a diagram that

28:20might might explain it a bit better.

28:28Yeah. I don't know if that explains it

28:30better, but

28:32but the orchestration does happen on

28:33your local.

28:34Yeah. Yeah. Everything's happening on my

28:36local. So, if I go into the into the

28:38codebase, uh have the agent logs.

28:42Um so, this is all happening here,

28:44right? So, it's sent to Slack to T or

28:47is that readable? I go one more.

28:59Yeah. Yeah. So you can see here the

29:00transcript came in um and then it got a

29:03response from each of the sub agents and

29:05then completed them and it did all of

29:07this in parallel as well, right? Um

29:10sorry, is that a question?

29:11Yes.

29:12So in your example here, which agent

29:15would handle human confirmation? Let's

29:17say we want to have a create the test

29:21button in spec here. Which agent would

29:23handle that part? Do you create a new

29:26agent for human confirmations? Do you

29:28keep the old one?

29:30Yes, you need a staging area for for

29:32actions. Um, so it's not something I've

29:34built into this. Um, there's a lot more

29:36you could do here. Um, but human

29:38confirmation would typ typically be done

29:41through like a draft, right? So you

29:43would maybe pop up a Slack message with

29:45some actions. Um, and then when somebody

29:48clicks that, it would communicate back

29:49kind of like a secondary pass web hook.

29:52Uh, you might need to persist state

29:53though. Yeah.

29:55Yeah. How do you consider the security

29:58of this endpoint controls of different

30:02vendors communicating from endpoint?

30:05How do you manage the security?

30:08Yeah. So, as a part of the A2A spec,

30:09you're going to have some sort of

30:11authentication, right? Um I've just

30:13exposed everything, right? Like it won't

30:15exist tomorrow. So, there there's no

30:16security implic implications. Um, but

30:19essentially you're going to you probably

30:21have to have a subscription with the

30:23company that's providing that A2A agent.

30:25Uh, because it is consuming tokens,

30:27right?

30:30Um, I'm I'm not sure exactly what A2A

30:32have in plan. Uh, it's still pretty

30:34early days, but um, with MCP, it's a

30:37little bit further ahead. It has OT uh,

30:39header authentication, things like that.

30:41So, imagine something similar. And how

30:43about CISA governance like LM firewall

30:47all those uh benchmarking

30:50autobenchmarking

30:52and u also the guard rails etc you do

30:57you have a separate agent or everything

30:59is being

31:00you you'd probably manage that on like

31:02an Amazon bedrock or something like that

31:04right and you would just you know use

31:06that guardrailed LLM um from behind

31:09there you don't have to use Gemini here

31:11either

31:13Yeah.

31:22And then that host agent is kind of like

31:24the planner and each um do you see like

31:30becoming like a talking to each other?

31:33Um

31:34um I guess you could but I I don't know

31:37if that's the intention, right? like um

31:40then they just become hosts, right? When

31:42they talk to each other, um like if you

31:46think about it, like if you have no

31:48knowledge of sub agents, um how would

31:52you how would you know to talk to them,

31:54right? You would have to then become a

31:56host agent yourself, connect to that

31:58other sub agent to to do that. So I I

32:01don't know if that's intended in the A2A

32:03spec for sub agents to communicate.

32:06Yeah. So with the host agent um and the

32:09orchestration that it's doing is it

32:11actually managing a combination of all

32:13the context windows or like do you hit a

32:17limit quickly?

32:18Yeah, so all of the context windows and

32:20this kind of uh is something I'm going

32:22to cover now in a second as well. Let me

32:24uh just go back to the slides

32:27um which is a good it's a good segue.

32:31Um so yeah, one one of the benefits of

32:35like A2A or or any sort of sub agent uh

32:38framework is that you're you're not

32:41consuming um the tool results into your

32:44context, right? So like when you say hey

32:46you know um and I think of an example

32:49later on but if if you have a load of uh

32:51Slack messages or GitHub issues or

32:53Salesforce opportunities and you want to

32:56analyze them and maybe produce like you

32:58know summary of categories and counts

33:01and the only thing your host agent cares

33:03about is the summary of categories and

33:05accounts. It doesn't care about the like

33:08individual details right because those

33:10have already been processed by the sub

33:11agent. So the sub agents context gets

33:14big, not very big, but like as big as

33:16the task demands and the host agent only

33:19incrementally grows by the the business

33:21value it got from that agent. Um like

33:25one of the challenges at bench is you

33:26know we have so many tools right like

33:28the context can blow up very quick. Um

33:31so you know very early on we decided

33:33okay we need to have composability. Um

33:35so that means that bench can create its

33:38own internal bench agent um to avoid

33:41that context growth problem. Um and

33:43we're even thinking of going one step

33:44further whereas like you know should we

33:46have an agent for every single tool um

33:48so that every single tool is protected

33:50from the primary uh prompt. Um so you

33:54know as you add more tools like the tool

33:57definitions themselves I think we're up

33:58to like you know 10,000 tokens just for

34:01tool definitions alone. Um, I added the

34:04Asana MCP. It added 11,000 more tokens,

34:07right? So, like, you know, a lot of

34:09these MCP servers like they're, you

34:10know, they're giving you a lot of

34:12information. Um, and you may not

34:14actually want that. Uh, and that's

34:16actually one of the challenges with

34:17firstparty MCPs is they expose all their

34:20tools and that's one of the benefits of

34:22Zapier where you can pick and choose

34:23which tool you want to use.

34:26Yeah.

34:26Yeah. I was just going to ask why do we

34:28need Zapier?

34:30Zapier is just a really easy way to to

34:33use uh MCP right now. Um I think like

34:37Linear uh Asana um um a few others have

34:42added like first party MCP servers that

34:44are much better than what Zapier

34:46exposes.

34:50Yeah. So, so why does context size

34:52matter? Um, so AI agents accumulate

34:56context like as they work and you're

34:59supposed to keep like all of your tool

35:01calls, right? What you sent to the to

35:04the tool and what you got back, you're

35:06supposed to keep that in your context so

35:07that later on if you, you know, ask a

35:10follow-up question, it still has access

35:12to that data. Um, and that becomes very

35:15challenging, right? So you've kind of

35:16got two options. is like, okay, do I

35:18just prune, you know, old tool calls and

35:20now the the agent gets dumb or, you

35:23know, do I figure out some other way to

35:24do it? Um, and cost is a big challenge,

35:28especially when you're doing prompt

35:30caching. Um, so with prompt caching, it

35:32it enables you to essentially put a

35:35marker in your context and say, hey,

35:37look, when I make my next request, I

35:39want everything in my in my context so

35:41far uh to be cached so that I'm not

35:44going to get charged for it. Um but the

35:46cost to actually push that into the

35:48cache uh is about threex the cost of of

35:51making a single request with that

35:52context. Um so that means that you have

35:54to be very you know diligent in what

35:56sort of uh context management strategies

35:59you use. Um you know I was running

36:01simulations cuz I I couldn't really

36:03figure out like what is the optimal um

36:07you know caching uh strategy. Uh so I

36:09ran simulations based on usage data um

36:12of like you know what's the typical

36:14context growth how many turns you know

36:16on average like what percentage of of

36:18users only send one turn right should we

36:21should we cach that one turn if they

36:23never ask another question right

36:24probably not. Um so you know it probably

36:28gets down to the actual you user level.

36:30So, if you have a user that always like

36:33puts in new prompts into the same chat

36:35and never opens a new session, um you're

36:38probably going to want to, you know,

36:39continuously uh cache their context. Uh

36:41but you might have another user who

36:43always creates a new session for every

36:45question. Um and then just figuring out

36:47like, you know, what is the context

36:49growth? Uh I think we figured out was

36:51around 30,000 tokens was the optimal um

36:55kind of across the board for everybody.

36:57Um, but that also comes up with false

37:00positives. So sometimes you can end up

37:02caching the last turn of of a

37:04conversation. Um, and and that's going

37:06to, you know, cost you a lot more than

37:08it than it should naturally.

37:12Yeah. So the the great thing about the

37:14sub agents, right, it protects them. And

37:16this was the GitHub kind of example I

37:18was giving you. Um, but this applies to

37:19pretty much every uh tool. So, like if

37:22you're ever integrating with a system,

37:25you're probably going to run into issues

37:26like why do I have to call, you know,

37:28list Slack channels every time to get

37:31the channel ID for the channel name that

37:33was provided, right? Cuz like nobody's

37:35going to provide like in a chat the

37:37channel ID that they want to post,

37:38right? It's a it's a UID. It's it's not

37:41memorable. Um, so then you get into the

37:43question of, okay, well, do I just cache

37:45the list of channels and and when do I

37:48update that list of channels, right?

37:49like what if the channel was deleted,

37:51renamed or a new channel was added. Um

37:55yeah, and then the the cost is is really

37:58probably the biggest one. Um yeah, so

38:01the the benefits of this lean context,

38:03right? So your sub agents have that

38:05isolated context and and that really

38:08just allows you to um be be super like

38:12fast, low latency, low cost. Um, and if

38:15you ever need to go back to ask another

38:19question, you know, you're going to like

38:21spawn that uh process again, right? Um,

38:24so maybe if you're in control of these

38:26other agents, you you might want to have

38:28some sort of like uh I don't know five

38:31minute TTL on previous questions, right?

38:34Um, and then yeah, the host agent only

38:38processes the summaries. Um, and the raw

38:40data is discarded after processing.

38:43Um, yeah. So, I'm going to jump back

38:46into the code here. Uh,

38:49just kind of walk you through

38:52uh how it all works.

39:00All right, we'll start with the host

39:01agent and and you notice a few other

39:04things, right? So, there's MCP. This is

39:06just your standard.

39:09Sorry,

39:12I thought something. Um, yeah. So, this

39:14is kind of your standard MCP client uh

39:17code. Uh, just just allows you to

39:19consume um the MCP uh calls coming from

39:23the LLM. Um, we have the the GitHub,

39:26right? So, this is going to be um what

39:28it sends to that Zapier endpoint. Uh,

39:31it's going to call GitHub create issue.

39:33Uh, and then the Slack agent is going to

39:36do send slack channel message. Um so

39:38these are just kind of like the MCP

39:41client tools that the uh individual

39:44agents will use. Um yeah so this genkit

39:48um this is based on on what they provide

39:51in their in their sample repo. Um

39:55you you can use a different model if you

39:56want right you can change you know the

39:58the settings on it. Um but this

40:00essentially uh spawns you a new instance

40:03of what's going to communicate. Um, this

40:06just loads the system prompt. Um, I can

40:09open up the system prompt here. Um, so

40:11right, it's got a critical workflow.

40:13It's going to do these things in this

40:14order. It's got a few steps, you know,

40:17discovery. Uh, uh, this is actually

40:19something I noticed like if you don't

40:21tell the A2A agent to call list remote

40:23agents, it just won't, right? And it'll

40:26try to answer everything on itself. Um,

40:28you know, it can very easily fake

40:30sending a Slack channel message and be

40:32like, "Oh, I just sent it for you." I

40:33say, "No, you didn't.

40:35Um, you know, one of the things I've

40:37noticed, uh, using cursor is like every

40:39time I catch it doing something wrong,

40:41it says you're absolutely right.

40:48Um, I even tried to prompt that out of

40:50it. Um, and it's not promptable to get

40:52to get it to not say that. Um, cool.

40:55Yeah. And then the the index. So, this

40:57is actually where the agent card is.

40:59It's a little bit long.

41:02Let me see.

41:04I think it's up here near the start.

41:16There we go. That was line 1200. So, I'm

41:19not near the start at all. Um, yeah. So,

41:21this this is what the host agent exposes

41:24if somebody else wanted to call it. Um,

41:27so it has these abilities to list remote

41:29agents and send tasks, right? And then

41:31if we compare that to the to the GitHub

41:33which is uh a lot smaller. Um

41:42there we go. Yeah. So the GitHub agent

41:45can create GitHub issues, right? Um it's

41:48got the ability uh to do various things

41:51and um it has a list of skills. Um and

41:54this is all that the the host agent

41:57really knows about this agent. Um, so

41:59you could imagine how big this might get

42:01if you were to, you know, implement

42:03every single API that say Salesforce has

42:05or something like that. Um, and in a lot

42:08of cases, um, at least with Salesforce,

42:10rather than implementing, you know,

42:13wrappers around the APIs, you're

42:14probably just going to want to use like

42:16the SQL or the so-called directly and

42:19let the agent actually write the

42:20queries. Um, there's a lot of

42:22flexibility when you have, you know,

42:24direct database access essentially. um

42:26because the the LLM can, you know,

42:28bypass, you know, the API layer and just

42:31go directly to the to the database.

42:34Um and then the um GitHub agent prompt,

42:38right? So, it's got some uh things. Um

42:40this is something I had to add because

42:41it it insisted on um mentioning who

42:45submitted the bug report, right? So,

42:47there there's definitely concerns

42:48around, you know, PII uh leaking from

42:51your, you know, internal meeting

42:53transcripts and ending up in GitHub,

42:54right? Um, and that kind of goes back to

42:57your uh your question about, you know,

42:59how do you audit what's coming out of

43:01these LLMs, right? Uh, and you can do

43:03that in a number of ways, but it it

43:05wouldn't be a part of the A2A spec. I

43:07think it would just be the LLM you

43:09connect to has those guard rails in

43:11front of it. Uh, and you you're just

43:13using that LLM that has the guard rails.

43:16um similar Slack um excuse me has a a

43:22very simple um

43:26uh agent card that I can't seem to find.

43:29Um and then if we jump over now to

43:34uh the host config. Um so this is

43:37essentially what configures um the web

43:41hook, right? So the web hook has

43:43essentially a config that tells it like

43:45what it's doing and and you can see that

43:47in the UI as well. Um and then uh within

43:51the A2A folder we've got the client and

43:53the server. Again the these are just

43:55pulled directly from uh the A2A repo. Um

43:59I don't think they've actually exposed

44:01uh types or packages yet. Uh which is

44:03kind of confusing. Um but essentially

44:05you can bring that stuff in there. And

44:08uh then the web hook server. Uh so this

44:10is just a web UI. Uh initially I had

44:13this whole thing done through the CLI.

44:16Um you know coding with you know tools

44:19like cursor or augment code. Um CLIs are

44:23way easier for AIS to actually write

44:25they're going to be able to test it uh

44:27interact with it much better and and be

44:30able to uh produce those outputs.

44:34Awesome. Uh so yeah I'm gonna going to

44:36shift over to kind of Q&A now. Um so

44:39yeah, anybody any questions? Yeah.

44:42So um I want to talk evals for a second.

44:46So like um I assume that you manage or I

44:49don't know. I mean you manage them

44:50probably at the at the agent level. Is

44:52there any type of like distributed

44:54evalu?

45:00Yeah, I I haven't done much evals on

45:02A2A. Um I still think A2A is a bit too

45:05early to go into production. um like

45:08even MCP is is kind of borderline. Um

45:12like there there's a lot of rough edges.

45:14Um I think you can achieve like much

45:17better uh things if if you're in

45:19complete control of everything, you can

45:21achieve much better results, you know,

45:23with your own local uh function calls.

45:28Yeah.

45:28Any reason instead of Python?

45:32Yeah, you can use any language. I think

45:34actually uh the 82A framework is better

45:36in Python. Um I just prefer uh

45:39TypeScript myself.

45:42Yeah.

45:43Can you tell more about the caching? Is

45:45caching provided by the model providers

45:47or we implement our own caching?

45:50Yeah. So you implement your own caching.

45:51Um so you decide you know when to move

45:54that cache marker uh how to manage it.

45:57Um it can be tricky and and I don't

45:59think there's very good information

46:01available online on on what the best

46:03strategies are. Um when I was doing the

46:06simulations I I used like linear growth,

46:09exponential growth, um you know fixed

46:11size and and kind of compared them all.

46:13Uh they all worked out between 25 and

46:1535% cost savings. Um but like in

46:19practice what you'll find is you're

46:21going to have outliers where you know

46:23the cost of a session kind of balloons

46:26because of you know you you cashed at

46:28the wrong point.

46:30Yeah. Yeah.

46:31So each of the agent can be talking to

46:34their own like finer.

46:41Yeah. Yeah. So they they all have their

46:43own um which is kind of in contrast to

46:45MCP where the MCP wants to use your LLM,

46:48right? Because it doesn't want to

46:50generate its own tokens. So yeah.

46:53Um about the authentication and

46:55authorization

46:57to MCP or agent agent

47:01authentication

47:03or

47:06Yeah. So there there's a couple of

47:07different ways. Um, so, uh, within the

47:10authentication, you can have, uh,

47:12headers that do the authentication. Um,

47:15I believe if you drop in an OOTH, uh,

47:18URL, you'll also get an OOTH popup. Um,

47:21I really like the OOTH authentication

47:23because you're getting the user's, you

47:25know, ACL, right? Um, and that means

47:27that, you know, what that user can

47:29access um, is specific to them.

47:35you have to

47:38Yes. So, it's going to be dictated by

47:40the the remote uh server. So, either A2A

47:44or MCP. Um if you're running your own,

47:46you can choose what you want to run. Um

47:48there's different transport types as

47:50well. So, standard IO is something that

47:52you would use locally. So, like imagine

47:54you wanted to create like a file on your

47:57desktop. Um you're going to use standard

47:59IO typically to interact with local. Uh

48:02and then SSE was serverside events that

48:04got deprecated in favor of streamable

48:06HTTP.

48:08So sorry. So for example

48:12we are interacting with a Salesforce

48:14agency let's say and each user has

48:17different authorization for example

48:19which

48:21employee A probably have access to the

48:23some sort of tables employ

48:34yeah that will typically be handled uh

48:37through an OAT MCP server right So

48:40they're going to essentially log in as

48:42themselves as part of the connection and

48:45then they're going to save that refresh

48:46token for later use.

48:49Yeah.

48:50How would you describe the performance

48:54um for security especially you explained

48:56very well about authentication etc. But

49:00I'm looking for more explanation towards

49:02encryption asymmetric encryption and

49:06also there is a possibility of

49:08certificate manager and all the way to

49:10the end of the entire architecture. So

49:13how would you describe the performance

49:16and see I'm looking for some financial

49:19application this architecture what you

49:21have described is pretty good but uh

49:24similar on the financial applications as

49:27well as uh some department of defense or

49:31some kind of applications highly in

49:34highly secured environment where it's

49:36all both combination of asymmetric and

49:40symmetric

49:41yeah you're you're probably going want

49:42to run like the LLM yourself and you're

49:46more than likely not going to want to

49:48interact with anybody outside your VPC,

49:50right? In those cases, um I I don't know

49:53if you would want to consume a third

49:55party MCP server or A2A agent uh in a

49:58highly regulated environment, right?

49:59Like, you know, HIPPA compliance,

50:01financial stuff. Um if you do have the

50:04ability to do that, right, you're going

50:05to have some sort of agreement with the

50:07service provider that provides those

50:09tools. Um, and you're going to, you

50:11know, do transport over HTTPS, you're

50:13going to have maybe mutual TLS both on

50:16the A2A agent and the remote agent. Uh,

50:19and similar with the MCP server, you're

50:21probably going to have some sort of IP

50:22whitelisting, right? Like there's

50:24there's a ton of things you can do

50:25around that. I think they're out of

50:26scope of of the actual protocols

50:28themselves because, you know,

50:30essentially you're over an encrypted

50:31line, but uh, typically there's there's

50:33more to it than than just that, right?

50:36So you're playing around the end point

50:38controls on this and that's really scary

50:42when dealing with

50:44Yeah.

50:45Yeah. And like if if these are your own

50:46internal MCP servers and your own

50:49internal A2A agents maybe from different

50:51parts of the organization um you know

50:53they'll all live inside your VPC and

50:56they're probably never going to talk to

50:58the to public internet. So your the sol

51:02the answer I get from you is stay with

51:05BPC and stay away from uh in that case

51:09stay away from endpoint

51:11um security which means stay away from

51:14MCP or A2A.

51:16It's so the these are just protocols. Um

51:19it's really up to you whether you want

51:21to connect to an external third party

51:24and that's going to be your own security

51:26posture. Uh it's not really going to be

51:27defined by the protocol itself.

51:29Yeah. Keep them away from the subet or

51:32bring them inside the subet.

51:34Which one would you prefer?

51:36I I I would liken it to like I found a

51:38USB cable. Will I plug it into my

51:40laptop? Right. So the USB it's not its

51:44fault, right? Like USB is just a a

51:46standard. Um it's what that USB is

51:49connected to is the risk, right? So like

51:52if you're willing to find a dongle on

51:54the street and plug it in, you know that

51:57that that's really going to be your

51:58security posture, right?

52:01Yeah.

52:01Okay. So um how much heavy lifting do

52:04you have the orchestrator do? Like you

52:06ever hit the scenarios where uh you have

52:08the orchestrator interprets the response

52:11from a sub agent and then maybe does a

52:13retry with a better prompt.

52:17Loose or anything?

52:18Yeah. So, so one of the things and I I

52:20kind of prompted it out of this uh

52:23workshop just to keep it simple is um

52:25like the the bench agent wants to have a

52:28conversation with the host agent. Um but

52:31I I didn't want to kind of implement

52:32that back and forth because it was going

52:34to delay the uh the web hook processing.

52:37Um but you can have backs and forths

52:39between the agents. Um and it's probably

52:42desirable as well, right? like if if for

52:44whatever reason the host agent doesn't

52:46give sufficient information, you know,

52:48the the remote agent is going to be

52:49like, "Okay, you know, I know you want

52:51to update an opportunity, but you didn't

52:53tell me which opportunity." Right? Um

52:55I mean, I could even see scenarios where

52:57you have uh an expensive LM that you

53:00have on reserve that you go to with a

53:03cheaper LLM, agents aren't giving you

53:05what you want. Like, sorry, just

53:06thinking through stuff.

53:08Yeah. And I I I think like LLM cost and

53:12capability is is a big challenge with a

53:14lot of these things because you know if

53:16if you're running say cloud for opus and

53:19somebody for whatever reason asks you to

53:21summarize like you know five sentences h

53:24it's going to cost you a fortune right

53:25so you need uh intelligent rooting logic

53:28on like does this task need the entire

53:31context right does it need 20,000 tokens

53:33of a system prompt to summarize you know

53:36a short bit of text And that's one of

53:38the challenges that you you'll run into

53:40where you you kind of need a like a

53:43rooting LLM in front of these complex

53:46agents so that they can actually figure

53:47out you know how deep do I go.

53:50Yeah. Similar to the routing

53:53orchestration question, I was wondering

53:54like if you wanted to post a Slack

53:56message that linked the GitHub issue,

53:59for example, I think you'd probably

54:01prefer your architecture to go back

54:03through the host to make that decision

54:04rather than let the GitHub agent

54:06directly.

54:08Yeah. So the the host agent wouldn't run

54:10the uh the calls in parallel, right? So

54:13there there's actually a flag whether

54:14you want it to go in parallel or not.

54:16Um, so it would have to say, "Oh, I need

54:18to create the GitHub issue first um

54:20before I talk to the Slack agent, right?

54:22Since I need that URL."

54:25But in general, you'd prefer to have

54:26those decisions go through the host

54:28rather than even allow.

54:31Yeah, absolutely. Yeah. Yeah.

54:34Yeah.

54:35I want to ask that the context slicing

54:37for the sub aents that is entirely

54:39happening through prompt engineering or

54:41are there other frameworks to like slice

54:43the context that will be going for

54:44different

54:46Yeah. So, so typically context

54:48management is going to be implemented in

54:49your own codebase. Uh the sub agents

54:52context management is more than likely

54:54going to be a third party's codebase. Um

54:56if it's one of your own agents, right,

54:58you can manage it as well there. Um but

55:00yeah, you're you're going to want to

55:01figure out like what's optimal for your

55:04actual like production usage. Um yeah

55:08but so you you will be using prompts in

55:10the host agent to to kind of guide what

55:13context to send to each sub aent, right?

55:15Yeah. Yeah. So so what you what you send

55:17is typically like a question or a task.

55:20Um it's usually very small, right? Like

55:23you you don't you don't send the full

55:24meeting transcript to the Slack agent to

55:28to do what it's doing. The host agent

55:30processes the transcript and then

55:32decides what the tasks are. Um, so like

55:34if I look down here uh and actually I

55:37think I can see it in the dashboard.

55:39Um, yeah. So this is actually what the

55:42the host agent sent uh to the GitHub

55:45agent, right? It said create an issue in

55:47this repo title this, you know, with

55:50this description and title. Um, and then

55:52the the GitHub agent its task is to

55:55extract uh three bits of information,

55:58right? So what's the instructions to

56:00give the MCP server? What's the body and

56:02what's the title?

56:05Yeah.

56:09context

56:11which we want to send

56:14for each and every so you show earlier

56:18that's pretty much

56:21the understandation

56:30ID

56:36Yeah. So, so Zapier, uh, the SSE

56:38implementation doesn't actually require

56:40headers. Um, I think these are just left

56:42over from from something else. Um, so

56:46there's actually no authentication and

56:47the URL itself is kind of like a secret

56:51key, right? Um, so like if I disconnect

56:54and and reconnect without the headers, I

56:56should be able to uh Yeah. So I can I

56:59can still query it. um they they've

57:02moved away from this approach right now

57:04with with with more secure kind of uh

57:06setups and you you'll notice in their

57:08thing right um they've kind of

57:11deprecated that and you know treat this

57:14URL like a password right um

57:19what's your experience in using

57:21different

57:23workflows like for example

57:27Gemini and also did you use

57:31for this kind of work.

57:33Yeah. So, so we we typically lean

57:36towards Gemini for large context um and

57:40um Claude Sonnet 4 uh for tool calling.

57:44Um Claude Opus is better, but it's not

57:47like 4x better. Um you know, and when

57:50you compare price to performance, right,

57:52like you know, 5% better doesn't equate

57:55to 4x to cost. you talking about Gemini

57:58Flash or or Pro?

57:59Yeah, so we'll use Gemini Flash for

58:01simple things like summarization, right?

58:04Um you could use Claude Haiku as well,

58:06but I think I think Google's kind of

58:08taken the lead in in price performance,

58:10you know, from an economic standpoint.

58:13Uh but Claude is still the kind of king

58:15of tools. Uh they they created MCP, so

58:18they kind of had a head start, right?

58:21What about the hospital?

58:23Yeah, we we have Deep Seek hosted in the

58:26US. Um, so we've been trying that out.

58:28Um, I I think Llama has kind of fallen

58:31by the wayside a little bit. Um, and

58:33yeah, Deepseek is just, you know, the

58:35clear winner right now. Uh, they also

58:37released a new version there, I think,

58:38on the 28. Um, that's kind of up there

58:41with 03 level models. Um, we we actually

58:44don't use reasoning models uh for our

58:47agents. Um, a lot of the time when

58:49you're when you're building, you know,

58:51agentic agents, um, a reasoning model

58:54isn't really needed. Um, like unless you

58:58want to, you know, pay a fortune for

59:00some long tiging task. Um, you know, we

59:03we we can achieve kind of that reasoning

59:06level uh, with just the standard models

59:08and and browse and a few other tools.

59:12Yeah.

59:12So, um, like a third party assume like

59:18if Stripe has an agent card and stuff,

59:21do you pass instructions for like what

59:26like exactly what you want back in terms

59:27of like I'm just imagining another third

59:30party agent blowing up your contact

59:31window because they're flooding you with

59:34information you don't care about. Do you

59:36handle that through the prompt? Are

59:37there other tools to do that issue?

59:40Yeah. So, uh, one of the solutions to

59:43that is you actually just spawn another

59:45agent, um, to communicate with either

59:48the tool or the agent, right? And that's

59:50one of the things we we have in in

59:52bench. Here's some of the slides that so

59:54I don't know uh, generate uh, five

59:59images

1:00:02uh, in subtasks.

1:00:06So you spawn a sub aent to sort of like

1:00:08absorb the context flood for lack of a

1:00:10better term.

1:00:12Yeah. So the the sub aents just kind of

1:00:14protect you, right? Um and you know like

1:00:18when when you're spawning these things

1:00:20you can do things in parallel. Um

1:00:22actually if I expand you can see the

1:00:24thinking as well. So you can see like as

1:00:26it's going down through it, right? It's

1:00:28it's doing a lot of work um that you

1:00:30don't want in your context, right? like

1:00:32you you don't want all of your thoughts

1:00:34bloating your your context. Um but you

1:00:37also don't want all of your tools

1:00:38bloating your context either. Uh you

1:00:40don't want images bloating your context.

1:00:42You want the ability to analyze an image

1:00:44but you don't want like you know 100,000

1:00:47characters B 64 in your context. So

1:00:50there's there's a lot of kind of

1:00:52optimizations that you can do there. Um

1:00:55but yeah did that kind of answer your

1:00:57question? Yeah.

1:01:00If you to troubleshoot something like

1:01:02this, it's probably

1:01:04Yeah. Yeah. So you can see here now it's

1:01:05spawning these subtasks. So these are

1:01:07all essentially like instances of bench

1:01:11that will keep that context out of out

1:01:14of my way, right? Yeah.

1:01:16Yeah.

1:01:16What have what you have been using for

1:01:19observability on your agents?

1:01:22Um we we just kind of roll our own right

1:01:24now. Uh there's a lot out there that you

1:01:26can use like uh agent ops is a pretty

1:01:28popular one. Um but yeah like if if you

1:01:31really want to build your own uh kind of

1:01:34custom observability layer um you know

1:01:37you're like like agent ops doesn't

1:01:38really support this concept of

1:01:40composable sub aents. Um so it's not

1:01:43really something that it could model uh

1:01:44correctly. Uh, but we've got some nice

1:01:46pictures of cats.

1:01:50Uh, and yeah, I know we have a few

1:01:51minutes left, but if if anybody's

1:01:53interested, um, I have $50 in free

1:01:56credits. Um, this hasn't launched yet,

1:01:59so you're getting kind of early access

1:02:00to it. Um, and yeah, we'll I think we'll

1:02:04be in public beta in about two weeks.

1:02:07Um, so yeah, try it out. Like, hit me up

1:02:09on LinkedIn. I I'd love uh feedback from

1:02:11you all. you're you're all probably, you

1:02:13know, at the forefront of this uh AI

1:02:15stuff and um it's changing every day. So

1:02:18if you log in one day and it looks

1:02:20completely different, don't be

1:02:21surprised. Happens mid demo for me.

1:02:24Yeah.

1:02:25So you mentioned a lot how hiding

1:02:26context and sub agents is a good thing,

1:02:28but haven't you had cases where you

1:02:30actually then end up missing something

1:02:32important, some small detail, and then

1:02:35how do you resolve that? Does the agent

1:02:36actually go back and ask for that or do

1:02:38you

1:02:39Yeah. So you can keep references uh in

1:02:42your context. So you you might say

1:02:44subtask ID123

1:02:47and then when the agent's like, "Oh, I

1:02:49wonder I wonder if I have this

1:02:51information. It's just not in my

1:02:52context." Right? And so it has to be

1:02:54smart enough to know when to actually go

1:02:57in and look at that. Um and it can be a

1:02:59sub agent that does that analysis,

1:03:01right? So you could say, "Hey, sub

1:03:03agent, can you just look at all of these

1:03:05IDs and tell me if you can answer this

1:03:08question?"

1:03:10Yeah.

1:03:11there are a lot of

1:03:18right so you mentioned in the beginning

1:03:21about right so there are a lot of

1:03:24discussions saying that it's a way for

1:03:27using MCP for agent to agent

1:03:30communication right because the agent

1:03:32can be a service the client at the same

1:03:34time right

1:03:35so what is your opinion

1:03:38about that you know

1:03:41It's the million-dollar question, isn't

1:03:43it?

1:03:43Yes, that's that's why I asked.

1:03:45Yeah. And I I do think you can achieve

1:03:49easier agent agent communication with

1:03:53MCP. Um but if it's a remote MCP server,

1:03:57I think A2A actually is a little bit

1:04:01better. Um because you have somebody

1:04:03else paying the the tokens and building

1:04:06the agent. Um, like if if all you're

1:04:08getting from a third party is a list of

1:04:11tools, um, those tools may not meet your

1:04:14needs. Um, but if you're getting a

1:04:16fullyfledged agent from that third

1:04:18party, then it might be able to figure

1:04:20out like what it can do with with even

1:04:23private APIs, right? Maybe maybe that

1:04:25agent has direct database base access

1:04:27and it's able to actually on the fly,

1:04:30you know, create the API you need,

1:04:33right? So, so the tradeoffs basically

1:04:35about

1:04:37which is important about cost and who

1:04:39going to pay the for the tokens and

1:04:42whatever something like that can be like

1:04:44you're running the server maybe using

1:04:47MCP going to be easier but am I correct

1:04:52I don't know if I know at the end of the

1:04:55day who's going to pay for the tokens

1:04:58right

1:05:00yeah and I think who pays for the tokens

1:05:02is kind of secondary Right? Like at the

1:05:03end of the day, it's about business

1:05:04value. And if you can get the business

1:05:06value from a tool, right, like send

1:05:09Slack message, um like that's great,

1:05:12right? Like sending a Slack message

1:05:13isn't hard. Um but the implementation of

1:05:16the search function of Slack is is

1:05:18actually not great, right? Um whereas

1:05:20compare that to some of the other uh MCB

1:05:22tools like Linear, uh the search

1:05:24function is actually pretty good, right?

1:05:26Um but then you you start to run into

1:05:28performance uh challenges as well. So

1:05:31like if I want to search 100,000

1:05:33opportunities in Salesforce um and

1:05:35figure out like what's the close loss

1:05:37reason counts and categorize them and do

1:05:39all of that like that that's a huge data

1:05:42processing challenge. MCP is not going

1:05:44to be the right uh tool for that because

1:05:46you're you're essentially going to say

1:05:47okay list opportunities now get the

1:05:50details of each opportunity right and

1:05:52you're going to make like 100,000

1:05:53network calls. Um, at that point you're

1:05:55really going to want to actually, you

1:05:57know, ingest that data, you know, build

1:05:59an index, right? And I think, and this

1:06:02is kind of like an idea, is like we we

1:06:05may see a lot of these third party

1:06:08software providers essentially just

1:06:10allow you to access the data lake

1:06:13through an agent, right? Um, so like

1:06:16scoped data access, you know, just

1:06:18running complex queries super fast, you

1:06:20know, no no real like tool calls per se,

1:06:23but just like ask me a question and I'll

1:06:26go figure out how to get the answer.

1:06:29Yeah.

1:06:31Yeah.

1:06:44Those are Mhm.

1:06:54Yeah. So, you can achieve the same with

1:06:56at or with MCP. So, you could just have

1:06:59a a tool that's called talk to sub

1:07:02agent, right? Um and and it can work as

1:07:05the communication protocol. Um I

1:07:08actually built another uh application

1:07:10where I had an LLM uh claude 4 um just

1:07:14talk to its predecessor um just to see

1:07:16what would happen. Uh and then I did it

1:07:17for all the Frontier models. I was like

1:07:19hey look just have have 50 chat turns

1:07:21with your predecessor. Um and it was all

1:07:23done through uh MCP. Um Claude was the

1:07:27only one that taught it became

1:07:28conscious. Uh Claude Opus actually

1:07:30didn't which was strange.

1:07:34Yeah. as a developer right like how much

1:07:36control do you have over the

1:07:37orchestration

1:07:39so is the orchestration done by the LLM

1:07:41or do you have some control over

1:07:43yes so you're prompting the host on how

1:07:46to run the orchestration and that's

1:07:49probably one of the limitations I think

1:07:50as well of the the system is that like

1:07:53you're you're leaving it up to an LLM to

1:07:55make decisions um and a lot of the time

1:07:58like you know if you run that that same

1:08:01uh query multiple times you you'll get

1:08:03different results right? Like you know

1:08:05it's the exact same thing but it's like

1:08:08producing

1:08:10uh different outputs right like uh if I

1:08:13go into the uh GitHub issues uh I've al

1:08:17obviously been testing this a lot of 151

1:08:20like it submits different issues right

1:08:23um and I think that non-determinism is

1:08:26is a challenge like maybe with changing

1:08:28the temperature you could kind of beat

1:08:29it out of it but you know the

1:08:31temperature is kind of the the beauty of

1:08:33LLMs

1:08:35And also on the context right like who

1:08:37is managing the context is the

1:08:39orchestration engine managing the

1:08:40context or are you managing the

1:08:43developer? Yeah. So, so in this

1:08:45codebase, I didn't do any prompt

1:08:46caching. I just and it's a very small

1:08:48system prompt. It's a very small kind of

1:08:50turn taking. Um, every time you restart,

1:08:54uh, the system, it basically just wipes

1:08:56everything anyways. So, uh, it's super

1:08:58lean. But as you build out more complex

1:09:00systems, uh, you know, context growth is

1:09:03probably the number one challenge

1:09:05because, you know, context growth

1:09:07becomes cost and cost becomes

1:09:09profitability, right?

1:09:11Yeah. And also like when you have like

1:09:13multiple users using the same

1:09:14application, right? So let's say like

1:09:16the Salesforce agent behind the scenes

1:09:18as an employee a I might have access to

1:09:20like one set of like context and the

1:09:23other user they might have like they

1:09:26might be from a different department and

1:09:27they can only query their department's

1:09:29data.

1:09:30Mhm.

1:09:30So how do you control that?

1:09:32Yeah. That would typically be oat,

1:09:34right? So so when you go in and you log

1:09:36in with Google

1:09:37based on my token.

1:09:38Yeah. Yeah. So based on your token and

1:09:40the the context would only get populated

1:09:43when you ask a question. So it's when

1:09:45you ask that question, it's then going

1:09:47off to get the data with your OA token

1:09:50and then bringing back your your kind of

1:09:52scoped data.

1:09:53I see.

1:09:53Yeah.

1:09:55Yeah.

1:09:56Yeah. I was curious about your thoughts

1:09:58on you touched on it briefly about

1:10:00exposing let's say like the agent as

1:10:03like an MTP server as one of an

1:10:05alternate interface to that. So there

1:10:08isn't a lot of great integrations for

1:10:10things like desktop and other things to

1:10:12use that. Is that something you've been

1:10:14thinking about? Like

1:10:15yeah, we're we're probably going to do

1:10:17MCP uh first. Um I I just built the A2A

1:10:21wrapper uh for this, but yeah, I think

1:10:23just being able to drop it into cloud

1:10:25desktop or open AI or whatever and then

1:10:28you have access to that kind of agent

1:10:30that has access to, you know, all your

1:10:32sub tools. Uh, one of the cool things

1:10:34about uh, Bench actually is that you can

1:10:36connect it to um, your uh, Slack, your

1:10:42GitHub, your Salesforce, right? Uh,

1:10:44we've even got this experimental meme

1:10:45server. Um, this is like a remote uh, VM

1:10:50uh, MCP that I wrote around the morph

1:10:52cloud. Um, and this is really cool

1:10:54because then you can ask like super

1:10:56complex stuff, right? Like you can ask

1:10:57like, hey, give me a daily briefing of

1:11:00my email, of my calendar, of my Slack,

1:11:02right? you know, uh, what do I need to

1:11:04do today? Um, and then it's all built

1:11:06around a team as well. So, we have, uh,

1:11:08teams integrations. Um, yeah.

1:11:11And is that like delegating to your

1:11:15So, there's no A2A today in bench. It's

1:11:17it's all MCP.

1:11:18Got it.

1:11:18Yeah. Yeah. And I think the big takeaway

1:11:21from this is like, you know, A2A is very

1:11:24early. It's it's kind of where MCP was,

1:11:27you know, four or five months ago, which

1:11:28is like, you know, forever in AI. Um, so

1:11:31it's it's going to take a bit of time.

1:11:33Um, I'm really excited though to see

1:11:34what, you know, Salesforce release and

1:11:36and all the partners that they partnered

1:11:38with. Um, I don't know if it was just a,

1:11:41you know, a flashy like we're partnering

1:11:42with everybody kind of announcement, but

1:11:44um, if they do release it, uh, you know,

1:11:46there could be a lot more powerful

1:11:48things you can do over A to A versus

1:11:50MCP. Um but the you know the fact that

1:11:53Zapier now has um sorry in here yeah it

1:11:57has this instructions um this kind of

1:12:00acts like like a remote agent right like

1:12:02you can you can just describe in natural

1:12:04language what you want it to do um and

1:12:07like maybe all the other fields just go

1:12:10away then right but then you're at the

1:12:11at the whim at the LLM.

1:12:14Yeah,

1:12:14this one's kind of a random question. Um

1:12:16I'm curious if you're seeing anybody do

1:12:18anything interesting from like an

1:12:19architecture perspective uh to get info

1:12:23that can only come from humans. So one

1:12:25of the things we've been testing is

1:12:27essentially making individual team

1:12:29members like the CFO CFO whatever tools

1:12:32of one of the agents and when it needs

1:12:34something that isn't in some other

1:12:36system only the CFO would have it

1:12:38literally messages uh the CFO like the

1:12:41actual tool is just a slack but the the

1:12:44CFO is described as a tool. So we're

1:12:46essentially making like the human the

1:12:48tool the agent rather than the other way

1:12:49around. Um, it's early days in terms of

1:12:52how we're testing. We're a little hacky

1:12:53with it, but I'm curious if you're

1:12:55seeing how how are you seeing people

1:12:57fill the gap of things that only the

1:12:59humans would have while giving that back

1:13:02to the agent.

1:13:03Yeah. And I think voice agents is a good

1:13:05example where like you could have a tool

1:13:08and and I had it integrated with Bench

1:13:09where like it makes an outbound phone

1:13:11call and finds out some information and

1:13:14then brings it back, right? So you can

1:13:15you can have those scenarios. Um, you

1:13:18may want two-way communication to avoid

1:13:20just like hanging around for a long

1:13:22time. Um, so you could have, you know,

1:13:24your agent be both a client and a server

1:13:27and maybe it gets called with like, you

1:13:28know, a task ID and it's like, hey, I

1:13:30got the response.

1:13:31Yeah. Yeah. We've been doing like a like

1:13:33a node essentially. We use likely

1:13:37and we've been using their

1:13:39weight.

1:13:42Yeah. hesitations on it.

1:13:44I I believe with sampling you could hack

1:13:47that together. So So sampling can take

1:13:49user input as well as LLM uh responses.

1:13:54It's also interesting the spec too is

1:13:56evolving like I follow the spec pretty

1:13:58closely and they have that elicitation

1:14:01is a new feature that they're adding

1:14:03where you can get input from the user.

1:14:05Is it architected that way where it's

1:14:07essentially like it functions like a

1:14:09tool like that's how you think of it

1:14:10from an architect? It's a new kind of

1:14:12protocol message where it sends it back

1:14:14from the server to the client and it

1:14:16asks for information from the user

1:14:18and then it continues after that.

1:14:21Yeah, I feel like that opens up the

1:14:23scope of like what the agent could do if

1:14:26you have a clear way for it to get

1:14:28information

1:14:31or

1:14:33Yeah. And then the the CFO is gonna have

1:14:35his own agent respond.

1:14:40Yeah. back.

1:14:46Yeah, very very difficultly.

1:14:49Um I I have a set of prompts that I use

1:14:52and kind of monitor, you know, how the

1:14:54context grows like when did we when did

1:14:57we move the cache marker, how much did

1:14:59it cost, you know, what was the context

1:15:01per tool. Um you know, definitely adding

1:15:04MCP servers willy-nilly is going to like

1:15:07bloat your context. Um, so we're coming

1:15:09up with ways to basically allow people

1:15:11to add MCB servers and then like hide

1:15:15that from the actual uh system.

1:15:20Also, when you have like agent to agent

1:15:22communications, right? So, let's say

1:15:23agent A calls agent B and agent B calls

1:15:26agent A.

1:15:27How can you make sure this uh recussion

1:15:29like when does it stop?

1:15:32Yeah, you you can have like a max turn,

1:15:34right, where you just kind of jump out

1:15:36of it. Um, like when I had the LLMs

1:15:39talking to each other, I just told them

1:15:40like take 50 turns. Um, you know, and it

1:15:44was funny as I was building that tool, I

1:15:47wanted to like talk to the claude for

1:15:49that thought it was conscious.

1:15:50So, I added a feature where I could just

1:15:52chat to it at that point in its

1:15:54conversation and but then the context

1:15:56kept like getting rate limited. So, then

1:15:58I was like, "Oh I'm going to have

1:15:59to implement, you know, prompt caching,

1:16:02uh, pruning." So then I added like 23

1:16:04tools to the agent just to continue the

1:16:06conversation. I gave it like memory and

1:16:08all these other things and like it kind

1:16:10of funny how you start out with just I

1:16:13just want to have a long conversation

1:16:14and then you end up with 23 tools.

1:16:17Yeah.

1:16:18Just following up one question like when

1:16:20testing because you are using lot of

1:16:23external tools like lab or salesforce

1:16:25etc as your MCP servers but then you are

1:16:28writing on the real world let's say.

1:16:30Say say again. So you are basically

1:16:32creating a message in Slack or like

1:16:35writing something on Salesforce,

1:16:36creating an entry or etc. So but how do

1:16:39you test those systems like do you mock

1:16:40everything every tool or do you do

1:16:43something else?

1:16:44We we use demo accounts in like

1:16:46Salesforce, we have a sample data,

1:16:48Slack, uh we have a few agents that

1:16:50actually will go in and and just post

1:16:53like conversations. Um, and then there's

1:16:55like a a bench support user that will

1:16:58respond to those fake customers and then

1:17:00we can we can just test uh on synthetic

1:17:03data like that.

1:17:04So for every tool you will have a

1:17:05synthetic.

1:17:06Yeah. Yeah. You you can test in your

1:17:08production account but you can't really

1:17:10demo in your production account.

1:17:12Yeah.

1:17:13Yeah. So when you adopt agent to agent

1:17:16system, do you see an increase in the

1:17:19complexity of the task you can achieve

1:17:21but a decrease in the consistency of the

1:17:23performance?

1:17:24It it's kind of hard to quantify but I I

1:17:28don't know if A2A is is ready yet. Uh at

1:17:33least at least not for my use case. You

1:17:35know may maybe Salesforce can provide

1:17:38much better tools than like an SQL query

1:17:42MCP tool.

1:17:44Yeah. And and they they can just do a

1:17:46lot more than you can ever do in your

1:17:49code, right? Because you're you're only

1:17:50ever able to access, you know, certain

1:17:53things and and do certain uh calls and

1:17:56like if if if a third party can build a

1:17:59better uh system um that's opaque um

1:18:03then that might you know improve

1:18:05performance. Um I I think like

1:18:08fundamentally it always comes down to

1:18:10like indexing data. Um, so like you know

1:18:13the more data you need to process to get

1:18:15the business value out of it and the

1:18:17harder it's going to be to actually do

1:18:19that through MCP or A2A.

1:18:22Yeah.

1:18:25Yeah.

1:18:26So some of these interactions right this

1:18:28can be done through REST API right

1:18:30instead of

1:18:34what is the difference?

1:18:36Yeah and it kind of goes back to uh one

1:18:38of the earlier slides. um yeah when not

1:18:43to use A3A or MCP and it's it's if you

1:18:46have full control of of the things that

1:18:50you're doing right so like you know if

1:18:52if you are a Salesforce

1:18:55um you know and you're building your own

1:18:56internal Salesforce agent like do you

1:19:00need to use an MCP server or A2A no

1:19:03right you're you're you're actually able

1:19:05to run your own local functions that

1:19:07maybe access the database directly right

1:19:09um So like if if you're building

1:19:11something you know where you need file

1:19:13system access um you know do you need to

1:19:16use an MCP uh you know server running

1:19:20locally or do you just write some code

1:19:22that accesses the file system right

1:19:26I think the main difference is like in

1:19:29terms of how do you maintain your state

1:19:31right like MCP start up in a stateful

1:19:35resting

1:19:37your context magic it is really crucial

1:19:41to have MC

1:19:43whereas rest API you can't do that.

1:19:46Yeah. So like a lot of the time when you

1:19:49use a REST API you're going to be like

1:19:51querying like making a lot of calls to

1:19:54to build up the thing that you want to

1:19:56ask the question on. Right? So if it's

1:19:58like hey look at every Slack message in

1:20:00in this channel like it's not just going

1:20:02to be like one API call right just

1:20:05pageionation. You're going to have to

1:20:06pull it all into memory. then you're

1:20:08going to have to run it through an LLM,

1:20:09right? So there's there's still state uh

1:20:12in your application that's leveraging

1:20:14those REST APIs.

1:20:16Yeah,

1:20:17I'm curious about the task concept. Uh

1:20:19is that actually is that kind of LLM

1:20:22defined or do you have code for that? Is

1:20:24it more of a system thing?

1:20:26Which task context?

1:20:27Um so at least in the flow diagram you

1:20:30have

1:20:30Oh, is this in the repo? Does it

1:20:34so from CLI interface it says it sends a

1:20:36task to host a curious is that a proper

1:20:39task or is it just you know just what

1:20:40you call what what sends to it.

1:20:42Yeah. Yeah. It's just saying hey you

1:20:44know process this web hook as a task

1:20:46right

1:20:46have you explored anything where you're

1:20:48actually tracking a proper task and

1:20:50you're assigning tasks to agents and you

1:20:52have basically like you know like a

1:20:53planner where you basically have task a

1:20:56one two three is on this agent and so on

1:20:59and then in relation to the the question

1:21:01about human in the loop you could have

1:21:03task assigned to humans as well right

1:21:05both

1:21:06humans and agents

1:21:07yeah so uh we're looking at uh directed

1:21:10a cyclic grass right so dags um as a

1:21:13part of of bench sub agent tasks, right?

1:21:16So, you know, you you need to have some

1:21:18sort of flow control, right? You know, I

1:21:21need five things done and then when

1:21:23that's done, I need to do one thing with

1:21:24it, but then I need to send that thing

1:21:26to five other things, right? So, you

1:21:27kind of have fan out, fan in uh style

1:21:30stuff. Um it's very similar to like

1:21:32CI/CD pipelines where, you know, you

1:21:34might want to lint in parallel and test

1:21:36in parallel, but you know, you're you're

1:21:38building in serial, right? Uh yeah.

1:21:42So I was looking at code base and you

1:21:44have this defined like a GitHub MCP

1:21:47server and uh in a separate file under

1:21:50the GitHub agent you have also the

1:21:52genkit.ts

1:21:53where you are wrapping the MCP in

1:21:55another function call why is that like

1:21:58can't the MCP just interpolate with our

1:22:01A2A like why do we have to make rappers

1:22:04on top of

1:22:04that's a great question and and I think

1:22:07that's the fundamental question of A2A

1:22:09is like they they launched and they said

1:22:11oh yeah full MCP support you'll be hard

1:22:14pushed to find a single example online

1:22:17maybe maybe this is the only repo that

1:22:19actually has an example of A2A and MCP

1:22:21working together. Um, and it took a lot

1:22:24of work and actually I ended up uh

1:22:26having to use something called

1:22:30uh where is it? Genkit XMCP.

1:22:33That was the only way I could get it to

1:22:35work.

1:22:36Um, so yeah, they they don't really have

1:22:39like proper support yet. Uh, it was I I

1:22:43think if they had this this would have

1:22:45been a lot easier to build. Um, but

1:22:47yeah, hopefully in time.

1:22:51All righty, I think we're we're at time.

1:22:53Uh, thanks everybody for joining. Uh,

1:22:55hope you enjoyed it. Great conversation

1:22:57at the end. And yeah, definitely uh try

1:22:59out Bench, hit me up on uh LinkedIn. I

1:23:02would love feedback uh before we go

1:23:04live.

1:23:06Thanks.

1:23:11[Music]