Building Agents with Amazon Nova Act and MCP - Du'An Lightfoot, Amazon (Full Workshop)

0:04[Music]

0:15Building agents with Amazon Nova ACT and

0:19MCP.

0:23I'm excited today because we're going to

0:25build intelligent autonomous AI systems

0:29that can help you build,

0:32scale, and improve your applications and

0:36business. My name is Dewan Lightoot and

0:39I'm joined by

0:45swap. Swap. Yeah,

0:51there we go. There you go. My name is

0:54Dwan Lifford and I'm joined by Hey, I'm

0:56Banjo Bami. I'm a solic architect here

0:58at AWS.

1:01Now, this is the AI engineer worldfare

1:05and I've been in tech over 15 years and

1:08right now is the most exciting time for

1:12me in my entire career. And one of the

1:16reasons for this excitement is agents.

1:19How many of you right now are been

1:21building agentic systems?

1:25I love it. So when we talk about agentic

1:29AI, I think it's important that we level

1:32set from an AWS perspective.

1:35There are three key terms we need to

1:37think about. First, the ability to plan.

1:42A agent gets a prompt. It gets an

1:45objective and it determines the actions

1:48that need to be taken. So it creates the

1:50plan and then it takes actions on those

1:53actions by using things like tools. Now

1:57the last piece, the third piece and

2:00probably the most interesting is the

2:02reasoning where the agent is able to

2:05evaluate the results and determine if it

2:10needs to update the plan and take

2:12additional actions until the objective

2:14is complete.

2:16This is an agent.

2:18Now when we actually break down the

2:21architecture,

2:23I think it's important to take a look at

2:24this because we have the user input, we

2:27have the agentic system, we have the

2:30possibility of some type of human in the

2:32loop and then we have the generator

2:35response. Now when we dive a little

2:38deeper, there's some components of this

2:40agentic system. We have the LLM.

2:43We have a knowledge base with external

2:46information that we want may want to

2:48provide. We have guardrails to say to

2:51the model don't do this or to ground the

2:54model with the truth from our knowledge

2:56base to say okay is this actual relevant

2:58information. Is this accurate to the

3:01information we're receiving from the

3:02knowledge base? And then we have access

3:04to additional tools, memory, or we may

3:07need to talk to additional agents or

3:09LLMs like Amazon Nova ACT through

3:12something like MCP.

3:14And we have the ability to design our

3:16own flows for these systems.

3:18Now, the most interesting piece that I

3:22think a lot of us are probably focused

3:24on when we're building these systems is

3:26around the continuous evaluation

3:28framework. Like how do we know if we're

3:30using the right LLM? How do we know if

3:33our prompt is consistent, accurate, or

3:36even optimized for the performance we're

3:38expecting? And then how do we even judge

3:41our system? How do we rate that and

3:43determine that that it's actually

3:44solving the problems that we need or

3:45intend?

3:47Now, once we have this, we need to log

3:49this information and then have some type

3:51of subject matter expert and determine

3:54how can we improve this system. And this

3:56is the iterative approach. So we're

3:58always trying to improve and optimize

4:00our agentic system.

4:04Now continuing on this continuing on

4:06with this story. Now there are some use

4:08cases that we should be building these

4:09systems for. Like if it's complex task

4:12and we don't know which tools should be

4:14used, how many tools should be used and

4:16we want the model to leverage his reason

4:18and capabilities. Well, this is a great

4:20use case for a gentic system. But if

4:24it's something that is just one step,

4:26our traditional if this then that

4:28approach is probably the best solution,

4:31right? We don't always need to provide

4:33some type of agentic system for

4:34something that can be done with a

4:36traditional solution.

4:40Now when we talk about agents on AWS,

4:43there are three approaches and

4:45perspectives we should think about.

4:47First is going to be to specialize using

4:49something like Amazon Q. How many of you

4:51are have used Amazon Q?

4:54The there's Amazon Q in the console to

4:57help solve your problems on AWS. In the

4:59console, there's Amazon Q developer

5:01inside of your IDE. And right now, one

5:04that I'm I think most excited about is

5:06Amazon QLI agent. How many of you have

5:09used that?

5:13For me, if you if you are into

5:15increasing your productivity, using a CI

5:18a CLI agent has helped me tremendously.

5:22You from editing a video, it can do

5:25that. Summarizing a document, reading my

5:28entire codebase. Like today for one of

5:31my demos, I had some code and I was

5:33trying to figure out why wasn't it

5:35working. I said, "Analyze this code and

5:36tell me what you see. Let me know the

5:39APIs that it's calling." Well, I looked

5:41up the APIs. Well, it didn't match my

5:43APIs in the API gateway. So, when the

5:45code was deployed, it wasn't deployed

5:46with the right APIs. So, the agent was

5:48able to help me save a ton of time by

5:51just analyzing the code and tell me what

5:53it saw because I never seen the code

5:54before, right? So, that's what these

5:56tools are able to um to help us do. The

5:58next is fully managed. If you're using

6:00Amazon Bedrock, you're able to leverage

6:02Amazon Bedrock agents to build and

6:05manage agents inside of AWS. And today

6:08what we're going to be focusing on is

6:09the DIY the do-it-yourself approach by

6:12using strands agents. This is allows you

6:15to not just leverage Amazon bedrock but

6:18also leverage models through other

6:22providers using light lm.

6:25Now when we talk about strands agents

6:27strands agent was announced about a

6:29month ago. I want to say something about

6:31a month ago. This is open source

6:33extremely lightweight. So if you use

6:35other agent FL frameworks, it's like

6:38that. But the implementation is you'll

6:41see in the code how easy it is to build

6:44a aentic system or a agent itself in a

6:47few lines of code and already get

6:49started. I built a multi- aent solution

6:51in about under 50 lines of code.

6:55And so when we break down strands

6:57agents, there are three components. We

6:59have a prompt, we have a LLM, and we

7:01have tools. So you create a function

7:04called let's say a get weather tool

7:07right you define your agent you give it

7:10a prompt and it's already implemented

7:13and you'll see in the code as Banjo goes

7:15through it here in a moment

7:19now taking it a step further as Danielle

7:22presented today on Amazon Nova Act these

7:25models are able to do some really cool

7:27things and this is another thing that

7:29I'm excited about Amazon Nova Act is a

7:32research preview model. And

7:36the capabilities of this allows you to

7:40use a prompt or give instructions and

7:43take complex tasks and do things like

7:45browse the internet to find research or

7:48to research or to search on Amazon.com

7:52to find the top list of widgets, right?

7:55And then return them and then add them

7:56to your cart. So you'll see how we can

7:58leverage this not just using the SDK for

8:01Amazon Nova ACT but also by leveraging

8:04MCP which leads us into the last piece

8:07which I think when we talking about

8:10agents I don't think we would be here

8:12today as fast as we have moved if it

8:15wasn't for MCP. How many of you are

8:18leveraging MCP today? Modern contest

8:20protocol. How many of you have built

8:22your own MCP servers?

8:25I built several um I got two that I use

8:28all the time. One, how many of you use

8:31Obsidian?

8:33Okay, so for my documentation, I built

8:35the Obsidian MCP server. This allows me

8:38to save all my documents, reference all

8:42my documents, and just my entire

8:44workflow is streamlined because of this

8:46MCP server I use right there. But I also

8:50use one for my bookmarks. I built the

8:52bookmark manager because every Friday

8:54I'm restarting my computer and I lose my

8:56bookmarks. I save them and I forget

8:58about them. But now I can just say save

9:00this bookmark. It gives it a

9:01description, gives it a title, give it a

9:03date and I can even add notes so I can

9:05remember where this bookmark. So now

9:07when I open up QCLI, I can say hey I'm

9:09looking on the top. I'm looking for um

9:11some information on MCP. Can you tell me

9:13all the bookmarks that I have? Then

9:15it'll find it. Can you tell me the ones

9:16I saved last week? And so these this is

9:18the power that we have today. But with

9:21that being said, I think it's time that

9:23we all start building. Banjo's going to

9:26take over. But if you you open your

9:29laptops and log on to this link, this is

9:32going to take you to a workshop

9:33environment where you have access to an

9:35Amazon account where Banjo is going to

9:38walk you through building out today's

9:40workshop. I thank you for your time.

9:42Cool. All right. So, uh, this is going

9:45to be a hands-on workshop. So, we've

9:47provisioned an AWS account for everybody

9:49here. So you don't have to install

9:50anything on your computer. Everything is

9:52going to be done through the browser.

9:54And I always say the hardest part of the

9:56workshop is just getting started. So

9:58some of my colleagues are also here. So

9:59raise your hand AWS folks that are here

10:01to support. Uh so we're going to take

10:03some time to just get logged into an

10:05environment. We're going to set up a VS

10:06Code server, enable models, uh get the

10:08Nova Act API key. So again, this is the

10:11hardest part of the workshop, just

10:12getting started. Uh so let's take some

10:14time to just get into the environment

10:16and I'll follow along as well.

10:19So, and this is uh again everything is

10:22you don't have to install anything on

10:23your computer. You don't have to use

10:24your own AWS account. Everything is

10:25provisioned for you. So, but while

10:27that's loading, let me briefly walk

10:29through the three modules of the

10:30workshop. Uh so, the workshop is really

10:32about how you can use Nova Act. So, the

10:34first module is just getting started

10:36with Nova Act. We're going to make an

10:37API call that the second part of the

10:40module is going to make an MCP server

10:41that can leverage a Nova Act. And then

10:44finally, we're going to use the strands

10:45agent to hook everything together. So

10:47that's kind of the the three steps

10:48you'll go through at this workshop and

10:50all the code is available via the link

10:52on GitHub so you can try it out on your

10:54own as well. Uh but yeah trying to get

10:57started here if you can't follow along

10:59I'm going to be doing up here so don't

11:00worry too much and again all the code is

11:02available so you can try it outline uh

11:04offline.

11:06Okay, so the first things first, uh, if

11:09you're following along, make sure to

11:10click this open AWS console button.

11:13Again, we provision the AWS account. You

11:15know, don't log into your own AWS

11:17account. Don't try to create a new one.

11:19Everything is uh previsioned here

11:20already. So, going to click clicking

11:23that button to open up your AWS account.

11:26So, logged into my AWS account.

11:35So the first thing we do in an AWS

11:36account, we're going to enable uh Amazon

11:38Bedrock models. So Amazon Bedrock think

11:41of as a serverless API to access

11:43different foundation models and you can

11:45build lots of gen applications in it. So

11:47it has capabilities like knowledge

11:49bases, guard rails, you can build agents

11:51on top of it for anything you need to

11:53build uh AI agents or geniary

11:56applications. Amazon Bedrock has

11:58capabilities for that. But for this

12:00workshop, we're just going to enable

12:01specific models. So going to enable

12:05specific models. You can click the

12:06Amazon models and we used the Claude 37,

12:103.5 IQ, and 3.5 sonnet. So those are the

12:14ones we're going to use for this

12:15workshop.

12:18And

12:19that's going to request access there.

12:27And again, all the instructions are also

12:28in this workshop as well. Uh, so we

12:31could follow along, but I'm just going

12:32to go through it just for sake of time.

12:35And then the next part once we get the

12:36model access, there's a VS Code server

12:38that has everything set up already. So

12:40I'm just going to go in there

12:47and if the URL and password is there,

12:49you can log in to your VS Code server

12:51with everything installed.

13:06and I'm also going to log into Amazon Q.

13:08So, Amazon Q is our U ID extends in to

13:11help you write code.

13:13U we have time you can sign up through a

13:16builder ID completely free. You don't

13:18need uh AWS account. You don't need to

13:20put in your credit card. You can just uh

13:22log in through there. I already have an

13:24account so just feed it up but it puts

13:26uh nice little AI agent there. I can ask

13:30questions, update code, etc. So it's uh

13:32I'll show you some examples of go

13:34through some of the code. So

13:39So who's gotten to this point setting up

13:41all the models workshops because this is

13:43once you get all this done then that's

13:45when the real fun begins. So does

13:46getting a pulse if I need to slow down

13:49or

13:50slow down a bit. Okay, I'll wait a bit

13:52again. raise your hand if you're stuck

13:53anywhere. Questions, we have uh agents

13:56that can come around and support you.

13:58So, I'm going to pause for a little bit.

14:04Any general questions while we're

14:06waiting?

14:09Oh, yeah. So, this workshop uh again all

14:11the code is available online. Uh this

14:14workshop available as well. So, you can

14:16also look through that. There's a

14:18website called workshops.awws. AWS

14:22and when you go there you can do

14:24something like uh Nova act and then it's

14:28the only worktop that shows up. So you

14:30can always go to workshops.Waww

14:32search nova act and this workshop will

14:34show up so you can see all the

14:35instructions all the code and run this

14:38uh on your own.

14:54Okay. And then the last thing, uh,

14:57because we're going to use Nova Act, we

14:58actually need to get a Novaact API key.

15:01So, if you go to Nova, Amazon.com,

15:04it this is a website that you can use

15:06the Amazon Nova model. So, you can do

15:08like chatting, generating images, uh,

15:10speaking with Nova, uh, generate videos,

15:13but then also this is where the the ACT

15:15API key is generated. So, if you're

15:19following along and you want to generate

15:20your key, again, it's free to log in.

15:23You can use your Amazon.com uh like when

15:25you order something on Amazon.com

15:27account to log into this and then you

15:29can just generate a key here and it'll

15:31be able to access that

15:49Oops. Okay. So, I'm going to walk

15:52through what module one is. Uh before I

15:55get has anybody got in here? Just quick

15:57pulse check. If not, you know, I'll

15:59continue. I know the Wi-Fi is slow, so

16:01it might be hard. So, I'll just continue

16:03on. Uh but yeah, the first one we got to

16:06see how Nova Act works. Uh how how the

16:09actual code looks like. Uh generated the

16:11key. uh need to export the key and then

16:14kind of running the first script which

16:15is actually going to open amazon.com uh

16:19and we're actually going to look for the

16:21first coffee maker. So let me see how

16:23that uh code looks like. Let's go here.

16:27Make this bigger.

16:29Oops.

16:36So very simple code uh with Nova Act

16:38it's again it's all on Python SDK I so I

16:42decide what a page to go to. So go to

16:44amazon.com I say I want you to search

16:46for a coffee maker. I say select the

16:49first result and I say get the title of

16:51that product page. So uh very simple if

16:53you ever done kind of web automation

16:55before of something like uh Selenium or

16:57Playright you probably have to like look

16:59for this diff tag you know look at this

17:01H1 tag grab this information. a lot of

17:03manual processes of actually inspecting

17:06the actual website. Here I'm just saying

17:08click the search bar, find something

17:10like I don't have to specify click this

17:12tag, do that. So it makes it much more

17:14easier to engage with the website as a

17:16natural human would instead of like

17:17looking through divs and trying to find

17:19this P tag specifically. So uh this is a

17:22great way to just uh you know use Nova

17:24Act right out of the box. So I'm going

17:27to uh run this

17:30so we can see

17:36examples.

17:38All right. All right. So, added my key.

17:41Going to what happens when I run this

17:43file.

17:59Give it a second.

18:04Oops, we failed. All right, let's start

18:06over.

18:10One.

18:16Ah, okay. I know.

18:23Got to run out of this. Let's start that

18:25over.

18:36Yeah, question.

18:38So just to explain why is Banjo running

18:40that command? It's running F XVFB. It's

18:43a frame buffer where it runs your X11

18:46system. What happens there? Nova act

18:49actually goes and clicks a mouse on a

18:51browser. That's why it needs to be run

18:52like that. Otherwise, it has no guey. So

18:54this is just kind of a way to emulate um

18:56a graphical user interface on this Linux

18:59box. Thank you, Darko. Yeah, since we're

19:01running everything in the the cloud on a

19:03browser, I'm saying, you know, open a

19:04browser again, but it's already in a

19:06browser. So that's why it crashed. So I

19:07had to put that frame buffer command. Uh

19:10and yeah, the workshop kind of walks

19:12through why we did that. But you can see

19:14uh what is going on when Nova act says,

19:17I'm going to search for a coffee maker.

19:18I'm at the Amazon homepage. My task is

19:20to search for this. So it's

19:21understanding what it's doing. I see the

19:23search spark has copy maker. I'm at the

19:26search spark here and now it actually

19:28puts the actual log of the actual HTML

19:31file. So it's taking screenshots. You

19:32can see what it looks like. It got the

19:34first results. I'm on the copy maker

19:36page. It selected it. And now I got the

19:39title of the now it says, you know,

19:41what's the title of this product page?

19:47All right. Got this Black Decker 12 copy

19:50maker. my task to return the title of

19:51the product page product title. It got

19:54that and it ended the session and then

19:56it also creates a a video log that I can

19:58actually look at to see what it did for

20:00each for everything it did in this webm

20:03file.

20:05Yeah. Question.

20:17Yeah.

20:22So the question was does it reason about

20:26the page in terms of pixels or in terms

20:29of text? Yeah. So it's actually looking

20:32through the actual uh the page itself.

20:35So you see in this video it sees it

20:36looks at the page. I can see what's in

20:38the page. So it's it's a large language

20:40model train. So it can actually see the

20:42actual the page is doing. So it's not

20:44looking at like like the H1 tag or

20:46whatnot. It understands the context of

20:48that particular page. It can see that's

20:50a search box. Okay, I'm going to go

20:52click through that search box. So yes,

20:53it understands the pixel level of what's

20:55on that actual page.

20:58So this is kind of the video. It's hard

21:00to see. Make it bigger.

21:03Sped up.

21:06So, it opens the page. It goes to the is

21:09able to type in coffee maker there. Um,

21:12it gets that information, clicks the

21:14button. So, even with all the ads and

21:16everything, the video can understand the

21:17task, clicks that, and it gets the

21:19information back. So, that's and that

21:21was a couple lines of code. So, you can

21:23extrapolate to other type of workflows

21:25you can do for searching through things.

21:28Sorry, I have a and I have a question.

21:30Yeah. So when you what I have

21:32experienced with these kind of

21:33frameworks is that when you run this on

21:34a server environment um services like

21:37Cloudflare will block the access and

21:39maybe do a capture challenge. How do we

21:42solve that using Q? Yeah. So with uh so

21:45using Amazon Nova so it doesn't do

21:47captures. It doesn't do things of that

21:48nature. So it's it's meant for like

21:50workflows you understand but yes it's

21:51not going to bypass captures and other

21:53things of that nature as well. So it's

21:55made for like going to amazon.com or

21:57look through a booking site. But if

21:58something that's like requires like a

22:00human or it wants to you can't bypass

22:02that

22:06you you wouldn't use Nova Act for that

22:08use case. If you need to pass a capture

22:10of something else that use another

22:11technology this is not meant to like

22:13overtake humans, you know, it's more

22:14like I'm helping them augment things but

22:16not if there's a capture involved then

22:18you have to use a different technology

22:19for that. It's also a preview. It's also

22:22a preview. Yes. So this is a research

22:24preview as well. So if that's a very

22:26good use case, you know, leave feedback

22:28on the Nova the website. So is is human

22:31and loop possible at all with it yet?

22:35Well, with this this one, it's no

22:37because I'm writing all the code here.

22:39So but again, this is Python code. So I

22:41could probably put in something here

22:42like, you know, ask something, make an

22:44API call here. So this is, you know,

22:46it's a Python code. you might be able to

22:48create some type of uh workflow that

22:50might augment like wait for a human

22:52response or whatnot because the browser

22:54is happening in like headless mode but

22:56could you make it work with a browser to

22:58human is also seeing at the same time?

23:00Yes. So it can pause and wait for

23:02somebody put in like a password or

23:03credentials or do a capture and then

23:05once it receives that works continue on

23:07the workflow. You could do yeah because

23:09right now I I ran it in headless mode

23:10but yes it can also run uh you know it

23:12would open up the browser. If I ran this

23:14on my MacBook, I would open up a Chrome

23:16browser and go through that session.

23:18Also, if you're running it and you

23:20wanted to bypass something that has two

23:22factor, if you're already logged into

23:25say Amazon.com and then you run a code,

23:28it's going to use your credentials in

23:29that browser session to continue on to

23:32perform that task. So, that's something

23:33that you can do as well.

23:36Cool.

23:38So, let me Oops.

23:42And then one other thing you can also do

23:44multi- uh you know parallel execution.

23:47So my last my next example is actually

23:49I'm trying to find multiple monitors and

23:51I wanted to compare them all at once. So

23:53I'll show you how that code looks like

24:04so I can check for the monitor extract

24:06information. I'm setting you know I want

24:09I'm defining what I want. So again, I'm,

24:11you know, saying I want to find the

24:12price, the rating, the size. Uh, go to

24:15amazon.com.

24:17Uh, I set it headless mode this time. So

24:19I don't need to do the frame buffer. I

24:21start multiple threads. It looks for

24:23each monitor simultaneously because each

24:25of these are individual tasks. So I can

24:27paralyze them instead of waiting it to

24:28to go through. I I define the list of

24:31monitors I want to go through, start the

24:33thread, and then it starts executing and

24:35finds the the results of the monitors.

24:37So I can run that in the background.

24:44So starting the three parallel threads

24:47and it's so again it's running in

24:48headless mode so it's going to be able

24:50to do this in the background where we

24:51can see kind of what the model is

24:53thinking how it navigates through the

24:54web page.

24:56Yep.

24:58I I have tried to use Nova uh in the

25:01past April and uh it worked for the

25:04first time but once I did it again it

25:07triggered the capture. Is this something

25:09that has been already resolved or is

25:11this happening because I think the

25:13website and it was Amazon in this case

25:15it was detecting it was a bot and uh is

25:18there like an LLM.txt

25:20or robots.txt txt that can declare it.

25:23So Nova actor is a GitHub repo. So you

25:25could go there and just grab that. But

25:27it's it's working now. Like I'm running

25:29it, you know, I just I this is just live

25:31code I'm doing right now. Like uh I just

25:34exported my API key, started running it.

25:36So uh you can try it in the workshop.

25:39Yeah, I mean it's it's ready to go.

25:40We're building right now.

25:44And you can kind of see that it's going

25:45on in the background what it's doing. Uh

25:47I've looked at this monitor, the Dell

25:49monitor. I'm at the Amazon homepage.

25:51It's like it's going through looking

25:53through the search results. It's saving

25:54things. So, you can see it's running in

25:56parallel. It got the information for the

25:58one of the first ones. So, it's going as

26:00I just set that up and it can execute

26:02that. So, if you have some type of uh I

26:05don't know like daily news thing. You

26:06need to go to the website and get news

26:08or something and like have a report Nova

26:10and there's no API for that. This is one

26:12way you can encodify how to do that kind

26:14of search and get the information.

26:17Have a question. Yeah. I'm wondering so

26:20how successful is this in terms of like

26:22more ambiguous test because I I ran the

26:24Amazon demo and that worked but I'm

26:25wondering could I just add Google there

26:28and and how like how vague and sort of

26:31how much does it know when it's

26:33navigating through like I was thinking

26:35like if I wanted to return a pair of

26:37sunglasses that broke would I would I be

26:39able to just say like start in Google

26:41and then find this company's website

26:43find a way to you know engage support

26:46open a ticket like how much sort of like

26:50how vague can you be and how smart is it

26:52currently would that would that yeah I

26:55mean the more instruction you give

26:56obviously the better but it's able to

26:57understand how to navigate a website

26:59that's what the model's trained on so if

27:00you say you know go to this sunglasses

27:02website it doesn't it probably wasn't

27:03trained on a specific sunglass website

27:05but it can understand that button

27:07support you know this but click a ticket

27:09so understand kind of the the general

27:11knowledge of how to navigate the website

27:12but if there's something very intricate

27:14about that website you're going to have

27:15to encode it in the text like make sure

27:17you click button an experts or whatever.

27:19So, it understands how to navigate

27:21websites. Got it. And does it understand

27:23when it's failed? Yeah, sometime

27:25sometimes I've seen it sometimes get

27:27stuck in a loop and like, oh no, I keep

27:29scrolling. I keep scrolling. I keep

27:30scrolling. It doesn't know when to stop.

27:32So, it again, this is in research

27:33preview, so things are getting better.

27:35The model's getting updated behind the

27:36scenes, but it's not like it's not AGI.

27:39So, that's got it. And one last

27:41question. Um, h how is it in terms of

27:44navigating like distrustful parts of the

27:46internet? I mean there's a lot on the

27:47internet that we see and we know is not

27:49to be trusted or it's something not to

27:51be followed. H how have you sort of

27:53worked around that problem? Yeah because

27:55again it is a model in the background so

27:56it's going to understand like if you're

27:57doing something it's not going to want

27:59to click that or might be there

28:01safeguards in place. So that's built

28:03into the model but again uh it is in

28:06research preview. You still have to

28:07explicitly say what buttons to press for

28:09certain actions but again the model it

28:11isn't LLM trained. is going to be able

28:13to understand the nuances and say if it

28:15can't take this action or can't do that

28:17that that could happen. But I haven't

28:19seen that use case. But if you keep

28:20pushing it, maybe you'll find those

28:21those things. Well, the thing I had in

28:23my mind is like if you go to a site

28:25where you have to download a link,

28:27sometimes there's an ad that says

28:28download a link and you know that that's

28:30just an ad trying to get your attention

28:32would the model know or is that some is

28:34that like for example like in the the

28:37Amazon.com it shows an ad for something

28:39but it said find the first thing. I was

28:40able to scroll past that ad and click

28:41something. So the model understands the

28:43task you give it. So yes, it can

28:45understand that. Thank you.

28:49All right. So this this just finished.

28:51Yeah, that's really quick. It showed it

28:53was able to find all the models, give me

28:54the size, the rating, the price for each

28:56of the monitors. So again, it it

28:58executed that on parallel. It got me the

29:00nice information. And that that's kind

29:02of the idea of like it can do parallel

29:04execution in the background. So you

29:05don't have to wait for it and don't see

29:07it actually clicking through the the

29:08task and you get your information.

29:11All right, one more question then we'll

29:12move on to the MCP part. So Nova is

29:16specifically meant to be used with a

29:18browser. Correct. Uh so Nova act. So

29:22Amazon Nova is a family of models on

29:24Amazon. So if you go to this website

29:26Nova Amazon.com

29:29uh you see there are different

29:30foundation models like Nova Pro,

29:32Premiere, Light Micro. These are like

29:34the text understanding models. So like

29:36your typical LLM calls. There's also an

29:39image model called Nova Canvas can

29:40generate images. There's a video real

29:43called Nova Real where can generate uh

29:45videos uh and then it's also a speech

29:47model text speech to speech called Nova

29:49Sonic. So Nova is a foundation of found

29:53uh foundation models by Amazon to do all

29:55these type of tasks and acts as just

29:57another one for browser automation. Are

29:59there plans to expand this like beyond

30:02the browser so that we can someday take

30:04actions in Slack or IDE or anything

30:09outside of the browser? Maybe some of

30:11the team is here so maybe talk with them

30:13later.

30:15Right. Thank you.

30:18All right. So I'm going to move on to

30:20the MCP part. Banjo. Yep. Nova Act is

30:24only available in US.

30:26Yes. Right now, Nova Act is only

30:28available in the US. It's in preview, so

30:30it's just getting started. So, if you

30:32log in from like a different uh account

30:35like address like UK or something, it

30:36might not it won't work. So, it only

30:38works in the US at the moment. Yes.

30:42Right. One more question over there and

30:44then I'm going to move on.

30:51Samsung

30:53might be Oh.

30:56[Laughter]

30:58Your Amazon.com is different. I don't

31:00know. Yeah, because it is opening up a

31:02different browser. So, it could have

31:03clicked something differently. Yeah. So,

31:07yeah, that's right. Look at the video

31:10preview video playback to see what your

31:12results were. Right. Yeah.

31:16Oh, one more. Okay. One more quick quick

31:19one. Are there plans to support

31:21persisting browsing data such as cookies

31:24in the cloud browser? So right now it's

31:27opening up its own uh browser, but you

31:29can also set like your own like Chromium

31:31profile and open up that browser. So

31:33everything you have saved there like if

31:35you want to log into your stuff, you can

31:36set your own custom browser, but by

31:38default it opens up a new like

31:40completely new browser without anything

31:41saved.

31:44All right. So I want to show uh I

31:47actually made an MCP server for Nova

31:49ACT. So a module tool is going through

31:52uh MCP and I can kind of show you what I

31:55did for the MCP server. Uh in fact we

31:57can use Amazon Q here. So I'm going to

32:00ask it uh

32:02can you tell me about the Nova

32:09ACT MCP server?

32:15Tell me what it does

32:19and oops.

32:23So tell me about the Nova act MCP

32:25server.

32:27So you can see it's going through um

32:30integrates Nova act browser and MCP. It

32:32has the browse session tool, browser

32:34action, execute parallel task, take

32:36screenshots, close browser, list

32:38results. So I created these different

32:40aspects of the MTCP server. So I can use

32:43something like claw desktop or cursor or

32:45Amazon QCLI to say you know open

32:47Amazon.com and find information for me.

32:49So it's it's portable. It understands uh

32:52so I don't have to actually write code.

32:54I can say so go to amazon.com and find

32:56me the co the first coffee maker. It

32:58will actually write all that code I did

32:59in the initial one to do that or the

33:02multimonitor. So I wrote wrote a bunch

33:04of code to do this. If I just said you

33:05know get me these three monitors to get

33:08the price. it would actually write all

33:09the Nova act code it needs to do that

33:11using the MCP server. So that's kind of

33:13the power of MCP that I just describe a

33:15task and then I can it will encode the

33:17actual browse action things it needs to.

33:20So and I also made an MCP client that

33:22can actually interpret that. So oops

33:26uh it connects to the MCP server it runs

33:29the code and it's able to use query

33:31bedrock. uh I am using a model so I'm

33:34using cloud 3.5 sonnet here because I'm

33:36is an MCP client and needs to have an LM

33:39behind that and then it's able to you

33:41know understand which tools to use uh

33:43run the code and open up the browser and

33:45whatnot. So let me just run the example

33:48here.

33:50So module two.

33:55So

33:57we are open the file. Just did that.

34:01We ask Amazon Q to explain the file to

34:03us. And now we're actually going to run

34:05it. So Python 3. And then I can open

34:09this up.

34:24Okay. So, let's be adventurous. So,

34:26somebody give me a query to try since

34:29anyone has an idea. Yeah, I'm going to

34:31just ask it and do something. So,

34:32someone give me an idea of what to run

34:34another act. Fix Wi-Fi.

34:39Uh, how would you fix the Can you find a

34:44website to find fix website? can find

34:48let's see website to fix Wi-Fi use

34:53headless mode

34:58I spelled it wrong but let's see

35:06all right goes to google.com how to fix

35:08Wi-Fi problems troubleshooting guide in

35:10the box and press enter return a list of

35:13the websites title descriptions

35:15All right,

35:17it's going through that. So, it open

35:19google.com. Uh, how to fix Wi-Fi

35:22problems. I see an empty search bar

35:24where I can type queries for search

35:25information. I should type how to fix

35:27Wi-Fi problem. So, you can see it's

35:28understanding what to do. It oh, it hit

35:31a recapture page. So, okay, the search

35:34results are not viewable. Blah blah. So,

35:38so see it looks like it got stuck on a

35:40recapture page. So, this is like a

35:41headless agent. So, someone asked a

35:42question about can I pass captures? What

35:44now? You see that it's it got stuck

35:45doing that. It looks like it's stuck in

35:48a loop now. So, it sees the capture

35:50again. So, I should skip the clip button

35:53to skip the capture window. The capture

35:55is still open. So, it it's probably

35:57going to be stuck here unless I close

35:58it. So, you can see there are

35:59limitations. It's not going to pass

36:01captures and whatnot. But that that was

36:03a good query to show that it Oh, did it

36:06fill it? It's still open. So, it's going

36:07to be stuck here. So, I'm just going to

36:08close it out. But you can see, you know,

36:10it it can't pass everything. and can't

36:12navigate through websites. So something

36:13like that was will not work. So that was

36:16a great test example to show.

36:20If I use the the bakedin one, you know,

36:22find that copy m under $50, it'll be

36:24able to go through that and use headless

36:25mode. But any questions on that? Seeing

36:28how the MCP server is working. I didn't

36:30have to write code. I just said do

36:32something. It actually wrote the code to

36:33to do it for me.

36:36Question over here.

36:41Yeah. Yeah. So, a question about if I

36:43can actually go into the browser and do

36:44it myself. Yeah. If I ran this locally

36:46on my machine, it'll actually be able to

36:48it'll open up the browser and I can

36:50actually click the button and it'll

36:51continue doing that. Right now, I'm

36:53running it within the browser. So, I'm

36:55de everything in headless mode. So, we

36:57can't interact with that.

37:02So, you can see it's able to find search

37:04under $50. It can actually look at the

37:06website. It's found search results on

37:09Amazon.com. So yeah, so that for that

37:11use case where we're not passing

37:12captures is able to continue and find

37:15the information there.

37:20So a question about can I actually order

37:22something? If I used my own browser

37:23session and like logged in at to my

37:26Amazon.com account and said yes, order

37:28this for me, you know, click through,

37:29it'll be able to understand that thing.

37:31But I would have to put in I would have

37:33to use my own browser sessions like I

37:35wouldn't want to log in by myself. Yeah.

37:44A question.

37:48If you give Nova act the authentication

37:50for Amazon for example like you give it

37:52your login details then can it log in

37:54and complete that action for you? Yeah,

37:56if I if I like say this is my username,

37:58this is my password, enter that into

38:00that field and you'll be able to

38:01understand, you know, this is a signin

38:03button and I have this information. But

38:05again, this is all Python code. So yeah,

38:07you can encode it, you can make it an

38:08environment variable so it won't read it

38:10directly. So a lot of ways to do that.

38:12Does it also like understand 2FA? Let's

38:13say it asks you to go to your Gmail and

38:16you will it then open the Gmail website,

38:19check the email if you're logged in

38:20again on your session and then input it

38:22or is it and you can well if there's no

38:24capture like we just thought of the

38:25capture. Yeah. So there's no nothing

38:27blocking. So but yeah again Nova Act is

38:30free to use. So there's a lot of

38:31creativity in this room. So I think we

38:33should have like a Nova Act hackathon. I

38:34think that would be you know do

38:36something crazy with Nova Act.

38:41All right. So, one more question. Yep.

38:44One more. Can I book a flight when my

38:47price alert is less than $100? It's like

38:49a continuously check. You could probably

38:52use something else for that. But yeah, I

38:54mean, nobody can open up that website.

38:55You can just have a query every day, you

38:57know, open Google flights and look at

38:59the quickest things and if something is

39:00below this threshold, you know, send me

39:02an email. So, again, this is all a

39:04Python script. So, you can set up

39:06something that triggers like once a day

39:08like in a lambda function and so yes,

39:10totally possible. So, Novax is very

39:12flexible and because it can run in

39:14headless mode, you don't need to have

39:15that UI. So, that's really what makes it

39:17helpful for interacting with websites

39:19that don't have a native API.

39:29Thanks. Yeah, this is pretty cool. I'm a

39:31little bit confused. Like, we have the

39:34Nova SDK SDK API key and we were also

39:38doing some stuff in bedrock. Ah, yeah.

39:40So how does this actually work? Yeah.

39:42Yeah. So in the the Nova API key

39:46separate but for this MCP client I did

39:49it actually needs a large thing with

39:50model to understand what's still

39:52happening. So if I go to claude oops I

39:57actually said I'm actually using claude

39:59sonnet 3.5 for my MCP server. So that's

40:02how because I just asked it you know

40:04find that website for me. How how does

40:06it know that without any of the code

40:08doing that? So it's using a large

40:10language model underneath the hood to

40:11actually find that information. So

40:13that's where we use bedrock for trying

40:16to find it in the code but

40:19sonnet. Yeah, I set the model ID. So

40:22you're an AI system helping you have

40:24tools you're using cloud 3.5 sonnet.

40:27You're making an API call a bedrock

40:28whenever something happens. So that's

40:30where the the LLM we're using. But Nova

40:32act is separate from that. So this MC

40:35like if you're using you know claw

40:36desktop it's running an LLM inside of

40:39that to be able to understand that for

40:40the MCP server

40:44question here

40:47question.

40:56Uh the question is uh does it integrate

40:58with browser plugins as well? Like could

41:00it integrate with LastPass? If you have

41:01the LastPass plugin, fill in the

41:03credentials through LastPass and then

41:05continue. I haven't tried that. But

41:06again, it does you can set up to use

41:08your own browser. So if you do that and

41:10that's integrated, it might be able to

41:12do that and click through that. But I

41:14have not tested that, but something to

41:15try out. Thank you.

41:18And the biggest problem you would face

41:20is two factor. Like even if you gave it

41:22a password like if you're using

41:24something like Google authenticator or

41:25something that would be like the biggest

41:27problem or capture but other than that

41:29if you provide it environmental variable

41:32or if you give it instructions on how to

41:33access LastPass in the browser it should

41:36be able to do it

41:38right and uh oh one more question then

41:40we'll go on to the last module.

41:53So clearly there are a lot of different

41:54uh agent architectures you could use. Um

41:57and I can imagine using this as uh like

42:01you have a coordinator agent set up

42:02somewhere that's running in the overall

42:04app and then when something pops up and

42:07says hey you need to go and look this up

42:09online go and check it. uh it should mod

42:12so my question is how modular it I mean

42:15it's just python so it should be pretty

42:16modular right is that the way in which

42:19you're imagining the architecture to be

42:22is just if I was coding a coordinator

42:25agent in lang chain or lang graph for

42:26example it would then call your sub

42:29agent and get and and run its stuff and

42:32then get and then get a textbased output

42:35that I throw into my message queue

42:38that's how it all integrates together is

42:39that right Yeah, that's one way you can

42:41do it. So, Nova act again, right? It's

42:42just Python. So, it could be a tool, it

42:44could be an API call. And the next

42:46module, we're actually going to show you

42:47how to actually make an agent from that.

42:49So, good good tea app right here. Uh, so

42:52Dan talked about the strands uh at the

42:55beginning. So, strands is a new agentic

42:57framework launched by uh AWS. So, let me

43:00open up the link. Uh, it's easy as a pip

43:03install strands and the first agent is

43:06like agent equals that. So it's very

43:08it's a model first uh way of interacting

43:11with agents. If you use a lot of agent

43:14frameworks in the past, there's a lot of

43:15bootstrapping and making sure everything

43:16is correct and like but that was

43:18necessary for kind of the older models

43:20like if you think back to like like

43:22Llama 2 for example, like how how far

43:24models have evolved since then. So but

43:26now we we can pass a lot of the you know

43:29bootstrapping we did previously. The

43:31agent can figure that out. So we don't

43:32need all these very uh heavy weights and

43:35like you know make sure everything's

43:36typed and every so whatnot. So here's a

43:39very simple example of how I actually

43:41spun up uh and also it has MCP native

43:44support. So in this example I actually

43:46have two MCP servers. Uh I have the AWS

43:50documentation and AWS diagrams MCP

43:53server. So if you go to this like AWS

43:55labs MCP, these are their official um

43:58AWS MCP servers and there's a bunch of

44:00different ones from like a cost

44:02analysis, Nova Canvas, diagramming,

44:04cloud form, uh lots of different ones

44:07here. Uh so again, it's all on GitHub,

44:10AWS Labs, MCP, but the example I do here

44:13is I'm actually uh I made like a

44:15solutions architect agent. Your role is

44:17to help customers understand is building

44:19on AWS. And I define these two MCP

44:22servers here. I give it the prompt and I

44:26say this agent has all the tools in the

44:28MCP server. It has a bedrock model. I'm

44:31using claude haiku here. And what's cool

44:33about strand is it can also use like

44:35light lm and o lama. So it has access to

44:38launch of different things or you can

44:39run it locally and of course it has

44:41access to Amazon bedrock. So that's what

44:43we're using here. So all those three

44:45things makes the agent, the tools, the

44:47model and the system prompt. And then I

44:49can say uh get the documentation for AWS

44:52Lambda and create a diagram of a website

44:54that uses Lambda. So let me run this

44:58code.

45:07CD.

45:18Okay. Okay, so it uses UV to install the

45:20MCP server locally. A lot of people I

45:23where does MCP run. This is running

45:25locally, but there are other ways to run

45:27it like in a lambda function and

45:28whatnot. But for just testing it out, it

45:30pulls down the the MCP server locally

45:33and runs it. And you can see it's

45:34already executing. So let's make this a

45:37bit bigger. Uh

45:42so it says, okay, I'm going to help you

45:44with that. First, I'm going to search

45:45the AWS Lambda documentation. uh read

45:47the documentation, then I'll create a

45:49diagram illustrating a static site. So

45:51you can see it does a post request to do

45:53the search. So the MCP server defines

45:55where everything is. I don't have to

45:56like feed it in the well architected

45:58framework. The AWS documentation is

46:00always updated. So it just knows called

46:02the search function. It got the Lambda

46:04welcome file. It it put that in. It's

46:07able to generate the diagram. It it

46:10generates the diagram. It tells us what

46:11is going on, how the workflow looks

46:13like. It tells me it saved the diagram

46:15to this location. I can open it up.

46:17Generated diagrams.

46:20Oops.

46:22And oh, it's very small. Let me see if I

46:25can make this bigger.

46:29There you go. So, it was able to

46:30generate the diagram for me. So, all

46:32through that about uh you know 40 lines

46:35of code. I have two MCP servers. I have

46:38my prompt and it's able to understand

46:40that get that and just generate

46:41something for me with that. Uh so that's

46:43very easy to get started with strands of

46:45building a agentic workflows. I know

46:47agent means a lot of different things to

46:49different people but you know you have

46:51tools the model the system prompt do

46:54some type of action and strands makes it

46:56extremely easy to do that. If I use

46:58other frameworks it could be a lot more

47:00code to do something like that

47:01especially integrating MCP natively like

47:03that. I'm going to pause here for any

47:05strands questions.

47:14It's coming.

47:18Um I know Bedrock already had it kind of

47:20agents SDK. So is strands replacing that

47:24or is this now the is this replacing

47:26that or is it supposed to complement

47:27that? Like is this the preferred way of

47:29creating agents with models in bedrock?

47:31Yeah. Well, when it comes to preferred

47:33way, it always comes down to your use

47:35case. So the bedrock agent has a lot

47:37more I guess opinionated ways to do

47:39things. It's you can do it through the

47:40console. It has built-in support right

47:43there in AWS. Well, strand is more as an

47:45open source framework. So you can

47:47download the code, you can use other

47:48models through that like light lama. If

47:51you use bedrock agent, you can't run

47:52that offline. So there's different use

47:54cases, different developer tooling. I

47:57mean me as a software engineer, I like

47:58you know code first doing things. So it

48:00does depend on your use case, what

48:01you're trying to do in your experience.

48:03Can can you show the code real quick?

48:05Yeah. Yeah, this is the code. Yeah, just

48:07show the agent.

48:10So, this is an open source framework. If

48:12you go where it says agent, you and it

48:15says model. Right now, we're using a

48:16bedrock model, but you can use another

48:18model with light LLM. Yep. So, you don't

48:20need AWS at all in that instance.

48:26Right.

48:32Yeah.

48:33So there's documentation anthropic

48:35lightm uh lot of different model

48:37providers lama open aai. So it's an open

48:40source framework so you can use it

48:41whatever you want. So but yeah that's

48:43the idea with strand open source model

48:45agent development kit.

48:47One question suppose I want to build a

48:50tax to SQL agent and I have um say 15

48:54tools already built in that I want this

48:57agent to be able to use. If I use this

49:00framework, um, how can I make sure that

49:04the agent know when to use the right

49:07tool and the sequence? Yeah, great

49:10question. Uh, so I didn't this example I

49:13have a weather agent. So one thing you

49:16said you already have tools. What I like

49:18about strange a lot is I can write a

49:20Python function I already have and let's

49:22put this tool decorator and that's it.

49:24You know, you don't have to put anything

49:25else. it understands this is the uh what

49:28you need to do and then when I'm going

49:30to that agent I have this tools and it

49:32has put in the the native tools we're

49:34going to be using http request is a as a

49:36standard tool in the strands framework

49:39so in this example I'm like asking what

49:41is the weather in Seattle and then also

49:43how many words are in this response uh

49:45this open API uh API weather.gov gov

49:48where you don't need an API key and it

49:50can find the information for you. So,

49:52I'm gonna just update this San Francisco

49:57and this

49:59show wrong, but it's a figure it out.

50:01Weather example, weather word count. And

50:04I was very specific, you know, find the

50:05weather first and then how many words

50:07are in the response. So, it's able to

50:08use that tool. It gets the forecast and

50:10then it knows to use that word count

50:12tool next. So, we're passing a lot of

50:14the information to the model. The models

50:16are very smart now. We don't have to say

50:18do this, do this, do this. The let the

50:20agent figure it out. That's kind of the

50:21goal of the agent. You give it the

50:23context and the tools necessary, it

50:25figures out the best way to solve the

50:26problem. But then wouldn't it be prone

50:29to hallucination when you give it 20

50:31tools and then because we've tried that

50:33with AWS bar know the similar things

50:37when you bind more I think more than 10

50:40tools it's going to sure there's always

50:43you know a balance but I again the

50:45models are much better like try using

50:47claw force on it. Is it hallucinating as

50:49much? Like these newer models are much

50:51better for understanding the concept and

50:52understanding what tools when. The older

50:55models sure they get confused. There's

50:57so many things. But I'm very confident

50:59on these newer models they can

51:00understand your use case and what tools

51:01available and figure out the best way to

51:03solve the problem. So then with this

51:05framework there wouldn't be a way for

51:06you to orchestrate a customized flow but

51:11more like you give the control to the

51:13agent. You could if you want to have

51:15like specific like do this specific way

51:18uh there are different ways in strands

51:20uh with something called workflow mode

51:23where you actually say uh you know this

51:26is the workflow I want to do research

51:28results analyze things write a final

51:30report if you have to do something very

51:32sequential a strands has that I won't

51:34have time to go through all the

51:36different you know ways to do multi-

51:37aent collaboration and whatnot but this

51:39for that particular like I wanted to do

51:41xyz first the workflow way can do that.

51:44So yes, then is it possible say um I I

51:49don't have a predefined workflow but I

51:51know it needs to figure out the right

51:53workflow then then that's what I just

51:55did there. You know I just gave it a

51:57sentence and figured it out. I see. I

51:58see. Okay, perfect. Thank you

52:02on it but um Cloud 4 has something

52:05called interle thinking. I believe

52:06that's what it's called where it can

52:08handle multiple tools processing much

52:11better than most models today. So if

52:13you're passing in 20 tools, it's able to

52:17work through the agentic loop to really

52:18figure out which tool to run. And it's

52:21also able to run parallel tool calls. So

52:24rather than just say, okay, here's the

52:27objective, let me run this tool. It can

52:29say here's the objective, let me run

52:30this tool, this tool, this tool, then

52:32this tool, and then process the results

52:34and determine what needs to happen next.

52:37So I would try a cloud for which he like

52:39Banjo mentioned.

52:42Then last example really quick. Uh again

52:45you know strands I made my Nova act MCP

52:48server and it can actually run that you

52:49know I define this is the MCP server use

52:52the Nova MCP you know use the cloud. So

52:55same type of thing I can have another

52:57agent you know use uh nova act as well.

53:00Uh so strands make it very easy to build

53:02these agentic workflows. Uh so that's

53:04really really enjoy the the developer

53:06experience of using strands and you know

53:08I already have the MCP server. We see

53:10the same exact example before. So once

53:12you have the MCP server, it's very easy

53:14to plug in into different uh

53:16architectures and strands makes it very

53:18easy to to accept that.

53:23Uh but yeah, those were the three

53:24modules really about how to use strands.

53:26Uh MCP then Amazon Nova ACT. Again, uh

53:31Strand is open source. You can download

53:32it pip install strands. Uh if you just

53:35type strandagents.com, it'll take you to

53:38the documentation.

53:39again also Nova act nova amazon.com it's

53:44free and log in

53:47and then think that's all the time we

53:49have but we do have a a survey uh and

53:52you can get AWS credit code by filling

53:54out this survey so I I have a question

53:56about that workflow thing in uh strands

53:59when you create these individual agents

54:01can you define which tools are passed on

54:04to each agents yeah yeah it's a great

54:06question Dark about different agents

54:07we're running out of time but I'll

54:09quickly show uh I have a multi- aent

54:12example I believe.

54:15Oh I think you had it in the docs. Yeah.

54:17Yeah, it's in the docs. Yeah. Yeah.

54:20Yeah. Each of these is a different

54:22agent. So you know this is an agent. You

54:24can have a different system prompt. You

54:25can have different tools. So you're just

54:27defining the agent and then yeah you can

54:29have different tools, different whatever

54:30there different models and then the

54:32workflow would just call that. So yes,

54:34completely customizable. So that's the

54:35good thing about Strand. It's very easy

54:36to customize and build scalable

54:38solutions like that. Thank you. And then

54:41again, uh here's the survey. You can get

54:44AWS credits for filling out this thing.

54:46Tell us how we did, what you liked, what

54:47you want to learn more. And now go

54:50build.

54:52[Applause]

55:03Yeah. Any other questions while we wait?

55:05I think we have a minute.

55:15Thanks for the presentation. Um so as

55:18these systems develop I think that it's

55:20reasonable to assume that um they would

55:23emerge as an increasingly effective

55:25vehicle for committing fraud online at

55:27scale which would push businesses to

55:30implement uh more things like capture

55:33which kind of decreases the surface area

55:35that tools like this would be

55:36applicable. So what is the long-term

55:39strategy for that? Well, you already saw

55:41we failed to capture today. Like, you

55:43know, we're not trying to b capture.

55:44We're not trying to break things. You

55:46know, a responsible AI is very important

55:48to Amazon. So, no, we're not trying to

55:49let this tool commit fraud. You know,

55:51you have to have an API key, so it could

55:53be monitored. So, use cases like that

55:55will be shut down.

56:00We

56:03think we're done. Yeah. So, thank you

56:05all.

56:08I think it's finished.

56:13Oh, we can keep going. We have more

56:15time. Oh,

56:17the clock the clock ran out. So, I

56:19thought we were kicked out. All right.

56:22Well, more questions then, I guess. I

56:25thought Yeah, another question.

56:35Um, so regarding Nova Act, let's say

56:38that I have a headless browser in the

56:41cloud. Is there a way to connect Nova

56:43act to my custom browser instance in the

56:45cloud? Yeah. Yeah. Yeah. You can there's

56:47a way to like put your own browser

56:48instance. So yeah, Nova supports that.

56:50So possible. Yeah. Thanks.

57:05Let me go to Novak GitHub page.

57:11And just some examples there.

57:34So yeah, there's a way to set up your

57:35own user agent for Nova app. Definitely

57:37possible.

57:47Questions. Yeah.

57:54Yeah. All right. Well, apparently I have

57:55still more time. So, I don't know if

57:56anyone actually got into the workshop.

57:58So, we can still uh

58:01build some stuff or I can try some other

58:03examples.

58:26Try to make Nova act.

58:37I tried to make a stream app with Nova

58:40Ax. So, we can try if that works.

58:43Oops.

59:05So, one example I tried, I tried to make

59:07a Streamllet app that uh look for like

59:10the top five uh PlayStation games on

59:12game FAQs and then create an image like

59:15a nice graph for me, but it it can fail.

59:18So, uh I think that's one of the issues

59:20there. I think it failed at one of the

59:22steps there.

59:23Uh

59:25let's see. Oh, that Nova app got an

59:28error. So,

59:30it couldn't navigate game faqs.com. So,

59:33it does it does fail at some of the

59:34things. So, that's you know, again,

59:36research preview. You have to be more

59:38specific on how it goes through things.

59:40Uh, but yeah, let me show you where the

59:42the code is just so you can have an

59:44example. Let me pull up the code.

59:52Yeah, let me try let me set up my local

59:54machine so we can see how it works.

59:55Yeah.

1:00:04Oh, yeah. Go for it. How much does Nova

1:00:07act depend on like uh semantic HTML and

1:00:11like good web design to actually work? I

1:00:13mean it understands the actual page so

1:00:15it can click through those things. But

1:00:17if the if the page like doesn't have

1:00:18like a search box or button and not be

1:00:20able to navigate. So as long as the p it

1:00:22can see the page, understand where to

1:00:24click and then click those correct

1:00:25buttons. So maybe a follow is there any

1:00:29like efforts to do like experimental

1:00:31like engagement on the page? So if it

1:00:34comes on a page that it's not familiar

1:00:35with, maybe it would try and act like a

1:00:38human would to like click on things or

1:00:40try things out depending what you you

1:00:42put in that prompt because again you're

1:00:44creating that workflow what it should

1:00:45do. So if you say, you know, explore

1:00:47this website and find things, it will

1:00:49it'll try to click through that. But

1:00:50again, it's up to kind of what that

1:00:52initial prompt is that you have for it.

1:00:55Yeah. When you're using overact, you're

1:00:56kind of giving it step-by-step

1:00:58instructions when you're using the SDK.

1:01:01So that way, if you kind of know it's an

1:01:03obscure website, you can give it those

1:01:06instructions that it need to perform

1:01:08rather than the MCP server um is using

1:01:11natural language to infer what needs to

1:01:13be done. So it's not specific

1:01:15instructions

1:01:16coming from you unless you provide it.

1:01:27Yeah. So, I'm going to run it locally on

1:01:28my machine just to show an example. Uh,

1:01:31let's see.

1:01:39Oh, let me hide my key for a second

1:01:42because it's been recorded.

1:01:55Python get coffee.

1:01:59Thanks for coming.

1:02:04All right. So, I'm just running it

1:02:05locally on my machine. So, without

1:02:06headless mode, so you can see it opens

1:02:08up the browser.

1:02:15It's able to type coffee maker.

1:02:19So what we're looking at now is not in

1:02:22headless mode. This is actually Nova act

1:02:24actually performing the task in a

1:02:26browser. So yeah

1:02:28a lot of questions about how does it

1:02:29work you know and we can try more

1:02:31complicated examples. I just wanted to

1:02:32show it could work on your machine

1:02:38and you can see the log. You know, I'm

1:02:40looking for and if I like change the

1:02:42page while it's doing something, it's

1:02:44going to like mess up. So, I'm going to

1:02:45click the page and see what it does.

1:02:46Like, so someone asked about click

1:02:48things of that nature. What's it going

1:02:50to do now?

1:03:00So see it crashed now because I brought

1:03:02I changed a different page didn't know

1:03:03what to do. So example you can interact

1:03:05with it when it's going through the

1:03:07motion as well. And then I believe I

1:03:10have an uh can the MCP server I set up a

1:03:14cloud instance.

1:03:17Oops.

1:03:26And then I have a my Nova act MCP server

1:03:29is there. So I'm able to actually you

1:03:30know I click this you can see all the

1:03:32tools it has available. So I can ask it

1:03:35to like navigate a website. So uh

1:03:39anyone have a complex example? You can

1:03:41see the MCP server. So I know some

1:03:44people have been asking some complex

1:03:45examples. So get go ahead and give me

1:03:47one

1:03:50here. You got you got one.

1:03:59You can try it. Do you have a specific

1:04:01website that has like drag and drop?

1:04:16Draw.io. Uh, let's go to draw.io. io and

1:04:21make a cool diagram. Use Nova act.

1:04:28Let's see what happens.

1:04:37All right. So, let's go to draw IO. All

1:04:39right. It opened the page.

1:04:45Do I have to accept something? Nope.

1:04:47It's going. Oops. All right. Open

1:04:49dry.io. Let's see.

1:04:55Make this smaller.

1:05:01Wait for page to load. Look at my

1:05:02initial setup for template selections.

1:05:05All right, it's going. Uh.

1:05:09Oh, it crashed.

1:05:11What happened? Oh, do I have to allow

1:05:15allow always?

1:05:17Oh, it took a screenshot.

1:05:26I need to continue the browser session

1:05:28to see what's available. Let's look at

1:05:29the screenshot. All right, it's opening

1:05:31up again. Uh, it's going to draw.io.

1:05:48Yeah, if I keep clicking away, it clicks

1:05:50back to the di the browser session. So,

1:05:54I need like two monitors.

1:05:57See, is it going to figure out how to

1:06:00use draw io?

1:06:04Wait for pay. Take screenshot. Look for

1:06:06template options. Come with blank pay.

1:06:08All right. It's so it's kind of I didn't

1:06:10give it any specific instructions. I

1:06:12just said make something cool. So maybe

1:06:14that's too hard to interpret for this

1:06:16website. Maybe I have to say click this

1:06:18click the square button and then drag

1:06:21the square to the center or something. I

1:06:22might have to been more explicit for

1:06:24that.

1:06:27It seems it seems to have frozen. All

1:06:29right, it's clicking something. All

1:06:31right, click new.

1:06:38Oh, okay. Hey, it's doing stuff

1:06:43again. It's not like super real time.

1:06:45It's going. It's not like

1:06:46instantaneously, but it it's it is

1:06:48clicking through the buttons, clicking

1:06:49through stuff.

1:06:51All right.

1:06:53Did it do anything? Oh, the CL. So, it

1:06:57looks like it fa So, yeah, looks like

1:07:00Claude failed that one. So, I won't

1:07:02blame Nova for that. But that's the

1:07:04that's the idea. So, thanks for trying

1:07:06to do something hard.

1:07:08All right, another question back there.

1:07:11Oh, yeah. Can we can Can we try another

1:07:13one? Yeah, let's try another one. Sure.

1:07:14Can we do um you know on Google Maps,

1:07:18find the top three rated coffee shops

1:07:21with within a mile radius of this hotel.

1:07:23Top three coffee shops

1:07:27shops near the Marriott Marquees in San

1:07:33Francisco.

1:07:39You'll figure it out.

1:07:46All right. Open Maps Google. Search Mary

1:07:48Marque San Francisco. Wait for results

1:07:50to load. So, it has a plan. It's going

1:07:51to do something. So, let's see. It

1:07:53opened Google Maps.

1:08:01All right. Type MQ San Francisco. So,

1:08:03it's able to type that.

1:08:11Okay. It searched. It found the Marquee.

1:08:20So, there's a copy button. Let's see if

1:08:22it clicks that. I'm curious.

1:08:37Looks like it's frozen.

1:08:39Give it a couple more seconds.

1:08:45What did it click?

1:08:47It got this 15 minutes. I was trying to

1:08:50type in that box. Okay.

1:09:03All right. Just type in coffee shops.

1:09:04All right. All right. It's going.

1:09:14All right. So, all right. It'll open the

1:09:15coffee shops and let's see if we can get

1:09:17those top three. There's a 48 47 another

1:09:2047. Let's see if it can get that.

1:09:39Did it crash?

1:09:41I think it did it, but I think I'm going

1:09:44to blame Claude. Cloud desktop crashes.

1:09:47might need a zip MCP client. Uh

1:09:52yeah,

1:09:56I think yeah, I think Claude Desktop

1:09:58doesn't like doing that. But again,

1:09:59because it's an MCP server, I can open

1:10:01up a different MCP client. So I can open

1:10:03like cursor, for example, and ask it

1:10:05questions through that

1:10:08cursor.

1:10:16Let me close this.

1:10:20And then

1:10:22you see it has the MCP tools. Oops, it

1:10:24has this up. Let me just open up a new

1:10:26one. I can do the same thing and use

1:10:31Nova act

1:10:41and then it's calling the MCP tool

1:10:43again. So that's the beauty of MCP. I

1:10:45already have this server. I can just use

1:10:47a different client. It can understand

1:10:48all the information it needs to and do

1:10:50the exact same command. So

1:10:53it's going to do the same thing. cursor

1:10:54might be smarter than cloud code.

1:10:58But yeah, it's able to do the exact same

1:10:59type of thing. So

1:11:04a question over here. Yeah, I just got a

1:11:06question.

1:11:08Yeah.

1:11:12Yes. So Novak question was where is

1:11:13Novak running? And yes, it's running in

1:11:15the cloud. So yeah, it's just you get

1:11:17that API key and it's doing the call

1:11:19behind the scenes in AWS cloud. Yeah. So

1:11:21then what what does it upload to the

1:11:23cloud? Well, it's asking the the

1:11:25questions and like you know go to Google

1:11:27Maps and then they say I understand that

1:11:29and it's actually clicking those buttons

1:11:31and doing the actions. So the the actual

1:11:34uh intent of what you're trying to do in

1:11:36the specific action

1:11:39and if I was using it locally,

1:11:43you couldn't use Nova act locally. It

1:11:46has to be uh connected to the internet

1:11:48to use it.

1:11:50Okay.

1:11:51But if I for example if I wanted to look

1:11:55my

1:11:57Yes.

1:12:01Ah yeah I I see what you're saying.

1:12:02Yeah. Yeah. If you I mean it is you know

1:12:05it's a API endpoint. It's been passed to

1:12:07AWS. So you know only pass information

1:12:09that you feel like it's not going to be

1:12:10we're not training the data or taking

1:12:12any of that nature but it's going to the

1:12:14AWS cloud and processing you know what

1:12:16to click on this button locally on your

1:12:18like browser.

1:12:24So looks like it's not. Yeah. See, now

1:12:26it's even certain the rating. It

1:12:28actually knows which rating to press. So

1:12:33So the Nova act is going to just give

1:12:35the plan, right? Yeah. Yeah. Well, when

1:12:39Nova act is is executing like in this

1:12:41MCP server example, I say, you know,

1:12:43find the top three copy stops in

1:12:45Marriott near the Marriott marquee and

1:12:47then I'm passing that information to the

1:12:49the LLM to understand that plan and then

1:12:51it uses Nova act to interact with the

1:12:54browser because like cursor or cloud

1:12:56code or Amazon Q, they can't interact

1:12:58with the specific uh you know website by

1:13:01itself. It uses it uses Nova act to do

1:13:03that, right? But like given a question

1:13:05though like how how does it come uh come

1:13:08up with a plan? Oh the MCP server like

1:13:10the the client so I picked the model in

1:13:13the example we had the MCP client we had

1:13:16this we showed the model I don't use

1:13:17cloud 3.5 that's coming up with the plan

1:13:20same thing here you know I asked you

1:13:22know help me find the top three copies

1:13:24of native Mario marquee this the model

1:13:26that uh cursor is using is coming up

1:13:29with that plan and then I'm using the

1:13:31nova act mcp server to act on it.

1:13:33Exactly. So this is the plan. Search for

1:13:35Mary Marquee. Click the mirror marquee

1:13:37you know search for the things and you

1:13:38see all this information Nova act

1:13:40returned and it actually it returned

1:13:42this time. So I think the problem was

1:13:44with claw desktop but it got the three

1:13:47top three copy stops there. Right. What

1:13:49are all the tools that uh Novaact can do

1:13:51today. Uh so the MCP server is what I

1:13:54wrote. So uh but the idea between Nova

1:13:57act it can interface with the web

1:13:59browser that that's the tool. The

1:14:00browser is the tool and it can anything

1:14:02that on the website can actually click

1:14:03through, go through the example, etc. I

1:14:06see.

1:14:09You got the repo. Do you got an

1:14:10architecture that shows the MCP just so

1:14:13they can see it? Yeah. So, I mentioned

1:14:16uh there's an official AWS MCP servers.

1:14:18So, uh this AWS Labs MCP and a lot of

1:14:22different um MCP servers here. For the

1:14:25one, the Nova Act one, I created my own

1:14:27one. uh go back to the nova act examples

1:14:30or where do the ah here when I use

1:14:35amazon to explain you know the am the

1:14:38mcp server for like what what's going on

1:14:40what tool was the browser session

1:14:42performing an action on the browser so

1:14:44this is a good uh thing to talk about so

1:14:47can you dive deeper

1:14:51on the browser action function and then

1:14:55we can see because this is how it's

1:14:56actually acting So, uh, Amazon Q browser

1:15:01action is designed to perform actions.

1:15:04It has this, uh, what's cool about it,

1:15:07it just does, oops, let's go into the

1:15:11code. It performs a single action in the

1:15:14Nova act browser. So, it's executing

1:15:16that action. It stores this act. act is

1:15:19like what Nova says you know click the

1:15:21search bar do this XY you know

1:15:25why the MCP client understands how to

1:15:27use this act that passes the correct

1:15:29action so we saw the example here one of

1:15:32the actions was like go to Google maps

1:15:35or click this button or do that search

1:15:38that's how it's able you know these

1:15:39actions and then the nova act MCP server

1:15:42is translating that to actually click

1:15:43that button so the MCP server provides

1:15:46all the interfaces it necessarily needs

1:15:48So then these MCP clients can interact

1:15:51and do actions and do things. Yeah. And

1:15:54Nova act is just the model in the

1:15:55background that's able to click those

1:15:57buttons.

1:16:00Extending this question, it so your MCP

1:16:03server so claw uh or um cursor running

1:16:07locally, right? It's calling your MCP

1:16:09server that's also running locally. Is

1:16:11your MCP and your MCP server is the one

1:16:13that spun up the I guess the Chromium

1:16:15instance, right? Is it is your MCP

1:16:18server taking screenshots of what you

1:16:20see in Chromium and shipping them to

1:16:22Nova to Nova Act? The screenshots are

1:16:25locally and then based on that like you

1:16:26can see it's actually getting all the

1:16:28information uh the final page

1:16:30information. So it's not storing your

1:16:31screenshot data and sending that

1:16:33everything that it's running locally and

1:16:35it's clicking those buttons based on

1:16:36what's on the browser sensing. Got it.

1:16:38But is is any of any of the information

1:16:40in Chromium does that any of that need

1:16:42to be sent into any form? Everything

1:16:44running Yes. running locally. I have the

1:16:46distinction. Okay, perfect. Thank you.

1:17:00And let me open up the

1:17:10Where's that looking?

1:17:15So one of the things about making MCP

1:17:17servers is you have to provide a lot of

1:17:18context. So uh for no act like I say you

1:17:22know when writing action for no action

1:17:24be descriptive of what to do you know

1:17:26click the hamburger menu icon go to

1:17:28order history don't find my order. So

1:17:30the more you know uh concise and

1:17:33prescriptive what you want to do it's

1:17:34better you know search for hotels in

1:17:36Houston sort by average customer like so

1:17:38the better specific it is uh that's how

1:17:41the MCP uh client is able to make those

1:17:43great requests and find the information

1:17:45so type coffee maker search block enter

1:17:48so so the more prescriptive you are of

1:17:50nova act the better results you're going

1:17:51to be and I encoded that all into this

1:17:53uh MCP server so the clients can

1:17:56leverage that

1:17:58so I think that's Probably one of the

1:18:00hardest things about making the MCP

1:18:01servers that's making sure you provide a

1:18:03next context of when to use the tool,

1:18:05how to use the tool, the inputs and

1:18:07outputs. But once you solve all that,

1:18:09it's very easy to plug and play to

1:18:11different MCP clients like we've done

1:18:13here.

1:18:39question. Yeah.

1:18:52Right. So when Nova act is doing

1:18:54something, it's passing back the log of

1:18:56everything it's doing. So you know what

1:18:58what steps it did. So the starting page,

1:19:00the act the results, the action result

1:19:02ID. So it's keeping a log of everything

1:19:04it did. Uh power. So it's able to get

1:19:06that JSON to understand what the ID what

1:19:08the result is. So you can see what it's

1:19:10doing so it can move on to the next

1:19:12step.

1:19:14Yep.

1:19:16Make this bigger.

1:19:21A question.

1:19:28Sorry, a quick question. Yeah. Is this

1:19:31able to do uh like uh automated UI

1:19:33testing because of this?

1:19:36Well, with Nova, you know, you can

1:19:38define like what you want it to do. So,

1:19:40you're going to have to define, you

1:19:41know, go to this button, click this,

1:19:43does this work? So, you can define that

1:19:45workflow. So I mentioned before like

1:19:47back in the day like if I'm writing

1:19:48selenium code I have to go click this H1

1:19:50tag do this like now you can just write

1:19:52in natural language you know click this

1:19:54button click that button so yes it can

1:19:56handle that use case uh specifically of

1:19:58like opening the browser checking these

1:19:59things and but you have to like you know

1:20:02this nova act search for coffee maker

1:20:04you know you have you specifically have

1:20:05to write what buttons to press.

1:20:08Thank you.

1:20:19See, guess if we have time, I can show

1:20:22some multi- aent collaboration with

1:20:24strands. That could be something cool.

1:20:26Uh, I think I have a repo for that. So,

1:20:31should be

1:20:34uh go to the AWS labs page. Where's that

1:20:39work?

1:20:44and then claude.

1:20:49Cool.

1:20:56Okay,

1:20:58I'm just going to copy this code and put

1:21:01it into our environment.

1:21:27So in this example, I'm actually going

1:21:29to show how strand says multi- aent

1:21:31collaboration. So one uh way I'm

1:21:34actually going to create a PowerPoint

1:21:36presentation based on uh you know a

1:21:38cloud migration request. I want to like

1:21:41move my u infrastructure on premise to

1:21:44the cloud. give me a presentation of how

1:21:46I would do that. And so for this, I

1:21:48created three different agents. I

1:21:50created a cost analysis agent. So I have

1:21:52a system prompt there, a solutions

1:21:54architect agent to map out what you're

1:21:56going to be doing. And then each of

1:21:58these uh tools is an actual agent. So

1:22:02this uh costbus has the docs MCP server,

1:22:05the cost analysis MCP server. It has its

1:22:08own prompt. The presentation agent has

1:22:10its own system prompt. It has a tool

1:22:12from there's a a PowerPoint MCP server

1:22:15that I'm using and then there's an

1:22:17architecture agent that also has you

1:22:18know its own specific tools system

1:22:21prompt etc. So uh different agents for

1:22:24different uh things in the workflow and

1:22:27then I have this orchestrator agent what

1:22:29I've called the migration orchestration

1:22:31agent. It has a prompt. I tell it what

1:22:33tools it has access to. And then the

1:22:35cool thing with strands is I make this

1:22:37orchestrator agent and then the tools or

1:22:39this other agents in that. So it knows

1:22:42when to call this agent for this

1:22:43particular tool when to do that and I

1:22:45say you know I want to migrate my work

1:22:48my uh workload. So write the right tools

1:22:52to find that. So I made a fictional

1:22:54company called shop easy e-commerce.

1:22:56They have onremise Java MySQL database.

1:23:00I want to zero down from migration like

1:23:02all this all these little constraints in

1:23:05there and I wanted to make a migration

1:23:07plan and a PowerPoint presentation that

1:23:09I can present to my executives of how

1:23:11this would work and I just assigned and

1:23:15I'm the orchestrator agent will find out

1:23:16what to do. I don't specifically say do

1:23:18this one first, do that first. We'll let

1:23:20the the agent figure that out. So let me

1:23:23run that strands

1:23:26and it should be multi- aent

1:23:34right so cloud partition agent as tools

1:23:39all right again so all the MCP server is

1:23:42running locally it downloads it's using

1:23:43the UX it start with the architecture

1:23:46design first generates a diagram

1:23:55going to use W. So take some time. It

1:23:58might fail but it would just update

1:24:00update itself.

1:24:02Making another judgment.

1:24:13All right. Think it couldn't generate

1:24:15the diagram there, but it's saying all

1:24:16right. That's going to this is what the

1:24:18diagram should have. This is what we're

1:24:19going to doing.

1:24:27Now it's going to do a cost analysis

1:24:29cost analysis on based of the things we

1:24:31did there. So it's it's a this workflow

1:24:33takes maybe a couple minutes to run. But

1:24:35you can see it's calling all these

1:24:36agents uh different things. It's

1:24:38understanding what to do, what actions

1:24:40to take first. It's finding pricing for

1:24:42EKS because it has the uh cost analysis

1:24:46tool and knows where to find that

1:24:48information. So it has the up-to-date

1:24:49pricing all the time finding for Aurora

1:24:52for its database. So it's able to

1:24:54understand all that information and get

1:24:55real time up-to-date information just

1:24:57because we have that uh pricing MCP

1:25:00server from the AWS labs example

1:25:04pricing

1:25:08is it oh cost analysis.

1:25:11Yeah, cost analysis, MCP server

1:25:13documentation, all the stuff you need

1:25:15for finding the right price on AWS. It

1:25:17has all that information and the agent

1:25:19was able to just use that once it's

1:25:22going to generate a report.

1:25:29So, it's still running. Again, this does

1:25:31take a while because I'm asked a very

1:25:33complex question, a lot of things going.

1:25:35Uh, so it does take a couple minutes to

1:25:37run through all that. It gets it monthly

1:25:39spend predictions, monthly savings, etc.

1:25:43So, it's able to understand all the

1:25:44information and get all up-to-date

1:25:46information based on the plan we've

1:25:48provided.

1:25:51And the last thing now wants to create

1:25:53an executive presentation. So, download

1:25:55the PowerPoint MCP server and now it's

1:25:57going to make a PowerPoint presentation

1:25:59based on that.

1:26:03So, adding the title slide. So, you

1:26:05know, add a placeholder. So generating

1:26:08powerpoints is a very popular use case

1:26:10and there's an MCP server that can go

1:26:11ahead and just do that add bullet points

1:26:14etc.

1:26:15So give it a couple another minute or