Beyond Code Coverage: Functionality Testing with Playwright MCP — Marlene Mhangami, Microsoft

0:15Okay.

0:17Hi everyone.

0:19Um

0:19my name is Maybelline and I am a senior

0:22developer advocate at both Microsoft and

0:25GitHub. So I work at I work in a group

0:28called Core AI, which looks at how

0:30developers are using AI across our

0:32products.

0:34Um so this is kind of new.

0:36To start off today, I wanted to show you

0:38some stats about GitHub from GitHub

0:41Octoverse

0:43uh last year's GitHub Octoverse report

0:45in 2025, which shows data about how

0:48developers are using GitHub. What we saw

0:52from our report was that more code was

0:55added to GitHub last year than ever

0:58before. So about a billion commits were

1:02pushed to the platform in 2025, which is

1:05GitHub's most active year ever. Okay?

1:09What we know now in 2026 is that this

1:12growth is accelerating. So a couple of

1:16We haven't actually released any

1:17official stats yet, but a couple of days

1:21ago our COO Kyle Daigle

1:23tweeted that we're seeing about 275

1:28million commits to the platform every

1:30week. And if we extrapolate that over

1:33time, we're going to see about 14

1:35billion commits by the end of the year.

1:37So that's 14 times the amount of growth

1:41we saw last year, commits we saw last

1:43year, which by the way last year again

1:45was our biggest year ever with a a

1:47billion commits.

1:49One thing that we know is that there's a

1:51growing share of these commits that are

1:54co-authored by AI agents. We haven't

1:56released the data yet, but we can

1:59actually track and see what you know,

2:01some Claude for example, co-signs,

2:03commits, and so does Copilot, but Codex

2:06doesn't, but we can also kind of track

2:07based off of

2:09uh some wording in in the code.

2:13I had a question when I saw all of this

2:16growth in terms of how much code we were

2:19seeing, and that question is, does AI

2:22actually make developers more

2:24productive? So, we're seeing all of this

2:26code, does it actually correlate with

2:28productivity?

2:30One of the best resources I've seen that

2:33tries to answer this question is

2:35actually from AI Engineer from a talk

2:37last year. And this graph shares

2:41findings from that talk that's from a

2:43Stanford University study of 120,000

2:45developers. And in this study

2:48um it found that while yes, AI can make

2:52developers more productive, it's

2:54actually how the developers are using AI

2:57that matters the most. So, this graph

3:00from the study actually shows us that

3:02clean code bases amplify AI gains and AI

3:07productivity, while unchecked AI in a

3:10code base is going to amplify entropy.

3:14To illustrate this point, the speaker

3:15from this talk gave a case study example

3:18of a company that used AI in an

3:20unchecked way in their database. And

3:23what you can see is that the number of

3:24PRs that the team was pushing out

3:26increased, but at the same time, the

3:29code quality

3:30that the team was was seeing decreased,

3:33and actually they spent a lot more time

3:35reworking that code, refactoring that

3:37code. And so, overall, though there was

3:40effective output increase of like 1%, AI

3:44didn't really improve the productivity

3:46from this team.

3:49So, what we learned from this study is

3:52that a lot of value that we are wanting

3:54to see as developers from AI relies or

3:57hinges on us having a clean code base.

4:01So, for developers that are using AI

4:02tools, we want to focus on things like

4:05good test coverage, type coverage, and

4:08things like good documentation,

4:09modularity, and so on.

4:12So, I'd actually also argue that we need

4:14to start standardizing some practices

4:16across our teams and across our

4:18industry. And this is something that's a

4:20bit of a controversial topic because

4:21some people at this conference believe

4:23in just closing their eyes and shipping.

4:25And that's also okay. But, I think in my

4:29ideal world and from this study we've

4:31seen, I would recommend standardized

4:33practices for keeping a code base clean.

4:36So, how can developers maintain create

4:39and maintain clean code? This question

4:41is actually not a new question. In our

4:43industry over time, we have seen several

4:46methods that have tried to make

4:49uh maintain maintaining a clean code

4:50base a central part of their philosophy.

4:54One of those approaches that I've

4:55actually seen a lot of developers that

4:57are doing agentic coding with coding

4:59agents talking about is test-driven

5:02development or TDD.

5:03Uh Simon Willison, who's very popular,

5:05just recently published a blog post

5:07about how he's using this specific

5:10flavor of TDD called red-green TDD. And

5:13here, what happens is a developer the

5:15first thing that happens is the

5:16developer gets an incoming feature

5:18request. As soon as they get the the

5:21request, they immediately start by

5:23writing a failing test because the

5:25feature doesn't exist. After that, the

5:28developer focuses on getting the test to

5:31pass. And in this green phase, when

5:33they're trying to get the test to pass,

5:35historically, you should not be focusing

5:37on the quality of the code. All you're

5:39focusing on is speed and getting the

5:42test to go green. So, in the past,

5:44developers maybe would copy code from

5:46Stack Overflow and so on and get the

5:48test pass. But then after that, the

5:51final phase of this is the refactor

5:53phase. And in this phase, you're just

5:55focusing on code quality. So you're

5:58taking that code that you made pass and

6:02refactoring it so that it follows all

6:04the best practices.

6:07So not everyone is a fan of TDD and like

6:10many things in this industry, TDD was

6:12pronounced dead in 2014.

6:16Um and one of the most common complaints

6:20that I've seen on the internet about TDD

6:22is that it focuses too much on code

6:24coverage with unit tests and that it

6:26doesn't actually test the system. So DHH

6:29created Rails published this blog post

6:32in in 2014 and was kind of talking about

6:34this that is an over focus on unit

6:37tests.

6:39And we know that when we over index on

6:42code coverage, there are several issues

6:44that come up. One of the issues is that

6:45there's a tendency test implementation

6:48details. So

6:50um take an example like we see on the

6:52screen where we are having an order

6:54calculation with a discount. If the test

6:57is tied directly to a method like

6:59calculate, just simply renaming the name

7:02of the test even if the functionality is

7:04still fine, it's going to break those

7:06unit tests. So that's not be going to be

7:09great. But even if we test specifically

7:12uh the behavior of the system like the

7:14final end result of of the price we're

7:16looking for um or we test on something

7:19like a stable contract like our API or a

7:22module that

7:23doesn't change but we export it, it

7:25should survive any refactors of our

7:28internal code.

7:30I would say that if you're interested in

7:31learning more about this and behavior

7:34driven uh TDD, I would recommend the

7:36talk by Ian Cooper called TDD where it

7:39all went wrong. It's very good talk.

7:42Another thing that we see is that in the

7:44age of AI

7:46that

7:48many developers are using AI to generate

7:51tests. What they've noticed is that AI

7:53sometimes generates self-affirming

7:56tests. So, while the code coverage tests

7:59might pass and you your unit test suite

8:01is all green, the behavior of the system

8:04is not being validated and that's where

8:06the problem lies.

8:09So, for the rest of this talk I'm going

8:10to be focusing on showing you how you

8:12can avoid these problems and start to

8:14test for functionality using Playwright.

8:18Playwright is an open-source testing

8:20framework that's built by Microsoft and

8:22it automates end-to-end testing in the

8:24browser by simulating user interactions.

8:27And the link that you see on the screen

8:29there is

8:30is the documentation.

8:32So, Playwright supports a number of

8:34different languages right now, Python,

8:36TypeScript, C#, and the example scripts

8:39that you can see on the screen is what a

8:41test would typically look like. So, you

8:44have that line that says page go to

8:46telling the the script that it starts at

8:50the toys play page is where we want to

8:52start. And then we're going to look for

8:54the placeholder search and then we're

8:56going to fill that search bar with that

8:59letter with the word Furby and that will

9:02actually do run the search for us

9:04automatically in the browser, for

9:05example. You can also use headed or

9:07headless mode. So, you don't necessarily

9:10have to look at the browser while your

9:12tests are running. You can actually just

9:13have them running in the background as

9:15well.

9:18Um why is it okay?

9:21So, going back to that idea of TDD, when

9:24we're using Playwright with AI, it

9:26actually should speed up the full

9:28process of TDD for us. So, a lot of

9:30developers in the past have really

9:32complained about how TDD is slow and

9:35that it it's not effective for teams

9:37that want to move fast, but if we have

9:39AI, then what happens is that red part

9:42and the green part are fast. So, we're

9:45focusing on getting our agents to

9:47generate these behavioral tests, the

9:49Playwright tests. Then we're focusing on

9:51getting the agent to quickly generate as

9:53fast as it can code that's going to make

9:55the test pass. And then I would

9:57recommend that developers are going to

9:59spend the most amount of time, so it

10:01grows bigger, on that refactoring stage.

10:04So, they're spending time looking at the

10:06code the agent has generated and making

10:09that code better.

10:12There's a number of ways you can connect

10:14your coding agents today with

10:16Playwright. One of those ways is through

10:18the Playwright MCP server. You can use

10:20the CLI tool if you'd like that instead,

10:23or you can use something called

10:25Playwright agents. And when you're using

10:27Playwright agents, you'll run the

10:29command that you can see on the screen.

10:31And once you run that command, it's

10:32going to install for you

10:34three agent.md files. So, the first one

10:37is going to be a planner, second is a

10:39generator, and the third is a healer.

10:42So, the planner will plan which tests to

10:44run, the

10:46generator is going to actually generate

10:48the the tests, and then the healer will

10:49fix those tests for you.

10:54So, I do want to show you a demo,

10:57and I am going to hope the demo gods

11:01are smiling today.

11:03So, we've all We'll give this a try.

11:07Oh. Oh, no.

11:10Okay.

11:11Here we go.

11:13So, I want to give us a scenario. The

11:15scenario is that Oh, you can't see my

11:19You are only looking at my PowerPoint

11:21right now, and I don't know how to stop

11:23that.

11:25Uh let me close the PowerPoint, maybe,

11:28and see if that will help.

11:32Um

11:34I now want to just show the screen.

11:36Sorry.

11:40Hopefully they'll give me

11:42more time.

11:44Okay, perfect. That that's working as

11:46expected. Okay.

11:47Perfect.

11:48So, the scenario that we're going to

11:50imagine today is imagine I'm a

11:52developer. I'm working at a toy company

11:54called Tailspin Toys.

11:55And a few days ago, I got an email from

11:57the search product management team, and

11:59they asked me to add some new search and

12:01filter features to the site. They asked

12:04me to add in a search bar with text

12:06search for simple searches and Azure AI

12:08search for more complex ones. And

12:10they've also asked me to add in a

12:11sidebar so customers can filter by

12:13category and price. So, I'd like for

12:16Co-pilot to help me with this task and

12:18also

12:19for us to use this

12:22uh TDD-first style of development. So,

12:25this is GitHub Co-pilot CLI. And the

12:28first thing that we can do is going to

12:30try to get the agent to get the

12:32information that we saw in that email

12:35and bring in the features here into our

12:37terminal. And

12:40uh for this, we're going to use

12:41something called Work IQ, which is

12:43Microsoft It's a skill that Microsoft

12:45has developed that lets developers

12:47connect to the M365 suite. So, Outlook,

12:51PowerPoint, whatever it is you would

12:52like. And to bring that information here

12:55into the terminal. So, if you're using

12:57the M365 suite for work, I can

12:59definitely recommend it.

13:00And what I will also mention with TDD,

13:03in the past when we've done things like

13:05unit tests, um typically people what

13:09would trigger writing a unit test is

13:11adding a new method to a class. But

13:13actually, in this new world, what we

13:16want to focus on is the behavior. So, we

13:17want to focus on a feature. So, if a

13:19feature request comes, that is what the

13:22trigger is for the test to be written.

13:24So, So we have our list of what needs to

13:28be

13:29actually developed.

13:31And I'm passing in a second uh I'm

13:34passing in the second prompt and I'm

13:36asking Copilot to help me develop these

13:38features using red green TDD to stop by

13:41writing the Playwright tests that fail

13:43for each feature. And I'm telling it not

13:44to commit the changes just for the sake

13:46of this example. And I do want to point

13:50out that the first thing the agent is

13:51going to do is it's going to start to

13:54examine my code base. So it's going to

13:56understand what's in my code base. I

13:58have the Playwright MCP server already

14:00installed in my CLI into co-Copilot CLI

14:04and it knows what it needs to do to

14:07create the tests to be able to test for

14:09these functions. So the agent is going

14:11to understand what the key code base is

14:13going to look like and then going to

14:15write the tests for it. Uh this process

14:18is actually going to take a while. So in

14:20the meantime, I'm going to switch over

14:22to a new tab.

14:25And I'm going to run the command to get

14:29Playwright the Playwright test. So

14:30earlier today I asked uh I got the agent

14:35to generate those failing tests and then

14:37I got it to do the green phase where the

14:39agent just creates the code to get the

14:41tests to pass and then now I'm asking my

14:44agent to go ahead and run the Playwright

14:46tests to actually test for that search

14:49bar and filter feature for us. So like I

14:52mentioned before, I have the Playwright

14:54MCP server already installed. You can

14:57see it installed here. And our agent is

15:00just going to look for the test file.

15:03And if everything works correctly, it's

15:05going to start writing some tests

15:06running some tests. So we see it's

15:08opened the correct page. It's typing in

15:11different inputs which is testing for

15:14the search bar is working. We saw Furby

15:16was correctly found. Simon was correctly

15:17found. And now it's clicking buttons. So

15:20also testing the category filter is

15:22working correctly. Again, my hands are

15:24not on the keyboard. This is all

15:26playwright and Copilot, so super cool.

15:29And now it's correctly finding all of

15:31the toys in the specific price range.

15:33So, when I run these functionality

15:35tests,

15:37all I can see actively that, okay, the

15:40agent has written this code. The code is

15:42working as I expected. The app is

15:45working as I expected. So, there's so

15:46many different ways that you can test

15:48your app by functionality and all of our

15:51tests pass.

15:52Uh now, once our tests have passed,

15:55that's when I would say we step into the

15:57next phase of actually going ahead and

15:59um

16:00and and running out our writing out

16:03refactors or refactoring the code the

16:04agent has created to generate these

16:07tests that pass.

16:09So, a final thing I will do is I will

16:13give you some best practices

16:16um

16:18with Playwright.

16:19The first thing I would say is that when

16:22Playwright runs those functionality

16:24tests, it's going to take screenshots of

16:27all the tests that it's run. I've gotten

16:29into the practice of adding those screen

16:31screenshots to a PR. So, if I've made

16:33some changes, I'll add them to a PR. The

16:35second thing is that you don't have to

16:36run it where it launches the browser

16:38like you saw in the example. You can run

16:40it in headless mode, so that it runs in

16:42the background. And then a final thing

16:44is I would say commit your code

16:46before you actually get it to fix the

16:49test. Or, you know, commit before it

16:52starts to make changes to your code

16:53because if you don't commit,

16:56it might not remember what it what

16:59happened in the past. So, that's

17:00something to do. And then I would also

17:01say to generate one feature one test per

17:04feature, as well.

17:06As a final note, these are some

17:08resources you could take a look at. Ah,

17:10I forgot to add the link to the GitHub

17:11GitHub repo, but all of the slides are

17:14going going available at that link

17:16there. You can check out the

17:17documentation and you can now connect

17:19with me on social media as well. So,

17:21yeah. Thanks everyone. I think that's

17:22all the time I HAVE TODAY.

17:29I THINK WE HAVE 2 minutes for questions.

17:32Does anyone have any questions

17:35about this? Yes, I see a question there.

17:49Yeah.

17:57We have a lot of

17:59uh

18:01I mean, I think if you have a lot of

18:03state management, I would focus on maybe

18:08I would recommend using Playwright

18:10agents where it downloads the specific

18:12agent.md file because that's going to

18:15have some specialized instructions that

18:17are better at handling state and things

18:19like that. So, I found that agents um

18:22Playwright agents specifically has a lot

18:24of good instructions already built into

18:26it that should help with that. Another

18:28thing that you could do is if you didn't

18:30want to use Playwright for everything,

18:32you could also just directly test your

18:34APIs. If there's an API available,

18:37that's something you could do.

18:39Um so, yeah. That's what I would

18:40recommend.

18:43Are there any other questions? Maybe one

18:44more question.

18:46Uh

18:48well, I'm not sure. Yeah.

18:58Yeah.

18:59Yes. Yes, it can. It can check your

19:02mobile versus on desktop. It should just

19:05work. Yeah. One more. Yeah.

19:14It's

19:14>> browser-based for the moment. Yeah, for

19:16the moment it's only browser-based.

19:17Yeah.

19:19Okay, I think that is all

19:23all the time I have for today. Thanks,

19:24everyone.

19:26Sorry about the no link to the GitHub.